Inside the Black Box: Raising Standards Through Classroom

Download Inside the Black Box: Raising Standards Through Classroom

Preview text

Kappan Classic

KAPPAN digital edition exclusive

Inside the Black Box: Raising Standards Through Classroom Assessment
Formative assessment is an essential component of classroom work and can raise student achievement.

By Paul Black and Dylan Wiliam
Raising the standards of learning that are achieved through school­ ing is an important national priority. In recent years, governments throughout the world have been more and more vigorous in making changes in pursuit of this aim. National, state, and district standards; target setting; enhanced programs for the external testing of students’ performance; surveys such as NAEP (National Assessment of Edu­ cational Progress) and TIMSS (Third International Mathematics and Science Study); initiatives to improve school planning and manage­ ment; and more frequent and thorough inspection are all means to­ ward the same end. But the sum of all these reforms has not added up to an effective policy because something is missing.
Learning is driven by what teachers and pupils do in classrooms. Teachers have to manage complicated and demanding situations, channeling the personal, emotional, and social pressures of a group of 30 or more youngsters in order to help them learn immediately and become better learners in the future. Standards can be raised only if teachers can tackle this task more effectively. What is missing from the efforts alluded to above is any direct help with this task. This fact was recognized in the TIMSS video study: “A focus on standards and accountability that ignores the processes of teaching and learning in classrooms will not provide the direction that teachers need in their quest to improve” (Stigler and Hiebert 1997).
In terms of systems engineering, present policies in the U.S. and in many other countries seem to treat the classroom as a black box. Certain inputs from the outside —pupils, teachers, other resources, management rules and requirements, parental anxieties, standards, tests with high stakes, and so on — are fed into the box. Some outputs are supposed to follow: pupils who are more knowl­ edgeable and competent, better test results, teachers who are reasonably satisfied, and so on. But what is hap­ pening inside the box? How can anyone be sure that a particular set of new inputs will produce better outputs if we don’t at least study what happens inside? And why is it that most of the reform initiatives mentioned in the first paragraph are not aimed at giving direct help and support to the work of teachers in classrooms?
PAUL BLACK is emeritus professor of science education at King’s College London, United Kingdom. DYLAN WILIAM is deputy director of the Institute of Education, University of London, United Kingdom.

This article was originally published as “Inside the Black Box: Raising Standards Through Classroom Assessment,” by Paul Black and Dylan Wiliam. Phi Delta Kappan 80, no. 2 (October 1998): 139-144, 146-148.
Deepen your understanding of this article with questions and activities from this month’s Kappan Professional Development Discussion Guide by Lois Brown Easton, free to members in the digital edition at kappanmagazine .org. V92 N1 Kappan 81

The answer usually given is that it is up to teach­ ers: They have to make the inside work better. This answer is not good enough, for two reasons. First, it is at least possible that some changes in the inputs may be counterproductive and make it harder for teachers to raise standards. Second, it seems strange, even unfair, to leave the most difficult piece of the standards-raising puzzle entirely to teachers. If there are ways in which policy makers and others can give
There is a wealth of research evidence that the
everyday practice of assessment in classrooms is beset
with problems and shortcomings.
direct help and support to the everyday classroom task of achieving better learning, then surely these ways ought to be pursued vigorously.
This article is about the inside of the black box. We focus on one aspect of teaching: formative as­ sessment. But we will show that this feature is at the heart of effective teaching.
We start from the self-evident proposition that teaching and learning must be interactive. Teachers need to know about their pupils’ progress and diffi­ culties with learning so that they can adapt their own work to meet pupils’ needs — needs that are often unpredictable and that vary from one pupil to an­ other. Teachers can find out what they need to know in a variety of ways, including observation and dis­ cussion in the classroom and the reading of pupils’ written work.
We use the general term assessment to refer to all those activities undertaken by teachers — and by their students in assessing themselves — that pro­ vide information to be used as feedback to modify teaching and learning activities. Such assessment be­ comes formative assessment when the evidence is ac­ tually used to adapt the teaching to meet student needs. (There is no internationally agreed-upon term here. “Classroom evaluation,” “classroom as­ sessment,” “internal assessment,” “instructional as­ sessment,” and “student assessment” have been used by different authors, and some of these terms have different meanings in different texts.)
There is nothing new about any of this. All teach­ ers make assessments in every class they teach. But there are three important questions about this process that we seek to answer:
1. Is there evidence that improving formative assessment raises standards?
2. Is there evidence that there is room for improvement?

3. Is there evidence about how to improve formative assessment?
In setting out to answer these questions, we have conducted an extensive survey of the research liter­ ature. We have checked through many books and through the past nine years’ worth of issues of more than 160 journals, and we have studied earlier re­ views of research. This process yielded about 580 ar­ ticles or chapters to study. We prepared a lengthy re­ view, using material from 250 of these sources, that has been published in a special issue of the journal Assessment in Education, together with comments on our work by leading educational experts from Aus­ tralia, Switzerland, Hong Kong, Lesotho, and the U.S. (Black and Wiliam 1998).
The conclusion we have reached from our re­ search review is that the answer to each of the three questions above is clearly yes. In the three main sec­ tions below, we outline the nature and force of the evidence that justifies this conclusion. However, be­ cause we are presenting a summary here, our text will appear strong on assertions and weak on the details of their justification. We maintain that these asser­ tions are backed by evidence and that this backing is set out in full detail in the lengthy review on which this article is founded.
We believe that the three sections below establish a strong case that governments, their agencies, school authorities, and the teaching profession should study very carefully whether they are seri­ ously interested in raising standards in education. However, we also acknowledge widespread evidence that fundamental change in education can be achieved only slowly — through programs of pro­ fessional development that build on existing good practice. Thus we do not conclude that formative as­ sessment is yet another “magic bullet” for education. The issues involved are too complex and too closely linked to both the difficulties of classroom practice and the beliefs that drive public policy. In a final sec­ tion, we confront this complexity and try to sketch out a strategy for acting on our evidence.
A research review published in 1986, concentrat­ ing primarily on classroom assessment work for chil­ dren with mild handicaps, surveyed a large number of innovations, from which 23 were selected (Fuchs and Fuchs 1986). Those chosen satisfied the condi­ tion that quantitative evidence of learning gains was obtained, both for those involved in the innovation and for a similar group not so involved. Since then, many more papers have been published describing similarly careful quantitative experiments. Our own

82 Kappan September 2010

review has selected at least 20 more studies. (The number depends on how rigorous a set of selection criteria are applied.) All these studies show that in­ novations that include strengthening the practice of formative assessment produce significant and often substantial learning gains. These studies range over age groups from 5-year-olds to university under­ graduates, across several school subjects, and over several countries.
For research purposes, learning gains of this type are measured by comparing the average improve­ ments in the test scores of pupils involved in an in­ novation with the range of scores that are found for typical groups of pupils on these same tests. The ra­ tio of the former divided by the latter is known as the effect size. Typical effect sizes of the formative assessment experiments were between 0.4 and 0.7. These effect sizes are larger than most of those found for educational interventions. The following exam­ ples illustrate some practical consequences of such large gains.
An effect size of 0.4 would mean that the average pupil involved in an innovation would record the same achievement as a pupil in the top 35% of those not so involved.
An effect size gain of 0.7 in the recent interna­ tional comparative studies in mathematics (Beaton et al. 1996) would have raised the score of a nation in the middle of the pack of 41 countries (e.g., the U.S.) to one of the top five.
Many of these studies arrive at another important conclusion: that improved formative assessment helps low achievers more than other students and so reduces the range of achievement while raising achievement overall. A notable recent example is a study devoted entirely to low-achieving students and students with learning disabilities, which shows that frequent assessment feedback helps both groups en­ hance their learning (Fuchs et al. 1997). Any gains for such pupils could be particularly important. Fur­ thermore, pupils who come to see themselves as un­ able to learn usually cease to take school seriously. Many become disruptive; others resort to truancy. Such young people are likely to be alienated from society and to become the sources and the victims of serious social problems.
Thus it seems clear that very significant learning gains lie within our grasp. The fact that such gains have been achieved by a variety of methods that have, as a common feature, enhanced formative assessment suggests that this feature accounts, at least in part, for the successes. However, it does not follow that it would be an easy matter to achieve such gains on a wide scale in normal classrooms. Many of the reports we have studied raise a number of other issues.
All such work involves new ways to enhance feed­

back between those taught and the teacher, ways that will require significant changes in classroom prac­ tice.
Underlying the various approaches are assump­ tions about what makes for effective learning — in particular the assumption that students have to be actively involved.
What is needed is a culture of success, backed by a
belief that all pupils can achieve.
For assessment to function formatively, the re­ sults have to be used to adjust teaching and learning; thus a significant aspect of any program will be the ways in which teachers make these adjustments.
The ways in which assessment can affect the mo­ tivation and self-esteem of pupils and the benefits of engaging pupils in self-assessment deserve careful at­ tention.
A poverty of practice. There is a wealth of re­ search evidence that the everyday practice of assess­ ment in classrooms is beset with problems and short­ comings, as the following selected quotations indicate.
“Marking is usually conscientious but often fails to offer guidance on how work can be improved. In a significant minority of cases, marking reinforces underachievement and underexpectation by being too generous or unfocused. Information about pupil performance received by the teacher is insufficiently used to inform subsequent work,” according to a United Kingdom inspection report on secondary schools (OFSTED 1996: 40).
“Why is the extent and nature of formative assess­ ment in science so impoverished?” asked a research study on secondary science teachers in the United Kingdom (Daws and Singh 1996: 99).
“Indeed they pay lip service to [formative assess­ ment] but consider that its practice is unrealistic in the present educational context,” reported a study of Canadian secondary teachers (Dassa, Vazquez-Abad, and Ajar 1993).
“The assessment practices outlined above are not common, even though these kinds of approaches are now widely promoted in the professional literature,” according to a review of assessment practices in U.S. schools (Neill 1997).
The most important difficulties with assessment revolve around three issues.
The first issue is effective learning. The tests used by teachers encourage rote and su­
perficial learning even when teachers say they want to develop understanding; many teachers seem un­ V92 N1 Kappan 83

aware of the inconsistency. The questions and other methods teachers use are
not shared with other teachers in the same school, and they are not critically reviewed in relation to what they actually assess.
For primary teachers particularly, there is a ten­ dency to emphasize quantity and presentation of work and to neglect its quality in relation to learn­ ing.
The second issue is negative impact. The giving of marks and the grading function are
overemphasized, while the giving of useful advice and the learning function are underemphasized.
Approaches are used in which pupils are com­ pared with one another, the prime purpose of which seems to them to be competition rather than per­ sonal improvement; in consequence, assessment feedback teaches low-achieving pupils that they lack “ability,” causing them to come to believe that they are not able to learn.
Feedback to any pupil should be about the
particular qualities of his or her work, with advice
on what he or she can do to improve, and should
avoid comparisons with other pupils.
The third issue is the managerial role of assess­ ments.
Teachers’ feedback to pupils seems to serve social and managerial functions, often at the expense of the learning function.
Teachers are often able to predict pupils’ results on external tests because their own tests imitate them, but at the same time teachers know too little about their pupils’ learning needs.
The collection of marks to fill in records is given higher priority than the analysis of pupils’ work to discern learning needs; furthermore, some teachers pay no attention to the assessment records of their pupils’ previous teachers.
Of course, not all these descriptions apply to all classrooms. Indeed, there are many schools and classrooms to which they do not apply at all. Never­ theless, these general conclusions have been drawn by researchers who have collected evidence — through observation, interviews, and questionnaires — from schools in several countries, including the U.S.
An empty commitment. The development of na­ tional assessment policy in England and Wales over the past decade illustrates the obstacles that stand in the way of developing policy support for formative assessment. The recommendations of a government

task force in 1988 (Task Group on Assessment and Testing 1988) and all subsequent statements of gov­ ernment policy have emphasized the importance of formative assessment by teachers. However, the body charged with carrying out government policy on assessment had no strategy either to study or to develop the formative assessment of teachers and did no more than devote a tiny fraction of its resources to such work (Daugherty 1995). Most of the avail­ able resources and most of the public and political attention were focused on national external tests. While teachers’ contributions to these “summative assessments” have been given some formal status, hardly any attention has been paid to their contri­ butions through formative assessment. Moreover, the problems of the relationship between teachers’ formative and summative roles have received no at­ tention.
It is possible that many of the commitments were stated in the belief that formative assessment was not problematic, that it already happened all the time and needed no more than formal acknowledgment of its existence. However, it is also clear that the po­ litical commitment to external testing in order to promote competition had a central priority, while the commitment to formative assessment was mar­ ginal. As researchers the world over have found, high-stakes external tests always dominate teaching and assessment. However, they give teachers poor models for formative assessment because of their limited function of providing overall summaries of achievement rather than helpful diagnosis. Given this fact, it is hardly surprising that numerous re­ search studies of the implementation of the educa­ tion reforms in the United Kingdom have found that formative assessment is “seriously in need of devel­ opment.” (Russell, Qualter, and McGuigan 1995). With hindsight, we can see that the failure to per­ ceive the need for substantial support for formative assessment and to take responsibility for developing such support was a serious error.
In the U.S. similar pressures have been felt from political movements characterized by a distrust of teachers and a belief that external testing will, on its own, improve learning. Such fractured relationships between policy makers and the teaching profession are not inevitable — indeed, many countries with en­ viable educational achievements seem to manage well with policies that show greater respect and sup­ port for teachers. While the situation in the U.S. is far more diverse than that in England and Wales, the effects of high-stakes state-mandated testing are very similar to those of the external tests in the United Kingdom. Moreover, the traditional reliance on multiple-choice testing in the U.S. — not shared in the United Kingdom — has exacerbated the nega­

84 Kappan September 2010

tive effects of such policies on the quality of class­ room learning.
The self-esteem of pupils. A report of schools in Switzerland states that “a number of pupils . . . are content to ‘get by.’ . . . Every teacher who wants to practice formative assessment must reconstruct the teaching contracts so as to counteract the habits ac­ quired by his pupils.” (Perrenoud 1991).
The ultimate user of assessment information that is elicited in order to improve learning is the pupil. There are negative and positive aspects of this fact. The negative aspect is illustrated by the preceding quotation. When the classroom culture focuses on rewards, “gold stars,” grades, or class ranking, then pupils look for ways to obtain the best marks rather than to improve their learning. One reported con­ sequence is that, when they have any choice, pupils avoid difficult tasks. They also spend time and en­ ergy looking for clues to the “right answer.” Indeed, many become reluctant to ask questions out of a fear of failure. Pupils who encounter difficulties are led to believe that they lack ability, and this belief leads them to attribute their difficulties to a defect in them­ selves about which they cannot do a great deal. Thus they avoid investing effort in learning that can lead only to disappointment, and they try to build up their self-esteem in other ways.
The positive aspect of students’ being the primary users of the information gleaned from formative as­ sessments is that negative outcomes — such as an obsessive focus on competition and the attendant fear of failure on the part of low achievers — are not inevitable. What is needed is a culture of success, backed by a belief that all pupils can achieve. In this regard, formative assessment can be a powerful weapon if it is communicated in the right way. While formative assessment can help all pupils, it yields par­ ticularly good results with low achievers by concen­ trating on specific problems with their work and giv­ ing them a clear understanding of what is wrong and how to put it right. Pupils can accept and work with such messages, provided that they are not clouded by overtones about ability, competition, and com­ parison with others. In summary, the message can be stated as follows: Feedback to any pupil should be about the particular qualities of his or her work, with advice on what he or she can do to improve, and should avoid comparisons with other pupils.
Self-assessment by pupils. Many successful in­ novations have developed self- and peer-assessment by pupils as ways of enhancing formative assessment, and such work has achieved some success with pupils from age 5 upward. This link of formative assess­

ment to self-assessment is not an accident; indeed, it is inevitable.
To explain this last statement, we should first note that the main problem that those who are develop­ ing self-assessments encounter is not a problem of reliability and trustworthiness. Pupils are generally honest and reliable in assessing both themselves and one another; they can even be too hard on them­ selves. The main problem is that pupils can assess themselves only when they have a sufficiently clear picture of the targets that their learning is meant to attain. Surprisingly, and sadly, many pupils do not have such a picture, and they appear to have become accustomed to receiving classroom teaching as an ar­ bitrary sequence of exercises with no overarching ra­ tionale. To overcome this pattern of passive recep­ tion requires hard and sustained work. When pupils do acquire such an overview, they then become more committed and more effective as learners. Moreover, their own assessments become an object of discus-
If there are ways in which policy makers and others can
give direct help and support to the everyday classroom
task of achieving better learning, then surely these
ways ought to be pursued vigorously.
sion with their teachers and with one another, and this discussion further promotes the reflection on one’s own thinking that is essential to good learning.
Thus self-assessment by pupils, far from being a luxury, is in fact an essential component of formative as­ sessment. When anyone is trying to learn, feedback about the effort has three elements: recognition of the desired goal, evidence about present position, and some understanding of a way to close the gap between the two (Sadler 1989). All three must be understood to some degree by anyone before he or she can take action to improve learning.
Such an argument is consistent with more gen­ eral ideas established by research into the way peo­ ple learn. New understandings are not simply swal­ lowed and stored in isolation; they have to be assim­ ilated in relation to preexisting ideas. The new and the old may be inconsistent or even in conflict, and the disparities must be resolved by thoughtful ac­ tions on the part of the learner. Realizing that there are new goals for the learning is an essential part of this process of assimilation. Thus we conclude: If formative assessment is to be productive, pupils should be trained in self-assessment so that they can understand the main purposes of their learning and thereby grasp what they need to do to achieve.
The evolution of effective teaching. The re­ V92 N1 Kappan 85

search studies referred to above show very clearly that effective programs of formative assessment in­ volve far more than the addition of a few observa­ tions and tests to an existing program. They require careful scrutiny of all the main components of a teaching plan. Indeed, it is clear that instruction and formative assessment are indivisible.
To begin at the beginning, the choice of tasks for classroom work and homework is important. Tasks have to be justified in terms of the learning aims that they serve, and they can work well only if opportu-
If formative assessment is to be productive, pupils
should be trained in self-assessment so that they can
understand the main purposes of their learning and
thereby grasp what they need to do to achieve.
nities for pupils to communicate their evolving un­ derstanding are built into the planning. Discussion, observation of activities, and marking of written work can all be used to provide those opportunities, but it is then important to look at or listen carefully to the talk, the writing, and the actions through which pupils develop and display the state of their understanding. Thus we maintain that opportunities for pupils to express their understanding should be designed into any piece of teaching, for this will ini­ tiate the interaction through which formative assess­ ment aids learning.
Discussions in which pupils are led to talk about their understanding in their own ways are important aids to increasing knowledge and improving under­ standing. Dialogue with the teacher provides the opportunity for the teacher to respond to and reori­ ent a pupil’s thinking. However, there are clearly recorded examples of such discussions in which teachers have, quite unconsciously, responded in ways that would inhibit the future learning of a pupil. What the examples have in common is that the teacher is looking for a particular response and lacks the flexibility or the confidence to deal with the un­ expected. So the teacher tries to direct the pupil to­ ward giving the expected answer. In manipulating the dialogue in this way, the teacher seals off any un­ usual, often thoughtful but unorthodox, attempts by pupils to work out their own answers. Over time the pupils get the message: They are not required to think out their own answers. The object of the ex­ ercise is to work out — or guess — what answer the teacher expects to see or hear.
A particular feature of the talk between teacher and pupils is the asking of questions by the teacher. This natural and direct way of checking on learning is often unproductive. One common problem is that,

following a question, teachers do not wait long enough to allow pupils to think out their answers. When a teacher answers his or her own question af­ ter only two or three seconds and when a minute of silence is not tolerable, there is no possibility that a pupil can think out what to say.
There are then two consequences. One is that, because the only questions that can produce answers in such a short time are questions of fact, these pre­ dominate. The other is that pupils don’t even try to think out a response. Because they know that the an­ swer, followed by another question, will come along in a few seconds, there is no point in trying. It is also generally the case that only a few pupils in a class an­ swer the teacher’s questions. The rest then leave it to these few, knowing that they cannot respond as quickly and being unwilling to risk making mistakes in public. So the teacher, by lowering the level of questions and by accepting answers from a few, can keep the lesson going but is actually out of touch with the understanding of most of the class. The ques­ tion/answer dialogue becomes a ritual, one in which thoughtful involvement suffers.
There are several ways to break this particular cy­ cle. They involve giving pupils time to respond; ask­ ing them to discuss their thinking in pairs or in small groups, so that a respondent is speaking on behalf of others; giving pupils a choice between different pos­ sible answers and asking them to vote on the options; asking all of them to write down an answer and then reading out a selected few; and so on. What is essen­ tial is that any dialogue should evoke thoughtful re­ flection in which all pupils can be encouraged to take part, for only then can the formative process start to work. In short, the dialogue between pupils and a teacher should be thoughtful, reflective, focused to evoke and explore understanding, and conducted so that all pupils have an opportunity to think and to express their ideas.
Tests given in class and tests and other exercises assigned for homework are also important means of promoting feedback. A good test can be an occasion for learning. It is better to have frequent short tests than infrequent long ones. Any new learning should first be tested within about a week of a first encounter, but more frequent tests are counterproductive. The quality of the test items — that is, their relevance to the main learning aims and their clear communica­ tion to the pupil — requires scrutiny as well. Good questions are hard to generate, and teachers should collaborate and draw on outside sources to collect such questions.
Given questions of good quality, it is essential to ensure the quality of the feedback. Research studies have shown that, if pupils are given only marks or grades, they do not benefit from the feedback. The

86 Kappan September 2010

worst scenario is one in which some pupils who get low marks this time also got low marks last time and come to expect to get low marks next time. This cy­ cle of repeated failure becomes part of a shared be­ lief between such students and their teacher. Feed­ back has been shown to improve learning when it gives each pupil specific guidance on strengths and weaknesses, preferably without any overall marks. Thus the way in which test results are reported to pupils so that they can identify their own strengths and weaknesses is critical. Pupils must be given the means and opportunities to work with evidence of their difficulties. For formative purposes, a test at the end of a unit or teaching module is pointless; it is too late to work with the results. We conclude that the feedback on tests, seatwork, and homework should give each pupil guidance on how to improve, and each pupil must be given help and an opportunity to work on the improvement.
All these points make clear that there is no one simple way to improve formative assessment. What is common to them is that a teacher’s approach should start by being realistic and confronting the question “Do I really know enough about the under­ standing of my pupils to be able to help each of them?”
Much of the work teachers must do to make good use of formative assessment can give rise to difficul­ ties. Some pupils will resist attempts to change ac­ customed routines, for any such change is uncom­ fortable, and emphasis on the challenge to think for yourself (and not just to work harder) can be threat­ ening to many. Pupils cannot be expected to believe in the value of changes for their learning before they have experienced the benefits of such changes. Moreover, many of the initiatives that are needed take more class time, particularly when a central pur­ pose is to change the outlook on learning and the working methods of pupils. Thus teachers have to take risks in the belief that such investment of time will yield rewards in the future, while “delivery” and “coverage” with poor understanding are pointless and can even be harmful.
Teachers must deal with two basic issues that are the source of many of the problems associated with changing to a system of formative assessment. The first is the nature of each teacher’s beliefs about learn­ ing. If the teacher assumes that knowledge is to be transmitted and learned, that understanding will de­ velop later, and that clarity of exposition accompa­ nied by rewards for patient reception are the essen­ tials of good teaching, then formative assessment is hardly necessary. However, most teachers accept the wealth of evidence that this transmission model does not work, even when judged by its own criteria, and so are willing to make a commitment to teaching

through interaction. Formative assessment is an es­ sential component of such instruction. We do not mean to imply that individualized, one-on-one teaching is the only solution; rather, we mean that what is needed is a classroom culture of questioning and deep thinking, in which pupils learn from shared discussions with teachers and peers. What emerges very clearly here is the indivisibility of instruction and formative assessment practices.
The other issue that can create problems for teachers who wish to adopt an interactive model of teaching and learning relates to the beliefs teachers hold about the potential of all their pupils for learn­ ing. To sharpen the contrast by overstating it, there is on the one hand the “fixed I.Q.” view — a belief that each pupil has a fixed, inherited intelligence that cannot be altered much by schooling. On the other hand, there is the “untapped potential” view — a be­ lief that starts from the assumption that so-called
Teachers will not take up ideas that sound
attractive, no matter how extensive the research
base, if the ideas are presented as general
principles that leave the task of translating them
into everyday practice entirely up to the
ability is a complex of skills that can be learned. Here, we argue for the underlying belief that all pupils can learn more effectively if one can clear away, by sen­ sitive handling, the obstacles to learning, be they cognitive failures never diagnosed or damage to per­ sonal confidence or a combination of the two. Clearly the truth lies between these two extremes, but the evidence is that ways of managing formative assessment that work with the assumptions of “un­ tapped potential” do help all pupils to learn and can give particular help to those who have previously struggled.
Changing the policy perspective. The assump­ tions that drive national and state policies for assess­ ment have to be called into question. The promo­ tion of testing as an important component for estab­ lishing a competitive market in education can be very harmful. The more recent shifting of emphasis to­ ward setting targets for all, with assessment provid­ ing a touchstone to help check pupils’ attainments, is a more mature position. However, we would ar­ gue that there is a need now to move further, to fo­ cus on the inside of the “black box” and so to explore the potential of assessment to raise standards directly as an integral part of each pupil’s learning work. V92 N1 Kappan 87

It follows from this view that several changes are needed. First, policy ought to start with a recogni­ tion that the prime locus for raising standards is the classroom, so that the overarching priority has to be the promotion and support of change within the classroom. Attempts to raise standards by reforming the inputs to and measuring the outputs from the black box of the classroom can be helpful, but they are not adequate on their own. Indeed, their help­ fulness can be judged only in light of their effects in classrooms.
Assessment, as it occurs in schools, is far
from a merely technical problem. Rather, it is
deeply social and personal.
The evidence we have presented here establishes that a clearly productive way to start implementing a classroom-focused policy would be to improve formative assessment. This same evidence also es­ tablishes that in doing so we would not be concen­ trating on some minor aspect of the business of teaching and learning. Rather, we would be concen­ trating on several essential elements: the quality of teacher/pupil interactions, the stimulus and help for pupils to take active responsibility for their own learning, the particular help needed to move pupils out of the trap of “low achievement,” and the devel­ opment of the habits necessary for all students to be­ come lifelong learners. Improvements in formative assessment, which are within the reach of all teach­ ers, can contribute substantially to raising standards in all these ways.
Four steps to implementation. If we accept the argument outlined above, what needs to be done? The proposals outlined below do not follow directly from our analysis of assessment research. They are consistent with its main findings, but they also call on more general sources for guidance (Black and Atkin 1996; Fullan 1991).
At one extreme, one might call for more research to find out how best to carry out such work; at the other, one might call for an immediate and largescale program, with new guidelines that all teachers should put into practice. Neither of these alterna­ tives is sensible: While the first is unnecessary be­ cause enough is known from the results of research, the second would be unjustified because not enough is known about classroom practicalities in the con­ text of any one country’s schools.
Thus the improvement of formative assessment cannot be a simple matter. There is no quick fix that can alter existing practice by promising rapid re­ wards. On the contrary, if the substantial rewards promised by the research evidence are to be secured,

each teacher must find his or her own ways of incor­ porating the lessons and ideas set out above into his or her own patterns of classroom work and into the cultural norms and expectations of a particular school community (Stigler and Hiebert 1997). This process is a relatively slow one and takes place through sustained programs of professional devel­ opment and support. This fact does not weaken the message here; indeed, it should be seen as a sign of its authenticity, for lasting and fundamental im­ provements in teaching and learning must take place in this way. A recent international study of innova­ tion and change in education, encompassing 23 proj­ ects in 13 member countries of the Organisation for Economic Co-operation and Development, has ar­ rived at exactly the same conclusion with regard to effective policies for change (Black and Atkin 1996). Such arguments lead us to propose a four-point scheme for teacher development.
1. Learning from development. Teachers will not take up ideas that sound attractive, no matter how extensive the research base, if the ideas are pre­ sented as general principles that leave the task of translating them into everyday practice entirely up to the teachers. Their classroom lives are too busy and too fragile for all but an outstanding few to un­ dertake such work. What teachers need is a variety of living examples of implementation, as practiced by teachers with whom they can identify and from whom they can derive the confidence that they can do better. They need to see examples of what doing better means in practice.
So changing teachers’ practice cannot begin with an extensive program of training for all; that could be justified only if it could be claimed that we have enough “trainers” who know what to do, which is certainly not the case. The essential first step is to set up a small number of local groups of schools — some primary, some secondary, some inner-city, some from outer suburbs, some rural — with each school committed both to a school-based develop­ ment of formative assessment and to collaboration with other schools in its local group. In such a process, the teachers in their classrooms will be working out the answers to many of the practical questions that the evidence presented here cannot answer. They will be reformulating the issues, per­ haps in relation to fundamental insights and certainly in terms that make sense to their peers in other class­ rooms. It is also essential to carry out such develop­ ment in a range of subject areas, for the research in mathematics education is significantly different from that in language, which is different again from that in the creative arts.
The schools involved would need extra support in order to give their teachers time to plan the ini­

88 Kappan September 2010

tiative in light of existing evidence, to reflect on their experience as it develops, and to offer advice about training others in the future. In addition, there would be a need for external evaluators to help the teach­ ers with their development work and to collect evi­ dence of its effectiveness. Video studies of classroom work would be essential for disseminating findings to others.
2. Dissemination. This dimension of the imple­ mentation would be in low gear at the outset —of­ fering schools no more than general encouragement and explanation of some of the relevant evidence that they might consider in light of their existing prac­ tices. Dissemination efforts would become more ac­ tive as results and resources became available from the development program. Then strategies for wider dissemination — for example, earmarking funds for inservice training programs — would have to be pur­ sued.
We must emphasize that this process will in­ evitably be a slow one. To repeat what we said above, if the substantial rewards promised by the evidence are to be secured, each teacher must find his or her own ways of incorporating the lessons and ideas that are set out above into his or her own patterns of class­ room work. Even with optimum training and sup­ port, such a process will take time.
3. Reducing obstacles. All features in the edu­ cation system that actually obstruct the development of effective formative assessment should be exam­ ined to see how their negative effects can be reduced. Consider the conclusions from a study of teachers of English in U.S. secondary schools.
Most of the teachers in this study were caught in conflicts among belief systems and institutional structures, agendas, and values. The point of fric­ tion among these conflicts was assessment, which was associated with very powerful feelings of being overwhelmed, and of insecurity, guilt, frustration, and anger. . . . This study suggests that assessment, as it occurs in schools, is far from a merely techni­ cal problem. Rather, it is deeply social and personal (Johnston et al 1995).
The chief negative influence here is that of short external tests. Such tests can dominate teachers’ work, and, insofar as they encourage drilling to pro­ duce right answers to short, out-of-context ques­ tions, they can lead teachers to act against their own better judgment about the best ways to develop the learning of their pupils. This is not to argue that all such tests are unhelpful. Indeed, they have an im­ portant role to play in securing public confidence in the accountability of schools. For the immediate fu­ ture, what is needed in any development program for formative assessment is to study the interactions between these external tests and formative assess­

ments to see how the models of assessment that ex­ ternal tests can provide could be made more helpful.
All teachers have to undertake some summative assessment. They must report to parents and pro­ duce end-of-year reports as classes are due to move on to new teachers. However, the task of assessing pupils summatively for external purposes is clearly different from the task of assessing ongoing work to monitor and improve progress. Some argue that these two roles are so different that they should be kept apart. We do not see how this can be done, given that teachers must have some share of responsibility for the former and must take the leading responsi­ bility for the latter (Wiliam and Black 1996). How­ ever, teachers clearly face difficult problems in rec­ onciling their formative and summative roles, and confusion in teachers’ minds between these roles can impede the improvement of practice.
The arguments here could be taken much further to make the case that teachers should play a far greater role in contributing to summative assess­ ments for accountability. One strong reason for giv­ ing teachers a greater role is that they have access to the performance of their pupils in a variety of con­ texts and over extended periods of time.
This is an important advantage because sampling pupils’ achievement by means of short exercises taken under the conditions of formal testing is fraught with dangers. It is now clear that perform­ ance in any task varies with the context in which it is presented. Thus some pupils who seem incompe­ tent in tackling a problem under test conditions can look quite different in the more realistic conditions of an everyday encounter with an equivalent prob­ lem. Indeed, the conditions under which formal tests are taken threaten validity because they are quite un­ like those of everyday performance. An outstanding example here is that collaborative work is very im­ portant in everyday life but is forbidden by current norms of formal testing (Wineburg 1997). These points open up wider arguments about assessment systems as a whole — arguments that are beyond the scope of this article.
4. Research. It is not difficult to set out a list of questions that would justify further research in this area. Although there are many and varied reports of successful innovations, they generally fail to give clear accounts of one or another of the important de­ tails. For example, they are often silent about the ac­ tual classroom methods used, the motivation and ex­ perience of the teachers, the nature of the tests used as measures of success, or the outlooks and expecta­ tions of the pupils involved.
However, while there is ample justification for proceeding with carefully formulated projects, we do not suggest that everyone else should wait for their V92 N1 Kappan 89

conclusions. Enough is known to provide a basis for active development work, and some of the most im­ portant questions can be answered only through a program of practical implementation.
Directions for future research could include a study of the ways in which teachers understand and deal with the relationship between their formative and summative roles or a comparative study of the predictive validity of teachers’ summative assess­ ments versus external test results. Many more ques­ tions could be formulated, and it is important for fu­ ture development that some of these problems be tackled by basic research. At the same time, experi­ enced researchers would also have a vital role to play in the evaluation of the development programs we have proposed.


The findings summarized above and the program

we have outlined have implications for a variety of

responsible agencies. However, it is the responsibil­

ity of governments to take the lead. It would be pre­

mature and out of order for us to try to consider the

relative roles in such an effort, although success

would clearly depend on cooperation among gov­

ernment agencies, academic researchers, and school-

based educators.

The main plank of our argument is that standards

can be raised only by changes that are put into di­

rect effect by teachers and pupils in classrooms.

There is a body of firm evidence that formative as­

sessment is an essential component of classroom

work and that its development can raise standards of

achievement. We know of no other way of raising

standards for which such a strong prima facie case

can be made. Our plea is that national and state pol­

icy makers will grasp this opportunity and take the

lead in this direction.



Beaton, Albert E., et al. Mathematics Achievement in the Middle School Years. Boston: Boston College, 1996.

Black, Paul J., and J. Myron Atkin. Changing the Subject: Innovations in Science, Mathematics, and Technology Education. London: Routledge for the Organisation for Economic Co-operation and Development, 1996.

Black, Paul, and Dylan Wiliam. “Assessment and Classroom Learning.” Assessment in Education (March 1998): 7-74.

Dassa Clement, Jesús Vazquez-Abad, and Djavid Ajar. “Formative Assessment in a Classroom Setting: From Practice to Computer Innovations.” Alberta Journal of Educational Research 39, no. 1 (1993): 116.

Daugherty, Richard. National Curriculum Assessment: A

Review of Policy, 1987-1994. London: Falmer Press, 1995.
Daws, Nicholas, and Birendra Singh. “Formative Assessment: To What Extent Is Its Potential to Enhance Pupils’ Science Being Realized?” School Science Review 77 (1996): 99.
Fuchs, Lynn S., and Douglas Fuchs. “Effects of Systematic Formative Evaluation: A Meta-Analysis.” Exceptional Children 53, no. 3 (November 1986): 199-208.
Fuchs, Lynn S., et al. “Effects of Task-Focused Goals on LowAchieving Students with and Without Learning Disabilities.” American Educational Research Journal 34, no. 3 (Fall 1997): 513-543.
Fullan, Michael G., with Suzanne Stiegelbauer. The New Meaning of Educational Change. London: Cassell, 1991.
Johnston, Peter. “Assessment of Teaching and Learning in Literature-Based Classrooms.” Teaching and Teacher Education 11, no. 4 (July 1995): 359.
Neill, D. Monty. “Transforming Student Assessment.” Phi Delta Kappan 79, no. 1 (September 1997): 35-36.
Office for Standards in Education (OFSTED). Subjects and Standards: Issues for School Development Arising from OFSTED Inspection Findings 1994-5: Key Stages 3 and 4 and Post-16. London: Her Majesty’s Stationery Office, 1996.
Perrenoud, Phillipe. “Towards a Pragmatic Approach to Formative Evaluation.” In Assessment of Pupils’ Achievement: Motivation and School Success, ed. Penelope Weston. Amsterdam: Swets and Zeitlinger, 1991: 92.
Russell, Terry A., Anne Qualter, and Linda McGuigan. “Reflections on the Implementation of National Curriculum Science Policy for the 5-14 Age Range: Findings and Interpretations from a National Evaluation Study in England.” International Journal of Science Education 17, no. 4 (1995): 481-492.
Sadler, D. Royce. “Formative Assessment and the Design of Instructional Systems.” Instructional Science 18, no. 2 (June 1989): 119-144.
Stigler, James W., and James Hiebert. “Understanding and Improving Classroom Mathematics Instruction: An Overview of the TIMSS Video Study.” Phi Delta Kappan 79, no. 1 (September 1997): 14-21.
Task Group on Assessment and Testing: A Report. London: Department of Education and Science and the Welsh Office, 1988.
Wiliam, Dylan, and Paul Black. “Meanings and Consequences: A Basis for Distinguishing Formative and Summative Functions of Assessment.” British Educational Research Journal 22, no. 5 (December 1996): 537-548.
Wineburg, Sam. “T.S. Eliot, Collaboration, and the Quandaries of Assessment in a Rapidly Changing World.” Phi Delta Kappan 79, no. 1 (September 1997): 59-65.

90 Kappan September 2010

Preparing to load PDF file. please wait...

0 of 0
Inside the Black Box: Raising Standards Through Classroom