No Child Gets Ahead - The evidence

In April I wrote a piece No Child Gets Ahead in which I argued that current implementations (and particularly in Texas) of the No Child Left Behind program is detrimental to the interests of the above average student. Let me also remind everyone that I consider the goals of NCLB laudable and important. Again, see that earlier rant for a defense of those goals.

Now there is increasing evidence that I am correct. Brighter students are not advancing at the rate one might normally expect of them. This was discussed in a New York Times opinion piece titled Smart Child Left Behind on August 28, 2009. The authors, Tom Loveless and Micheal Petrilli, refer at first to report by the Center for Educational Policy published in June 2009.

The Rosy CEP report

The CEP report asks the question in its title, Is the emphasis on proficiency shortchanging higher- and lower-achieving students? Their answer is no. But Loveless and Petrilli argue that the CEP's report is deeply flawed. After reading the report, I entirely agree that it is broken beyond repair. The most egregious error in that study is the exclusive use of state proficiency test scores. State proficiency tests are designed to measure skill at the grade proficiency level. They never test anything above grade level (which is where advanced students are). My anecdotal experience is that pretty much everyone in my daughter's gifted and talented program hit the ceiling (score 100%) of the state TAKS tests. State proficiency tests are not designed to measure learning beyond the grade level proficiency levels, and simply don't work to measure learning for the high level students. The CEP report pretty much spells out the flaw without realizing it

The main measure of student achievement for this study consists of data from the state tests in reading (or English language arts) and mathematics used for NCLB accountability. Although no large-scale test provides a complete picture of student achievement, we have analyzed state test results because these tests are given to nearly all students in a state, are intended to reflect each state’s academic content standards, and are designed to assess whether students have met their states’ expectations for performance at a particular grade level. [Emphasis mine.]

It appears that the CEP report measures success in a state by looking at state results in terms of the percentage of students (with each state) scoring proficient or above. This notion of counting the number of students who exceed a certain (minimal) standard as a way of seeing whether you are serving the higher performing students is entirely missing the point of the exercise. The question we are asking is Does the way we measure school success shortchange the top students?. The CEP's answer appears to be, Well if we count success according to the way NCLB measure it, then we have success..

Another astounding flaw in the CEP analysis is their use of states as their level of analysis. For them, a gain in a small state completely off sets a lose in a large state, even if it means a decline for millions of more students than there is a gain for. This is truly blushworthy error. even though they fully acknowledge it (on page 18). I could go on. These are not minor technical quibbles. These problems completely and utterly undermine the CEP conclusions.

Why worry

Before I go on to cite the evidence for my assertion that NCLB does shortchange the better students, let me spell out why I and so many others worry that it would do exactly that. I've outlined these reasons in my earlier rant and when combined with the actual level of these proficiency standards (see my rants, Race to the Bottom and No Comparison) there really is a concern. As I said before, people and systems do respond to incentive systems, so we should look very clearly at what we incentivize.

Two Scenarios

The NCLB incentive system rewards schools and districts for the number of students who pass (minimal) proficiency tests. The margin of passing or failing (how high above or below the passing cut-off) counts for nothing. Imagine a class with three students: Alice, Bob, and Charlie. And suppose that the proficiency level is considered met if a student scores a 70 on the crucial test. (Obviously I'm grossly simplifying the examples for the purposes of illustration.)

Now consider scenario 1: Alice scores a 78 and passes. Bob scores and 70 and passes, Charlie scores a 50 and fails. In this scenario, the class has two passes and one failure. That is what will be counted in determining the school's, district's, and (probably) teacher's rating.

Now consider scenario 2: Alice scores a 95 and passes, Bob scores a 69 and fails, Charlie scores a 62 and fails. This class has one pass and two failures. The school, district, and teacher will be marked down severely for this.

All of the incentives (and they are powerful incentives) of NCLB push for scenario 2 above scenario 1. But let's look which class is serving the students better. Both Alice and Charlie do much better in scenario 1 than they do in scenario 2. While Bob does slightly worse in scenario 1 then in 2.

Of course you may object that I could have set up an example where the class that did better on NCLB criteria would also be the one that we would all agree better served the students. But my example illustrates real choices that schools and teachers make every day.

Suppose that you are a teacher and you are confident that Alice will pass the exam with little extra effort from you. With more effort from you, she might learn a great deal, but she is already on a clear target to pass the exam. And suppose that Bob is a student who looks like he will pass the exam, but only if you put extra effort into preparing him. Finally, as a teacher, with all of your pre-tests and such, you determine that even with an extraordinary effort on your part, Charlie is unlikely to pass the exam. If you want to keep your job, and the school wants to keep property values high in its district, then you will focus your effort on Bob.

Response to Response to Intervention

In the excellent teacher training program that I am currently enrolled in, we have been studying the mechanisms by which we identify and help the struggling student, Bob, before he falls too far behind. It is a program (or framework) called Response to Intervention (RtI). It really looks like it should be effective at identifying students like Bob earlier and getting the teacher to devote more time to Bob's needs. But of course any additional time spent on Bob will be time taken away from Alice and Charlie (unless additional staff are provided or the school day is lengthened).

This is one thing that people always seem to forget. Anyone who says that we need to spend more time doing X (where X is "with struggling students", "in the library", "teaching math", "practicing bus evacuations", "taking tests", etc) needs to remember that that means spending less time doing something else. This applies to money as well as time. An additional dollar spent on X is a dollar taken away from something else. It's easy to say what we should spend more time or money on, but it's very hard to answer the question of where that time or money comes from.

How to find out

I've already explained that if we want to test whether the incentives set up by NCLB create a disservice to above average students, we can't measure that by counting how many states have an increase in the number of students reaching the proficient level in that state. So how do we check? First of all we will need to use measures that are (a) comparable across states, and (b) which accurately measure the skills of the above average student. Ideally, we would like to have (c) where the progress of individual students from year to year is measured.

Getting data that is comparable across states is difficult. NCLB allows each state to set its own minimum and proficient standards. Because both politicians and educators like to be able to boast about how well their students are doing, there is pressure to set these standards low. (There are also some good reasons to set them low.) As a consequence of this, there is an incentive to shy away from mechanisms that allow state standards to be compared with one another or have students from one state compared with those of another. (See my earlier rant, No Comparison.)

We also need achievement results that don't suffer from a ceiling effect. That is, it should assess the full range of student achievement including those students near the top. This can be difficult for a number of reasons. First of all, the state assessments for NCLB are completely unsuited for this; so any tests would need to be in addition to those required for NCLB. Secondly, most testing to see whether students have learned the material presented in class; thus they rarely can test students who are above grade level.

Fortunately, there have been an number of attempts to collect such data. In an earlier post, I discussed the Measure of Academic Progress produced by the Northwest Evaluation Association. This time, I will be looking at The Nation's Report Card: Writing 2007 produced by the National Center for Education Statistics (part of the US Department of Education). They developed a scale which should include most advanced students, and sampled school children from across the country. Details of their method can be found in the report. For our purposes, merely showing a chart on page 9 of their report should make the point.

Before NCLB went into effect nation wide (2002) there was no growth in 8th grade writing skills at the lowest levels, while there were gains at the highest level. After NCLB went into effect, there were gains significant at the lowest levels and stagnation at the upper levels. Now I admit that I did troll through reports to find the most dramatic example. But for all grades studied and in all areas we find that NCLB has led wonderful gains at the lower levels. These are important and valued achievements. At the same time, it has lead to a flattening of growth at the higher levels.

In all fairness?

As I've said elsewhere, gains in one place often have costs elsewhere. If we have to have a trade off of improvements for the top students or improvements for the bottom students, maybe we are redressing a prior imbalance by focussing on the struggling student. I will address this issue in a later post. Here I will say that the situation before NCLB was destructive and unjust, with the below average abandoned. NCLB needs to be credited with fixing that. But the current situation, in which the above average student is ignored by the educational system, is little better. But whether you think that the current situation is right or wrong, I hope that everyone realizes that it does shortchange the above average students. In future posts, I will try to elaborate on how I think we can develop an accountability system that establishes incentives which serve all students.