Left Behind

Yesterday I went to my first UW Madison Econ department seminar which played host to Derek Neal. He has some Chicago school data before and after the implementation of high stakes testing and in his paper entitled Left Behind By Design: Proficiency Counts and Test-Based Accountability.
His thesis is captured well with this quote.

"We were told to cross off the kids who would never pass. We were told to cross off the kids who, if we handed them the test tomorrow, they would pass. And then the kids who were left over, those were the kids we were supposed to focus on."*

They find evidence that high stakes testing, when the standard is set high enough to be unreachable for some results in no improvements in test scores in the tails of the distribution.

Some interesting observations:
1. High stakes testing leads to teaching more intensively to to the students close to the proficiency standard while ignoring the students in the tails of the distribution. Much like teaching to the test ignores important skills that are unrelated to those required for the test.

2. The higher the proficiency standards the more children are left behind, as the probability of them becoming proficiency decreases with rising standards, the teacher's effort shifts up the ability distribution. So if you really don't want to leave any children behind, you need to make sure the standards are attainable for all students. Of course this will result in less teaching effort being allocated to the upper tail of the distribution.

3. Even a value added metric is difficult to design in that we often treat test scores as cardinal measures when they are likely no better than ordinal. Thus the teaching effort may not be uniformly distributed across student ability if test gains are easy to get in certain areas of the ability distribution.

No Child Left Behind (NCLB) is up for renewal, and it doesn't sound like they have any hope of getting the incentives right.

It still seems to me to a value added approach would be a better direction for NCLB to go. It certainly minimizes the problems associated with merely comparing proficiency rates across very different schools. And it is more in line with what we mean when we talk about learning. Under NCLB, a school that has the brightest students and 100% proficiency would be considered successful even though the students might not be learning anything.