Nostalgia for No Child Left Behind and Other Education Policy White Elephants

“Evidence is the antidote in education,” writes journalist Holly Korbey in a recent post of her blog, The Bell Ringer. I agree. Still, the skeptic in me asks, “which evidence?” She seemed to answer that question later in the same post: “all the available evidence.”

Well … OK. Does she really mean it? … ALL THE AVAILABLE EVIDENCE?

I have no reason to believe that she wrote insincerely. Nonetheless, I would argue that the answer is no. The paradox turns on what one considers to be “all.”

Read the post through and one discovers that her version of all the evidence confines itself within boundaries. For example, we learn that an “evidence revolution” began only thirty-five years ago, even in the hard sciences and medicine. Before then, “decisions about treatment were often based on ‘someone’s opinion or conventional wisdom.’”

Yet, the Salk polio vaccine I received as a child was not formulated based on opinion and conventional wisdom. Those many thousands of lines of children throughout the country’s schools in the 1950s and 1960s assembled eagerly, but not to be injected with folk belief Before 35 years ago, there were Marie Curie, Antoine Lavoissier, James Faraday, Archimedes, Eratosthenes, and … you get the drift.

Korbey continues, “The evidence revolution … happened in education beginning two decades ago.”

What was so special about the late 1990s and early 2000s to make it such a profound moment of demarcation in the progress of science and education research? I nominate two suspects. First, the launch of the World Wide Web. Once one began to expect all information could be found on the Web, the bulk of the world’s research—pre 1990s—disappeared for many. For some, absence of (conveniently retrieved online) evidence equals evidence of absence.

Second, the arrival en masse of economists to study education subfields they had previously ignored, such as testing, accountability, curriculum, and instruction. It is a popular belief among some of those economists that no research of any worthwhile quality existed prior to their arrival on the scene.

Korbey promotes the US Education Department’s What Works Clearinghouse as a reliable source of education research. But the WWC restricted its search for evidence to randomized experimental designs conducted within the prior two decades. They apparently did all their source searching on the Web. They also found Reading Recovery to be a successful program, based on their review “of the available evidence.”

Even now, a great abundance of scientific research documents remain unavailable on the World Wide Web. One must get off one’s rear end, transport oneself to an academic library, and ask a librarian for instructions on how to use a microfiche or microfilm reader. Hopefully, the librarian will know how and, hopefully, the fiche or film remains available. Sure, many pre-1990s journals have been digitized. But too often “digitized” really means photocopied or digitized with an optical reader, which means the research is there to read but is not searchable, a distinction that may matter little when researchers shun libraries anyway.

Or one must contact a school district, state education department, or contracting evaluation organization and ask directly for their technical reports. Not every organization makes the effort to promote their program evaluations to make it easy for others to find.

Korbey’s colleague at The Argument, Kelsey Piper, also seems selective in the research she reads. She tells her readers that the research of economists can be trusted above all others and specifically recommends the work of Thomas Dee, of Stanford University.

Economists’ work, she asserts, is more trustworthy because “Virtually all well-regarded economics journals recommend or require full sharing of the data and code that went into a paper.” That means that if you have training in econometrics, access to statistical software, and a month of free time available to rebuild a study from scratch, you are free to try to replicate an economist’s study.

Short of that, you’re out of luck. It is far more common, for example, in some other fields to respond to a piece of research without such a prohibitive investment. Many journals in psychology, for example, but few in economics, belong to COPE and PubPeer, which allow others to send feedback to a journal article and initiate a debate with just a few minutes of typing.

In my own reading, I have found economists often to be parochial, reading and citing only other economists, which is particularly a problem in education whose subject matter leans more often toward psychology and sociology than economics. Economists tend to have little to no training or experience in curriculum and instruction or testing and measurement, for example, yet may nonetheless profess expertise.

Piper, however, wants to “bring in the economists” to fix education research.

“So it’s possible that in order to fix anything else, you simply need more competent and qualified people doing research in education. But as long as the field of education has low standards, it won’t attract very many researchers who are capable enough to instead work in a more respected field.

“We need to convince more economists to do work on education, using modern econometrics and adhering to the standards of their own field, which has adopted norms to combat fraud, p-hacking, data concealment, and grossly underpowered studies. There are some; while Stanford brought us Jo Boaler, it also has applied economist Thomas Dee, whose education policy work I find valuable and interesting (and which, tellingly, cuts across ideological lines in its findings).”

For his part, Dee has made a name for himself by defending the No Child Left Behind (NCLB) Act and the District of Columbia’s value-added measurement program, IMPACT.

For NCLB, Dee and his co-author, Brian Jacob, declare “Accountability has produced substantial gains in math skills but not in reading.” They compare gains on the National Assessment of Educational Progress (NAEP) after 2003 between states which had “accountability” programs before 2003 to states which lacked such programs before 2003. (“Accountability” in their study means accountability for schools, not for students.) For the latter states, NCLB testing allegedly represented the introduction of accountability.

The latter group of states showed more of an improvement in mathematics and Dee and Jacob suggest that the cause must be the addition of the NCLB program.

Their conclusion does not square with some of the most basic evidence, however. Figure 1 displays the annualized achievement gains (in score scale points) on the NAEP mathematics exams (4th and 8th grades combined) for four historical periods: 1992–2003 — the pre-NCLB period; 2003–2013 — the NCLB, but pre-Common Core period; 2013–2019 — the NCLB and Common Core period; and 2019–2024, the NCLB, Common Core, plus COVID19 period.

Figure 1. US Average Annual Gain in NAEP Average Mathematics Score, Grades 4 and 8 Combined: 1992–2024

One may notice, first, a steady decline over the decades as federal mandates accumulated.

But notice also the decline from the first historical period to the second. If NCLB had increased math achievement, as Dee and Jacob assert, the second bar should be higher than the first. Their analysis assumed that the achievement gain baseline for their comparison was zero. It was not. The country was doing just fine, or at least better, before federal intervention. Math achievement had been improving before 2003 with the states in charge.

NCLB slowed the nation’s math achievement gains. While the states adding “accountability” anew accelerated their own math achievement, as Dee asserts, the other states decelerated their own math achievement gains at an even greater rate. NCLB nudged one set of schools while stifling another set with extra testing they did not need.

When Congress passed the 2002 No Child Left Behind (NCLB) Act with its testing requirement over 7 grades and 3 subjects most U.S. states had just spent many years developing content standards and standards-based tests for most core subjects. Some states were caught in the middle of the process when the NCLB Act passed. There was little, if any, federal consideration for how the NCLB testing structure would mesh (or not) with preexisting state testing structures.

Moreover, contrary to what Dee and Jacob assert, the NCLB tests differed substantially from most states’, except in Texas, whose idiosyncratic testing program served loosely as the NCLB model. Most contemporary state tests focused on student performance, often with real consequences for individual students, such as grade promotion (or not), graduation (or not), and access to special programs (or not). By contrast, the NCLB tests focused on schools, rewarding or punishing them based on their average test-score trends. Students faced no individual consequences, or “stakes” with the NCLB tests. Neither did most teachers, for that matter.

The NCLB tests were also administered more frequently—every year from Grades 3 through 8—whereas most states administered their laboriously developed tests at only one or two transition years of students’ academic careers, such as the end or beginning of a level of education (e.g., primary, middle, high).

The federal government required administration of the NCLB tests in seven grade levels from the early 2000s and still does now, over two decades later, via its annual reporting requirement. During the Great Recession (~2008–2011) when many state budgets necessitated some triage among their expensive and often ill-fitting assortment of tests, only the state-designed tests could legally be excised, regardless of relative quality or state academic needs. Ultimately, the NCLB tests did not increase the amount of testing so much as crowd out the typically better aligned state-designed tests.

Over a century’s worth of research studies inform us that three testing characteristics on average can strongly increase student achievement: the action of testing itself (the “testing effect”), student stakes, and feedback to the students. The NCLB tests would reprise the first factor but ignore the latter two, which had been commonly present among preexisting state tests, foreseeably diminishing testing’s learning benefits. Most states phased out their own state-designed testing programs by the early to mid-2010s, coincident with the start of the long U.S. student achievement decline.

Still more problems with the Dee and Jacob analysis can be found itemized in a Brookings Institution publication which published fellow economists’ critiques along with their article. The critiques label the study “a creative and valiant, if not completely successful” effort, but mostly assert omitted variable bias and other fatal flaws. With no encouragement or support, many researchers would have left well enough alone—a valiant effort for inconclusive results.

Undeterred, Dee and Jacob disseminated their study, and their claim that NCLB increased math achievement, far and wide. One can find the full study, or an excerpt, or a summary at the journals Educational Evaluation and Policy Analysis, Policy Analysis and Management, Education Next, The 74, and for the Russell Sage Foundation; at websites for the National Bureau for Economic Research, the Brookings Institution, the Thomas B. Fordham Institute, the University of Michigan, the Urban Institute, and the American Economic Association; and in ERIC, GitHub, ResearchGate, SciSpace, Semantic Scholar, RePEc, jstor; … and elsewhere.

Such wide dissemination of such marginal evidence is made possible by the group cohesion of the Education Reform Citation Cartel, and lots of money and in-kind services from the authors’ universities, the federal government, and some wealthy foundations. Yet, the economists’ core analysis lies hidden behind mathematics and jargon impenetrable to 99% of the population, including those journalists and policymakers who accept the Cartel’s work as incontestable dogma.

Journalists and policymakers embrace blind faith in something they do not understand. Warren Buffet is famous for saying “Investment must be rational; if you can’t understand it, you can’t be rational about it.” He became astonishingly rich, and well-respected. Would that policymakers and journalists follow his advice and muster a reasonable amount of skepticism toward what celebrity scholars tell them.

Is this how we should make education policy? Policy by prestige bias and saturation of the information dissemination channels? A tight clique of well-endowed scholars overwhelms research dissemination venues and platforms with their work, no matter how weak and obscure, while at best ignoring and at worst dismissing, rival research and evidence from the many thousands outside of their clique?

Thomas Dee would pair with another economist co-author, James Wyckoff, to claim that Washington DC’s “more rigorous approach” to teacher accountability works well, too in “A Lasting Impact: High-stakes teacher evaluations drive student success in Washington, D.C.”

Criticisms of this study come in two flavors: its tautological structure and the vapidness of the value-added measure (VAM) used for the IMPACT score.

1. Tautology.
The Dee-Wyckoff study is largely tautological. Teachers who score low on IMPACT are more likely to leave the DC Public Schools. They are more likely to leave because a low IMPACT score lowers their career prospects inside DCPS. That’s true whether the IMPACT score is valid or not, or fair or not. The study is widely interpreted as validating the IMPACT program—that IMPACT scores genuinely measure teacher quality—when it does no such thing.

2. Value-added measurement.
Some years ago, the Washington Post published a map of the District of Columbia shaded by average teacher scores on the District’s VAM, IMPACT. The shadings were highly correlated geographically with socioeconomic status. Did this mean that the most effective teachers just by coincidence taught in the wealthier parts of town?

Imagine yourself as a District teacher in the poorest part of town. If your students arrive two grade levels behind, they will likely learn little during your year of trying to teach them the current grade-level content. You can choose to teach them “where they are,” the subject matter content that they should have learned in earlier years. They may learn more that way, but they will not do any better on the VAM tests, which cover current year content.

Moreover, too many VAM models use a previous year’s scores as baseline that are, at least three months old before the new teacher even meets a student. Teachers can be judged for a student’s year of learning despite being in the same room with that student for less than five percent of the student’s year.

Then there is the matter of the test content. If you are an elementary school teacher expected to teach writing, social studies, and science, but only reading and math are tested, you face a moral conflict. Do you teach all the subject matter, as you are legally and ethically obligated to do? If you do, you may end up having your VAM score compared to that of another teacher who ditched the other subjects to goose her kid’s math and reading scores.

Finally, the tests do not “count” for the student, who may or may not exert an effort to perform well. Many studies of student effort on “no stakes” tests reveal that motivation varies by a variety of factors, such as age, gender, ethnicity, and question type. More factors over which a teacher has no control.

VAM models like DC’s IMPACT make little common sense. Moreover, no less an authority than the American Statistical Association advises against using VAM. Were VAM to be widely accepted, however, economists would enjoy a windfall of contract work conducting VAM studies.

Dee and Wyckoff disseminated their paper widely. One can find the full study, or an excerpt, or a summary at the journals Educational Evaluation and Policy Analysis, Policy Analysis and Management, Education Finance and Policy, Education Next; at the newspapers Education Week, EdSource, Washington Post; at university websites at Stanford, Brown, UVA, Stanford; at think tanks the Hoover Institution, Brookings Institution, American Enterprise Institute, Cato Institute, CALDER Center, National Bureau of Economic Research; at the advocacy organizations Fordham Institute, National Council on Teacher Quality, FutureEd, TNTP; at the research aggregators RePEc, jstor, Semantic Scholar, Research Gate, GitHub, The Free Library, SSRN; and elsewhere.

Coincidentally, another group of economists from outside the Education Reform Citation Cartel published a paper also at the National Bureau of Economic Research, which found mostly statistical noise in VAM scores. In their study, VAM scores were no better at predicting student outcomes than were a student height measure, another factor over which a teacher has no control. Their article was published or mentioned in only several venues, far fewer than Dee and Wycoff’s.

Policymakers and journalists mislead us when they talk only to those they perceive to be at the top of the status order. Indeed, often, those elite researchers suggesting new reforms were also largely responsible for the old reforms that didn’t work. But they likely will not admit that to the policymakers and journalists to whom they promote themselves.

I once encouraged journalists to get more involved in covering education research and to pay more attention to economists’ work. I suppose that I’ve gotten my wish. We are now beneficiaries of deep coverage of education research from several national journalists. Deep, but not broad. Most of the journalists covering education research talk to the same small group of academic economists and political scientists.

There always exists rival research and rival evidence. But elite and career-strategic scholars may not tell you about it. Indeed, more than just ignore it, they may declare that other scholars and other evidence do not even exist. Or, if they acknowledge the other, they may characterize it as so dreadful it is not even worth citing. Instead, they encourage policymakers and journalists to join them inside their echo chamber of back scratching colleagues—their mutual admiration society, their clique, their citation cartel.

This entry was posted in Censorship, Education journalism, Education policy, Education Reform, information suppression, K-12, research ethics, Richard P. Phelps and tagged , , , , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *