Think Tank Thoughtlessness:

A Once-in-a-Generation Opportunity Squandered


by R.P. Phelps

Tweet



I have long argued that some of the best research on standardized testing, conducted by the world’s most knowledgeable testing experts, has been systematically suppressed in the United States by education insiders who control the main arterial routes of education research dissemination. For a few years now, however, another group of education researchers with an information dissemination capacity of comparable power—the Republican Party-aligned think tanks(1) (i.e., the Other Blob)—has taken an interest in testing policy. The potential for a long-overdue extension in the breadth of information on testing made generally available to the public emerged for a brief moment when the think tanks discovered the issue ...then quickly faded.


Another essay discusses the think tankers’ assertions of the nonexistence of most of the research literature on standardized testing, exaggerated claims to originality in their own work, and a propensity to censorship and suppression of information. This particular essay focuses on just one set among their erroneous research claims.

 

"In 1996, [our city] became one of the first large, urban school districts to implement high-stakes testing, introducing a comprehensive accountability program that incorporated incentives for both students and teachers.

 

"In 1996, [our city] began a national trend when it coupled a new school-level accountability program with an accountability initiative with high-stakes consequences for students. ...Over the past five years, virtually every major school system and many states ...have instituted elements of [our city's] policy." (2002;2)

 

“As the first large urban school district to introduce a comprehensive accountability system, [our city] provides an exceptional case study of the effects of high-stakes testing—a reform strategy that will become omnipresent as the No Child Left Behind Act is implemented nationwide.” (2003;3)


If these statements were true, then the authors' research in that city would have been mighty important. Pretty much anyone who has worked in the testing business for more than a few years, though, knows these claims are preposterous. But, they were published in a mainstream education journal for, ...well, perhaps for obvious reasons.


Their city, it turns out, administered an off-the-shelf norm-referenced test (NRT) in several grades for high-stakes, starting in the late 1990s (yes, of course, it was called a criterion-referenced test, wink, wink, nudge, nudge). Far from being a trend-setter, by using an NRT for high-stakes, they turned back the clock a few decades. Scarcely any jurisdictions use NRTs for high-stakes anymore, and none should. It is simply not fair (nor, perhaps, even legal) to attach consequences to a poor test performance when the test is not tightly aligned to a standardized curriculum.


But, even if the test in the authors’ city had been standards-based and well-aligned, it would hardly have been very original. Even before the state minimum competency era of the late 1970s and 1980s, many school districts, and some individual schools, particularly in the Southwest and West, used standardized tests in high-stakes accountability programs. In some cases, the tests were administered every year, at several or all grade levels, as grade promotion requirements.


During the state minimum competency era, a quarter of a century ago, many school districts purchased standardized tests to “fill in” the grade levels the state tests did not affect. The fill in tests were usually, though not always, purchased from the test publisher responsible for the state tests, making the across-grade-levels alignment easier. Indeed, the policy of local districts purchasing additional standardized tests to supplement state testing was probably more the rule than the exception.


There exists a unique data base of school-district testing programs for the academic year 1990-91, developed at the U.S. General Accounting Office. Their purpose was to determine the nationwide extent of standardized test use and its cost. Their questionnaires, however, were fairly detailed, and asked separately about each test’s stakes and grade levels. The survey achieved a 70 percent response rate—no small feat given the burden imposed on the respondents —from a 5 percent sample of U.S. school districts.


I looked through the data base for school districts that administered high-stakes tests in at least five grade levels. To disprove the claims of the think tankers, I needed to find only some school districts that administered high-stakes tests in several grades, several years before their district was “one of the first” and “began a national trend.” Of course, I found more—a dozen in New York State alone, and 24 total.


Number of annual standardized tests

Number of those tests that were high-stakes

State(s) of school district(s)

14

12

NY

11

at least 8

NY

10

at least 8

NY (3)

10

at least 7

NY

9

at least 7

NY

9

at least 6

NY (3)

9

5

NY

7

7

OH, TX, AL, GA, IL

6

5

IL

5

5

PA, VA, MN, IN, LA, IA, NY


And, this was in a 5 percent sample. Extrapolate twenty-fold, in order to represent 100 percent of U.S. school districts, and there would appear to have been about 500 U.S. school districts with high-stakes standardized testing in at least five grades in the academic year 1990-91, and half as many with high-stakes standardized testing in at least seven grades. And, this assumes that every survey respondent was willing to make the effort to fill out a separate, several-page questionnaire for each and every test they administered, as they were requested to do.


Number of those tests that were high-stakes

Number of school districts in GAO sample

Number of school districts in United States (estimate)

12

1

20

at least 8

4

80

at least 7

7

140

at least 6

3

60

5

9

180

 

 

 

Total

24

480


Extravagant claims of the Other Blob’s think tankers are shown, yet again, to be uninformed, self-promotional, and wrong. It makes one wonder ...is there any limit to what some of them are willing to claim? Does their braggadocio have any boundaries?


Their posturing would be funny if it were not so tragic. They possess massive marketing power and their reports, along with their claims, spread far, wide, and deep. They have repeatedly asserted over the course of several years now that most of the research literature on testing does not exist. This encourages people not to look for that work which, typically, is of better quality than theirs’. Theirs' may just be breast-beating assertions designed to make their work appear more important than it is, but the collateral damage they inflict on our society in the meantime must be substantial.


Several years ago, the GOP-aligned think tanks were presented an historic opportunity with enormous implications to benefit U.S. education. They had the resources to blast open the seal of censorship the vested interests have used to hide a huge research literature on standardized testing’s benefits. Instead, the think tankers have chosen to reinforce the censorial efforts of testing opponents, despite the critical need of their Republican politician clients for exactly the opposite behavior. The Other Blob researchers did this, apparently, for no more reason than.... Well, you guess what the reason is.


To some extent, the think tankers’ limited field of vision may be induced by academic field myopia. For whatever reason, the Other Blob relies almost exclusively on economists for education research advice. Rummaging through the first eight or nine decades and tens of thousands of research studies on standardized testing, one will find few conducted by economists, but tens of thousands conducted by psychologists, program evaluators, and education practitioners.


Just in the past few years, however, a small group of academic economists has become highly incentivized by the Republicans to conduct research on the topic. And, lo and behold, these Physicists of the Social Sciences now declare that scholarly research on standardized testing began only a few years ago, coincident to the moment when they got involved.


One of these scholars seems to be crafting a career out of conducting studies on topics that hundreds of psychologists (but few economists) already have studied to death. He broadcasts his results to the world through the Other Blob’s dissemination network touted as original treatments of important topics, and deposits them in the archives of the National Bureau of Economic Research as such.


Personally, I don’t believe that academic discipline myopia is a valid excuse for not knowing the research literature on a topic, particularly in a field that so obviously has been dominated by other academic disciplines for decades. Regardless, the scholar in question also was informed directly, by me, years ago that a huge research literature existed that he was ignoring. He voluntarily chose to continue to ignore it. He also chose to continue to declare his research to be the first on whatever topic he chose to work on.


If these policy wonks are successful, the likely outcome will be that when the NCLB Act starts fading into obscurity and the think tankers turn their attention to the newest hot topic, they will leave behind a public understanding of standardized testing poorer even than that which existed when they joined the issue a few years ago, which was highly skewed by vested interest censorship. But, for the few of them, curriculum vitae will be full to bursting with claims of alleged testing expertise and pioneering research work.


________________________


1. These would be the American Enterprise Institute, the Brookings Institution, Education Sector, the Thomas P. Fordham Foundation, the Manhattan Institute, the Program on Education Policy & Governance at Harvard University, and three groups at Stanford University (the Hoover Institution, the Koret Task Force, and CREDO).


2. Roderick, Melissa, Brian Jacob and Anthony Bryk. 2002. “The impact of high-stakes testing in Chicago on student achievement in the promotional gate grades.” Educational Evaluation and Policy Analysis, 24(4):333-57.


3. Jacob, Brian A. 2003. “High Stakes in Chicago” Education Next. Winter, p.66.