HOME:  Dismissive Reviews in Education Policy Research
	Author	Co-author(s)	Dismissive Quote	type	Title	Source	Link1
1	Laura S. Hamilton		"Despite the widespread enthusiasm for assessment-based reforms, many of the current and proposed uses of large-scale assessments are based on unverified assumptions about the extent to which they will actually lead to improved teaching and learning, and insufficient attention has been paid to the characteristics of assessment programs that are likely to promote desired outcomes."	Denigrating	Testing What Has Been Taught, p.47	American Educator, Winter 2010-2011
2	Laura S. Hamilton		"Can assessments meaningfully be aligned to standards, … What would the key features of an assessment system designed to increase student learning and improve instruction be? While current assessment knowledge is not sufficient to fully answer these questions, in this article I offer an overview of what is known and several suggestions for improving our approach to assessment."	Denigrating	Testing What Has Been Taught, p.47	American Educator, Winter 2010-2011
3	Laura S. Hamilton		"There is no research evidence to tell us definitively how to build an assessment system that will promote student learning and be resistent to the negative consequences that are common in high-stakes testing programs."	Dismissive	Testing What Has Been Taught, p.49	American Educator, Winter 2010-2011
4	Laura S. Hamilton		“Research on the effects of various assessment-design features is limited, so any effort that relies heavily on assessment as a tool for school improvement should be carried out with caution."	Denigrating	Testing What Has Been Taught, p.50	American Educator, Winter 2010-2011
5	Laura S. Hamilton	Brian M. Stecher, Kun Yuan	“A few studies have attempted to examine how the creation and publication of standards, per se, have affected practices.” p. 3	Dismissive	Standards-Based Reform in the United States: History, Research, and Future Directions	Center on Education Policy, December, 2008	http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf
6	Laura S. Hamilton	Brian M. Stecher, Kun Yuan	“The research evidence does not provide definitive answers to these questions.” p. 6	Denigrating	Standards-Based Reform in the United States: History, Research, and Future Directions	Center on Education Policy, December, 2008	http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf
7	Laura S. Hamilton	Brian M. Stecher, Kun Yuan	“He [Poynter 1994] also noted that ‘virtually all of the arguments, both for and against standards, are based on beliefs and hypotheses rather than on direct empirical evidence’ (p. 427). Although a large and growing body of research has been conducted to examine the effects of SBR, the caution Porte rexpressed in 1994 about the lack of empirical evidence remains relevant today.” pp. 34-35	Dismissive	Standards-Based Reform in the United States: History, Research, and Future Directions	Center on Education Policy, December, 2008	http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf
8	Laura S. Hamilton	Brian M. Stecher, Kun Yuan	“Arguably the most important test of quality is whether the standards promote high-quality instruction and improved student learning, but as we discuss later, there is very little research to address that question.” p. 37	Dismissive	Standards-Based Reform in the United States: History, Research, and Future Directions	Center on Education Policy, December, 2008	http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf
9	Laura S. Hamilton	Brian M. Stecher, Kun Yuan	“[T]here have been a few studies of SBR as a comprehensive system. . . . [T]here is some research on how the adoption of standards, per se, or the alignment of standards with curriculum influences school practices or student outcomes.” p. 38	Dismissive	Standards-Based Reform in the United States: History, Research, and Future Directions	Center on Education Policy, December, 2008	http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf
10	Laura S. Hamilton	Brian M. Stecher, Kun Yuan	“The lack of evidence about the effects of SBR derives primarily from the fact that the vision has never been fully realized in practice.” p. 47	Dismissive	Standards-Based Reform in the United States: History, Research, and Future Directions	Center on Education Policy, December, 2008	http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf
11	Laura S. Hamilton	Brian M. Stecher, Kun Yuan	“[A]lthough many conceptions of SBR emphasize autonomy, we currently know relatively little about the effects of granting autonomy or what the right balance is between autonomy and prescriptiveness.” p. 55	Dismissive	Standards-Based Reform in the United States: History, Research, and Future Directions	Center on Education Policy, December, 2008	http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf
12	Laura S. Hamilton	Brian M. Stecher, Kun Yuan	“One of the primary responsibilities of the federal government should be to ensure ongoing collection of evidence demonstrating the effects of the policies, which could be used to make decisions about whether to continue on the current course or whether small adjustments or a major overhaul are needed.” p. 55	Dismissive	Standards-Based Reform in the United States: History, Research, and Future Directions	Center on Education Policy, December, 2008	http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf
13	Laura S. Hamilton	Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz	“The shortcomings of the studies make it difficult to determine the size of teacher effects, but we suspect that the magnitude of some of the effects reported in this literature are overstated.” p. xiii	Denigrating	Evaluating Value-Added Models for Teacher Accountability	Rand Corporation, 2003	https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf
14	Laura S. Hamilton	Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz	“Using VAM to estimate individual teacher effects is a recent endeavor, and many of the possible sources of error have not been thoroughly evaluated in the literature.” p. xix	Dismissive	Evaluating Value-Added Models for Teacher Accountability	Rand Corporation, 2003	https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf
15	Laura S. Hamilton	Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz	“Empirical evaluations do not exist for many of the potential sources of error we have identified. Studies need to be conducted to determine how these factors contribute to estimated teacher effects and to determine the conditions that exacerbate or mitigate the impact these factors have on teacher effects.” p. xix	Dismissive	Evaluating Value-Added Models for Teacher Accountability	Rand Corporation, 2003	https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf
16	Laura S. Hamilton	Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz	“This lack of attention to teachers in policy discussions may be attributed in part to another body of literature that attempted to determine the effects of specific teacher background characteristics, including credentialing status (e.g., Miller, McKenna, and McKenna, 1998; Goldhaber and Brewer, 2000) and subject matter coursework (e.g., Monk, 1994).” p. 8	Dismissive	Evaluating Value-Added Models for Teacher Accountability	Rand Corporation, 2003	https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf
17	Laura S. Hamilton	Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz	“To date, there has been little empirical exploration of the size of school effects and the sensitivity of teacher effects to modeling of school effects.” p. 78	Dismissive	Evaluating Value-Added Models for Teacher Accountability	Rand Corporation, 2003	https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf
18	Laura S. Hamilton	Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz	“There are no empirical explorations of the robustness of estimates to assumptions about prior-year schooling effects.“ p. 81	Dismissive	Evaluating Value-Added Models for Teacher Accountability	Rand Corporation, 2003	https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf
19	Laura S. Hamilton	Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz	“There is currently no empirical evidence about the sensitivity of gain scores or teacher effects to such alternatives.” p. 89	Dismissive	Evaluating Value-Added Models for Teacher Accountability	Rand Corporation, 2003	https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf
20	Laura S. Hamilton	Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz	“Empirical evaluations do not exist for many of the potential sources of error we have identified. Studies need to be conducted to determine how these factors contribute to estimated teacher effects and to determine the conditions that exacerbate or mitigate the impact these factors have on teacher effects.” p. 116	Dismissive	Evaluating Value-Added Models for Teacher Accountability	Rand Corporation, 2003	https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf
21	Laura S. Hamilton	Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz	“Although we expect missing data are likely to be pervasive, there is little systematic discussion of the extent or nature of missing data in test score databases.” p. 117	Dismissive	Evaluating Value-Added Models for Teacher Accountability	Rand Corporation, 2003	https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf
22	Brian M. Stecher	Laura S. Hamilton	"The business model of setting clear targets, attaching incentives to the attainment of those targets, and rewarding those responsible for reaching the targets has proven successful in a wide range of business enterprises. But there is no evidence that these accountability principles will work well in an educational context, and there are many reasons to doubt that the principles can be applied without significant adaptation."	Dismissive	Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable	Rand Review, Spring 2002	https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html
23	Brian M. Stecher	Laura S. Hamilton	" The lack of strong evidence regarding the design and effectiveness of accountability systems hampers policymaking at a critical juncture."	Denigrating	Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable	Rand Review, Spring 2002	https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html
24	Brian M. Stecher	Laura S. Hamilton	"Nonetheless, the evidence has yet to justify the expectations. The initial evidence is, at best, mixed. On the plus side, students and teachers seem to respond to the incentives created by the accountability systems	Dismissive	Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable	Rand Review, Spring 2002	https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html
25	Brian M. Stecher	Laura S. Hamilton	"Proponents of accountability attribute the improved scores in these states to clearer expectations, greater motivation on the part of the students and teachers, a focused curriculum, and more-effective instruction. However, there is little or no research to substantiate these positive changes or their effects on scores."	Dismissive	Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable	Rand Review, Spring 2002	https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html
26	Brian M. Stecher	Laura S. Hamilton	"One of the earliest studies on the effects of testing (conducted in two Arizona schools in the late 1980s) showed that teachers reduced their emphasis on important, nontested material."	Dismissive	Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable	Rand Review, Spring 2002	https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html
27	Brian M. Stecher	Laura S. Hamilton	"Test-based accountability systems will work better if we acknowledge how little we know about them, if the federal government devotes appropriate resources to studying them, and if the states make ongoing efforts to improve them."	Dismissive	Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable	Rand Review, Spring 2002	https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html
28	Brian M. Stecher	Laura S. Hamilton, Stephen P. Klein, Eds.	"High-stakes testing may also affect parents (e.g., their attitudes toward education, their engagement with schools, and their direct participation in their child's learning) as well as policymakers (their beliefs about system performance, their judgements about program effectiveness, and their allocation of resources). However, these issues remain largely unexamined in the literature."	Dismissive	Consequences of large-scale, high-stakes testing on school and classroom practice	Chapter 4 in Making sense of test-based accountability in education, 2002, p.79	https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf
29	Brian M. Stecher	Laura S. Hamilton, Stephen P. Klein, Eds.	"As described in chapter 2, there was little concern about the effects of testing on teaching prior to the 1970s."	Dismissive	Consequences of large-scale, high-stakes testing on school and classroom practice	Chapter 4 in Making sense of test-based accountability in education, 2002, p.81	https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf
30	Brian M. Stecher	Laura S. Hamilton, Stephen P. Klein, Eds.	"In light of the changes that occurred in the uses of large-scale testing in the 1980s and 1990s, researchers began to investigate teachers' reactions to external assessment. The initial research on the impact of large-scale testing was conducted in the 1980s and the 1990s."	Dismissive	Consequences of large-scale, high-stakes testing on school and classroom practice	Chapter 4 in Making sense of test-based accountability in education, 2002, p.83	https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf
31	Brian M. Stecher	Laura S. Hamilton, Stephen P. Klein, Eds.	"Researchers have not documented the desirable consequences of testing … as clearly as the undesirable ones. More importantly, researchers have not generally measured the extent or magnitude of the shifts in practice that they identified as a result of high-stakes testing."	Dismissive	Consequences of large-scale, high-stakes testing on school and classroom practice	Chapter 4 in Making sense of test-based accountability in education, 2002, pp.99–100	https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf
32	Laura S. Hamilton	Brian M. Stecher, Stephen P. Klein	"Although test-based accountability has shown some compelling results, the issues are complex, the research is new and incomplete, and many of the claims that have received the most attention have proved to be premature and superficial."	Denigrating	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Summary, p.xiv
33	Laura S. Hamilton	Brian M. Stecher, Stephen P. Klein	"The research evidence does not provide definitive information about the actual costs of testing but the information that is available suggests that expenditures for testing have grown in recent years."	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Introduction, p.9
34	Laura S. Hamilton	Daniel M. Koretz	"There is currently no substantial evidence on the effects of published report cards on parents’ decisionmaking or on the schools themselves."	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 2: Tests and their use in test-based accountability systems, p.44
35	Laura S. Hamilton	Brian M. Stecher	"So test-based accountability remains controversial because there is inadequate evidence to make clear judgments about its effectiveness in raising test scores and achieving its other goals."	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.122
36	Laura S. Hamilton	Brian M. Stecher	"Unfortunately, the complexity of the issues and the ambiguity of the existing research do not allow our recommendations to take the form of a practical “how-to” guide for policymakers and practitioners."	Denigrating	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.123
37	Laura S. Hamilton	Brian M. Stecher	"Another part of the interpretive question is the need to gather information in other subject areas to portray a more complete picture of achievement. The scope of constructs that have been considered in research to date has been fairly narrow, focusing on the subjects that are part of the accountability systems that have been studied. Many legitimate instructional objectives have been ignored in the literature to date."	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.127
38	Laura S. Hamilton	Brian M. Stecher	"States should also conduct ongoing analyses of the performance of groups whose members may not be numerous enough to permit separate reporting. English-language learners and students with disabilities are increasingly being included in high-stakes testing systems, and, as discussed in Chapter Three, little is currently known about the validity of scores for these groups."	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.131
39	Laura S. Hamilton	Brian M. Stecher	"It would be especially helpful to know what changes in instruction are made in response to different kinds of information and incentives. In particular, we need to know how teachers interpret information from tests and how they use it to modify instruction."	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.133
40	Laura S. Hamilton	Brian M. Stecher	It seems clear that aligning the components of the system and providing appropriate professional development should, at a minimum, increase teachers’ political support for test-based accountability policies .... Although there is no empirical evidence to suggest that this strategy will reduce inappropriate responses to high-stakes testing,... Additional research needs to be done to determine the importance of alignment for promoting positive effects of test-based accountability.	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.135
41	Laura S. Hamilton	Brian M. Stecher	"… we currently do not know enough about test-based accountability to design a system that is immune from the problems we have discussed	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.136
42	Laura S. Hamilton	Brian M. Stecher	"There is some limited evidence that educators’ responses to test based accountability vary according to the characteristics of their student populations,…"	Denigrating	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.138
43	Laura S. Hamilton	Brian M. Stecher	"... there is very limited evidence to guide thinking about political issues."	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.139
44	Laura S. Hamilton	Brian M. Stecher	"First, we do not have an accurate assessment of the additional costs."	Denigrating	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.141
45	Laura S. Hamilton	Brian M. Stecher	"Part of the reason these issues are rarely considered may be that no one has produced a good estimate of the cost of an improved accountability system in comparison with its benefits."	Denigrating	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.141
46	Laura S. Hamilton	Brian M. Stecher	"Nevertheless, our knowledge of the costs of alternative accountability systems is still somewhat limited. Policymakers need to know how much it would cost to change their current systems to be responsive to criticisms such as those described in this book. These estimates need to consider all of the associated costs, including possible opportunity costs associated with increased testing time and increased test preparation time."	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.142
47	Laura S. Hamilton	Brian M. Stecher	"However, there is still much about these systems that is not well understood. Lack of research-based knowledge about the quality of scores and the mechanisms through which high-stakes testing programs operate limits our ability to improve these systems. As a result, our discussions also identified unanswered questions..."	Dismissive	Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6: Improving test-based accountability, p.143
48	Daniel M. Koretz	Daniel F. McCaffrey, Laura S. Hamilton	"Few efforts are made to evaluate directly score gains obtained under high-stakes conditions, and conventional validation tools are not fully adequate for the task.", p. 1	Dismissive	Toward a framework for validating gains under high-stakes conditions	CSE Technical Report 551, CRESST/Harvard Graduate School of Education, CRESST/RAND Education, December 2001
49	Laura S. Hamilton		“Despite the number of studies investigating affective aspects of test taking, little is known about how students perceive the kinds of extended performance assessments currently being developed for state and local testing programs.” - Abstract	Denigrating	An Investigation of Students' Affective Responses to Alternative Assessment Formats	Paper presented at the Annual Meeting of the National Council on Measurement in Education (New Orleans, LA, April 5-7, 1994)	http://files.eric.ed.gov/fulltext/ED376203.pdf
50	Laura S. Hamilton		“As stated earlier, this study was not intended to produce results that could be generalized to other tasks or to other samples of students, but to identify questions that might be addressed by future studies and to suggest possible hypotheses.” p. 23	Dismissive	An Investigation of Students' Affective Responses to Alternative Assessment Formats	Paper presented at the Annual Meeting of the National Council on Measurement in Education (New Orleans, LA, April 5-7, 1994)	http://files.eric.ed.gov/fulltext/ED376203.pdf
	IRONIES:
	Laura S. Hamilton	Brian M. Stecher, Stephen P. Klein	"Greater knowledge about testing and accountability can lead to better system design and more-effective system management." p.xiv		Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Summary, p.xiv
	Laura S. Hamilton	Brian M. Stecher	"Incremental improvements to existing systems, based on current research on testing and accountability, should be combined with long-term research and development efforts that may ultimately lead to a major redesign of these systems. Success in this endeavor will require the thoughtful engagement of educators, policymakers, and researchers in discussions and debates about tests and testing policies."		Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002	Chapter 6, Improving test-based accountability, pp.143-144
			Cite selves or colleagues in the group, but dismiss or denigrate all other work
			Falsely claim that research has only recently been done on topic.