| HOME: Dismissive Reviews in Education Policy Research | |||||||||
| Author | Co-author(s) | Dismissive Quote | type | Title | Source | Link1 | Funders | Notes | |
| 1 | John F. Pane | "Practitioners and policymakers seeking to implement personalized learning, lacking clearly defined evidence-based models to adopt, are creating custom designs for their specific contexts. Those who want to use rigorous research evidence to guide their designs will find many gaps and will be left with important unanswered questions about which practices or combinations of practices are effective. It will likely take many years of research to fill these gaps". | Dismissive | Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped, p.1 | Rand Corporation Perspective, October 2018 | https://www.rand.org/pubs/perspectives/PE314.html | Funded by the William and Flora Hewlett Foundation, UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is monitoring the extent to which the two consortia’s assessment development efforts are likely to produce tests that measure and support goals for deeper learning. And, Rand Corporation funders | Pane devotes considerable text to claims that no prior research exists, except for another Rand study, and then, on p.7 admits that there exist some relevant mastery learning studies from the 1980s. He implies, however, that there were only one or a few. In fact, there were hundreds. There have also been thousands of studies of personalized instruction in conjunction with studies in special education, tutoring, teachers' aides, tracking, etc. | |
| 2 | John F. Pane | "The purpose of this Perspective is to offer strategic guidance for designers of personalized learning programs to consider while the evidence base is catching up." | Dismissive | Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped, p.1 | Rand Corporation Perspective, October 2018 | https://www.rand.org/pubs/perspectives/PE314.html | Funded by the William and Flora Hewlett Foundation, UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is monitoring the extent to which the two consortia’s assessment development efforts are likely to produce tests that measure and support goals for deeper learning. And, Rand Corporation funders | Pane devotes considerable text to claims that no prior research exists, except for another Rand study, and then, on p.7 admits that there exist some relevant mastery learning studies from the 1980s. He implies, however, that there were only one or a few. In fact, there were hundreds. There have also been thousands of studies of personalized instruction in conjunction with studies in special education, tutoring, teachers' aides, tracking, etc. | |
| 3 | John F. Pane | "This guidance draws on theory, basic principles of learning science, and the limited research that does exist on personalized learning and its component parts." | Dismissive | Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped, p.1 | Rand Corporation Perspective, October 2018 | https://www.rand.org/pubs/perspectives/PE314.html | Funded by the William and Flora Hewlett Foundation, UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is monitoring the extent to which the two consortia’s assessment development efforts are likely to produce tests that measure and support goals for deeper learning. And, Rand Corporation funders | Pane devotes considerable text to claims that no prior research exists, except for another Rand study, and then, on p.7 admits that there exist some relevant mastery learning studies from the 1980s. He implies, however, that there were only one or a few. In fact, there were hundreds. There have also been thousands of studies of personalized instruction in conjunction with studies in special education, tutoring, teachers' aides, tracking, etc. | |
| 4 | John F. Pane | "Thus far, the research evidence on personalized learning as an overarching schoolwide model is sparse." | Dismissive | Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped, p.4 | Rand Corporation Perspective, October 2018 | https://www.rand.org/pubs/perspectives/PE314.html | Funded by the William and Flora Hewlett Foundation, UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is monitoring the extent to which the two consortia’s assessment development efforts are likely to produce tests that measure and support goals for deeper learning. And, Rand Corporation funders | Pane devotes considerable text to claims that no prior research exists, except for another Rand study, and then, on p.7 admits that there exist some relevant mastery learning studies from the 1980s. He implies, however, that there were only one or a few. In fact, there were hundreds. There have also been thousands of studies of personalized instruction in conjunction with studies in special education, tutoring, teachers' aides, tracking, etc. | |
| 5 | John F. Pane | "A team of RAND Corporation researchers conducted the largest and most-rigorous studies of student achievement effects to date." | 1stness | Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped, p.4 | Rand Corporation Perspective, October 2018 | https://www.rand.org/pubs/perspectives/PE314.html | Funded by the William and Flora Hewlett Foundation, UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is monitoring the extent to which the two consortia’s assessment development efforts are likely to produce tests that measure and support goals for deeper learning. And, Rand Corporation funders | Pane devotes considerable text to claims that no prior research exists, except for another Rand study, and then, on p.7 admits that there exist some relevant mastery learning studies from the 1980s. He implies, however, that there were only one or a few. In fact, there were hundreds. There have also been thousands of studies of personalized instruction in conjunction with studies in special education, tutoring, teachers' aides, tracking, etc. | |
| 6 | John F. Pane | "While we await the answers to those questions, substantial enthusiasm around personalized learning persists. Educators, policy makers, and advocates are moving forward without the guidance of conclusive research evidence." | Dismissive | Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped, p.5 | Rand Corporation Perspective, October 2018 | https://www.rand.org/pubs/perspectives/PE314.html | Funded by the William and Flora Hewlett Foundation, UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is monitoring the extent to which the two consortia’s assessment development efforts are likely to produce tests that measure and support goals for deeper learning. And, Rand Corporation funders | Pane devotes considerable text to claims that no prior research exists, except for another Rand study, and then, on p.7 admits that there exist some relevant mastery learning studies from the 1980s. He implies, however, that there were only one or a few. In fact, there were hundreds. There have also been thousands of studies of personalized instruction in conjunction with studies in special education, tutoring, teachers' aides, tracking, etc. | |
| 7 | John F. Pane | "In the absence of comprehensive, rigorous evidence to help select the personalized learning components most likely to succeed, what is the path forward? I suggest a few guiding principles aimed at using existing scientific knowledge and the best available resources." | Denigrating | Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped, p.5 | Rand Corporation Perspective, October 2018 | https://www.rand.org/pubs/perspectives/PE314.html | Funded by the William and Flora Hewlett Foundation, UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is monitoring the extent to which the two consortia’s assessment development efforts are likely to produce tests that measure and support goals for deeper learning. And, Rand Corporation funders | Pane devotes considerable text to claims that no prior research exists, except for another Rand study, and then, on p.7 admits that there exist some relevant mastery learning studies from the 1980s. He implies, however, that there were only one or a few. In fact, there were hundreds. There have also been thousands of studies of personalized instruction in conjunction with studies in special education, tutoring, teachers' aides, tracking, etc. | |
| 8 | John F. Pane | "However, more work is necessary to establish causal evidence that the concept leads to improved outcomes for students" | Dismissive | Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped, p.9 | Rand Corporation Perspective, October 2018 | https://www.rand.org/pubs/perspectives/PE314.html | Funded by the William and Flora Hewlett Foundation, UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is monitoring the extent to which the two consortia’s assessment development efforts are likely to produce tests that measure and support goals for deeper learning. And, Rand Corporation funders | Pane devotes considerable text to claims that no prior research exists, except for another Rand study, and then, on p.7 admits that there exist some relevant mastery learning studies from the 1980s. He implies, however, that there were only one or a few. In fact, there were hundreds. There have also been thousands of studies of personalized instruction in conjunction with studies in special education, tutoring, teachers' aides, tracking, etc. | |
| 9 | Jennifer
L. Steele, Matthew W. Lewis, Lucrecia Santibañez, Susannah Faxon-Mills, Mollie Rudnick, Brian M. Stecher, Laura S. Hamilton |
"Despite taking on considerable momentum in the field, competency-based systems have not been extensively researched." p.2 | Dismissive | Competency-Based Education in Three Pilot Programs Examining Implementation and Outcomes | Rand Education, 2014 | https://www.rand.org/content/dam/rand/pubs/research_reports/RR700/RR732/RAND_RR732.pdf | "The research described in this report was sponsored by the Bill & Melinda Gates Foundation" | ||
| 10 | Jennifer
L. Steele, Matthew W. Lewis, Lucrecia Santibañez, Susannah Faxon-Mills, Mollie Rudnick, Brian M. Stecher, Laura S. Hamilton |
"Recent studies have described the experiences of educators working to undertake competency-based reforms or have highlighted promising models, but these studies have not systematically examined the effects of these models on student learning or persistence." p.2 | Denigrating | Competency-Based Education in Three Pilot Programs Examining Implementation and Outcomes | Rand Education, 2014 | https://www.rand.org/content/dam/rand/pubs/research_reports/RR700/RR732/RAND_RR732.pdf | "The research described in this report was sponsored by the Bill & Melinda Gates Foundation" | ||
| 11 | Jennifer
L. Steele, Matthew W. Lewis, Lucrecia Santibañez, Susannah Faxon-Mills, Mollie Rudnick, Brian M. Stecher, Laura S. Hamilton |
"… there are no studies that would allow us to attribute outperformance to the competency-based education systems alone," p.2 | Dismissive | Competency-Based Education in Three Pilot Programs Examining Implementation and Outcomes | Rand Education, 2014 | https://www.rand.org/content/dam/rand/pubs/research_reports/RR700/RR732/RAND_RR732.pdf | "The research described in this report was sponsored by the Bill & Melinda Gates Foundation" | ||
| 12 | Jennifer
L. Steele, Matthew W. Lewis, Lucrecia Santibañez, Susannah Faxon-Mills, Mollie Rudnick, Brian M. Stecher, Laura S. Hamilton |
"Because it is one of the first studies we are aware of since the late 1980s that has attempted to estimate the impact of competency-based models on students’ academic outcomes," p.4 | 1stness | Competency-Based Education in Three Pilot Programs Examining Implementation and Outcomes | Rand Education, 2014 | https://www.rand.org/content/dam/rand/pubs/research_reports/RR700/RR732/RAND_RR732.pdf | "The research described in this report was sponsored by the Bill & Melinda Gates Foundation" | ||
| 13 | Jennifer
L. Steele, Matthew W. Lewis, Lucrecia Santibañez, Susannah Faxon-Mills, Mollie Rudnick, Brian M. Stecher, Laura S. Hamilton |
"In part, the lack of recent research on competency-based education may be due to variability around the concept of competency-based education itself." p.10 | Dismissive | Competency-Based Education in Three Pilot Programs Examining Implementation and Outcomes | Rand Education, 2014 | https://www.rand.org/content/dam/rand/pubs/research_reports/RR700/RR732/RAND_RR732.pdf | "The research described in this report was sponsored by the Bill & Melinda Gates Foundation" | ||
| 14 | Kun Yuan, Vi-Nhuan Le | "… there has been no systematic empirical examination of the extent to which other widely used achievement tests emphasize deeper learning." p.xi | Dismissive | Measuring
Deeper Learning Through Cognitively Demanding Test Items |
Rand Corporation Research Report, 2014 | https://www.rand.org/content/dam/rand/pubs/research_reports/RR400/RR483/RAND_RR483.pdf | "The research described in this report was sponsored by the William and Flora Hewlett Foundation" | ||
| 15 | Pete Wilmoth | "The increasing availability of computers and Internet access makes technology based education an enticing option, both inside and outside the classroom. However, school districts have adopted many such tools without compelling evidence that they are effective in improving student achievement." | Dismissive | Cognitive Tutor: Encouraging Signs for Computers in the Classroom | The RAND Blog, November 19, 2013 | "The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A070185 to the RAND Corporation." | |||
| 16 | Pete Wilmoth | "To help fill this evidence gap, a RAND research team … " | Dismissive | Cognitive Tutor: Encouraging Signs for Computers in the Classroom | The RAND Blog, November 19, 2013 | "The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A070185 to the RAND Corporation." | |||
| 17 | Pete Wilmoth | "As one of the first large-scale assessments of a blended learning approach, this study suggests promise for using technology to improve student achievement." | 1stness | Cognitive Tutor: Encouraging Signs for Computers in the Classroom | The RAND Blog, November 19, 2013 | "The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A070185 to the RAND Corporation." | |||
| 18 | John F. Pane, Beth Ann Griffin, Daniel F. McCaffrey, and Rita Karam, | "These tools allow self-paced instruction and provide students with customized feedback. These features, it is widely held, will improve student engagement and improve proficiency. However, evidence to support these claims remains scarce." p.2 | Dismissive | Does an Algebra Course with Tutoring Software Improve Student Learning? | Rand Corporation Brief, 2013 | "The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A070185 to the RAND Corporation." | |||
| 19 | John F. Pane, Beth Ann Griffin, Daniel F. McCaffrey, and Rita Karam, | "To make headway in addressing this knowledge gap, a team of RAND researchers …" p.3 | Dismissive | Does an Algebra Course with Tutoring Software Improve Student Learning? | Rand Corporation Brief, 2013 | "The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A070185 to the RAND Corporation." | |||
| 20 | Susannah Faxon-Mills, Laura S. Hamilton, Mollie Rudnick, Brian M. Stecher | "In particular, there is still much to learn about how changes in testing might influence the education system and how tests of deeper content and more complex skills and processes could best be used to promote the Foundation’s goals for deeper learning." p.1 | Dismissive | New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement | Rand Corporation Research Report, 2013 | "Funding to support the research was provided by the William and Flora Hewlett Foundation." "Marc Chun at the Hewlett Foundation first approached us about reviewing the literature on the impact of assessment, and he was very helpful in framing this investigation." | |||
| 21 | Susannah Faxon-Mills, Laura S. Hamilton, Mollie Rudnick, Brian M. Stecher | "Given the gaps in evidence regarding the link between testing and student outcomes … " p.1 | Dismissive | New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement | Rand Corporation Research Report, 2013 | "Funding to support the research was provided by the William and Flora Hewlett Foundation." "Marc Chun at the Hewlett Foundation first approached us about reviewing the literature on the impact of assessment, and he was very helpful in framing this investigation." | |||
| 22 | Susannah Faxon-Mills, Laura S. Hamilton, Mollie Rudnick, Brian M. Stecher | "The first step for each of these research areas was to identify relevant material from previous literature reviews on these topics, including those conducted by RAND researchers (e.g., Hamilton, Stecher, and Klein, 2002; Hamilton, 2003; Stecher, 2010) and by the National Research Council (e.g., Koenig, 2011). p.5 | Dismissive | New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement | Rand Corporation Research Report, 2013 | "Funding to support the research was provided by the William and Flora Hewlett Foundation." "Marc Chun at the Hewlett Foundation first approached us about reviewing the literature on the impact of assessment, and he was very helpful in framing this investigation." | |||
| 23 | Susannah Faxon-Mills, Laura S. Hamilton, Mollie Rudnick, Brian M. Stecher | "… we paid particular attention to sources from the past ten years, since these studies were less likely to have been included in previous literature reviews." p.5 | Dismissive | New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement | Rand Corporation Research Report, 2013 | "Funding to support the research was provided by the William and Flora Hewlett Foundation." "Marc Chun at the Hewlett Foundation first approached us about reviewing the literature on the impact of assessment, and he was very helpful in framing this investigation." | |||
| 24 | Susannah Faxon-Mills, Laura S. Hamilton, Mollie Rudnick, Brian M. Stecher | "Time and resource constraints limited the extent of our literature reviews, but we do not think this had a serious effect on our findings. Most importantly, we included all the clearly relevant studies from major sources that were available for electronic searching. In addition, many of the studies we reviewed also included comprehensive reviews of other literature, leading to fairly wide coverage of each body of literature." p.8 | Dismissive | New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement | Rand Corporation Research Report, 2013 | "Funding to support the research was provided by the William and Flora Hewlett Foundation." "Marc Chun at the Hewlett Foundation first approached us about reviewing the literature on the impact of assessment, and he was very helpful in framing this investigation." | |||
| 25 | Susannah Faxon-Mills, Laura S. Hamilton, Mollie Rudnick, Brian M. Stecher | "However, the amount of research on test attributes is limited, and the research has been conducted in a wide variety of contexts involving a wide variety of tests. Thus, while the findings are interesting, few have been replicated." p.22 | Dismissive | New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement | Rand Corporation Research Report, 2013 | "Funding to support the research was provided by the William and Flora Hewlett Foundation." "Marc Chun at the Hewlett Foundation first approached us about reviewing the literature on the impact of assessment, and he was very helpful in framing this investigation." | |||
| 26 | Susannah Faxon-Mills, Laura S. Hamilton, Mollie Rudnick, Brian M. Stecher | "It is important to recognize that the literature on how school characteristics, such as urbanicity and governance, affect educators’ responses to testing is sparse." p.29 | Dismissive | New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement | Rand Corporation Research Report, 2013 | "Funding to support the research was provided by the William and Flora Hewlett Foundation." "Marc Chun at the Hewlett Foundation first approached us about reviewing the literature on the impact of assessment, and he was very helpful in framing this investigation." | |||
| 27 | Susannah Faxon-Mills, Laura S. Hamilton, Mollie Rudnick, Brian M. Stecher | "… there is little empirical evidence that provides guidance on the amount and types of professional development that would promote constructive responses to assessment. | Dismissive | New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement | Rand Corporation Research Report, 2013 | "Funding to support the research was provided by the William and Flora Hewlett Foundation." "Marc Chun at the Hewlett Foundation first approached us about reviewing the literature on the impact of assessment, and he was very helpful in framing this investigation." | |||
| 28 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | "He also noted that virtually all of the arguments, both for and against standards, are based on beliefs and hypotheses rather than on direct empirical evidence” (p. 427) Although a large and growing body of research has been conducted to examine the effects of SBA, the caution Porter expressed in 1994 about the lack of empirical evidence remains relevant today." pp.157-158 | Denigrating | Standards-Based Accountability in the United States: Lessons Learned and Future Directions | Education Inquiry, 3(2), June 2012, 149-170 | https://www.academia.edu/15201890/Standards_Based_Accountability_in_the_United_States_Lessons_Learned_and_Future_Directions_1 | "Material in this paper has been adapted from a paper commissioned by the Center on Education Policy: Hamilton, L.S., Stecher, B.M., & Yuan, K. (2009) Standards-based Reform in the United States: History, Research, and Future Directions. Washington, DC: Center on Education Policy. Portions of this work were supported by the National Science Foundation under Grant No. REC-0228295." | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 29 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | "High-quality research on the effects of SBA is difficult to conduct for a number of reasons,…." p.158 | Dismissive | Standards-Based Accountability in the United States: Lessons Learned and Future Directions | Education Inquiry, 3(2), June 2012, 149-170 | https://www.academia.edu/15201890/Standards_Based_Accountability_in_the_United_States_Lessons_Learned_and_Future_Directions_1 | "Material in this paper has been adapted from a paper commissioned by the Center on Education Policy: Hamilton, L.S., Stecher, B.M., & Yuan, K. (2009) Standards-based Reform in the United States: History, Research, and Future Directions. Washington, DC: Center on Education Policy. Portions of this work were supported by the National Science Foundation under Grant No. REC-0228295." | Access to anonymized student data is granted all the time. Externally administered high-stakes testing is widely reviled among US educationists. It strains credulity that one can not find one or a few districts out of the many thousands to cooperate in a study to discredit testing. |
| 30 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | "Even when the necessary data have been collected by states or other entities, it is often difficult for researchers to obtain these data because those responsible for the data refuse to grant access, either because of concerns about confidentiality or because they are not interested in having their programmes scrutinised by. researchers. Thus, the amount of rigorous analysis is limited." p.158 | Dismissive | Standards-Based Accountability in the United States: Lessons Learned and Future Directions | Education Inquiry, 3(2), June 2012, 149-170 | https://www.academia.edu/15201890/Standards_Based_Accountability_in_the_United_States_Lessons_Learned_and_Future_Directions_1 | "Material in this paper has been adapted from a paper commissioned by the Center on Education Policy: Hamilton, L.S., Stecher, B.M., & Yuan, K. (2009) Standards-based Reform in the United States: History, Research, and Future Directions. Washington, DC: Center on Education Policy. Portions of this work were supported by the National Science Foundation under Grant No. REC-0228295." | Access to anonymized student data is granted all the time. Externally administered high-stakes testing is widely reviled among US educationists. It strains credulity that one can not find one or a few districts out of the many thousands to cooperate in a study to discredit testing. |
| 31 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | "These evaluation findings reveal the challenges inherent in trying to judge the quality of standards. Arguably the most important test of quality is whether the standards promote high-quality instruction and improved student learning but, as we discuss later, there is very little research to address that question." p.158 | Dismissive | Standards-Based Accountability in the United States: Lessons Learned and Future Directions | Education Inquiry, 3(2), June 2012, 149-170 | https://www.academia.edu/15201890/Standards_Based_Accountability_in_the_United_States_Lessons_Learned_and_Future_Directions_1 | "Material in this paper has been adapted from a paper commissioned by the Center on Education Policy: Hamilton, L.S., Stecher, B.M., & Yuan, K. (2009) Standards-based Reform in the United States: History, Research, and Future Directions. Washington, DC: Center on Education Policy. Portions of this work were supported by the National Science Foundation under Grant No. REC-0228295." | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 32 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | "In fact, the bulk of research relevant to SBA has focused on the links between high-stakes tests and educators’ practices rather than standards and practices." p.159 | Dismissive | Standards-Based Accountability in the United States: Lessons Learned and Future Directions | Education Inquiry, 3(2), June 2012, 149-170 | https://www.academia.edu/15201890/Standards_Based_Accountability_in_the_United_States_Lessons_Learned_and_Future_Directions_1 | "Material in this paper has been adapted from a paper commissioned by the Center on Education Policy: Hamilton, L.S., Stecher, B.M., & Yuan, K. (2009) Standards-based Reform in the United States: History, Research, and Future Directions. Washington, DC: Center on Education Policy. Portions of this work were supported by the National Science Foundation under Grant No. REC-0228295." | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 33 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | "The existing evidence does not provide definitive guidance regarding the SBA system features that would be most likely to promote desirable outcomes." p.163 | Dismissive | Standards-Based Accountability in the United States: Lessons Learned and Future Directions | Education Inquiry, 3(2), June 2012, 149-170 | https://www.academia.edu/15201890/Standards_Based_Accountability_in_the_United_States_Lessons_Learned_and_Future_Directions_1 | "Material in this paper has been adapted from a paper commissioned by the Center on Education Policy: Hamilton, L.S., Stecher, B.M., & Yuan, K. (2009) Standards-based Reform in the United States: History, Research, and Future Directions. Washington, DC: Center on Education Policy. Portions of this work were supported by the National Science Foundation under Grant No. REC-0228295." | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 34 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | “A few studies have attempted to examine how the creation and publication of standards, per se, have affected practices.” p. 3 | Dismissive | Standards-Based Reform in the United States: History, Research, and Future Directions | Center on Education Policy, December, 2008 | http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf | Institute of Education Sciences, US Education Department | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 35 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | “The research evidence does not provide definitive answers to these questions.” p. 6 | Denigrating | Standards-Based Reform in the United States: History, Research, and Future Directions | Center on Education Policy, December, 2008 | http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf | Institute of Education Sciences, US Education Department | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 36 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | “He [Poynter 1994] also noted that ‘virtually all of the arguments, both for and against standards, are based on beliefs and hypotheses rather than on direct empirical evidence’ (p. 427).” pp. 34-35 | Dismissive, Denigrating | Standards-Based Reform in the United States: History, Research, and Future Directions | Center on Education Policy, December, 2008 | http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf | Institute of Education Sciences, US Education Department | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 37 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | "Although a large and growing body of research has been conducted to examine the effects of SBR, the caution Poynter expressed in 1994 about the lack of empirical evidence remains relevant today.” pp. 34-35 | Dismissive | Standards-Based Reform in the United States: History, Research, and Future Directions | Center on Education Policy, December, 2008 | http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf | Institute of Education Sciences, US Education Department | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 38 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | “Arguably the most important test of quality is whether the standards promote high-quality instruction and improved student learning, but as we discuss later, there is very little research to address that question.” p. 37 | Dismissive | Standards-Based Reform in the United States: History, Research, and Future Directions | Center on Education Policy, December, 2008 | http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf | Institute of Education Sciences, US Education Department | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 39 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | “[T]here have been a few studies of SBR as a comprehensive system. . . . [T]here is some research on how the adoption of standards, per se, or the alignment of standards with curriculum influences school practices or student outcomes.” p. 38 | Dismissive | Standards-Based Reform in the United States: History, Research, and Future Directions | Center on Education Policy, December, 2008 | http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf | Institute of Education Sciences, US Education Department | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 40 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | “The lack of evidence about the effects of SBR derives primarily from the fact that the vision has never been fully realized in practice.” p. 47 | Dismissive | Standards-Based Reform in the United States: History, Research, and Future Directions | Center on Education Policy, December, 2008 | http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf | Institute of Education Sciences, US Education Department | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 41 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | “[A]lthough many conceptions of SBR emphasize autonomy, we currently know relatively little about the effects of granting autonomy or what the right balance is between autonomy and prescriptiveness.” p. 55 | Dismissive | Standards-Based Reform in the United States: History, Research, and Future Directions | Center on Education Policy, December, 2008 | http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf | Institute of Education Sciences, US Education Department | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 42 | Laura S. Hamilton | Brian M. Stecher, Kun Yuan | “One of the primary responsibilities of the federal government should be to ensure ongoing collection of evidence demonstrating the effects of the policies, which could be used to make decisions about whether to continue on the current course or whether small adjustments or a major overhaul are needed.” p. 55 | Dismissive | Standards-Based Reform in the United States: History, Research, and Future Directions | Center on Education Policy, December, 2008 | http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf | Institute of Education Sciences, US Education Department | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 43 | Laura S. Hamilton | Brian M. Stecher, Julie A. Marsh, Jennifer Sloan McCombs, Abby Robyn, Jennifer Lin Russell, Scott Naftel, Heather Barney | "For many educators, the utility of SBA was demonstrated in a few pioneering states in the 1990s. Two of the most prominent examples of SBA occurred in Texas and North Carolina, where scores on state accountability tests rose dramatically after the introduction of SBA systems (Grissmer and Flanagan, 1998)." p.4 | Standards-Based Accountability Under No Child Left Behind: Experiences of Teachers and Administrators in Three States | Rand Corporation, 2007 | https://www.rand.org/pubs/monographs/MG589.html | "This research was sponsored by the National Science Foundation under grant number REC-0228295." | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). | |
| 44 | Laura S. Hamilton | Brian M. Stecher, Julie A. Marsh, Jennifer Sloan McCombs, Abby Robyn, Jennifer Lin Russell, Scott Naftel, Heather Barney | "However, the paths through which SBA [standards-based accountability] changes district, school, and classroom practices and how these changes in practice influence student outcomes are largely unexplored. There is strong evidence that SBA leads to changes in teachers’ instructional practices (Hamilton, 2004; Stecher, 2002)." p.5 | Dismissive | Standards-Based Accountability Under No Child Left Behind: Experiences of Teachers and Administrators in Three States | Rand Corporation, 2007 | https://www.rand.org/pubs/monographs/MG589.html | "This research was sponsored by the National Science Foundation under grant number REC-0228295." | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 45 | Laura S. Hamilton | Brian M. Stecher, Julie A. Marsh, Jennifer Sloan McCombs, Abby Robyn, Jennifer Lin Russell, Scott Naftel, Heather Barney | "Much less is known about the impact of SBA at the district and school levels and the relationships among actions at the various levels and student outcomes. This study was designed to shed light on this complex set of relationships…" p.5 | Dismissive | Standards-Based Accountability Under No Child Left Behind: Experiences of Teachers and Administrators in Three States | Rand Corporation, 2007 | https://www.rand.org/pubs/monographs/MG589.html | "This research was sponsored by the National Science Foundation under grant number REC-0228295." | Relevant pre-2000 studies of the effects of standards, alignment, goal setting, setting reachable goals, etc. include those of Mitchell (1999); Morgan & Ramist (1998); the *Southern Regional Education Board (1998); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); the Florida Office of Program Policy Analysis (1997); Pomplun (1997); Schmoker (1996); Aguilera & Hendricks (1996); Banta, Lund, Black & Oblander (1996); Bottoms & Mikos (1995); *Bamburg & Medina (1993); Bishop (1993); the U. S. General Accounting Office (1993); Eckstein & Noah (1993); Mattsson (1993); Brown (1992); Heyneman & Ransom (1992); Whetton (1992); Anderson, Muir, Bateson, Blackmore & Rogers (1990); Csikszentmihalyi (1990); *Levine & Lezotte (1990); LaRoque & Coleman (1989); Hillocks (1987); Willingham & Morris (1986); Resnick & Resnick (1985); Ogle & Fritts (1984); *Natriello & Dornbusch (1984); Brooke & Oxenham (1984); Rentz (1979); Wellisch, MacQueen, Carriere & Dick (1978); *Rosswork (1977); Estes, Colvin & Goodwin (1976); Wood (1953); and Panlasigui & Knight (1930). |
| 46 | Julie A. Marsh, John F. Pane, and Laura S. Hamilton | "Unlike past studies of data use in schools, this paper brings together information systematically gathered from large, representative samples of educators at the district, school, and classroom levels in a variety of contexts." p.1 | Dismissive, Denigrating | Making Sense of Data-Driven Decision Making in Education | Rand Corporation Occassional Paper, 2006 | ||||
| 47 | Julie A. Marsh, John F. Pane, and Laura S. Hamilton | "Although
a few studies have tried to link DDDM to changes in school culture or
performance (Chen et al., 2005; Copland, 2003; Feldman and Tung, 2001;
Schmoker and Wilson, 1995; Wayman and Stringfield 2005), most of the
literature focuses on implementation. In addition, previous work has tended
to describe case studies of schools or has taken the form of advocacy or technical assistance (such as the “how to” implementation guides described by Feldman and Tung, 2001)." p.4 |
Dismissive, Denigrating | Making Sense of Data-Driven Decision Making in Education | Rand Corporation Occassional Paper, 2006 | ||||
| 48 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "...the federal government and the nation’s school systems have made and are continuing to make significant investments toward the improvement of mathematics education. However, the knowledge base upon which these efforts are founded is generally weak." p.iii | Denigrating | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 49 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "New curricular materials have been developed along with training and coaching programs intended to provide teachers with the knowledge and skills needed to use those materials. However, these efforts have been supported by only a limited and uneven base of research and research-based development, which is part of the reason for the limited success of those efforts." p. xi | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 50 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "More important, the intense debates over the past decade seem to be based more often on ideology than on evidence." p.xiii | Denigrating | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 51 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "However, despite more than a century of efforts to improve school mathematics in the United States, investments in research and development have been virtually nonexistent." p.xiv | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 52 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "There has never been a long-range programmatic effort to fund research and development in mathematics education, nor has funding been organized to focus on knowledge that would be usable in practice." p.xiv | Denigrating | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 53 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "Despite the strong history of work in this area, we lack research about what is happening today in algebra classrooms; how innovations in algebra teaching and learning can be designed, implemented, and assessed; and how policy decisions shape student learning and affect equity." p.xxi | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 54 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "Because most studies have focused on algebra at the high school level, we lack knowledge about younger students’ learning of algebraic ideas and skills." p.xxi | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 55 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "Little is known about what happens when algebra is viewed as a K–12 subject, what happens when it is integrated with other subjects, or what happens when it emphasizes a wider range of concepts and processes." p.xxi | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 56 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "Research could inform the perennial debates surrounding the algebra curriculum: what to include, emphasize, reduce, or omit.." p.xxi | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 57 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "For the most part, these debates are poorly informed because research evidence is lacking." p.xxiv | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 58 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "Despite more than a century of efforts to improve school mathematics in the United States, efforts that have yielded numerous research studies and development projects, investments in research and development have been inadequate." p.5 | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 59 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "Federal agencies (primarily the National Science Foundation and the U.S. Department of Education) have contributed funding for many of these efforts. But the investments have been relatively small, and the support has been fragmented and uncoordinated." p.5 | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 60 | Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "There has never been a long-range programmatic effort devoted solely to funding research in mathematics education, nor has research (as opposed to development) funding been organized to focus on knowledge that would be usable in practice. Consequently, major gaps exist in the knowledge base and in knowledge-based development." p.5 | Dismissive | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |
| 61 | Laura S. Hamilton | Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz | “The shortcomings of the studies make it difficult to determine the size of teacher effects, but we suspect that the magnitude of some of the effects reported in this literature are overstated.” p. xiii | Denigrating | Evaluating Value-Added Models for Teacher Accountability | Rand Corporation, 2003 | https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf | Rand Corporation funders | Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done. |
| 62 | Laura S. Hamilton | Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz | “Using VAM to estimate individual teacher effects is a recent endeavor, and many of the possible sources of error have not been thoroughly evaluated in the literature.” p. xix | Dismissive | Evaluating Value-Added Models for Teacher Accountability | Rand Corporation, 2003 | https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf | Rand Corporation funders | Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done. |
| 63 | Laura S. Hamilton | Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz | “Empirical evaluations do not exist for many of the potential sources of error we have identified. Studies need to be conducted to determine how these factors contribute to estimated teacher effects and to determine the conditions that exacerbate or mitigate the impact these factors have on teacher effects.” p. xix | Dismissive | Evaluating Value-Added Models for Teacher Accountability | Rand Corporation, 2003 | https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf | Rand Corporation funders | Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done. |
| 64 | Laura S. Hamilton | Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz | “This lack of attention to teachers in policy discussions may be attributed in part to another body of literature that attempted to determine the effects of specific teacher background characteristics, including credentialing status (e.g., Miller, McKenna, and McKenna, 1998; Goldhaber and Brewer, 2000) and subject matter coursework (e.g., Monk, 1994).” p. 8 | Dismissive | Evaluating Value-Added Models for Teacher Accountability | Rand Corporation, 2003 | https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf | Rand Corporation funders | Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done. |
| 65 | Laura S. Hamilton | Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz | “To date, there has been little empirical exploration of the size of school effects and the sensitivity of teacher effects to modeling of school effects.” p. 78 | Dismissive | Evaluating Value-Added Models for Teacher Accountability | Rand Corporation, 2003 | https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf | Rand Corporation funders | Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done. |
| 66 | Laura S. Hamilton | Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz | “There are no empirical explorations of the robustness of estimates to assumptions about prior-year schooling effects.“ p. 81 | Dismissive | Evaluating Value-Added Models for Teacher Accountability | Rand Corporation, 2003 | https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf | Rand Corporation funders | Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done. |
| 67 | Laura S. Hamilton | Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz | “There is currently no empirical evidence about the sensitivity of gain scores or teacher effects to such alternatives.” p. 89 | Dismissive | Evaluating Value-Added Models for Teacher Accountability | Rand Corporation, 2003 | https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf | Rand Corporation funders | Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done. |
| 68 | Laura S. Hamilton | Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz | “Empirical evaluations do not exist for many of the potential sources of error we have identified. Studies need to be conducted to determine how these factors contribute to estimated teacher effects and to determine the conditions that exacerbate or mitigate the impact these factors have on teacher effects.” p. 116 | Dismissive | Evaluating Value-Added Models for Teacher Accountability | Rand Corporation, 2003 | https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf | Rand Corporation funders | Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done. |
| 69 | Laura S. Hamilton | Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz | “Although we expect missing data are likely to be pervasive, there is little systematic discussion of the extent or nature of missing data in test score databases.” p. 117 | Dismissive | Evaluating Value-Added Models for Teacher Accountability | Rand Corporation, 2003 | https://www.rand.org/content/dam/rand/pubs/monographs/2004/RAND_MG158.pdf | Rand Corporation funders | Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done. |
| 70 | Brian M. Stecher | Laura S. Hamilton | "The business model of setting clear targets, attaching incentives to the attainment of those targets, and rewarding those responsible for reaching the targets has proven successful in a wide range of business enterprises. But there is no evidence that these accountability principles will work well in an educational context, and there are many reasons to doubt that the principles can be applied without significant adaptation." | Dismissive | Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable | Rand Review, Spring 2002 | https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html | Rand Corporation funders | See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm . This list includes 24 studies completed before 2000 whose primary focus was to measure the effect of “test-based accountability.” A few dozen more pre-2000 studies also measured the effect of test-based accountability although such was not their primary focus. Include qualitative and program evaluation studies of test-based accountability, and the count of pre-2000 studies rises into the hundreds. |
| 71 | Brian M. Stecher | Laura S. Hamilton | " The lack of strong evidence regarding the design and effectiveness of accountability systems hampers policymaking at a critical juncture." | Denigrating | Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable | Rand Review, Spring 2002 | https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html | Rand Corporation funders | See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm . This list includes 24 studies completed before 2000 whose primary focus was to measure the effect of “test-based accountability.” A few dozen more pre-2000 studies also measured the effect of test-based accountability although such was not their primary focus. Include qualitative and program evaluation studies of test-based accountability, and the count of pre-2000 studies rises into the hundreds. |
| 72 | Brian M. Stecher | Laura S. Hamilton | "Nonetheless, the evidence has yet to justify the expectations. The initial evidence is, at best, mixed. On the plus side, students and teachers seem to respond to the incentives created by the accountability systems | Dismissive | Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable | Rand Review, Spring 2002 | https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html | Rand Corporation funders | See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm . This list includes 24 studies completed before 2000 whose primary focus was to measure the effect of “test-based accountability.” A few dozen more pre-2000 studies also measured the effect of test-based accountability although such was not their primary focus. Include qualitative and program evaluation studies of test-based accountability, and the count of pre-2000 studies rises into the hundreds. |
| 73 | Brian M. Stecher | Laura S. Hamilton | "Proponents of accountability attribute the improved scores in these states to clearer expectations, greater motivation on the part of the students and teachers, a focused curriculum, and more-effective instruction. However, there is little or no research to substantiate these positive changes or their effects on scores." | Dismissive | Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable | Rand Review, Spring 2002 | https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html | Rand Corporation funders | In fact, the evidence "that testing can improve education" is voluminous. See, for example, Phelps, R. P. (2005). The rich, robust research literature on testing’s achievement benefits. In R. P. Phelps (Ed.), Defending standardized testing (pp. 55–90). Mahwah, NJ: Psychology Press. Or, see https://journals.sagepub.com/doi/abs/10.1177/0193841X19865628#abstract |
| 74 | Brian M. Stecher | Laura S. Hamilton | "One of the earliest studies on the effects of testing (conducted in two Arizona schools in the late 1980s) showed that teachers reduced their emphasis on important, nontested material." | Dismissive | Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable | Rand Review, Spring 2002 | https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html | Rand Corporation funders | Hundreds, perhaps thousands, of studies of the effects of testing predate CRESST's in the 1980s. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm |
| 75 | Brian M. Stecher | Laura S. Hamilton | "Test-based accountability systems will work better if we acknowledge how little we know about them, if the federal government devotes appropriate resources to studying them, and if the states make ongoing efforts to improve them." | Dismissive | Putting Theory to the Test: Systems of "Educational Accountability" Should be Held Accountable | Rand Review, Spring 2002 | https://www.rand.org/pubs/periodicals/rand-review/issues/rr-04-02/theory.html | Rand Corporation funders | See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm . This list includes 24 studies completed before 2000 whose primary focus was to measure the effect of “test-based accountability.” A few dozen more pre-2000 studies also measured the effect of test-based accountability although such was not their primary focus. Include qualitative and program evaluation studies of test-based accountability, and the count of pre-2000 studies rises into the hundreds. |
| 76 | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein | "Although test-based accountability has shown some compelling results, the issues are complex, the research is new and incomplete, and many of the claims that have received the most attention have proved to be premature and superficial." | Denigrating | Summary, p.xiv | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Hundreds, perhaps thousands, of studies of the effects of testing predate CRESST's in the 1980s. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm | |
| 77 | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein | "The research evidence does not provide definitive information about the actual costs of testing but the information that is available suggests that expenditures for testing have grown in recent years." | Dismissive | Introduction, p.9 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | No. See, for example, Phelps, R.P. (2000, Winter). Estimating the cost of systemwide student testing in the United States. Journal of Education Finance, 25(3) 343–380; Danitz, T. (2001, February 27). Special report: States pay $400 million for tests in 2001. Stateline.org. Pew Center for the States; Hoxby, C.M. (2002). The cost of accountability, in W. M Evers & H.J. Walberg (Eds.), School Accountability, Stanford, CA: Hoover Institution Press; U.S. GAO. (1993, January). Student testing: Current extent and expenditures, with cost estimates for a national examination. GAO/PEMD-93-8. Washington, DC: US General Accounting Office; Picus, L.O., & Tralli, A. (1998, February). Alternative assessment programs: What are the true costs? CSE Technical Report 441, Los Angeles: CRESST; Phelps, R.P. (1998). Benefit-cost analysis of systemwide student testing, Paper presented at the annual meeting of the American Education Finance Association, Mobile, AL. | |
| 78 | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein | "The General Accounting Office (1993) … estimate was $516 million … The estimate does not include time for more-extensive test preparation activities." p.9 | Denigrating | Introduction, p.9 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | As a matter of fact the GAO report did include those costs -- all of them. The GAO surveys very explicitly instructed respondents to "include any and all costs related" to each test, including any and all test preparation time and expenses. | |
| 79 | Laura S. Hamilton, Daniel M. Koretz | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "There is currently no substantial evidence on the effects of published report cards on parents’ decisionmaking or on the schools themselves." | Dismissive | Chapter 2: Tests and their use in test-based accountability systems, p.44 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | For decades, consulting services have existed that help parents new to a city select the right school or school district for them. |
| 80 | Vi-Nhuan Le, Stephen P. Klein | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Research on the inflation of gains remains too limited to indicate how prevalent the problem is." | Dismissive | Chapter 3: Technical criteria for evaluating tests, p. 68 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | In fact the test prep, or test coaching, literature is vast and dates back decades, with meta-analyses of the literature dating back at least to the 1970s. There's even a What Works Clearinghouse summary of the (post World Wide Web) college admission test prep research literature: https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/wwc_act_sat_100416.pdf . See also: Gilmore (1927) DeWeerdt (1927) French (1959) French & Dear (1959) Ortar (1960) Marron (1965) ETS (1965). Messick & Jungeblut (1981) Ellis, Konoske, Wulfeck, & Montague (1982) DerSimonian and Laird (1983) Kulik, Bangert-Drowns & Kulik (1984) Powers (1985) Samson (1985) Scruggs, White, & Bennion (1986) Jones (1986). Fraker (1986/1987) Halpin (1987) Whitla (1988) Snedecor (1989) Bond (1989). Baydar (1990) Becker (1990) Smyth (1990) Moore (1991) Alderson & Wall (1992) Powers (1993) Oren (1993). Powers & Rock (1994) Scholes, Lane (1997) Allalouf & Ben Shakhar (1998) Robb & Ercanbrack (1999) McClain (1999) Camara (1999, 2001, 2008) Stone & Lane (2000, 2003) Din & Soldan (2001) Briggs (2001) Palmer (2002) Briggs & Hansen (2004) Cankoy & Ali Tut (2005) Crocker (2005) Allensworth, Correa, & Ponisciak (2008) Domingue & Briggs (2009) Koljatic & Silva (2014) Early (2019) Herndon (2021) |
| 81 | Vi-Nhuan Le, Stephen P. Klein | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Relatively little is known about how testing accomodations affect score validity, and the few studies that have been conducted on the subject have had mixed results." | Dismissive | Chapter 3: Technical criteria for evaluating tests, p. 71 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | |
| 82 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "High-stakes testing may also affect parents (e.g., their attitudes toward education, their engagement with schools, and their direct participation in their child's learning) as well as policymakers (their beliefs about system performance, their judgements about program effectiveness, and their allocation of resources). However, these issues remain largely unexamined in the literature." | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, p. 79 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf | US National Science Foundation | Parents and other adults are typically reached.through public opinion polls. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm . Among the hundreds of polls conducted between 1958 and 2008, a majority of them included parents in particular or adults in general. |
| 83 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "As described in chapter 2, there was little concern about the effects of testing on teaching prior to the 1970s." | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, p. 81 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf | US National Science Foundation | Rubbish. Entire books were written on the topic, for example: C.C. Ross, Measurement in Today’s Schools, 1942; G.M. Ruch, G.D. Stoddard, Tests and Measurements in High School Instruction, 1927; C.W. Odell, Educational Measurement in High School, 1930. Other testimonies to the abundance of educational testing and empirical research on test use starting in the first half of the twentieth century can be found in Lincoln & Workman 1936, 4, 7; Butts 1947, 605; Monroe 1950, 1461; Holman & Docter 1972, 34; Tyack 1974, 183; and Lohman 1997, 88. |
| 84 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "In light of the changes that occurred in the uses of large-scale testing in the 1980s and 1990s, researchers began to investigate teachers' reactions to external assessment. The initial research on the impact of large-scale testing was conducted in the 1980s and the 1990s." | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, p. 83 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf | US National Science Foundation | Hundreds, perhaps thousands, of studies of the effects of testing predate CRESST's in the 1980s. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm |
| 85 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "The bulk of the research on the effects of testing has been conducted using surveys and case studies." | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, p. 83 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf | US National Science Foundation | This is misleading. True, many of the hundreds of studies on the effects of testing have been surveys and case studies. But, many, and more by my count, have been randomized experiments. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; |
| 86 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Data on the incidence of cheating [on educational tests] are scarce…" | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, p. 96 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Actually, there have been, in surveys, in which respondents freely admit that they cheat and how. Moreover, news reports of cheating, by students or educators, have been voluminous. See, for example, Caveon Test Security's "Cheating in the News" section on its web site. |
| 87 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Less is known about changes in policies at the district and school levels in response to high-stakes testing, but mixed evidence of some impact has appeared." | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, p. 96 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Relevant pre-2000 studies of the effects of testing on at-risk students, completion, dropping out, curricular offerings, attitudes, etc. include those of Schleisman (1999); the *Southern Regional Education Board (1998); Webster, Mendro, Orsak, Weerasinghe & Bembry (1997); Jones (1996); Boylan (1996); Jones, 1993; Jacobson (1992); Grisay (1991); Johnstone (1990); Task Force on Educational Assessment Programs [Florida] (1979); Wellisch, MacQueen, Carriere & Duck (1978); Enochs (1978); Pronaratna (1976); and McWilliams & Thomas (1976). |
| 88 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Although numerous news articles have addressed the negative effects of high-stakes testing, systematic research on the subject is limited." | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, p. 98 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Relevant pre-2000 studies of the effects of testing on at-risk students, completion, dropping out, curricular offerings, attitudes, etc. include those of Schleisman (1999); the *Southern Regional Education Board (1998); Webster, Mendro, Orsak, Weerasinghe & Bembry (1997); Jones (1996); Boylan (1996); Jones, 1993; Jacobson (1992); Grisay (1991); Johnstone (1990); Task Force on Educational Assessment Programs [Florida] (1979); Wellisch, MacQueen, Carriere & Duck (1978); Enochs (1978); Pronaratna (1976); and McWilliams & Thomas (1976). |
| 89 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Research regarding the effects of test-based accountability on equity is very limited." | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, p. 99 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf | US National Science Foundation | |
| 90 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Researchers have not documented the desirable consequences of testing … as clearly as the undesirable ones." | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, p. 99 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf | US National Science Foundation | See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm |
| 91 | Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | " … researchers have not generally measured the extent or magnitude of the shifts in practice that they identified as a result of high-stakes testing." | Dismissive | Chapter 4: Consequences of large-scale, high-stakes testing on school and classroom practice, pp. 99–100 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/content/dam/rand/pubs/monograph_reports/2002/MR1554.pdf | US National Science Foundation | The 1993 GAO study did. See, also: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm |
| 92 | Lorraine M. McDonnell | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "...this chapter can only describe the issues that are raised when one looks at testing from a political perspective. Because of the lack of systematic studies on the topic." | Dismissive | Chapter 5: Accountability as seen through a political lens, p.102 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Parents and other adults are typically reached.through public opinion polls. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm . Among the hundreds of polls conducted between 1958 and 2008, a majority of them included parents in particular or adults in general. |
| 93 | Lorraine M. McDonnell | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "...public opinion, as measured by surveys, does not always provide a clear and unambiguous measure of public sentiment." | Denigrating | Chapter 5: Accountability as seen through a political lens, p.108 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Parents and other adults are typically reached.through public opinion polls. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm . Among the hundreds of polls conducted between 1958 and 2008, a majority of them included parents in particular or adults in general. |
| 94 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "So test-based accountability remains controversial because there is inadequate evidence to make clear judgments about its effectiveness in raising test scores and achieving its other goals." | Dismissive | Chapter 6: Improving test-based accountability, p.122 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | In fact, the evidence "that testing can improve education" is voluminous. See, for example, Phelps, R. P. (2005). The rich, robust research literature on testing’s achievement benefits. In R. P. Phelps (Ed.), Defending standardized testing (pp. 55–90). Mahwah, NJ: Psychology Press. Or, see https://journals.sagepub.com/doi/abs/10.1177/0193841X19865628#abstract |
| 95 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Unfortunately, the complexity of the issues and the ambiguity of the existing research do not allow our recommendations to take the form of a practical “how-to” guide for policymakers and practitioners." | Denigrating | Chapter 6: Improving test-based accountability, p.123 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | In fact, the evidence "that testing can improve education" is voluminous. See, for example, Phelps, R. P. (2005). The rich, robust research literature on testing’s achievement benefits. In R. P. Phelps (Ed.), Defending standardized testing (pp. 55–90). Mahwah, NJ: Psychology Press. Or, see https://journals.sagepub.com/doi/abs/10.1177/0193841X19865628#abstract |
| 96 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Additional research is needed to identify the elements of performance on tests and how these elements map onto other tests …." | Denigrating | Chapter 6: Improving test-based accountability, p.127 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | |
| 97 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Another
part of the interpretive question is the need to gather
information in other subject areas to portray a more complete picture of achievement. The scope of constructs that have been considered in research to date has been fairly narrow, focusing on the subjects that are part of the accountability systems that have been studied. Many legitimate instructional objectives have been ignored in the literature to date." |
Denigrating | Chapter 6: Improving test-based accountability, p.127 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Many studies of the effects of testing predate CRESST's in the 1980s and cover all subject fields, not just reading and math. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm |
| 98 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "States should also conduct ongoing analyses of the performance of groups whose members may not be numerous enough to permit separate reporting. English-language learners and students with disabilities are increasingly being included in high-stakes testing systems, and, as discussed in Chapter Three, little is currently known about the validity of scores for these groups." | Dismissive | Chapter 6: Improving test-based accountability, p.131 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Difficult to believe given that the federal government has for decades generously funded research into testing students with disabilities. See, for example, https://nceo.info/ and Kurt Geisinger's and Janet Carlson's chapters in Defending Standardized Testing and Correcting Fallacies in Educational and Psychological Testing. |
| 99 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "It would be especially helpful to know what changes in instruction are made in response to different kinds of information and incentives. In particular, we need to know how teachers interpret information from tests and how they use it to modify instruction." | Dismissive | Chapter 6: Improving test-based accountability, p.133 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Relevant studies of the effects of tests and/or accountability program on motivation and instructional practice include those of the *Southern Regional Education Board (1998); Johnson (1998); Schafer, Hultgren, Hawley, Abrams Seubert & Mazzoni (1997); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); Tuckman & Trimble (1997); Clarke & Stephens (1996); Zigarelli (1996); Stevenson, Lee, et al. (1995); Waters, Burger & Burger (1995); Egeland (1995); Prais (1995); Tuckman (1994); Ritchie & Thorkildsen (1994); Brown & Walberg, (1993); Wall & Alderson (1993); Wolf & Rapiau (1993); Eckstein & Noah (1993); Chao-Qun & Hui (1993); Plazak & Mazur (1992); Steedman (1992); Singh, Marimutha & Mukjerjee (1990); *Levine & Lezotte (1990); O’Sullivan (1989); Somerset (1988); Pennycuick & Murphy (1988); Stevens (1984); Marsh (1984); Brunton (1982); Solberg (1977); Foss (1977); *Kirkland (1971); Somerset (1968); Stuit (1947); and Keys (1934). *Covers many studies; study is a research review, research synthesis, or meta-analysis. |
| 100 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | It seems clear that aligning the components of the system and providing appropriate professional development should, at a minimum, increase teachers’ political support for test-based accountability policies .... Although there is no empirical evidence to suggest that this strategy will reduce inappropriate responses to high-stakes testing,... Additional research needs to be done to determine the importance of alignment for promoting positive effects of test-based accountability. | Dismissive | Chapter 6: Improving test-based accountability, p.135 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Relevant studies of the effects of tests and/or accountability program on motivation and instructional practice include those of the *Southern Regional Education Board (1998); Johnson (1998); Schafer, Hultgren, Hawley, Abrams Seubert & Mazzoni (1997); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); Tuckman & Trimble (1997); Clarke & Stephens (1996); Zigarelli (1996); Stevenson, Lee, et al. (1995); Waters, Burger & Burger (1995); Egeland (1995); Prais (1995); Tuckman (1994); Ritchie & Thorkildsen (1994); Brown & Walberg, (1993); Wall & Alderson (1993); Wolf & Rapiau (1993); Eckstein & Noah (1993); Chao-Qun & Hui (1993); Plazak & Mazur (1992); Steedman (1992); Singh, Marimutha & Mukjerjee (1990); *Levine & Lezotte (1990); O’Sullivan (1989); Somerset (1988); Pennycuick & Murphy (1988); Stevens (1984); Marsh (1984); Brunton (1982); Solberg (1977); Foss (1977); *Kirkland (1971); Somerset (1968); Stuit (1947); and Keys (1934). *Covers many studies; study is a research review, research synthesis, or meta-analysis. |
| 101 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "… we currently do not know enough about test-based accountability to design a system that is immune from the problems we have discussed | Dismissive | Chapter 6: Improving test-based accountability, p.136 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | In fact, the evidence "that testing can improve education" is voluminous. See, for example, Phelps, R. P. (2005). The rich, robust research literature on testing’s achievement benefits. In R. P. Phelps (Ed.), Defending standardized testing (pp. 55–90). Mahwah, NJ: Psychology Press. Or, see https://journals.sagepub.com/doi/abs/10.1177/0193841X19865628#abstract |
| 102 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "There is some limited evidence that educators’ responses to test based accountability vary according to the characteristics of their student populations,…" | Denigrating | Chapter 6: Improving test-based accountability, p.138 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | There was and is far more than "limited" evidence. See, for example: Hundreds, perhaps thousands, of studies of the effects of testing predate CRESST's in the 1980s. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm |
| 103 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "... there is very limited evidence to guide thinking about political issues." | Dismissive | Chapter 6: Improving test-based accountability, p.139 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Parents and other adults are typically reached.through public opinion polls. See, for example: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm . Among the hundreds of polls conducted between 1958 and 2008, a majority of them included parents in particular or adults in general. |
| 104 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "First, we do not have an accurate assessment of the additional costs." | Denigrating | Chapter 6: Improving test-based accountability, p.141 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Yes, we did and we do. See, for example, Phelps, R.P. (2000, Winter). Estimating the cost of systemwide student testing in the United States. Journal of Education Finance, 25(3) 343–380; Danitz, T. (2001, February 27). Special report: States pay $400 million for tests in 2001. Stateline.org. Pew Center for the States; Hoxby, C.M. (2002). The cost of accountability, in W. M Evers & H.J. Walberg (Eds.), School Accountability, Stanford, CA: Hoover Institution Press; U.S. GAO. (1993, January). Student testing: Current extent and expenditures, with cost estimates for a national examination. GAO/PEMD-93-8. Washington, DC: US General Accounting Office; Picus, L.O., & Tralli, A. (1998, February). Alternative assessment programs: What are the true costs? CSE Technical Report 441, Los Angeles: CRESST; Phelps, R.P. (1998). Benefit-cost analysis of systemwide student testing, Paper presented at the annual meeting of the American Education Finance Association, Mobile, AL. |
| 105 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "However, many of these recommended reforms are relatively inexpensive in comparison with the total cost of education. This equation is seldom examined." | Denigrating | Chapter 6: Improving test-based accountability, p.141 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | Wrong. See, for example, Phelps, R.P. (2000, Winter). Estimating the cost of systemwide student testing in the United States. Journal of Education Finance, 25(3) 343–380; Phelps, R.P. (1998). Benefit-cost analysis of systemwide student testing, Paper presented at the annual meeting of the American Education Finance Association, Mobile, AL. |
| 106 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Part of the reason these issues are rarely considered may be that no one has produced a good estimate of the cost of an improved accountability system in comparison with its benefits." | Denigrating | Chapter 6: Improving test-based accountability, p.141 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | No. See, for example, Phelps, R.P. (2000, Winter). Estimating the cost of systemwide student testing in the United States. Journal of Education Finance, 25(3) 343–380; Danitz, T. (2001, February 27). Special report: States pay $400 million for tests in 2001. Stateline.org. Pew Center for the States; Hoxby, C.M. (2002). The cost of accountability, in W. M Evers & H.J. Walberg (Eds.), School Accountability, Stanford, CA: Hoover Institution Press; U.S. GAO. (1993, January). Student testing: Current extent and expenditures, with cost estimates for a national examination. GAO/PEMD-93-8. Washington, DC: US General Accounting Office; Picus, L.O., & Tralli, A. (1998, February). Alternative assessment programs: What are the true costs? CSE Technical Report 441, Los Angeles: CRESST; Phelps, R.P. (1998). Benefit-cost analysis of systemwide student testing, Paper presented at the annual meeting of the American Education Finance Association, Mobile, AL. |
| 107 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "Nevertheless, our knowledge of the costs of alternative accountability systems is still somewhat limited. Policymakers need to know how much it would cost to change their current systems to be responsive to criticisms such as those described in this book. These estimates need to consider all of the associated costs, including possible opportunity costs associated with increased testing time and increased test preparation time." | Dismissive | Chapter 6: Improving test-based accountability, p.142 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | No. See, for example, Phelps, R.P. (2000, Winter). Estimating the cost of systemwide student testing in the United States. Journal of Education Finance, 25(3) 343–380; Danitz, T. (2001, February 27). Special report: States pay $400 million for tests in 2001. Stateline.org. Pew Center for the States; Hoxby, C.M. (2002). The cost of accountability, in W. M Evers & H.J. Walberg (Eds.), School Accountability, Stanford, CA: Hoover Institution Press; U.S. GAO. (1993, January). Student testing: Current extent and expenditures, with cost estimates for a national examination. GAO/PEMD-93-8. Washington, DC: US General Accounting Office; Picus, L.O., & Tralli, A. (1998, February). Alternative assessment programs: What are the true costs? CSE Technical Report 441, Los Angeles: CRESST; Phelps, R.P. (1998). Benefit-cost analysis of systemwide student testing, Paper presented at the annual meeting of the American Education Finance Association, Mobile, AL. |
| 108 | Laura S. Hamilton, Brian M. Stecher | Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, Eds. | "However, there is still much about these systems that is not well understood. Lack of research-based knowledge about the quality of scores and the mechanisms through which high-stakes testing programs operate limits our ability to improve these systems. As a result, our discussions also identified unanswered questions..." | Dismissive | Chapter 6: Improving test-based accountability, p.143 | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | https://www.rand.org/pubs/monograph_reports/MR1554.html | US National Science Foundation | In fact, the evidence "that testing can improve education" is voluminous. See, for example, Phelps, R. P. (2005). The rich, robust research literature on testing’s achievement benefits. In R. P. Phelps (Ed.), Defending standardized testing (pp. 55–90). Mahwah, NJ: Psychology Press. Or, see https://journals.sagepub.com/doi/abs/10.1177/0193841X19865628#abstract |
| 109 | Daniel M. Koretz | Daniel F. McCaffrey, Laura S. Hamilton | "Although high-stakes testing is now widespread, methods for evaluating the validity of gains obtained under high-stakes conditions are poorly developed. This report presents an approach for evaluating the validity of inferences based on score gains on high-stakes tests. It describes the inadequacy of traditional validation approaches for validating gains under high-stakes conditions and outlines an alternative validation framework for conceptualizing meaningful and inflated score gains.", p.1 | Denigrating | Toward a framework for validating gains under high-stakes conditions | CSE Technical Report 551, CRESST/Harvard Graduate School of Education, CRESST/RAND Education, December 2001 | https://files.eric.ed.gov/fulltext/ED462410.pdf | Office of Research and Improvement, US Education Department | In fact the test prep, or test coaching, literature is vast and dates back decades, with meta-analyses of the literature dating back at least to the 1970s. There's even a What Works Clearinghouse summary of the (post World Wide Web) college admission test prep research literature: https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/wwc_act_sat_100416.pdf . See also: Gilmore (1927) DeWeerdt (1927) French (1959) French & Dear (1959) Ortar (1960) Marron (1965) ETS (1965). Messick & Jungeblut (1981) Ellis, Konoske, Wulfeck, & Montague (1982) DerSimonian and Laird (1983) Kulik, Bangert-Drowns & Kulik (1984) Powers (1985) Samson (1985) Scruggs, White, & Bennion (1986) Jones (1986). Fraker (1986/1987) Halpin (1987) Whitla (1988) Snedecor (1989) Bond (1989). Baydar (1990) Becker (1990) Smyth (1990) Moore (1991) Alderson & Wall (1992) Powers (1993) Oren (1993). Powers & Rock (1994) Scholes, Lane (1997) Allalouf & Ben Shakhar (1998) Robb & Ercanbrack (1999) McClain (1999) Camara (1999, 2001, 2008) Stone & Lane (2000, 2003) Din & Soldan (2001) Briggs (2001) Palmer (2002) Briggs & Hansen (2004) Cankoy & Ali Tut (2005) Crocker (2005) Allensworth, Correa, & Ponisciak (2008) Domingue & Briggs (2009) Koljatic & Silva (2014) Early (2019) Herndon (2021) |
| 110 | Daniel M. Koretz | Daniel F. McCaffrey, Laura S. Hamilton | "Few efforts are made to evaluate directly score gains obtained under high-stakes conditions, and conventional validation tools are not fully adequate for the task.", p. 1 | Dismissive | Toward a framework for validating gains under high-stakes conditions | CSE Technical Report 551, CRESST/Harvard Graduate School of Education, CRESST/RAND Education, December 2001 | https://files.eric.ed.gov/fulltext/ED462410.pdf | Office of Research and Improvement, US Education Department | In fact the test prep, or test coaching, literature is vast and dates back decades, with meta-analyses of the literature dating back at least to the 1970s. There's even a What Works Clearinghouse summary of the (post World Wide Web) college admission test prep research literature: https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/wwc_act_sat_100416.pdf . See also: Gilmore (1927) DeWeerdt (1927) French (1959) French & Dear (1959) Ortar (1960) Marron (1965) ETS (1965). Messick & Jungeblut (1981) Ellis, Konoske, Wulfeck, & Montague (1982) DerSimonian and Laird (1983) Kulik, Bangert-Drowns & Kulik (1984) Powers (1985) Samson (1985) Scruggs, White, & Bennion (1986) Jones (1986). Fraker (1986/1987) Halpin (1987) Whitla (1988) Snedecor (1989) Bond (1989). Baydar (1990) Becker (1990) Smyth (1990) Moore (1991) Alderson & Wall (1992) Powers (1993) Oren (1993). Powers & Rock (1994) Scholes, Lane (1997) Allalouf & Ben Shakhar (1998) Robb & Ercanbrack (1999) McClain (1999) Camara (1999, 2001, 2008) Stone & Lane (2000, 2003) Din & Soldan (2001) Briggs (2001) Palmer (2002) Briggs & Hansen (2004) Cankoy & Ali Tut (2005) Crocker (2005) Allensworth, Correa, & Ponisciak (2008) Domingue & Briggs (2009) Koljatic & Silva (2014) Early (2019) |
| 111 | Sheila Barron | "Although this is a topic researchers ... talk about often, very little has been written about the difficulties secondary analysts confront." p.173 | Dismissive | Difficulties associated with secondary analysis of NAEP data, chapter 9 | Grading the Nation's Report Card, National Research Council, 2000 | https://www.nap.edu/catalog/9751/grading-the-nations-report-card-research-from-the-evaluation-of | National Research Council funders | In their 2009 Evaluation of NAEP for the US Education Department, Buckendahl, Davis, Plake, Sireci, Hambleton, Zenisky, & Wells (pp. 77–85) managed to find quite a lot of research on making comparisons between NAEP and state assessments: several of NAEP's own publications, Chromy 2005), Chromy, Ault, Black, & Mosquin (2007), McLaughlin (2000), Schuiz & Mitzel (2005), Sireci, Robin, Meara, Rogers, & Swaminathan (2000), Stancavage, Et al (2002), Stoneberg (2007), WestEd (2002), and Wise, Le, Hoffman, & Becker (2004). | |
| 112 | Sheila Barron | "...few articles have been written that specifically address the difficulties of using NAEP data." p.173 | Dismissive | Difficulties associated with secondary analysis of NAEP data, chapter 9 | Grading the Nation's Report Card, National Research Council, 2000 | https://www.nap.edu/catalog/9751/grading-the-nations-report-card-research-from-the-evaluation-of | National Research Council funders | In their 2009 Evaluation of NAEP for the US Education Department, Buckendahl, Davis, Plake, Sireci, Hambleton, Zenisky, & Wells (pp. 77–85) managed to find quite a lot of research on making comparisons between NAEP and state assessments: several of NAEP's own publications, Chromy 2005), Chromy, Ault, Black, & Mosquin (2007), McLaughlin (2000), Schuiz & Mitzel (2005), Sireci, Robin, Meara, Rogers, & Swaminathan (2000), Stancavage, Et al (2002), Stoneberg (2007), WestEd (2002), and Wise, Le, Hoffman, & Becker (2004). | |
| 113 | Brian M. Stecher | Stephen P. Klein | "In constrast, relatively little has been published on the costs of such measures [performance tests] in operational programs. An Office of Technology Assessment (1992) … (Hoover and Bray) …." | Dismissive | The Cost of Science Performance Assessments in Large-Scale Testing Programs, p.1 | Educational Evaluation and Policy Analysis, Spring 1997, 19(1) | "This article is based on work supported by the National Science Foundation under Grant No. MDR-9154406." p.12 | The January 1993 GAO report on testing costs included such information. CRESST has spent a quarter century denigrating that report. | |
| 114 | Brian M. Stecher | Stephen P. Klein | "However, empirical and observational data suggest much more needs to be done to understand what hands-on tasks actually measure. Klein et al. (1996b) … Shavelson et al. (1992) … Hamilton (1994) …." pp.9-10 | Dismissive | The Cost of Science Performance Assessments in Large-Scale Testing Programs, p.1 | Educational Evaluation and Policy Analysis, Spring 1997, 19(1) | "This article is based on work supported by the National Science Foundation under Grant No. MDR-9154406." p.12 | Article references only works by other CRESST authors and completely ignores the career-tech education literature, where such studies are most likely to be found. | |
| 115 | Brian M. Stecher | Stephen P. Klein | "Future research will no doubt shed more light on the validity question, but for now, it is not clear how scores on hands-on performance tasks should be interpreted." p.10 | Dismissive | The Cost of Science Performance Assessments in Large-Scale Testing Programs, p.1 | Educational Evaluation and Policy Analysis, Spring 1997, 19(1) | "This article is based on work supported by the National Science Foundation under Grant No. MDR-9154406." p.12 | Article references only works by other CRESST authors and completely ignores the career-tech education literature, where such studies are most likely to be found. | |
| 116 | Brian M. Stecher | Stephen P. Klein | "Advocates of performance assessment believe that the use of these measures will reinforce efforts to reform curriculum and instruction. … Unfortunately, there is very little research to confirm either the existence or the size of most off these potential benefits. Those few studies ... Klein (1995) ... Javonovic, Solanno-Flores, & Shavelson, 1994; Klein et al., 1996a)." p.10 | Dismissive | The Cost of Science Performance Assessments in Large-Scale Testing Programs, p.1 | Educational Evaluation and Policy Analysis, Spring 1997, 19(1) | "This article is based on work supported by the National Science Foundation under Grant No. MDR-9154406." p.12 | Article references only works by other CRESST authors and completely ignores the career-tech education literature, where such studies are most likely to be found. | |
| 117 | Robert L. Linn | Daniel M. Koretz, Eva Baker | “’Yet we do not have the necessary comprehensive dependable data. . . .’ (Tyler 1996a, p. 95)” p. 8 | Dismissive | Assessing the Validity of the National Assessment of Educational Progress | CSE Technical Report 416 (June 1996) | http://www.cse.ucla.edu/products/reports/TECH416.pdf | Office of Research and Improvement, US Education Department | In their 2009 Evaluation of NAEP for the US Education Department, Buckendahl, Davis, Plake, Sireci, Hambleton, Zenisky, & Wells (pp. 77–85) managed to find quite a lot of research on making comparisons between NAEP and state assessments: several of NAEP's own publications, Chromy 2005), Chromy, Ault, Black, & Mosquin (2007), McLaughlin (2000), Schuiz & Mitzel (2005), Sireci, Robin, Meara, Rogers, & Swaminathan (2000), Stancavage, Et al (2002), Stoneberg (2007), WestEd (2002), and Wise, Le, Hoffman, & Becker (2004). |
| 118 | Robert L. Linn | Daniel M. Koretz, Eva Baker | "“There is a need for more extended discussion and reconsideration of the approach being used to measure long-term trends.” p. 21 | Dismissive | Assessing the Validity of the National Assessment of Educational Progress | CSE Technical Report 416 (June 1996) | http://www.cse.ucla.edu/products/reports/TECH416.pdf | Office of Research and Improvement, US Education Department | There was extended discussion and cosideration. Simply put, they did not get their way because others disagreed with them. |
| 119 | Robert L. Linn | Daniel M. Koretz, Eva Baker | "“Only a small minority of the articles that discussed achievement levels made any mention of the judgmental nature of the levels, and most of those did so only briefly.” p. 27 | Denigrating | Assessing the Validity of the National Assessment of Educational Progress | CSE Technical Report 416 (June 1996) | http://www.cse.ucla.edu/products/reports/TECH416.pdf | Office of Research and Improvement, US Education Department | All achievement levels, just like all course grades, are set subjectively. This information was never hidden. |
| 120 | Daniel M. Koretz | Robert L. Linn, Stephen Dunbar, Lorrie A. Shepard | “Evidence relevant to this debate has been limited.” p. 2 | Dismissive | The Effects of High-Stakes Testing On Achievement: Preliminary Findings About Generalization Across Tests | Originally presented at the annual meeting of the AERA and the NCME, Chicago, April 5, 1991 | http://nepc.colorado.edu/files/HighStakesTesting.pdf | Office of Research and Improvement, US Education Department | See, for example, https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm |
| IRONIES: | |||||||||
| Rand Corporation | "All RAND [monographs/occasional papers/etc.] undergo rigorous peer review to ensure that they meet high standards for research quality and objectivity." | ||||||||
| Rand Corporation | "The research team found considerable research on the effects of testing, with studies describing a wide variety of effects on teachers’ and students’ activities in the classroom." | Can New Tests Lead to Better Teaching and Deeper Learning? | Rand Corporation Brief, 2014 | ||||||
| Susannah Faxon-Mills, Laura S. Hamilton, Mollie Rudnick, Brian M. Stecher | "We found considerable research on the effects of testing in U.S. schools, including studies of high-stakes testing, performance assessment, and formative assessment." p. viii | New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement | Rand Corporation Research Report, 2013 | "Funding to support the research was provided by the William and Flora Hewlett Foundation." "Marc Chun at the Hewlett Foundation first approached us about reviewing the literature on the impact of assessment, and he was very helpful in framing this investigation." | |||||
| Deborah Loewenberg Ball | Jo Boaler, Phil Daro, Andrew Porter, & 14 others | "High-quality work depends on open debate unconstrained by orthodoxies and political agendas. It is crucial that the composition of the panels and the extended research communities be inclusive, engaging individuals with a wide range of views and skills." p.xxiii | Mathematical Proficiency for All Students | Rand Corporation, 2003 | https://www.rand.org/pubs/monograph_reports/MR1643.html | Office of Research and Improvement, US Education Department | |||
| Laura S. Hamilton | Brian M. Stecher, Stephen P. Klein | "Greater knowledge about testing and accountability can lead to better system design and more-effective system management." p.xiv | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | Summary, p.xiv | |||||
| Laura S. Hamilton | Brian M. Stecher | "Incremental improvements to existing systems, based on current research on testing and accountability, should be combined with long-term research and development efforts that may ultimately lead to a major redesign of these systems. Success in this endeavor will require the thoughtful engagement of educators, policymakers, and researchers in discussions and debates about tests and testing policies." | Making Sense of Test-Based Accountability in Education, Rand Corporation, 2002 | Chapter 6, Improving test-based accountability, pp.143-144 | |||||
| Brian M. Stecher | Stephen P. Klein | "Additional information about the impact of performance assessments on curriculum and instruction would provide policymakers with valuable data on the benefits that may accrue from this relatively expensive form of assessment." p.11 | The Cost of Science Performance Assessments in Large-Scale Testing Programs, p.1 | Educational Evaluation and Policy Analysis, Spring 1997, 19(1) | |||||
| Author cites (and accepts as fact without checking) someone elses dismissive review | |||||||||
| Cite selves or colleagues in the group, but dismiss or denigrate all other work | |||||||||
| Falsely claim that research has only recently been done on topic. | |||||||||
| 1) [as of July 4, 2021] SCOPE funders include: Bill & Melinda Gates Foundation; California Education Policy Fund; Carnegie Corporation of New York; Center for American Progress; Community Education Fund, Silicon Valley Community Foundation; Ford Foundation; James Irvine Foundation; Joyce Foundation; Justice Matters; Learning Forward; Metlife Foundation; National Center on Education and the Economy; National Education Association; National Public Education Support Fund; Nellie Mae Education Foundation; NoVo Foundation; Rose Foundation;S. D. Bechtel, Jr. Foundation; San Francisco Foundation; Sandler Foundation; Silver Giving Foundation; Spencer Foundation; Stanford University; Stuart Foundation; The Wallace Foundation; William and Flora Hewlett Foundation; William T. Grant Foundation | |||||||||