HOME: Dismissive Reviews in Education Policy Research

Author

Co-author(s)

Dismissive Quote

type

Title

Source

Funders

Link1

Notes

Notes2

Goldring, T., Jacob, B.,Kreisman,D., Ricks, M.

"Our paper presents the first causal evidence about the effects of CTE funding, complementing a fairly sparse descriptive literature (Klein 2001; Foster, Klein, and Elliott 2014, e.g.,)." p.3

1stness, Dismissive

Loopholes and the Incidence of Public Services:Evidence from Funding Career & Technical Education

NBER Working Paper 32390,April 2024

NBER funders

http://www.nber.org/papers/w32390

Berne, J.S., Jacob, B.A, Musaddiq, T.,Shapiro, A. Weiland,C.

"We provide the first estimates of the impact of Michigan’s TK program." abstract

1srness

The Effect of Early Childhood Programs on Third-Grade Test Scores: Evidence from Transitional Kindergarten in Michigan

NBER Working Paper 32236; March 2024

This research was funded by the Smith Richardson Foundation, as well as training grants R305B20011 and R305B170015 from the U.S. Department of Education’s Institute of Education Sciences.

Peter De Vlieger

Brian A. Jacob, Kevin Stange

"Yet relatively little is known about the impact of instructor effectiveness on student performance in higher education"

Dismissive

Measuring up: Assessing instructor effectiveness in higher education

Education Next, SUMMER 2017 / VOL. 17, NO. 3

Harvard PEPG and Thomas B. Fordham Institute

https://educationnext.org/measuring-up-assessing-instructor-effectiveness-higher-education/

In fact, the research literature on testing in higher education is long and deep. Consider, for example, the work of Trudy Banta, Patricia Cross, and Thomas Angelo. See also the large number of higher education studies in this meta analysis: https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 ; https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm

Peter De Vlieger

Brian A. Jacob, Kevin Stange

"This lack of research is largely the result of data and methodological challenges."

Dismissive

Measuring up: Assessing instructor effectiveness in higher education

Education Next, SUMMER 2017 / VOL. 17, NO. 3

Harvard PEPG and Thomas B. Fordham Institute

https://educationnext.org/measuring-up-assessing-instructor-effectiveness-higher-education/

Jacob, B. & Rothstein, J.

While the literature in psychometrics (the field concerned with the theory and methodology of psychological measurement) has explored many if not all of the issues that we discuss, economists and other applied researchers are generally unaware of them and frequently misuse test score measures, with potentially serious consequences for their analyses, page 3-4.

dismissive

The Measurement of Student Ability in Modern Assessment Systems

NBER Working Paper 22434; July 2016

NBER funders

Dee, T.S., Dobbie, W., Jacob, B.A., Rockoff, J.

However, despite widespread concerns over test validity and the manipulation of scores, we know little about the factors that lead educators to manipulate student test scores (e.g., accountability policies versus individual students traits). Page 1

dismissive

The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations

NBER Working Paper 22165; April 2016

NBER funders

Actually, there have been, in surveys, in which respondents freely admit that they cheat and how. Moreover, news reports of cheating, by students or educators, have been voluminous. See, for example, Caveon Test Security's "Cheating in the News" section on its web site. The most famous test score inflation study of all time -- John J. Cannells "Lake Wobegon Effect" study -- is largely about cheating. See: https://nonpartisaneducation.org/Review/Books/CannellBook1.htm https://nonpartisaneducation.org/Review/Books/Cannell2.pdf; See also Gregory J. Cizek's Cheating on Tests: https://www.goodreads.com/book/show/5084641-cheating-on-tests ; and Caveon Test Security's resource pages: https://www.caveon.com/resources/.

Dee, T.S., Dobbie, W., Jacob, B.A., Rockoff, J.

Furthermore, there is also little empirical evidence on whether test score manipulation has any long-run consequences for students’ educational outcomes and performance gaps by race, ethnicity, and gender, page 1.

dismissive

The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations

NBER Working Paper 22165; April 2016

NBER funders

Dee, T.S., Dobbie, W., Jacob, B.A., Rockoff, J.

However, Diamond and Persson (2016) find no evidence of the negative human capital effects we document in our setting, page 3.

dismissive

The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations

NBER Working Paper 22165; April 2016

NBER funders

Jacob, B., Rockoff, J.E., Taylor, E.S., Lindy, B., Rosen, R.

Selecting more effective teachers among job applicants during the hiring process could be a highly cost-effective means of improving educational quality, but there is little research that links information gathered during the hiring process to subsequent teacher performance, abstract.

dismissive

Teacher Applicant Hiring and Teacher Performance: Evidence from DC Public Schools

NBER Working Paper 22054; March 2016

Generous financial support was provided by the Smith Richardson Foundation. & NBER funders

Jacob, B., Rockoff, J.E., Taylor, E.S., Lindy, B., Rosen, R.

Limited progress has not been due to a lack of attention by academics, but rather to the use of small samples, the focus on a small set of teacher characteristics found in administrative data, and the lack of high quality performance measures on teachers, page 2.

denigrating

Teacher Applicant Hiring and Teacher Performance: Evidence from DC Public Schools

NBER Working Paper 22054; March 2016

Generous financial support was provided by the Smith Richardson Foundation. & NBER funders

Jacob, B., Dynarski, S., Frank, K., Schneider, B.

The intent of the Michigan Merit Curriculum is to increase academic preparation for college and enhance career readiness. But the theoretical effects of the initiative are ambiguous, page 3.

denigrating

Are Expectations Alone Enough? Estimating the Effect of a Mandatory College-Prep Curriculum in Michigan

NBER Working Paper 22013; February 2016

The Institute of Education Sciences provided generous support through Grant R305E100008. & NBER funders

Peter De Vlieger

Brian A. Jacob, Kevin Stange

"Instructors are a chief input into the higher education production process, yet we know very little about their role in promoting student success."

Dismissive

Measuring Instructor Effectiveness in Higher Education, abstract

"This paper was prepared for the NBER Conference “Productivity in Higher Education” held on June 1, 2016." Published by NBER November 30,2016.

NBER funders

Peter De Vlieger

Brian A. Jacob, Kevin Stange

"Professors and instructors are a chief input into the higher education production process, yet we know very little about their role in promoting student success."

Dismissive

Measuring Instructor Effectiveness in Higher Education, p.1

"This paper was prepared for the NBER Conference “Productivity in Higher Education” held on June 1, 2016." Published by NBER November 30,2016.

NBER funders

Peter De Vlieger

Brian A. Jacob, Kevin Stange

"Yet relatively little is known about the importance of or correlates of instructor effectiveness in postsecondary education."

Dismissive

Measuring Instructor Effectiveness in Higher Education, p.4

"This paper was prepared for the NBER Conference “Productivity in Higher Education” held on June 1, 2016." Published by NBER November 30,2016.

NBER funders

Peter De Vlieger

Brian A. Jacob, Kevin Stange

"Yet little is known about instructor effectiveness in postsecondary education, in part due to difficulties with outcome measurement and self-selection."

Dismissive

Measuring Instructor Effectiveness in Higher Education, p.4

"This paper was prepared for the NBER Conference “Productivity in Higher Education” held on June 1, 2016." Published by NBER November 30,2016.

NBER funders

Alia Wong (journalist)

Thomas S. Dee, Will Dobbie, Brian A. Jacob, & Jonah Rockoff

"The prevalence of test-score manipulation in the United States is well-documented. ... What hasn’t been well documented are the causes and consequences of such manipulation."

Dismissive

Why Would a Teacher Cheat? Educators often choose to inflate students' scores on standardized tests, and the motivations—and effects—indicate that a little deception isn't always a bad thing.

The Atlantic, April 27, 2016

NBER funders

https://www.theatlantic.com/education/archive/2016/04/why-teachers-cheat/480039/

Brian A. Jacob

Thomas S. Dee, Will Dobbie, Jonah Rockoff

"…despite widespread concerns over test validity and the manipulation of scores, we know little about the factors that lead educators to manipulate student scores (e.g., accountability policies versus individual students traits).

Dismissive

The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations, p.1

National Bureau of Economic Research, Working Paper 22165, April 2016

NBER funders

https://www.nber.org/papers/w22165

Brian A. Jacob

Thomas S. Dee, Will Dobbie, Jonah Rockoff

"…there is little empirical evidence on whether test score manipulation has any long-run consequences for students' educational outcomes and performance gaps by race, ethnicity, and gender."

Dismissive

The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations, p.1

National Bureau of Economic Research, Working Paper 22165, April 2016

NBER funders

https://www.nber.org/papers/w22165

Brian A. Jacob

Thomas S. Dee, Will Dobbie, Jonah Rockoff

"Our results contribute to an emerging literature that documents both the moral hazard that can be created by test-scoring procedures…. In early work, Jacob and Levitt (2003) find...

Dismissive

The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations, p.3

National Bureau of Economic Research, Working Paper 22165, April 2016

NBER funders

https://www.nber.org/papers/w22165

Brian A. Jacob

Thomas S. Dee, Will Dobbie, Jonah Rockoff

"Our results contribute to an emerging literature that documents both the moral hazard that can be created by test-scoring procedures…. In early work, Jacob and Levitt (2003) find...

Dismissive

The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations, p.3

National Bureau of Economic Research, Working Paper 22165, April 2016

NBER funders

https://www.nber.org/papers/w22165

Brian Jacob

Susan Dynarski, Kenneth Frank, Barbara Schneider

"A large body of research has found that increasing high school graduation requirements (whether in the form of additional courses or exit exams) results in lower graduation rates among the most disadvantaged students. There is less evidence on how such policies affect student achievement, but the existing research generally does not find large gains in student performance." p.2

Dismissive

Are Expectations Alone Enough? Estimating the Effect of a Mandatory College-Prep Curriculum in Michigan

EPI Working Paper 01-2016, January, 2016 & NBER Working Paper No. 22013

"The U.S. Department of Education’s Institute of Education Sciences provided generous support through Grant R305E100008." & NBER funders

https://edpolicy.umich.edu/files/01-2016_are-expectations-alone-enough.pdf

See, for example, https://www.tandfonline.com/doi/full/10.1080/15305058.2011.602920 https://nonpartisaneducation.org/Review/Resources/QuantitativeList.htm ; https://nonpartisaneducation.org/Review/Resources/SurveyList.htm ; https://nonpartisaneducation.org/Review/Resources/QualitativeList.htm

For example, from Table 2, Chapter 3 of Correcting Fallacies: "The many studies of district and state minimum competency or diploma testing programs popular from the 1960s through the 1980s found positive effects for students just below the cut score and mixed effects for students far below and anywhere above. Researchers have included Fincher, Jackson, Battiste, Corcoran, Jacobsen, Tanner, Boylan, Saxon, Anderson, Muir, Bateson, Blackmore, Rogers, Zigarelli, Schafer, Hultgren, Hawley, Abrams, Seubert, Mazzoni, Brookhart, Mendro, Herrick, Webster, Orsak, Weeasinghe, and Bembry"

For example, from Table 3, Chapter 3 of Correcting Fallacies: "Relevant pre-2000 studies of the effects of minimum-competency testing and the problems with a single passing score include those of Frederiksen (1994); Winfield (1990); Ligon, Johnstone, Brightman, Davis, et al. (1990); Losack (1987); Mangino & Babcock (1986); Serow (1982); Brunton (1982); Paramore, et al. (1980); Ogden (1979); and Findley (1978)."

Brian Jacob

Susan Dynarski, Kenneth Frank, Barbara Schneider

"In addition, a small but growing body of research shows that taking certain core courses, especially those in math and science, can have significant, positive effects on long-term labor-market outcomes (Betts & Rose, 2004; Cortes, Goodman, & Nomi, 2015; Goodman, 2012; Levine & Zimmerman, 1995)." p.3

Dismissive

Are Expectations Alone Enough? Estimating the Effect of a Mandatory College-Prep Curriculum in Michigan

EPI Working Paper 01-2016, January, 2016 & NBER Working Paper No. 22013

"The U.S. Department of Education’s Institute of Education Sciences provided generous support through Grant R305E100008." & NBER funders

https://edpolicy.umich.edu/files/01-2016_are-expectations-alone-enough.pdf

Brian Jacob

Susan Dynarski, Kenneth Frank, Barbara Schneider

"Moreover, there is substantial evidence that high school exit exams, a closely related policy, increase drop-out rates, particularly among low-income students (Dee & Jacob, 2007; Jacob, 2001; Jenkins, Kulick, & Warren, 2006; Papay, Murnane, & Willett, 2010), and little evidence that they improve student achievement (Dee & Jacob, 2007; Grodsky, Kalogrides, & Warren, 2009)." p.4

Dismissive

Are Expectations Alone Enough? Estimating the Effect of a Mandatory College-Prep Curriculum in Michigan

EPI Working Paper 01-2016, January, 2016 & NBER Working Paper No. 22013

"The U.S. Department of Education’s Institute of Education Sciences provided generous support through Grant R305E100008." & NBER funders

https://edpolicy.umich.edu/files/01-2016_are-expectations-alone-enough.pdf

Brian Jacob

Susan Dynarski, Kenneth Frank, Barbara Schneider

"… with the exception of the working paper by Buddin and Croft (2014), this article is (to the best of our knowledge) the first analysis of a state attempt to require college-preparatory courses for all students." p.5

1stness

Are Expectations Alone Enough? Estimating the Effect of a Mandatory College-Prep Curriculum in Michigan

EPI Working Paper 01-2016, January, 2016 & NBER Working Paper No. 22013

"The U.S. Department of Education’s Institute of Education Sciences provided generous support through Grant R305E100008." & NBER funders

https://edpolicy.umich.edu/files/01-2016_are-expectations-alone-enough.pdf

Brian A. Jacob

Jonah Rockoff, Eric Taylor, Ben Lindy, Rachel Rosen

"...despite many decades of research, little progress has been made in establishing rigorous methods to select individuals likely to become successful teachers. ... More recent research has shown some promising results ... (Rockoff et al. 2011) ... (Boyd et al. 2008). Only one concurrent study (Goldhaber et al. 2014) examines the extent to which teacher performance can be predicted using data collected as part of an actual hiring process.",

Denigrating

Teacher Applicant Hiring and Teacher Performance: Evidence from DC Public Schools

New York Federal Reserve Bank

https://www.newyorkfed.org/medialibrary/media/research/education_seminar_series/jrtlr_teach_dc_23_feb_2015.pdf

The authors should have looked in the Industrial/Organizational Psychology (i.e., Personnel Psychology), for example Hunter & Schmidt's meta-analyses of the use of test instruments in personnel selection.

Brian A. Jacob

Jonah Rockoff, Eric Taylor, Ben Lindy, Rachel Rosen

"Selecting more effective teachers among job applicants during the hiring process could be a highly cost-effective means of improving educational quality, but there is little research that links information gathered during the hiring process to subsequent teacher performance.", Abstract

Dismissive

Teacher Applicant Hiring and Teacher Performance: Evidence from DC Public Schools

New York Federal Reserve Bank

https://www.newyorkfed.org/medialibrary/media/research/education_seminar_series/jrtlr_teach_dc_23_feb_2015.pdf

Brian A. Jacob

“And, yet, there is little empirical evidence on whether such incentives will change teacher behavior or improve student achievement.” p. 2

Dismissive

The Effect of Employment Protection on Teacher Effort

University of Michigan & NBER, March 2012

NBER funders

https://cep.lse.ac.uk/seminarpapers/07-05-13-BJ.pdf

Brian A. Jacob

“In addition, this analysis contributes to the economic literature on employment protection more generally. To the best of my knowledge, it is one of the few empirical studies of the impact of employment protection on worker effort...” p. 4

1stness

The Effect of Employment Protection on Teacher Effort

University of Michigan & NBER, March 2012

NBER funders

https://cep.lse.ac.uk/seminarpapers/07-05-13-BJ.pdf

In response to similar claims made by one of Jacob's colleagues, the teacher union expert Myron Lieberman showcased bibliographies on the topic with the number of references exceeding a thousand.

Brian A. Jacob

“ . . . the only study to directly examine this issue in the public sector.” p. 4

1stness

The Effect of Employment Protection on Teacher Effort

University of Michigan & NBER, March 2012

NBER funders

https://cep.lse.ac.uk/seminarpapers/07-05-13-BJ.pdf

In response to similar claims made by one of Jacob's colleagues, the teacher union expert Myron Lieberman showcased bibliographies on the topic with the number of references exceeding a thousand.

Brian A. Jacob

“Surprisingly few studies have examined the impact of employment protection on worker behavior.” p. 5

Dismissive

The Effect of Employment Protection on Teacher Effort

University of Michigan & NBER, March 2012

NBER funders

https://cep.lse.ac.uk/seminarpapers/07-05-13-BJ.pdf

In response to similar claims made by one of Jacob's colleagues, the teacher union expert Myron Lieberman showcased bibliographies on the topic with the number of references exceeding a thousand.

Brian A. Jacob

“Two recent reviews of pay-for-performance in education conclude that the existing evidence on merit pay is limited and shows mixed results (Springer and Podgursky 2008, Lavy 2008).” p. 6

Dismissive

The Effect of Employment Protection on Teacher Effort

University of Michigan & NBER, March 2012

NBER funders

https://cep.lse.ac.uk/seminarpapers/07-05-13-BJ.pdf

Relevant studies of the effects of varying types of incentive or the optimal structure of incentives include those of Kelley (1999); the *Southern Regional Education Board (1998); Trelfa (1998); Heneman (1998); Banta, Lund, Black & Oblander (1996); Brooks-Cooper, 1993; Eckstein & Noah (1993); Richards & Shen (1992); Jacobson (1992); Heyneman & Ransom (1992); *Levine & Lezotte (1990); Duran, 1989; *Crooks (1988); *Kulik & Kulik (1987); Corcoran & Wilson (1986); *Guskey & Gates (1986); Brook & Oxenham (1985); Oxenham (1984); Venezky & Winfield (1979); Brookover & Lezotte (1979); McMillan (1977); Abbott (1977); *Staats (1973); *Kazdin & Bootzin (1972); *O’Leary & Drabman (1971); Cronbach (1960); Hurlock (1925), and Zeng (2001). *Covers many studies; study is a research review, research synthesis, or meta-analysis. Other researchers who, even prior to 2000, studied test-based incentive programs include Homme, Csanyi, Gonzales, Rechs, O’Leary, Drabman, Kaszdin, Bootzin, Staats, Cameron, Pierce, McMillan, Corcoran, Roueche, Kirk, Wheeler, Boylan, and Wilson.

Moreover, the mastery learning/mastery testing experiments conducted from the 1960s through today varied incentives, frequency of tests, types of tests, and many other factors to determine the optimal structure of testing programs. Researchers included such notables as Bloom, Carroll, Keller, Block, Burns, Wentling, Anderson, Hymel, Kulik, Tierney, Cross, Okey, Guskey, Gates, and Jones.

Brian A. Jacob

"On this topic, debate has been vigorous but research almost nil,…"

Dismissive

Principled principals: New evidence from Chicago shows they fire the least effective teachers

Education Next, Fall 2011 / Vol. 11, No. 4

Harvard PEPG and Thomas B. Fordham Institute

https://educationnext.org/principled-principals/

Thomas S. Dee

Brian A. Jacob

"Though the reauthorization of NCLB is currently under consideration, the empirical evidence on the imnpact of NCLB on student achievement is, to date, extremely limited."

Dismissive

The Impact of No Child Left Behind on Student Achievement, p.419

Journal of Public Policy Analysis & Management, 30(3), 418–446 (2011)

"An earlier version of this work was also presented byJacob as the David N. Kershaw Lecture at the Annual meeting of the Association of Public Policyand Management (November 2008)"

Thomas S. Dee

Brian A. Jacob

"In a recent review of this diverse evaluation literature, Figlio and Ladd (2008) suggest that three studies (Carnoy & Loeb, 2002; Jacob, 2005; Hanushek & Raymond, 2005) are the “most methodologically sound” (Ladd, 2007)."

Denigrating

The Impact of No Child Left Behind on Student Achievement, p.420–421

Journal of Public Policy Analysis & Management, 30(3), 418–446 (2011)

"An earlier version of this work was also presented byJacob as the David N. Kershaw Lecture at the Annual meeting of the Association of Public Policyand Management (November 2008)"

Thomas S. Dee

Brian A. Jacob

“Few studies have implemented regression-based research designs that attempt to isolate the effects of school accountability policies on district, school, and classroom practices from the potentially confounding effects of other determinants.” p. 181

Denigrating

The Impact of No Child Left Behind on Students, Teachers, and Schools

Brookings Papers on Economic Activity, Fall 2010

Brookings Institution Funders, "An earlier version of this work was also presented byJacob as the David N. Kershaw Lecture at the Annual meeting of the Association of Public Policyand Management (November 2008)"

https://www.brookings.edu/~/media/Projects/BPEA/Fall%202010/2010b_bpea_dee.PDF

No. See, for example, Phelps, R.P. (2000, Winter). Estimating the cost of systemwide student testing in the United States. Journal of Education Finance, 25(3) 343–380; Danitz, T. (2001, February 27). Special report: States pay $400 million for tests in 2001. Stateline.org. Pew Center for the States; Hoxby, C.M. (2002). The cost of accountability, in W. M Evers & H.J. Walberg (Eds.), School Accountability, Stanford, CA: Hoover Institution Press; U.S. GAO. (1993, January). Student testing: Current extent and expenditures, with cost estimates for a national examination. GAO/PEMD-93-8. Washington, DC: US General Accounting Office; Phelps, R.P. (1998). Benefit-cost analysis of systemwide student testing, Paper presented at the annual meeting of the American Education Finance Association, Mobile, AL.

Thomas S. Dee

Brian A. Jacob

"Overall, the extant literature offers at best suggestive evidence on how accountability reforms may have influenced school spending." p.176

Dismissive, Denigrating

The Impact of No Child Left Behind on Students, Teachers, and Schools

Brookings Papers on Economic Activity, Fall 2010

https://www.brookings.edu/~/media/Projects/BPEA/Fall%202010/2010b_bpea_dee.PDF

Thomas S. Dee

Brian A. Jacob

“However, there is surprisingly little research on the relationship between school accountability and spending, despite an extensive literature on education finance more generally.” p. 175

Dismissive

The Impact of No Child Left Behind on Students, Teachers, and Schools

Brookings Papers on Economic Activity, Fall 2010

https://www.brookings.edu/~/media/Projects/BPEA/Fall%202010/2010b_bpea_dee.PDF

Thomas S. Dee

Brian A. Jacob

"In a recent review of this diverse evaluation literature, David Figlio and Helen Ladd (2007) suggest that three studies (Carnoy and Loeb 2002, Jacob 2005, and Hanushek and Raymond 2005) are the “most methodologically sound” (Ladd 2007, p. 9)." p.152

Denigrating

The Impact of No Child Left Behind on Students, Teachers, and Schools

Brookings Papers on Economic Activity, Fall 2010

https://www.brookings.edu/~/media/Projects/BPEA/Fall%202010/2010b_bpea_dee.PDF

Brian A. Jacob

“In contrast, there has been remarkably little research on the demand side of the teacher labor market. For example, few studies have examined how principals hire or fire teachers, or how changes in personnel policies might influence teacher quality.” p. 2

Dismissive

Do Principals Fire the Worst Teachers?

NBER Working Paper No. 15715, February 2010

NBER funders

https://www-personal.umich.edu/~baBrian A. Jacob/w15715_teacher_firing.pdf

Brian A. Jacob

“This void is not unique to education research. There is a vast economics literature on employee compensation, for example, but relatively few empirical studies that examine the factors that employers consider when hiring or dismissing workers.” p. 3

Dismissive

Do Principals Fire the Worst Teachers?

NBER Working Paper No. 15715, February 2010

NBER funders

https://www-personal.umich.edu/~baBrian A. Jacob/w15715_teacher_firing.pdf

Brian A. Jacob

“While many studies mention the determinants of job displacement, few studies attempt to carefully explore employer preferences for worker characteristics. One important exception is the literature on employer discrimination.” p. 6

Dismissive

Do Principals Fire the Worst Teachers?

NBER Working Paper No. 15715, February 2010

NBER funders

https://www-personal.umich.edu/~baBrian A. Jacob/w15715_teacher_firing.pdf

Brian A. Jacob

Specifically, I document the extent to which student performance trends on state assessments differ from those on the National Assessment of Educational Progress (NAEP). While such divergence has been documented in several studies ... there has been no systematic analysis of this issue nationwide..., p.2

Dismissive

Test-based accountability and student achievement: An investigation of differential performance on NAEP and state assessment

CLOSUP Working Paper Series, Number 17, February 2009, U. Michigan

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://closup.umich.edu

In their 2009 Evaluation of NAEP for the US Education Department, Buckendahl, Davis, Plake, Sireci, Hambleton, Zenisky, & Wells (pp. 77–85) managed to find quite a lot of research on making comparisons between NAEP and state assessments: several of NAEP's own publications, Chromy 2005), Chromy, Ault, Black, & Mosquin (2007), McLaughlin (2000), Schuiz & Mitzel (2005), Sireci, Robin, Meara, Rogers, & Swaminathan (2000), Stancavage, Et al (2002), Stoneberg (2007), WestEd (2002), and Wise, Le, Hoffman, & Becker (2004).

Brian A. Jacob

To fill this gap in the research literature, …, p.3

Dismissive

Test-based accountability and student achievement: An investigation of differential performance on NAEP and state assessment

CLOSUP Working Paper Series, Number 17, February 2009, U. Michigan

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://closup.umich.edu

Brian A. Jacob

More importantly, there has been little research on the reasons why student performance differs between NAEP and local assessments, p.11

Dismissive

Test-based accountability and student achievement: An investigation of differential performance on NAEP and state assessment

CLOSUP Working Paper Series, Number 17, February 2009, U. Michigan

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://closup.umich.edu

Brian A. Jacob

However, this work does not explicitly examine the issue of test score inflation. Jacob (2005) makes an effort to fill this gap. P.12

Dismissive

Test-based accountability and student achievement: An investigation of differential performance on NAEP and state assessment

CLOSUP Working Paper Series, Number 17, February 2009, U. Michigan

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://closup.umich.edu

In fact the test prep, or test coaching, literature is vast and dates back decades, with meta-analyses of the literature dating back at least to the 1970s. There's even a What Works Clearinghouse summary of the (post World Wide Web) college admission test prep research literature: https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/wwc_act_sat_100416.pdf . See also: Messick & Jungeblut (1981) Ellis, Konoske, Wulfeck, & Montague (1982) DerSimonian and Laird (1983) Kulik, Bangert-Drowns & Kulik (1984) Fraker (1986/1987) Halpin (1987) Whitla (1988) Snedecor (1989) Becker (1990) Smyth (1990) Moore (1991) Alderson & Wall (1992) Powers (1993) Powers & Rock (1994) Scholes, Lane (1997) Allalouf & Ben Shakhar (1998) Robb & Ercanbrack (1999) McClain (1999) Camara (1999, 2001, 2008) Stone & Lane (2000, 2003) Din & Soldan (2001) Briggs (2001) Palmer (2002) Briggs & Hansen (2004) Cankoy & Ali Tut (2005) Crocker (2005) Allensworth, Correa, & Ponisciak (2008) Domingue & Briggs (2009) Early (2019)

Brian A. Jacob

Jonah Rockoff, Thomas J. Kane, Douglas O. Staiger

“Research on the relationship between teachers' characteristics and teacher effectiveness has been underway for over a century, yet little progress has been made in linking teacher quality with factors observable at the time of hire.” p. 1

Dismissive

Can You Recognize an Effective Teacher When You Recruit One?

NBER Working Paper 14485, November 2008

We are grateful to the Spencer Foundation and the Carnegie Corporation for generous
financial support.

https://www.dartmouth.edu/~dstaiger/Papers/w14485.pdf

Just like there exists a massive literature on "effective schools" there exists a subset on "effective teachers."

Brian A. Jacob

Jonah Rockoff, Thomas J. Kane, Douglas O. Staiger

“However, most research on teacher effectiveness has examined a relatively small set of teacher characteristics, such as graduate education and certification . . . researchers’ lack of success in predicting new teacher performance may be driven by a narrow focus on commonly available data.” p. 1

Denigrating

Can You Recognize an Effective Teacher When You Recruit One?

NBER Working Paper 14485, November 2008

We are grateful to the Spencer Foundation and the Carnegie Corporation for generous
financial support.

https://www.dartmouth.edu/~dstaiger/Papers/w14485.pdf

Just like there exists a massive literature on "effective schools" there exists a subset on "effective teachers."

Brian A. Jacob

Jonah Rockoff, Thomas J. Kane, Douglas O. Staiger

“While many studies have been conducted, few definitive conclusions have been made. One reason has been the widespread but controversial use of the Minnesota Multiphasic Personality Inventory. …” p. 8

Dismissive

Can You Recognize an Effective Teacher When You Recruit One?

NBER Working Paper 14485, November 2008

We are grateful to the Spencer Foundation and the Carnegie Corporation for generous
financial support.

https://www.dartmouth.edu/~dstaiger/Papers/w14485.pdf

Just like there exists a massive literature on "effective schools" there exists a subset on "effective teachers."

Brian A. Jacob

Jonah Rockoff, Thomas J. Kane, Douglas O. Staiger

“However, there is little work examining the relationship between self-efficacy and student learning.” p. 9

Dismissive

Can You Recognize an Effective Teacher When You Recruit One?

NBER Working Paper 14485, November 2008

We are grateful to the Spencer Foundation and the Carnegie Corporation for generous
financial support.

https://www.dartmouth.edu/~dstaiger/Papers/w14485.pdf

Just like there exists a massive literature on "effective schools" there exists a subset on "effective teachers."

Brian A. Jacob

Jonah Rockoff, Thomas J. Kane, Douglas O. Staiger

“In addition to being one of the first studies of teacher value-added and its correlation with principal evaluations, this paper also finds a significant positive relationship between teachers’ sense of self-efficacy and student achievement growth.” p. 10

1stness

Can You Recognize an Effective Teacher When You Recruit One?

NBER Working Paper 14485, November 2008

We are grateful to the Spencer Foundation and the Carnegie Corporation for generous
financial support.

https://www.dartmouth.edu/~dstaiger/Papers/w14485.pdf

Tennessee's TVAAS value-added measurement system had been running a decade when they wrote this and did much of what these authors claim had never been done.

Brian A. Jacob

Jonah Rockoff, Thomas J. Kane, Douglas O. Staiger

“While use of commercial selection instruments has grown considerably, there is little systematic evidence on the power of these instruments for predicting teacher effectiveness.” p. 11

Dismissive

Can You Recognize an Effective Teacher When You Recruit One?

NBER Working Paper 14485, November 2008

We are grateful to the Spencer Foundation and the Carnegie Corporation for generous
financial support.

https://www.dartmouth.edu/~dstaiger/Papers/w14485.pdf

Thomas S. Dee, Brian A. Jacob

Adam Gamoran, Ed

"Despite their widespread acceptance, these student-focused reforms have been the subject of little rigorous inquiry." p.155

Denigrating

Do High School Exit Exams Influence Educational Attainment or Labor Market Performance?

Standards-Based Reform and the Poverty Gap: Lessons for No Child Left Behind, Brookings Institution, 2007

Brookings Institution Funders

In fact, many, if not most, large-scale testing and accountability programs in the past have been evaluated. The evaluation reports tended to end up on shelves in district and state research bureaus. Some declare there to be no research after looking only in the most easily accessible locations for the most easily retrieved evidence.

Just some of the relevant pre-2008 studies of the effects of minimum-competency or exit exams and the problems with a single passing score include those of Alvarez, Moreno, & Patrinos (2007); Grodsky & Kalogrides (2006); Audette (2005); Orlich (2003); StandardsWork (2003); Meisels, et al. (2003); Braun (2003); Rosenshine (2003); Tighe, Wang, & Foley (2002); Carnoy & Loeb (2002); Baumert & Demmrich (2001); Rosenblatt & Offer (2001); Phelps (2001); Toenjes, Dworkin, Lorence, & Hill (2000); Wenglinsky (2000); Massachusetts Finance Office (2000); DeMars (2000); Bishop (1999, 2000, 2001, & 2004); Grissmer & Flanagan(1998); Strauss, Bowes, Marks, & Plesko (1998); Frederiksen (1994); Ritchie & Thorkildsen (1994); Chao-Qun & Hui (1993); Potter & Wall (1992); Jacobson (1992); Rodgers, et al. (1991); Morris (1991); Winfield (1990); Ligon, Johnstone, Brightman, Davis, et al. (1990); Winfield (1987); Koffler (1987); Losack (1987); Marshall (1987); Hembree (1987); Mangino, Battaille, Washington, & Rumbaut (1986); Michigan Department of Education (1984); Ketchie (1984); Serow (1982); Indiana Education Department (1982); Brunton (1982); Paramore, et al. (1980); Ogden (1979); Down(2) (1979); Wellisch (1978); and Findley (1978).

Thomas S. Dee, Brian A. Jacob

Adam Gamoran, Ed

"In this study, we present new empirical evidence … This study makes three broad contributions to the extant literature.." p.155

1stness

Do High School Exit Exams Influence Educational Attainment or Labor Market Performance?

Standards-Based Reform and the Poverty Gap: Lessons for No Child Left Behind, Brookings Institution, 2007

Brookings Institution Funders

Thomas S. Dee, Brian A. Jacob

Adam Gamoran, Ed

"However, despite the prominence of standards-based reforms, these long-standing state-level experiements have actually been the subject of relatively little empirical scrutiny." p.159

Dismissive

Do High School Exit Exams Influence Educational Attainment or Labor Market Performance?

Standards-Based Reform and the Poverty Gap: Lessons for No Child Left Behind, Brookings Institution, 2007

Brookings Institution Funders

Thomas S. Dee, Brian A. Jacob

Adam Gamoran, Ed

"The evidence that does exist is generally mixed and unsatisfactory." p.159

Denigrating

Do High School Exit Exams Influence Educational Attainment or Labor Market Performance?

Standards-Based Reform and the Poverty Gap: Lessons for No Child Left Behind, Brookings Institution, 2007

Brookings Institution Funders

Brian A. Jacob

Lars Lefgren

“The few studies that examine the correlation between principal evaluations and other measures of teacher performance, such as parent or student satisfaction, find similarly weak relationships (Peterson 1987, 2000).” p. 6, note 4

Dismissive

Can Principals Identify Effective Teachers? Evidence on Subjective Performance Evaluation in Education

June 2007, [eventually published under this title in JEL Vol. 26.2008, 1, p. 101-136]

https://economics.byu.edu/Documents/Lars%20Lefgren/papers/principals.pdf

Just like there exists a massive literature on "effective schools" there exists a subset on "effective teachers."

Brian A. Jacob

"While such divergence has been documented in several studies, there has been no systematic analysis of this issue nationwide."

Dismissive

TEST-BASED ACCOUNTABILITY AND STUDENT ACHIEVEMENT: AN INVESTIGATION OF DIFFERENTIAL PERFORMANCE ON NAEP AND STATE ASSESSMENTS, p.2

NBER Working Paper 12817, January 2007

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://www.nber.org/papers/w12817

Relevant studies of the effects of tests and/or accountability program on motivation and instructional practice: Goslin (1967), *Southern Regional Education Board (1998); Johnson (1998); Schafer, Hultgren, Hawley, Abrams Seubert & Mazzoni (1997); Miles, Bishop, Collins, Fink, Gardner, Grant, Hussain, et al. (1997); Tuckman & Trimble (1997); Clarke & Stephens (1996); Zigarelli (1996); Stevenson, Lee, et al. (1995); Waters, Burger & Burger (1995); Egeland (1995); Prais (1995); Tuckman (1994); Ritchie & Thorkildsen (1994); Brown & Walberg, (1993); Wall & Alderson (1993); Wolf & Rapiau (1993); Eckstein & Noah (1993); Chao-Qun & Hui (1993); Plazak & Mazur (1992); Steedman (1992); Singh, Marimutha & Mukjerjee (1990); *Levine & Lezotte (1990); O’Sullivan (1989); Somerset (1988); Pennycuick & Murphy (1988); Stevens (1984); Marsh (1984); Brunton (1982); Solberg (1977); Foss (1977); *Kirkland (1971); Somerset (1968); Stuit (1947); and Keys (1934). *Covers many studies; study is a research review, research synthesis, or meta-analysis.

Brian A. Jacob

"To fill this gap in the research literature, over a period of several years I collected…"

Dismissive

TEST-BASED ACCOUNTABILITY AND STUDENT ACHIEVEMENT: AN INVESTIGATION OF DIFFERENTIAL PERFORMANCE ON NAEP AND STATE ASSESSMENTS, p.3

NBER Working Paper 12817, January 2007

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://www.nber.org/papers/w12817

Brian A. Jacob

"I attempt to go beyond previous studies that merely document aggregate performance difference across exams to explore why performance trends may differ across these exams." p.3

Denigrating

TEST-BASED ACCOUNTABILITY AND STUDENT ACHIEVEMENT: AN INVESTIGATION OF DIFFERENTIAL PERFORMANCE ON NAEP AND STATE ASSESSMENTS

NBER Working Paper 12817, January 2007

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://www.nber.org/papers/w12817

Brian A. Jacob

"A growing body of evidence suggests that test-based accountability programs increase student achievement (Richards and Sheu 1992, Grissmer et. al. 2000, Deere and Strayer 2001, Jacob 2002, Carnoy and Loeb 2002, Hanushek and Raymond 2002)." p.9

Denigrating

TEST-BASED ACCOUNTABILITY AND STUDENT ACHIEVEMENT: AN INVESTIGATION OF DIFFERENTIAL PERFORMANCE ON NAEP AND STATE ASSESSMENTS

NBER Working Paper 12817, January 2007

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://www.nber.org/papers/w12817

Brian A. Jacob

Denigrating

TEST-BASED ACCOUNTABILITY AND STUDENT ACHIEVEMENT: AN INVESTIGATION OF DIFFERENTIAL PERFORMANCE ON NAEP AND STATE ASSESSMENTS

NBER Working Paper 12817, January 2007

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://www.nber.org/papers/w12817

Brian A. Jacob

"More importantly, there has been little research on the reasons why student performance differs between NAEP and local assessments."

Dismissive

TEST-BASED ACCOUNTABILITY AND STUDENT ACHIEVEMENT: AN INVESTIGATION OF DIFFERENTIAL PERFORMANCE ON NAEP AND STATE ASSESSMENTS, p.11

NBER Working Paper 12817, January 2007

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://www.nber.org/papers/w12817

Brian A. Jacob

"Jacob (2005) makes an effort to fill this gap. He found…"

Dismissive

TEST-BASED ACCOUNTABILITY AND STUDENT ACHIEVEMENT: AN INVESTIGATION OF DIFFERENTIAL PERFORMANCE ON NAEP AND STATE ASSESSMENTS, p.12

NBER Working Paper 12817, January 2007

Funding for this project was generously provided by the U.S. Department of Education NAEP Secondary Analysis Grant (#R902B030024).

https://www.nber.org/papers/w12817

Lars Lefgren

Brian A. Jacob

"While principals can and do judge teachers’ performance, however, there is little good evidence on the accuracy of their judgments. The research reported in this paper fills this gap."

Dismissive, 1stness

When principals rate teachers: the best--and the worst--stand out

Education Next, Spring 2006 / Vol. 6, No. 2

Harvard PEPG and Thomas B. Fordham Institute

https://educationnext.org/whenprincipalsrateteachers/

Thomas S. Dee

Brian A. Jacob

"However, these pervasive reforms [high school exit exams] have been the subject of little rigorous empirical scrutiny." p.3

Dismissive

Do High School Exit Exams Influence Educational Attainment or Labor Market Performance?

NBER Working Paper 12199, April 2006

NBER funders

https://www.nber.org/papers/w12199

Thomas S. Dee

Brian A. Jacob

"The literature that is available provides contradictory evidence on the fundamental consequences of these state initiatives." p.3

Denigrating

Do High School Exit Exams Influence Educational Attainment or Labor Market Performance?

NBER Working Paper 12199, April 2006

NBER funders

https://www.nber.org/papers/w12199

Thomas S. Dee

Brian A. Jacob

"However, despite the prominence of standards-based reforms, these long-standing state-level experiments have actually been the subject of relatively little empirical scrutiny." p.7

Dismissive

Do High School Exit Exams Influence Educational Attainment or Labor Market Performance?

NBER Working Paper 12199, April 2006

NBER funders

https://www.nber.org/papers/w12199

Thomas S. Dee

Brian A. Jacob

"The evidence that does exist is generally mixed and unsatisfactory. This is largely due to two issues: the lack of consistent and reliable data on outcomes such as achievement or educational attainment, and the difficulty of differentiating between effects due to the exit exams and effects due to other related policies or conditions in the state at the time." p.7

Denigrating

Do High School Exit Exams Influence Educational Attainment or Labor Market Performance?

NBER Working Paper 12199, April 2006

NBER funders

https://www.nber.org/papers/w12199

Brian A. Jacob

Lars Lefgren

“An important feature of most Section 8 programs including Gautreaux and MTO is that they involve voluntary relocation. Only a few studies examine forced relocation.” p. 1

Dismissive

Principals as Agents: Subjective Performance Measurement in Education

Kennedy School of Government Faculty Research Working Paper Series, RWP05-040, June 2005

Harvard Kennedy School

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.9537&rep=rep1&type=pdf

Brian A. Jacob

Lars Lefgren

“The paper also speaks to the broader literature on subjective performance assessment. While such evaluations are central to promotion, retention and compensation decisions in most industries, they have received relatively little attention in the economics literature (Prendergast 1999).” p. 5

Dismissive

Principals as Agents: Subjective Performance Measurement in Education

Kennedy School of Government Faculty Research Working Paper Series, RWP05-040, June 2005

Harvard Kennedy School

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.9537&rep=rep1&type=pdf

Brian A. Jacob

Lars Lefgren

Dismissive

Principals as Agents: Subjective Performance Measurement in Education

Kennedy School of Government Faculty Research Working Paper Series, RWP05-040, June 2005

Harvard Kennedy School

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.9537&rep=rep1&type=pdf

Brian A. Jacob

"The recent federal education bill, No Child Left Behind, requires states to test students in grades three to eight each year, and to judge school performance on the basis of these test scores. While intended to maximize student learning, there is little empirical evidence about the effectiveness of such policies."

Dismissive

Accountability, Incentives and Behavior: The Impact of High-Stakes Testing in the Chicago Public Schools, Abstract, 2002.

Journal of Public Economics, Volume 89, Issues 5Ð6, June 2005, Pages 761-796.

Funding for this research was provided by the Spencer Foundation

https://www.nber.org/papers/w8968.pdf

https://www.sciencedirect.com/science/article/pii/S0047272704001549

Brian A. Jacob

"Despite its increasing popularity within education, there is little empirical evidence on test-based accountability (also referred to as high-stakes testing)."

Dismissive

Accountability, Incentives and Behavior: The Impact of High-Stakes Testing in the Chicago Public Schools, p. 2, 2002.

Journal of Public Economics, Volume 89, Issues 5Ð6, June 2005, Pages 761-796.

Funding for this research was provided by the Spencer Foundation

https://www.nber.org/papers/w8968.pdf

https://www.sciencedirect.com/science/article/pii/S0047272704001549

Brian A. Jacob

"...most studies of school-based accountability do not utilize individual students data and thus cannot examine many outcomes of interest or investigate how effects vary across students."

Denigrating

Accountability, Incentives and Behavior: The Impact of High-Stakes Testing in the Chicago Public Schools, p. 2, 2002.

Journal of Public Economics, Volume 89, Issues 5Ð6, June 2005, Pages 761-796.

Funding for this research was provided by the Spencer Foundation

https://www.nber.org/papers/w8968.pdf

https://www.sciencedirect.com/science/article/pii/S0047272704001549

Brian A. Jacob

"...the federal government has moved to ensure a minimal level of testing and reporting that only a decade ago would have been unthinkable."

Dismissive

Accountability, Incentives and Behavior: The Impact of High-Stakes Testing in the Chicago Public Schools, p. 2, 2002.

Journal of Public Economics, Volume 89, Issues 5Ð6, June 2005, Pages 761-796.

Funding for this research was provided by the Spencer Foundation

https://www.nber.org/papers/w8968.pdf

https://www.sciencedirect.com/science/article/pii/S0047272704001549

Brian A. Jacob

Lars Lefgren

"As standards and accountability have become increasingly prominent features of the educational landscape, educators have relied more on remedial programs such as summer school and grade retention to help low-achieving students meet minimum academic standards. Yet the evidence on the effectiveness of such programs is mixed, and prior research suffers from selection bias."

Denigrating

Remedial Education and Student Achievement: A Regression-Discontinuity Analysis, Abstract

The Review of Economics and Statistics, February 2004, Vol. 86, No. 1, Pages: 226-244

https://www.mitpressjournals.org/doi/abs/10.1162/003465304323023778#.WRSzmcm1vHF

For example, from Table 2, Chapter 3 of Correcting Fallacies, "Developmental (i.e., remedial) education researchers have conducted many studies to determine what works best to keep students from failing in their “courses of last resort,” after which there are no alternatives. Researchers have included Boylan, Roueche, McCabe, Wheeler, Kulik, Bonham, Claxton, Bliss, Schonecker, Chen, Chang, and Kirk."

Brian A. Jacob

Lars Lefgren

"However, researchers have paid considerably less attention to remedial programs designed to improve the performance of low achieving students, including summer school and grade retention (Eide and Showalter (forthcoming)." p.1

Dismissive

Remedial Education and Student Achievement: A Regression-Discontinuity Analysis

The Review of Economics and Statistics, February 2004, Vol. 86, No. 1, Pages: 226-244

https://www.mitpressjournals.org/doi/abs/10.1162/003465304323023778#.WRSzmcm1vHF

Brian A. Jacob

Lars Lefgren

"However, prior studies do not adequately address the potential biases introduced by the non-random selection into summer school and retention."

Denigrating

Remedial Education and Student Achievement: A Regression-Discontinuity Analysis

The Review of Economics and Statistics, February 2004, Vol. 86, No. 1, Pages: 226-244

https://www.mitpressjournals.org/doi/abs/10.1162/003465304323023778#.WRSzmcm1vHF

Steven D. Levitt

Brian A. Jacob

"These scandals have aroused public concern, but there has been little hard evidence on the extent of cheating by school personnel on the type of tests required by recently enacted accountability legislation."

Dismissive

To catch a cheat

Education Next, Winter 2004 / Vol. 4, No. 1

Harvard PEPG and Thomas B. Fordham Institute

https://educationnext.org/tocatchacheat/

See, for example, https://nonpartisaneducation.org/Review/Articles/v6n3.htm ; https://nonpartisaneducation.org/Review/Books/CannellBook1.htm

Brian A. Jacob

Steven D. Levitt

“There has been very little previous empirical analysis of teacher cheating. ...Our paper represents the first systematic attempt to (1) identify the overall prevalence of teacher cheating empirically and (2) analyze the factors that predict cheating.” p. 845

1stness

Rotten Apples: An Investigation of the Prevalence and Predictors of Teacher Cheating

Quarterly Journal of Economics, August 2003

Financial support was provided by the National Science Foundation and the Sloan Foundation.

https://pricetheory.uchicago.edu/levitt/Papers/Brian A. JacobLevitt2003.pdf

See, for example, https://nonpartisaneducation.org/Review/Articles/v6n3.htm ; https://nonpartisaneducation.org/Review/Books/CannellBook1.htm

Brian A. Jacob

Steven D. Levitt

“Finally, this paper fits into a small but growing body of research focused on identifying corrupt or illicit behavior on the part of economic actors (see Porter and Zona [1993], Fisman [2001], Di Tella and Schargrodsky [2001], and Duggan and Levitt [2002]).” p. 871

Dismissive

Rotten Apples: An Investigation of the Prevalence and Predictors of Teacher Cheating

Quarterly Journal of Economics, August 2003

Financial support was provided by the National Science Foundation and the Sloan Foundation.

https://pricetheory.uchicago.edu/levitt/Papers/Brian A. JacobLevitt2003.pdf

See, for example, https://nonpartisaneducation.org/Review/Articles/v6n3.htm ; https://nonpartisaneducation.org/Review/Books/CannellBook1.htm

Brian A. Jacob

“Despite this shift, there is relatively little evidence on the impact of public housing or housing vouchers on educational outcomes.” p. 1

Dismissive

Public Housing, Housing Vouchers and Student Achievement: Evidence from Public Housing Demolitions in Chicago

Working Paper 9652, April 2003

https://core.ac.uk/download/pdf/6707824.pdf

Brian A. Jacob

“An important feature of most Section 8 programs including Gautreaux and MTO is that they involve voluntary relocation. Only a few studies examine forced relocation.”, p.1

Dismissive

Public Housing, Housing Vouchers and Student Achievement: Evidence from Public Housing Demolitions in Chicago

Working Paper 9652, April 2003

https://core.ac.uk/download/pdf/6707824.pdf

Brian A. Jacob

"Chicago’s experience with accountability provides some lessons for other districts and states as they begin to implement the mandates of No Child Left Behind."

Dismissive

High Stakes in Chicago

Education Next, v.1., p.66, 2003.

Harvard PEPG and Thomas B. Fordham Institute

https://educationnext.org/highstakesinchicago/

https://nonpartisaneducation.org/Foundation/ThinkTankThoughtlessness.htm

per usual for such studies, there were no controls for any variation in test administration procedures. There have been "large, urban school districts" with high-stakes testing since at least the 1940s.

Brian A. Jacob

"As the first large urban school district to introduce a comprehensive accountability system, [our city] provides an exceptional case study of the effects of high-stakes testing--a reform strategy that will become omnipresent as the No Child Left Behind Act is implemented nationwide."

1stness

High Stakes in Chicago

Education Next, v.1., p.66, 2003.

Harvard PEPG and Thomas B. Fordham Institute

https://educationnext.org/highstakesinchicago/

https://nonpartisaneducation.org/Foundation/ThinkTankThoughtlessness.htm

Brian A. Jacob

Melissa Roderick, Anthony Bryck

"There has been little investigation of whether the purported benefits of these policies (of standardized grade promotion testing)-in the form of increased achievement on standardized tests-actually occur. ...We know very little about whether the introduction of high-stakes testing, particularly when combined with extra resources and with school accountability measures, will increase achievement on standardized tests for all students prior to the promotional gate (both those who are promoted as well as those who may later be retained)."

Dismissive

The impact of high-stakes testing in Chicago on student achievement in the promotional gate grades.

Educational Evaluation and Policy Analysis, 24(4):333-57, 2002.

https://nonpartisaneducation.org/Foundation/ThinkTankThoughtlessness.htm

Brian A. Jacob

Melissa Roderick, Anthony Bryck

"In 1996, [our city's schools] became one of the first large, urban school districts to implement high-stakes testing, introducing a comprehensive accountability program that incorporated incentives for both students and teachers."

1stness

The impact of high-stakes testing in Chicago on student achievement in the promotional gate grades.

Educational Evaluation and Policy Analysis, 24(4):333-57, 2002.

https://nonpartisaneducation.org/Foundation/ThinkTankThoughtlessness.htm

Brian A. Jacob

Melissa Roderick, Anthony Bryck

"In 1996, [our city] began a national trend when it coupled a new school-level accountability program with an accountability initiative with high-stakes consequences for students. ...Over the past five years, virtually every major school system and many states...have instituted elements of [our city's] policy."

1stness

The impact of high-stakes testing in Chicago on student achievement in the promotional gate grades.

Educational Evaluation and Policy Analysis, 24(4):333-57, 2002.

https://nonpartisaneducation.org/Foundation/ThinkTankThoughtlessness.htm

Brian A. Jacob

Lars Lefgren

"As standards and accountability have become an increasingly prominent feature of the educational landscape, educators have relied more on remedial programs such as summer school and grade retention to help low-achieving students meet minimum academic standards. Yet the evidence on the effectiveness of such programs is mixed, and prior research suffers from selection bias."

Denigrating

REMEDIAL EDUCATION AND STUDENT ACHIEVEMENT: A REGRESSION-DISCONTINUITY ANALYSIS, Abstract

NBER WORKING PAPER SERIES, Working Paper 8918

NBER funders

https://www.nber.org/papers/w8918

For example, from Table 3, Chapter 3 of Correcting Fallacies: "Relevant pre-2000 studies of the effects of testing on at-risk students, completion, dropping out, curricular offerings, attitudes, etc. include those of Schleisman (1999); the *Southern Regional Education Board (1998); Webster, Mendro, Orsak, Weerasinghe & Bembry (1997); Jones (1996); Boylan (1996); Jones, 1993; Jacobson (1992); Grisay (1991); Johnstone (1990); Task Force on Educational Assessment Programs [Florida] (1979); Wellisch, MacQueen, Carriere & Duck (1978); Enochs (1978); Pronaratna (1976); and McWilliams & Thomas (1976)."

Brian A. Jacob

Lars Lefgren

"Aware of the importance of education, economists have spent considerable effort examining what factors affect academic achievement. There is a large literature on the importance of financial resources in determining educational outcomes. However, researchers have paid considerably less attention to remedial programs designed to improve the performance of low achieving students, including summer school and grade retention (Eide and Showalter forthcoming)."

Dismissive

REMEDIAL EDUCATION AND STUDENT ACHIEVEMENT: A REGRESSION-DISCONTINUITY ANALYSIS, p.1

NBER WORKING PAPER SERIES, Working Paper 8918

NBER funders

https://www.nber.org/papers/w8918

The authors admit that much previous work has been done on the topic, (e.g., "In a detailed synthesis of 93 summer school evaluations, Cooper et. al. (2000)…"). They imply that they have read all of the research literature in detail, but cite none of it specifically. Then, they claim all of it is inferior to their work.

Brian A. Jacob

Lars Lefgren

"In contrast, the majority of retention studies find that the practice of requiring students to repeat a grade decreases self-esteem, school adjustment, and academic achievement, and increases dropout rates. However, prior studies fail to account for the selection of students into these programs, thus potentially overstating the benefits of summer school and exaggerating the harm of retention."

Denigrating

REMEDIAL EDUCATION AND STUDENT ACHIEVEMENT: A REGRESSION-DISCONTINUITY ANALYSIS, p.2

NBER WORKING PAPER SERIES, Working Paper 8918

NBER funders

https://www.nber.org/papers/w8918

Brian A. Jacob

Lars Lefgren

"Both summer school and grade retention have a long history within American education, dating back to the introduction of mass public education in the mid-nineteenth century (Shepard and Smith 1989). Both practices have been widely implemented and have received considerable attention from researchers. However, prior studies do not adequately address the potential biases introduced by the non-random selection into summer school and retention." p.3

Denigrating

REMEDIAL EDUCATION AND STUDENT ACHIEVEMENT: A REGRESSION-DISCONTINUITY ANALYSIS, p.3

NBER WORKING PAPER SERIES, Working Paper 8918

NBER funders

https://www.nber.org/papers/w8918

Brian A. Jacob

Lars Lefgren

"However, even the most careful of these studies relied on comparisons between students who chose to attend summer school with those who chose not to attend. If the most motivated students (or those with the most motivated, supportive parents) attend summer school, then the estimated summer school treatment effect will be biased upward. In addition, these studies do not examine whether these benefits are sustained in subsequent years." p.4

Denigrating

REMEDIAL EDUCATION AND STUDENT ACHIEVEMENT: A REGRESSION-DISCONTINUITY ANALYSIS, p.3

NBER WORKING PAPER SERIES, Working Paper 8918

NBER funders

https://www.nber.org/papers/w8918

Brian A. Jacob

"There is less evidence on whether, and to what extent, accountability programs lead to test score inflation." p.2

Dismissive

"Test-Based Accountability and Student Achievement Gains: Theory and Evidence"

Taking Account of Accountability: Assessing Politics and Policy, John F. Kennedy School of Government. Harvard University, June 10 - 11, 2002

In fact the test prep, or test coaching, literature is vast and dates back decades, with meta-analyses of the literature dating back at least to the 1970s. There's even a What Works Clearinghouse summary of the (post World Wide Web) college admission test prep research literature: https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/wwc_act_sat_100416.pdf . See also: Gilmore (1927) French & Dear (1959) Ortar (1960) Marron (1965) ETS (1965). Messick & Jungeblut (1981) Ellis, Konoske, Wulfeck, & Montague (1982) DerSimonian and Laird (1983) Kulik, Bangert-Drowns & Kulik (1984) Powers (1985) Samson (1985) Scruggs, White, & Bennion (1986) Jones (1986). Fraker (1986/1987) Halpin (1987) Whitla (1988) Snedecor (1989) Bond (1989). Baydar (1990) Becker (1990) Smyth (1990) Moore (1991) Alderson & Wall (1992) Powers (1993) Oren (1993). Powers & Rock (1994) Scholes, Lane (1997) Allalouf & Ben Shakhar (1998) Robb & Ercanbrack (1999) McClain (1999) Camara (1999, 2001, 2008) Stone & Lane (2000, 2003) Din & Soldan (2001) Briggs (2001) Palmer (2002) Briggs & Hansen (2004) Cankoy & Ali Tut (2005) Crocker (2005) Allensworth, Correa, & Ponisciak (2008) Domingue & Briggs (2009) Koljatic & Silva (2014) Early (2019)

Brian A. Jacob

"The Chicago Public Schools (ChiPS) was one of the first large, urban school districts to implement high-stakes testing. In 1996-97," p.6

1stness

"Test-Based Accountability and Student Achievement Gains: Theory and Evidence"

Taking Account of Accountability: Assessing Politics and Policy, John F. Kennedy School of Government. Harvard University, June 10 - 11, 2002

Harvard Kennedy School

https://nonpartisaneducation.org/Foundation/ThinkTankThoughtlessness.htm

Brian A. Jacob

“Nearly 20 years later, the debate surrounding MCT [minimum competency tests] remains much the same, consisting primarily of opinion and speculation.... A lack of solid empirical research has allowed the controversy to continue unchecked by evidence or experience... This paper... makes several improvements on the current literature by...” p. 99

Denigrating

Getting Tough? The Impact of High School Graduation Exams

Educational Evaluation and Policy Analysis, Summer 2001, Vol. 23, No. 2, pp. 99-121

Google cache of https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.407.5840&rep=rep1&type=pdf

Brian A. Jacob

“The lack of empirical research on the achievement effects of mandatory graduation exams is striking, particularly in light of their growing popularity across the nation. The few studies that have examined the impact of MCT on student achievement tend to focus on younger children in low stakes testing environments.” p. 101

Dismissive

Getting Tough? The Impact of High School Graduation Exams

Educational Evaluation and Policy Analysis, Summer 2001, Vol. 23, No. 2, pp. 99-121

Google cache of https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.407.5840&rep=rep1&type=pdf

Brian A. Jacob

"Winfield (1990) … focuses exclusively on school-level exams whose relative salience for students is unclear. Controlling for a variety of individual, school, and regional variables (though not prior achievement), …" p. 101

Denigrating

Getting Tough? The Impact of High School Graduation Exams

Educational Evaluation and Policy Analysis, Summer 2001, Vol. 23, No. 2, pp. 99-121

Google cache of https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.407.5840&rep=rep1&type=pdf

State-mandated exams are administered at the school level and thus are school-level exams. Moreover, it would appear that Winfield did contol for prior achievement, in her group of control variables she called "academic behaviors."

Brian A. Jacob

“...the evidence on graduation exams and achievement is limited and mixed, ....” p. 101

Dismissive

Getting Tough? The Impact of High School Graduation Exams

Educational Evaluation and Policy Analysis, Summer 2001, Vol. 23, No. 2, pp. 99-121

Google cache of https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.407.5840&rep=rep1&type=pdf

100

Brian A. Jacob

Kreitzer, Madaus, and Haney (1989) . . . .note that the positive correlation between state graduation test requirements and dropout rates is thought provoking, but that there is no solid empirical evidence for a causal link between test policy and dropout rates.” pp. 101–102

Dismissive

Getting Tough? The Impact of High School Graduation Exams

Educational Evaluation and Policy Analysis, Summer 2001, Vol. 23, No. 2, pp. 99-121

Google cache of https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.407.5840&rep=rep1&type=pdf

See https://nonpartisaneducation.org/Review/Reviews/v16n4.htm
For example, from Table 3, Chapter 3 of Correcting Fallacies: "Relevant pre-2000 studies of the effects of testing on at-risk students, completion, dropping out, curricular offerings, attitudes, etc. include those of Schleisman (1999); the *Southern Regional Education Board (1998); Webster, Mendro, Orsak, Weerasinghe & Bembry (1997); Jones (1996); Boylan (1996); Jones, 1993; Jacobson (1992); Grisay (1991); Johnstone (1990); Task Force on Educational Assessment Programs [Florida] (1979); Wellisch, MacQueen, Carriere & Duck (1978); Enochs (1978); Pronaratna (1976); and McWilliams & Thomas (1976)."

101

Brian A. Jacob

“More important, states and districts that implement such exams may have other policies or characteristics that act to reduce the probability of dropping out. Few studies have rigorously addressed this question.” p. 102

Dismissive

Getting Tough? The Impact of High School Graduation Exams

Educational Evaluation and Policy Analysis, Summer 2001, Vol. 23, No. 2, pp. 99-121

Google cache of https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.407.5840&rep=rep1&type=pdf

102

Brian A. Jacob

"School reforms designed to hold students and teachers accountable for student achievement have become increasingly popular in recent years. Yet there is little empirical evidence on how such policies impact student or teacher behavior, or how they ultimately affect student achievement." Abstract

Dismissive

THE IMPACT OF HIGH-STAKES TESTING ON STUDENT ACHIEVEMENT: EVIDENCE FROM CHICAGO, June 2001

"Funding for this research was provided by the Spencer Foundation."

103

Brian A. Jacob

"Despite the increasing popularity of high-stakes testing, there is little evidence on how such policies influence student or teacher behavior, or how they ultimately affect student achievement." p.3

Dismissive

THE IMPACT OF HIGH-STAKES TESTING ON STUDENT ACHIEVEMENT: EVIDENCE FROM CHICAGO, June 2001

"Funding for this research was provided by the Spencer Foundation."

"Others have considered the role of tests in incentive programs. These researchers have included Homme, Csanyi, Gonzales, Rechs, O’Leary, Drabman, Kaszdin, Bootzin, Staats, Cameron, Pierce, McMillan, Corcoran, and Wilson. International organizations, such as the World Bank or the Asian Development Bank, have studied the effects of testing on education programs they sponsor. Researchers have included Somerset, Heynemann, Ransom, Psacharopoulis, Velez, Brooke, Oxenham, Bude, Chapman, Snyder, and Pronaratna.
Moreover, the mastery learning/mastery testing experiments conducted from the 1960s through today varied incentives, frequency of tests, types of tests, and many other factors to determine the optimal structure of testing programs. Researchers included such notables as Bloom, Carroll, Keller, Block, Burns, Wentling, Anderson, Hymel, Kulik, Tierney, Cross, Okey, Guskey, Gates, and Jones."

"What about: Brooks-Cooper, C. (1993), Brown, S. M. & Walberg, H. J. (1993), Heneman, H. G., III. (1998), Hurlock, E. B. (1925), Jones, J. et al. (1996), Kazdin, A. & Bootzin, R. (1972), Kelley, C. (1999), Kirkpatrick, J. E. (1934), O’Leary, K. D. & Drabman, R. (1971), Palmer, J. S. (2002), Richards, C. E. & Shen, T. M. (1992), .Rosswork, S. G. (1977), Staats, A. (1973), Tuckman, B. W. (1994), Tuckman, B. W. & Trimble, S. (1997), Webster, W. J., Mendro, R. L., Orsack, T., Weerasinghe, D. & Bembry, K. (1997, September).

104

Brian A. Jacob

"Chicago was one of the first large, urban school districts to implement a comprehensive high-stakes accountability policy. Beginning in 1996,..." p.3

1stness

THE IMPACT OF HIGH-STAKES TESTING ON STUDENT ACHIEVEMENT: EVIDENCE FROM CHICAGO, June 2001

"Funding for this research was provided by the Spencer Foundation."

https://nonpartisaneducation.org/Foundation/ThinkTankThoughtlessness.htm

per usual for such studies, there were no controls for any variation in test administration or security

105

Brian A. Jacob

"While several studies found a positive association between student achievement and minimum competency testing (Bishop 1998, Frederisksen 1994, Neill 1998, Winfield, 1990), a recent study with better controls for prior student achievement finds no effect (Jacob 2001)." p.7

Denigrating

THE IMPACT OF HIGH-STAKES TESTING ON STUDENT ACHIEVEMENT: EVIDENCE FROM CHICAGO, June 2001

"Funding for this research was provided by the Spencer Foundation."

106

Brian A. Jacob

"Another alternative is to mandate particular curriculum programs or instructional practices. ... Unfortunately, there is little evidence that such mandated programmatic reforms have significant effects on student learning (Jacob and Lefgren 2001b)." p.36

Dismissive

THE IMPACT OF HIGH-STAKES TESTING ON STUDENT ACHIEVEMENT: EVIDENCE FROM CHICAGO, June 2001

"Funding for this research was provided by the Spencer Foundation."

IRONIES:

Jacob, B. & Rothstein, J.

"While the literature in psychometrics (the field concerned with the theory and methodology of psychological measurement) has explored many if not all of the issues that we discuss, economists and other applied researchers are generally unaware of them and frequently misuse test score measures, with potentially serious consequences for their analyses," page 3-4.

The Measurement of Student Ability in Modern Assessment Systems

NBER Working Paper 22434; July 2016

Brian Jacob is a member of the Academic Freedom Alliance < https://academicfreedom.org/about/ > "The AFA commits to two means of protecting academic freedom. First, our members will defend faculty members’ freedom of thought and expression in their work as researchers and writers or in their lives as citizens, within established ethical and legal bounds; freedom to design courses and conduct classes using reasonable pedagogical judgment; and freedom from ideological tests, affirmations, and oaths."
Apparently, dismissing and denigrating others' research is consistent with "[defending] faculty members' freedom of thought and expression." ... so long as those whose work is dismissed and denigrated are not specifically identified.

Author cites (and accepts without checking) someone elses dismissive review

Cite selves or colleagues in the group, but dismiss or denigrate all other work

Falsely claim that research has only recently been done on topic.