Summary of and Comments on the Middle School Mathematics Initiative
Senior Associate Commissioner
Massachusetts Department of Education
Attached is a copy of the Donahue Institute’s evaluation of the Department’s Middle School Mathematics Initiative. This two-year curriculum intervention and research project was designed and carried out in collaboration with faculty at the University of Massachusetts-Lowell. Its purpose was to help teachers in under-performing middle schools improve student achievement in mathematics through the use of mathematics coaches emphasizing a systematic approach to lesson planning and implementation appropriate for any mathematics class. The results of this Initiative are positive. However, they are also sobering, as I shall explain below after noting the key features and findings of this project.
(1) Coaching: For this initiative, the Department employed six highly experienced mathematics teachers as mathematics specialists, or coaches, in order to assess the value of coaching in improving student learning in mathematics. We were especially interested in assessing the value of coaching because it is a highly recommended pedagogical strategy today for strengthening teachers’ effectiveness in their classrooms despite the fact that it is a very expensive strategy with no body of scientifically based research evidence attesting to its efficacy
(2) Lesson Planning: The basic task of the six specialists was to train 50 teachers in grades 6, 7, and 8 in eight school districts in lesson planning and implementation over the course of more than one year (24 teachers in the first year of the study continued into the second year of the study). They occasionally suggested new or widely accepted teaching practices and engaged in other activities.
(3) Pre-Post Tests of Student Achievement: All students in the intervention and comparison classes (over 1000 students in each group each year) were given pre-post tests consisting of items similar to released MCAS grade 6 mathematics items addressing basic arithmetical operations. The Department sought to determine learning gains during the academic year and to pinpoint students’ achievement level in arithmetical skill and understanding more precisely than we can from MCAS tests, which are given only at the end of two-year grade spans.
(4) Mathematics Course Work for Teachers: As part of the first year of the project, 15 teachers voluntarily took a middle school mathematics course at the University of Massachusetts-Lowell. As part of the second year, 36 teachers took a Department-sponsored middle school mathematics course taught in four locations by three mathematics professors using both a common syllabus and a pre-post test that they had developed. This feature was intended to help the Department explore the relationship between teacher knowledge in mathematics and gains in student learning.
Findings: First, students in the MSMI classrooms had change scores that were significantly higher than those of similar students in classrooms with no intervention, even though there was a much higher percentage of students identified as Limited English Proficient in the MSMI classrooms. Second, teachers’ lesson planning ability was related to change scores, that is, students of teachers with higher scores on lesson planning made significantly more improvement than students of teachers with lower lesson planning scores. We also found that students of teachers with more teaching experience achieved higher gains than students of teachers with less teaching experience.
Discussion: As noted earlier, the results of the project were statistically significant; overall, the MSMI students had greater gain scores than the comparison students, and teachers’ lesson planning ability was related to these gains. But, with respect to practical significance, many questions can be raised.
The grades 6, 7, and 8 students in both intervention and comparison classes could achieve a maximum of 20 points on a test of basic arithmetical operations that included word problems all pitched to a grade 6 level. On average, both groups of students got about 8 or 9 points at the beginning of the year and about 11 or 12 points at the end of the year. This is a modest gain, even if statistically significant. The Department also examined the results of the grades 6 and 8 mathematics MCAS tests given in the spring of 2002 and found that the students in intervention and comparison groups had average scaled scores of 225.7 and 231.4, respectively, in grade 6 and 222.2 and 227.8, respectively, in grade 8, confirming that the average student in the MSMI project was still a low-performing student by the end of the year.
The results of both the MCAS tests and the pre-post test used in the MSMI thus provide only modest support for the efficacy of mathematics coaches in improving mathematical learning in low-achieving students. Although the participating teachers in this project all found their work with the mathematics specialists beneficial to their teaching, these benefits did not translate directly into meaningful increases in mathematical achievement for the low-performing middle school students in the project. It is possible that the teachers needed more time working with a specialist and/or with lesson planning.
It is worth noting that the study provides no support for any other interventions taking place in the MSMI and comparison classes in these schools. Most low-performing schools today receive targeted assistance of varying kinds (whether for the whole school and the regular classroom teacher or for the low-performing or ESL student through the Title I or bilingual education teacher), and this was the case in this study. Both the intervention and comparison groups as a whole in this project could be considered mixed models. The only clear difference between the two groups was the Department’s own carefully defined model of coaching.
None of the above remarks should be construed as comments on the quality of the specialists or the mathematics courses provided for the teachers. According to the teachers, the two most positive aspects of the Initiative were (1) having specialists, all of whom were highly rated, and (2) the training on lesson planning and implementation. The teachers also judged the mathematics courses they took to be useful and showed gains on the pre-post test that was given them. They did recommend a separate mathematics course for grade 6 teachers in the future, not one course enrolling teachers in grades 6, 7, and 8.
The Department will continue to support strong mathematics course work for pre-service and in-service teachers even though MSMI students in the classes of teachers who took the mathematics course in the second year of the study showed no greater gains overall than the other MSMI students. Increased teacher knowledge may not quickly or even necessarily lead to gains in student learning for low-achieving middle school students. We also plan to use, and encourage others to use, two important instruments for professional development in mathematics that were developed for this Initiative: (1) the middle school mathematics course syllabus developed by the mathematicians who taught the course, together with a pre-post test and classroom applications related to the Massachusetts Mathematics Curriculum Framework, and (2) a protocol for tracking observations of middle school mathematics classroom teaching practices. The Department will also continue to support lesson planning and implementation if only on the basis of common sense. And we will continue to explore ways to help schools retain new teachers.
Where Should We Go? During the course of the study and in discussions of its results with specialists and teachers at a Title I conference, several factors affecting student learning were identified as needing exploration: (1) student reading level, (2) use of grade level textbooks in a standards-based environment, (3) grouping practices, and (4) student absenteeism. All bear further study.
(1) Student reading level is a factor affecting academic performance in every subject in the curriculum. The students in both the intervention and comparison classes in the MSMI study are below average in reading as well as in mathematics. According to the 2002 grade 7 English language arts MCAS assessments, the mean scale score of the intervention students in grade 7 was 236.3, and the mean scale score of the comparison students in grade 7 was 238.9, compared to the state average of 242.0. Given the emphasis in current school mathematics programs on reading and writing to solve word problems, poor readers and writers are apt to encounter more difficulty in mathematics classes today than they did in the past. More skilled mathematics teaching may not be able to surmount middle school student limitations as readers and writers. The extent to which low-performing students in middle school mathematics are hampered by their limitations as readers and writers warrants systematic exploration.
(2) In standards-oriented schools, it is understandable why administrators purchase grade level textbooks for the middle school; the grade 8 MCAS mathematics test is based on grade 8 standards and if they are to prepare students for the grade 8 MCAS they feel obligated to address the standards on which the grade 8 test is based. However, unlike the widespread availability of developmentally appropriate below grade-level reading materials (often called high interest/low vocabulary), there seem to be few if any below grade-level mathematics materials available to teach skills students have not yet acquired but which are needed for problem solving in the grade level textbooks. We do not know how middle school teachers are using grade-level textbooks for their below-grade level students, or what they use in their place if they use substitutes. It is also not clear to what extent teachers may be remiss in adapting textbook materials to address students’ needs.
(3) There is a relatively large body of research on the effects on achievement of grouping students with varying skill levels in different ways. Some evidence suggests that low-performing students in mathematics learn more when they are in more homogeneous groups with materials geared to their needs. In classes with a wide range of student achievement, it is not clear how well teachers can address the specific weaknesses of low-performing students, especially if they are using grade-level materials.
(4) Student absentee rates for 2001-2002 were not available at the time this report was written, but we have them for the first year of the project. In grade 8 for the first year of the project, in the MSMI schools, 598 out of 2,654 students (23%) were absent 11 to 20 days for the year, while 20% (an additional 525 students) were absent more than 20 days. Absentee rates in the comparison schools were slightly higher. It is possible that absentee rates were similar in 2001-2002. High absenteeism directly affects student learning.
Two possibilities for improving middle school learning in mathematics, other than coaching, surfaced during this study and deserve to be explored as well: (1) well-planned, long-term study groups for middle school mathematics teachers and (2) looping. Well-planned study groups give teachers regular opportunities to discuss their lesson plans, their teaching, their students’ work, and their textbook materials, and can be defined as good professional development—sequential, subject-focused, and collegial. They can also build in possibilities for teachers to observe each other teaching. Such activities may have an indirect influence on student learning if they alleviate teacher “burnout” and promote teacher retention.
Looping is the practice of having a teacher follow the same group of students over several grades. It is a common practice in mathematics teaching (and foreign language teaching) in the upper elementary and middle school in many European and Asian countries, and it would not be difficult to try out in our middle schools for grades 5, 6, 7, and possibly 8. The mathematics teacher would know his or her students well by the end of the first school year and would also know exactly what the students had learned or not learned. Less time would thus be wasted at the beginning of the next school year, and much more continuity could be built into the teacher’s classroom curriculum. Looping has been tried primarily in the primary grades but deserves more piloting and evaluation in middle schools, especially since it is no more expensive than grade by grade changes in staffing assignments.
Access this resource in .pdf format
Citation: Stotsky, S. (2017). Summary of and Comments on the Middle School Mathematics Initiative, Nonpartisan Education Review / Resources. Retrieved [date] from http://nonpartisaneducation.org/Review/Resources/SStotskyMSMI.pdf