Student Growth Using Teach to One Roadmaps

In this study, we analyzed the fall and winter 2022 MAP scores of 208 students from one urban middle school. Employing a quasi-experimental matched paired approach, we aimed to identify discrepancies in MAP growth between pairs of closely resembling students. For each Roadmaps user active for at least half the semester, we sought a non-active counterpart in the same grade with similar fall MAP scores, ideally sharing the same class. With these pairs, we assessed the divergence in MAP growth from fall to winter scores.

Out of the 208 student pool, we identified 44 viable matches meeting predefined similarity criteria. Following adjustments for grade, initial fall scores, and classroom dynamics, active Roadmaps participants exhibited 5.6 (p < .01) RIT points of additional MAP growth in their winter assessments as compared to their matched peers. That is the equivalent of an additional one and half semesters of learning demonstrated by students using this program.

Roadmaps is a web based tech product developed by Teach to One that is primarily used in a supplementary educational environment for middle school math. The program is used to identify what math skills a student knows using a diagnostic assessment and then recommends the next skills that connect with what they already know. This program is built on the work of New Classroom’s to pursue educational environments that are personalized to the needs of each student. 

The analysis outlined in this paper documents a case study that uses fall and winter NWEA MAP assessment scores of two hundred students from one middle school who may or may not have used the program. We use each MAP score as a snapshot of their knowledge to assess how students active in the program learned through the semester as compared to their peers. 

MAP is a growth oriented assessment that provides an integer based scoring system to benchmark where a student lands as compared to national standards. This system uses the  RIT or Rasch UniT model to provide a linear scale of student achievement. A student’s MAP score represents the level questions that they answer correctly 50% of the time. 

This assessment provides greater granularity in student achievement as opposed to typical standards-based state tests which determine if students meet grade level expectations. This distinction is important as many of the students in this school started well below grade level expectations. Standards-based testing is not going to see all the progress that students make in catching up to their grade, while MAP captures that growth. 

Using the results of this assessment, it was identified that over half of the students in this school began the year below the 20th percentile of national averages. 25% fell into the 20-40th percentiles. The last 32 students tested around the national average with the highest reaching up to the 82nd percentile. Resulting from this, a rigorous supplemental program was implemented through the fall of 2022 where students had their choice in Roadmaps or another program. Regardless of their choice, the teachers and administrators created a growth oriented culture that celebrated skill acquisition and pushed the students to maximize their opportunity. Given this environment, we saw an opportunity for a natural experiment comparing the students that used Roadmaps to those that did not. Following this semester, the school pivoted to exclusively preparing the students for the end of year state test, ending the environment that supported this analysis. 

Taking 61 students who were active for at least half the semester in Roadmaps, we used a k-nearest neighbor matching process (with replacement) [1] to find the most similar partner who was not active in the program. Using the methodology outlined in Austin, [2] matches were limited to those within a suitable level of similarity, leaving unmatched students to be dropped. 

These matches were created over three similarity factors. First, each pair had to be in the same grade. Second, they must have a similar Fall MAP score. Third, if available we prioritized students being in the same classroom environment. This matching process is done to minimize the potential for confounding variables. Matching on grade and classroom environment reduces the potential for biases from different educational methodologies and experiences. Controlling for the fall starting point leads to matches of students with similar aptitudes and attitudes towards math up to that point of their education. 

Student progress over the semester was tracked with RIT point growth, calculated by subtracting their fall from the winter MAP scores. The resulting values were then compared for each pair and averaged to calculate the mean difference in RIT growth between active and non-active students.