3.

How do teachers implement the enVisionMATH curriculum?

4.

What are teachers’ perceptions of the quality and utility of the enVisionMATH program?

# Participants and Settings

PRES Associates recruited eight schools to participate in the first year of the study (2007–08), including sites in NH, MA, KY, TN, CO, MT, OH, and NC. Fifty-nine teachers and 1,197 students were represented from these states. Two schools (NC and CO) had to withdraw from the study during the second year (2008–09) to comply with district mathematics adoptions, prohibiting students from being randomly assigned to treatment or control conditions. Therefore, the final analytic sample was comprised of 44 teachers and 708 students. The study sample included representation of all ethnic, socioeconomic, special education status, and mathematical ability levels.

Measures

Multiple measures were used to assess student achievement and program implementation. In order to measure program implementation and teacher perceptions, evaluators collected data through observations and interviews with math teachers. Math teachers also completed monthly implementation logs. This background information provided researchers with a detailed data source on what was occurring in treatment and control classrooms in terms of math instruction and allowed researchers to identify areas of overlap in terms of content taught and activities. Evaluators also conducted biannual classroom observations and interviews with classroom teachers. The observation data provided critical insight into the nature of use and the effectiveness of the math materials used with treatment and control students.

Evaluators employed three student measures to assess changes in students’ math skills over the course of the study. Teachers administered each assessment in fall 2007, spring 2008, and spring 2009. Evaluators selected the Metropolitan Achievement Test (MAT8) as a norm-referenced assessment of problem-solving and computation, the Group Mathematics Assessment and Diagnostic Evaluation (GMADE) – Concepts & Communication subtest of language and vocabulary, and the Balanced Assessment of Mathematics (BAM) as an open-ended, performance-based assessment. These assessments have broad visibility and acceptance in the field, demonstrate high technical merit, and align well with the enVisionMATH program. The assessments were given to all treatment and control students.

The MAT8 is a group-administered, norm-referenced test that assesses content and process skills that are relevant to students’ everyday lives. The Math Computation, and Math Concepts and Problem-solving subtests were selected for administration. The Concepts and Problem-solving subtest measure a student’s facility for applying mathematics to many different kinds of problems and evaluating his or her results. The Math Computation subtest measures students’ ability to complete arithmetic operations.

The GMADE is a norm-referenced, standards-based assessment of mathematical skills. The Concepts and Communication subtest was selected for administration. This subtest uniquely addresses the language, vocabulary, and representations of mathematics. The MAT8 and GMADE tests were scored by PRES researchers following the standardized scoring procedures (including raw score conversions) as outlined in the publisher’s technical/scoring manual.

6

Two versions of the BAM were used as part of this RCT—a published version for the 3rd, 4th, and 5th grades and one created for 2nd graders by PRES researchers. The assessments were designed as performance assessments in order to provide students with an opportunity to show what they know and understand. The BAM was scored by PRES researchers. In order to facilitate comparisons between the two tests, percent correct was the metric used for study analyses.

Additionally, teacher and student surveys were developed to gather information on attitudes that may be affected by their math program. Surveys were completed in the Fall and Spring of each year.

# Student Performance Results

Results for enVisionMATH Students

PRES Associates determined that students who used enVisionMATH demonstrated statistically significant gains in math achievement over the two-year study period. Moreover, significant gains in achievement were evidenced after just one year of implementation as well. Specifically, students using enVisionMATH significantly improved in the areas of math concepts and problem-solving, math computation, math vocabulary, and communication in math.

The results also provided evidence of accelerated growth rates during the second year of enVisionMATH usage in the areas of math concepts and problem-solving, and math vocabulary skills. This suggests the cumulative effects of enVisionMATH become stronger over time.

Figure 1—enVisionMATH Students’ Math Performance at Pre- and Posttesting: Metropolitan Achievement Test (MAT8) which tests understanding of math concepts and problem solving.

675

650

671.7

669.7

MAT8 Scale Score

625

600

575

550

615.8

637.8

586.2

639.1

Pretest (Fall 2007) Posttest (Spring 2008) Posttest (Spring 2009)

500

## Concepts & Problem Solving

## Computation

There was significant improvement in enVisionMATH students’ understanding of math concepts and problem-solving and math computational skills.

7