Understanding the New IES Report on the D.C. Opportunity Scholarship Program

The Institute of Education Science (IES) recently released a report that analyzed first-year outcomes for students enrolling in D.C. Opportunity Scholarship Program (OSP) since its 2011 reauthorization.

The OSP provides students from low-income families in D.C. with scholarships worth up to $8,452 for K–8 students and $12,679 for high school students. The IES research team examined three cohorts of students and analyzed the impact of their first years in the program from 2012 to 2014. The analysis examined effects on both the offer of a scholarship and actual use of a scholarship.

Key findings from the study:

• After students’ initial years in OSP since reauthorization in 2011, students receiving and using scholarships scored 7.3 percentile points lower in math a year after they applied to the program than students who did not receive a scholarship; differences in reading test scores were not statistically significant overall, but they were lower for students attending non-SINI schools (“schools in need of improvement”) and for students entering grades K–5.
• Both students who were offered scholarships and their parents were more likely to give their school an “A” or “B” than the control group (8.2 and 4.3 percentage points higher, respectively), but the differences were not statistically significant.
• Parents of children who used and who were offered scholarships were significantly more likely than the control group to rate their school as “very safe”—the most basic level in Maslow’s hierarchy of needs.

It’s a safe bet that school choice opponents will tout these results as reason to abandon the program and build a case against educational choice. Context is necessary, however, for evaluating these results and potential implications.

Before Jumping to Conclusions…

Before jumping to conclusions, it’s important to note that the study is not a simple comparison of scholarship students versus district school students.

Instead, the study compares students who used (or were merely offered) a scholarship against a “control group” consisting of students who applied for but were not offered a scholarship through the OSP lottery. However, 42 percent of the control group attended a charter school and 10 percent attended a private school without a scholarship, meaning that the majority of the non-scholarship students nevertheless exercised some form of school choice.

Additionally, there is a substantial gap in resources between school sectors. The average OSP voucher ($9,422 in FY 2015–16) is less than half the per-student amount received per pupil at D.C. charter schools (over $21,000) and less than one-third of the per-student amount received in the D.C. district schools (over $30,000).

It’s not hard to imagine that the results might be different if Congress invested as much in the OSP as it invests in D.C. district and charter schools. While money itself cannot guarantee improvement, using it in productive ways can be impactful.

Previous research found positive impacts from the OSP. Enacted in 2004, a team of researchers led by Patrick Wolf from the University of Arkansas rigorously studied the OSP during its initial four years of operation. The key findings from this four-year evaluation:

• The OSP significantly increased the likelihood of graduating from high school by 12 percentage points for students who received a voucher offer than students in the control group.
• Although the researchers reported “no conclusive evidence that the OSP affected student achievement overall, or for the high priority group of students who applied from ‘Schools in Need of Improvement,’” a more recent analysis of the OSP data from the initial four-year evaluation found that students who used vouchers scored significantly higher in reading in their second, third, and fourth years in the program.
• The OSP had a positive impact on parental satisfaction and perceptions about school safety.

Reconciling Results from All Studies of D.C.’s Voucher Program

So how can one reconcile different results from both studies?

First, it is important to recall the dramatic increase in the exercise of choice among D.C. families between the initial study and the most recent one.

In the 2003-04 academic year, fewer than 14,000 D.C. students attended charter schools while more than 61,500 attended district schools. In 2015–16, nearly 39,000 D.C. students attended charter schools and 1,301 received OSP scholarships while about 48,000 attended district schools. As noted above, the control group in the most recent study was more likely to be attending a school of choice than a district school, whereas the control group in the initial study was primarily composed of district school students.

It is likely no coincidence that the dramatic increase in families exercising choice coincided with a dramatic increase in parental satisfaction.

The first IES study of the OSP, published in 2007, found that 74 percent of scholarship parents gave their child’s school an “A” or “B” compared to only 55 percent of parents in the control group. A decade later, the control group’s satisfaction had risen to 72.4 percent, an increase of more than 17 percentage points. The greater availability of school choice means parents are more likely to find a school that meets their child’s needs.

The increased parental satisfaction with the wider availability of school choice may also reflect improvements in the alternatives to private schools, particularly charter schools.

Increasing quality in the “counterfactual” (i.e., the options available to the control group that did not receive a scholarship) would translate into reduced “participant effects” (i.e., gains on test scores for scholarship students relative to their non-scholarship peers). Indeed, the results from the National Assessment on Educational Progress (NAEP) suggest that D.C.’s district and charter schools have improved over time.

Then again, the improved math and reading test scores among the charter and district schools may reflect a worrisome narrowing of the curriculum. The IES report noted that district and charter school principals reported more instruction time spent on math and reading than private school principals. It may be the case that there is more “teaching to the test” going on in these schools.

While tests can be informative, parents know their kids are more than scores.

Politicians should avoid using test scores as the sole basis for policy decisions, especially because, as Jay Greene has shown, there is an increasing disconnect between test scores and important measures of later-in-life outcomes. Moreover, evidence suggests that a more holistic education is better in the long run than a narrow focus on reading and math.

As researchers and policymakers continue to evaluate this program, it’s important to keep context in mind. Evaluations of the program before 2011 suggest that students benefited from the program in its early years, and the program remains popular among D.C. families.

Policymakers should also keep in mind the central purpose of educational choice: empowering families with the ability to choose the learning environment that best meets their child’s individual needs. Test scores can help inform parental choices, but they should not determine whether parents have right to choose.