School Choice Research is Not a Rorschach Test

In 1921, Swiss psychoanalyst Hermann Rorschach published Pscychodiagnostik, wherein he produced 10 cards blotted with ink that he believed were a window into an individual’s subconscious. In the nearly 100 years since, the term “Rorschach Test” has come to denote a phenomenon whose interpretation is derived entirely by the observer.

In recent years, the research on private school choice has become its own kind of Rorschach Test. Folks who don’t like school choice can highlight this study or that that “proves” school choice doesn’t work. Folks who like school choice have done the reverse. Unfortunately, that’s not how science is supposed to work. We’ve got to look at all of the available empirical evidence and weigh it out.

That is what we are doing with this slide show. We have compiled what we understand to be every high-quality study on private school choice that looks at the effects that it has on student academic achievement, attainment, civics, segregation and on neighboring public schools.

What does this research show?  We think a few things:

1. Given enough time, school choice programs create small, positive test score gains for participating students. Of the 18 random assignment studies conducted, 13 have found positive outcomes for either the full sample or at least one sub-sample of students studied. Three found no visible effect for any group of students, and three found negative outcomes for all or some students.
2. School choice programs appear to increase graduation rates for participating students. Three studies have examined these effects so far and found positive effects on educational attainment for at least one subgroup of students. No studies have found negative
3. There is virtually no evidence that school choice harms neighboring public schools. In fact, students tend to experience small gains on test scores there. And school choice programs achieve these benefits with fewer public resources. Of the 33 studies that examine the competitive effects of school choice programs, 31 found positive effects, one saw no visible effect and one found negative effects. Moreover, 40 fiscal analyses have been conducted on school choice programs. All but three found these programs generated net fiscal savings overall for taxpayers, and three found the programs were revenue neutral for taxpayers.
4. Similarly, we see no evidence that students who participate in school choice programs are alienated from their communities or show less public-spiritedness than their public school-educated peers. In fact, research too appears to show the contrary. Of the 12 studies looking at civic values and practices outcomes, eight found positive effects. Four found no visible effect, and none found negative effects.
5. School choice ameliorates segregation. It does not exacerbate it. Of the 10 studies that have examined school choice’s effect on integration in schools, nine found positive effects. One was unable to detect any effects, and none found negative effects.

All of that said, we want to be crystal clear on several issues.

First, we do not believe that student test scores are the be-all-end-all of our education system. We know from our own surveys that families value numerous aspects of a school’s educational environment over its test scores. Test scores are one of the only tools that we have to make objective comparisons between schools, and they do give us some information about aspects of education that we care about, so we tend to rely on them a lot. But, high test scores are not education’s sine qua non.

Second, we recognize these studies include a wide variety of programs. Some of these programs are large, some are small. Some are statewide, and some focus only on one city. Some of these programs have been around for years, and some are new. Context matters, and not all programs are created equal. That should be taken into account when interpreting findings.

Third, we realize that this is not the universe of all studies that have been conducted on school choice. Where possible, we have used the highest-quality research design (randomized control trials). Because of the incredibly large number of times that private school choice programs have been studied, we have the luxury of limiting parts of the analysis to just RCTs.

Finally, it’s important to recognize that there are potential trade-offs with all research designs, even RCTs. We prefer RCTs over other methods because they have higher internal validity than non-gold-standard methods, but they may not necessarily have higher external validity compared to other methods.

A study with high internal validity boosts our confidence that effects we observe are attributable to the program itself and not to other extraneous factors—that is, we are confident that the program caused the outcome we observed. A study with high external validity allows us to generalize the results to other students in other programs. Competitive effects studies may have high external validity, but they have lower internal validity than RCTs, meaning we don’t have as much confidence as we would with RCTs that there is a causal relationship between school choice programs and test score gains by students who remain in public schools.

Here at EdChoice, we have a commitment to intellectual honesty and transparency. As more studies are added to this growing body of research, we will update our documents appropriately. Beyond that, in the slightly longer term, we hope to compile an open-access database of school choice research. Sign up for EdChoice emails to be the first to know when we make this incredible tool available!