Are School Voucher Studies Biased from Front to Back?

Determining if a policy actually causes some impact on people’s lives is an incredibly difficult task.  Humans are complex creatures, and the world that we live in has lots and lots of interconnected moving parts. It is a challenge to see how altering one small part of it, say by offering a student a voucher, changes the rest.

One of the central challenges of making this determination is bias, that is, a worry when making comparisons that the differences you observe are driven by factors not accounted for in the model you have used to estimate them.

One cause of bias in school voucher studies would be on the front end, called selection bias, wherein the students in the “treatment” and the “control” groups vary in ways that are not accounted for that will ultimately affect their performance.  If students who receive vouchers are more motivated, have parents that take a greater interest in their studies, or have some greater innate ability, comparing them to those that do not might generate a result that looks like it is from the voucher, but is actually explained by these unaccounted for differences.

Another cause of bias could be on the back end, driven by non-random attrition of students from the program. Even after making sure that the treatment and the control group are equivalent to one another, if voucher schools kicked out students at a higher rate and thus only “counted” the students who actually persisted (whom we would imagine would be better performing), again, the perceived impact of the voucher would be driven not by the quality of the education that a given school provided but by the sorting mechanism the school imposed.

Noted statistician Gene Glass brought this latter concern up as a criticism of the voucher literature on twitter recently:

This is a serious concern. If we had reason to believe that either non-observed factors on the front or back end were driving the generally positive impacts of school voucher programs, it would cause us to question all of them.

Luckily, numerous voucher studies have taken the necessary steps to safeguard both against selection bias on the front end and bias introduced by attrition on the back end.

On the front end, researchers have been able to randomly assign students vouchers. The procedure is relatively straightforward, a large number of interested students apply (which mitigates concerns of differential motivation) and the only thing that determines whether they get a voucher or not is random chance. This is the same procedure used in medical trials and high-level social science work across the world. It is the strongest tool we have to determine the causal influence of an intervention.

Luckily, numerous voucher studies have taken the necessary steps to safeguard both against selection bias on the front end and bias introduced by attrition on the back end.

As I have detailed previously, this method is superior even to careful matching designs that compare those students who use vouchers to students who do not use vouchers but fit the same demographic profile.

On the back end, studies have controlled for difference in attrition by following, for the entirety of the study, both students who won the lottery and students who lost, regardless of what they ended up doing. This is called the “intent to treat” sample. Some students who lost voucher lotteries had families who were able to scrape together the money to get them into a private school. Some students who won decided not to take the voucher and attended public school instead. Others were kicked out of their respective schools and ended up going somewhere else. But, regardless of these decisions, winning the voucher lottery put children into the treatment group, and losing the voucher lottery put children in the control group. None of these individuals machinations matter. The estimate is unbiased.

To be extra, super, triple sure, researchers can “rebalance” the outcome samples to adjust for any students who not only dropped out of a school, but out of data collection entirely.  Again, if a significant number of students simply stopped allowing their results to be collected, and that attrition was non-random, that attrition from the study itself could bias the results. Luckily, there are corrections for this.

The upshot of all of this is that if some students drop-out (or are kicked out) of voucher schools and are negatively affected, that will show up in the estimate of the effect of the voucher.

In other words, dropouts will not bolster the voucher group results—as many opponents of school choice like to say they will—because the study does not leave out the negative academic outcomes of kids who drop out of the voucher program.

One study that used this method was the Institute for Education Science’s evaluation of the DC Opportunity Scholarship Program. Following the intent to treat sample found no statistically significant difference in performance between voucher and non-voucher students on standardized test scores and a 12-percentage point increase in the probability that students who won the voucher lottery would graduate from high school. We can have great confidence that the result seen in these estimates is due to the actual effect of the program, not by non-observed factors, on the front end or the back end. If you’re interested in the rebalancing, head to Appendix A (the Research Methodology section), particularly the section starting on Page A-21 “Adjustments for Nonresponse.”

This study made it through the Institute for Education Sciences’ incredibly rigorous peer-review process. An academic paper version of the findings was published in the prestigious peer-reviewed Journal of Policy Analysis and Management. This is serious, careful research that any unbiased observer can trust.

The Friedman Foundation has periodically collected other randomized control trial studies whose results are available here.

Determining the effect of an educational intervention in our complex world is challenging, but possible.  Using rigorous methods can give us confidence that the differences that we observe between those who receive an intervention and those that do not are meaningful and happened as a result of that intervention. Contra Professor Glass, school vouchers have been repeatedly evaluated using these methods. No Joke.