New UVA Study of Limited Help in Understanding School Vouchers

I know y’all are going to find this hard to believe, but a press release touting a recent study was over the top and could lead casual readers to misunderstand the paper’s findings. Since this never happens, I’ll give you a moment to take a few deep breaths and come to grips with what happened.

OK, are we good? Can we proceed? Alright.

The study in question was published in Educational Researcher by the University of Virginia’s Robert Pianta and Arya Ansari. It is relatively straightforward. Pianta and Ansari used data from the National Institute of Child Health and Human Development’s Study of Early Child Care and Youth Development survey to compare the outcomes of 15-year-olds in the sample who had attended private schools to 15-year-olds in the sample who had not.

Initially, they found a large private school advantage, but after including a series of control variables for demographic characteristics, the two groups were statistically indistinguishable from one another.

Like I said, pretty straightforward. But that’s when things went off the rails. In UVA’s press release, Pianta decided to use his findings to attack school voucher programs. But does the study actually measure the effects of school voucher programs? It does not.

If that is not enough, the strategy that the researchers used is particularly ill-suited for trying to understand how school voucher programs affect children.

The single biggest concern in evaluating school choice programs is selection bias. Researchers have to answer the question “Are the students who receive vouchers different from those who don’t in ways that might affect the outcomes that we’re trying to measure?” More motivated students, for example, might be more likely to seek vouchers and also more likely to do better in school. When we observe that they do better in school, is it because of the voucher or because of their motivation? We can’t always say.

To cope with this problem, the gold standard of school choice research utilizes random assignment.  Everyone who wants a voucher gets their name thrown in a hopper and random chance is the only thing that differs between those who get a voucher and those who don’t. That’s how we know that any differences between the two groups can be attributed to the program.

The authors of this paper did not do this.

Now, this is not a problem if you’re trying to answer the research question that they set out to answer.  If you want to know if there are differences in performance between those who attend private schools and those who attend public schools at age 15, controlled for demographics—Pianta and Ansari’s method is a perfectly reasonable way to do it. We can’t always perform random assignment studies or create detailed matched-sample comparisons following students over many years. Sometimes you have to work with what you have.

That said, when you are using methods like this, you have to be really careful in how you present the findings. You can’t take the next step and make causal claims about private or public schools driving these differences. You especially can’t take an even further step and make causal claims about school voucher programs driving these differences.

What’s more, if there are better methods and better data out there, you need to defer to the superior research on the question. That is perhaps the most frustrating part of this whole saga.

In the paper itself, the authors take pains to describe the extant literature on private school choice programs.  At the end of their conclusion, they even write, “ … there is some evidence from the experimental literature that voucher programs may produce a slight benefit for the achievement of poor children to the extent they enroll steadily.” Where was this in the press release?

If you are interested in seeing what the research has to say about private school choice, check out this collection of all of the studies that actually try and untangle the messy questions involved.

So, to recap:

• Attending private school and receiving a school voucher aren’t the same thing. You can’t just leap from one to the other.
• Simply observing differences (even if you make some statistical corrections) between two groups of students does not get around selection bias. If that is your strategy, you cannot make causal claims without some serious caveats.
• And finally, there is a lot of research on the specific effects that school choice programs have on children, and that is a much better source of information when it comes to thinking about school choice policy.