Friday Freakout: Cherry-Picking Better Than Nose-Picking, But Not By Much

School choice opponents have no shortage of unfounded arguments aimed at blocking parents from choosing the best educational fit for their own kids.

One of their favorite talking points is that researchers “cherry-pick” data-driven reports that highlight only the most positive effects from school choice.

Time and again, the Friedman Foundation and other choice advocates have shown that’s not the case.

In fact, just last month we released the most recent edition of A Win-Win Solution, a report that examines 100 high-quality school choice studies in four areas: improving academic outcomes for students and schools, saving taxpayers money, reducing segregation in schools and improving students’ civic values.

Out of 100 studies, 87 found positive effects; ten were neutral; and only three studies found any instance of negative effects.

Let’s repeat those numbers for dramatic effect: out of 100 studies, all but three find students benefit in one way or another from school choice programs across the nation.

If this were the track record for an issue our opponents support, you can bet they would share it. Even the harshest critic would agree these are pretty darn good results.

Yet the very folks who accuse us of cherry-picking research do just that—cherry-pick data—by pumping up a rare negative finding as gospel that all school choice is harmful.

Witness the state teachers’ union in Indiana in a recent blog post:

Researchers studied more than 3,000 voucher students and 500,000 public school students in Indiana. Time trend comparisons of test scores for students who initially attended public schools, received vouchers and transferred to private schools versus time trends for students who continued to attend public schools found that “a student who had entered a private school with a math score at the 50th percentile declined to the 44th percentile after one year.”

Reading that, you might imagine the Indiana voucher program is spiraling out of control, with thousands of Hoosier kids falling behind on standardized tests.

Is it the truth? Not even close.

What happened in this case is a researcher started examining some data in a not-yet-complete, not-yet-published report, then referenced some data from that incomplete report in his paper on the topic. Then a special interest group with an anti-choice agenda selected a single piece of data—as in just one number!—from that reference in that paper to condemn all forms of school choice in a blog post.

It was a grownup game of telephone.

The sky is not falling in Indiana or Louisiana, the other state referenced in the paper. The research cited simply is not consistent with dozens of high-quality studies done on school choice programs across the United States.

It’s important to note that we’re not saying the negative results are wrong. They just don’t match up with piles of other research showing positive effects from school choice.

Unlike the anti-choice crowd, we’re not just willing to admit that some studies are going to show statistical decreases in test scores among school choice users, but we’re also willing to learn from that intel.

School choice programs are often intended for use by students and families who are unable to access the best educational fit in the traditional way: by ZIP Code or tuition, which are only available to those with the means to move or pay. Sometimes these students have fallen behind in their traditional public schools and need extra help catching up or adjusting to a new school curriculum and environment.

Looking only at a group of school switchers, as the Indiana report did, is like comparing two athletes running a one-mile race.

Let’s say Athlete 1, who changed schools, starts from the same place as Athlete 2, who stayed in the same school (though it is often the case that voucher students are academically behind before ever switching schools).

The learning loss that might result from school switching is like making Athlete 1 retie her shoes two times and jump a hurdle before she can start the race, while Athlete 2 is able to run right after the starting pistol fires.

It’s no surprise that Athlete 2 is still in the lead a quarter-mile into the race. But the longer the race goes on, the better picture we get about how the athletes compare to each other despite their starting gap.

The same goes for student performance in school choice programs. The two Louisiana Scholarship Program studies mentioned previously, for instance, show that in voucher students’ first year at a new school, their scores dropped slightly, but in their second year in the program, their scores ticked back upward. Until third and fourth year data become available, researchers cannot definitively determine whether voucher students’ rate of improvement is faster or slower than it would be if they’d stayed in their assigned schools.

More important, we know (and even school choice opponents agree) test scores are not the end-all-be-all for families assessing their schooling options.

Our most recent Indiana Parent Survey, which will be published by July 2016, finds that the top two reasons families selected a private school for their children were “better academics” and “morals/character/values instruction.”

Test scores matter, but parents agree that quality in the classroom and quality of school life and culture are way more important measures for their kids’ development and future.

Despite mounting evidence and public opinion to the contrary, folks who’ve already made up their mind that putting parents in the educational driver’s seat is a bad thing will be out there scouring the research looking for the outlying pieces of data that show school choice “isn’t working.” In other words, picking only the ripest cherries and leaving reality on the trees.

It’s up to school choice advocates to continue sharing the truth.