SUMMARY: The right number is 17 percent. At least, that’s the result of a Stanford University study that looked at the educational outcomes of 70 percent of the nation’s charter school students. While the Stanford study has been widely quoted in the press, it has been criticized by one Stanford researcher, who claims that a statistical error invalidates the study’s results.
ANALYSIS: In June 2009, Stanford University’s Center for Research on Education Outcomes, or CREDO released what is, so far, the largest-ever study of outcomes among students in charter schools. The study looked at schools in the District of Columbia and 15 states with charters — including high-population states such as California, Texas and Florida. The researchers examined the test scores of charter school students in those states, and compared them to “virtual twins” – groups of public school students who are demographically similar to the students in each state’s charter school system.
The researchers found that only 17 percent of charter school students performed measurably better than their virtual twins. More than twice as many — 37 percent — actually performed worse in charter schools. And for 46 percent of students, there was no noticeable difference.
According to the study, black and Hispanic students, as a group, did significantly worse in charters than in traditional public schools. Students in poverty, as a group, did better, as did students who don’t speak English as their first language. The study also found that while charters had a beneficial effect on elementary school students, charter students at the middle and high school levels performed significantly worse than their public school counterparts.
The study also included an important caveat: Charter school performance varied greatly depending on how charter schools were governed. Results were worst in states with a cap on the total number of allowable charter schools, and states without a central authority for approving charters. In states with a central approving authority, and schools with no cap, charters performed well.
The Stanford study is not without its critics. In August 2009, another Stanford professor, Caroline M. Hoxby, released a memo criticizing “a serious statistical mistake” that, according to Hoxby, put charter schools in the CREDO study at a disadvantage compared to their public-school counterparts.
By comparing real students in real charter schools to an “average” student from a similar demographic group, Hoxby’s paper argued, the CREDO study was creating a much larger margin of error for measurements of traditional public school students, as compared to students in charters. In effect, Hoxby is arguing that the comparison between charter students and their “virtual twins” is an apples-and-oranges comparison.
The original CREDO paper, Hoxby’s response and CREDO’s reply are all available at http://credo.stanford.edu.
Taylor Bright, spokesman for the Sparks campaign, said the candidate was unaware of the debate about the CREDO numbers when he began using the statistic in his speeches.
Even so, he said, Sparks is standing by the numbers.
“The researchers are willing to stand behind their study, and so are we,” he said.
While Sparks appears to have misquoted the study occasionally, using the 15 percent figure instead of 17 percent, he was quoting research-based numbers from a reliable source, in this case an academic institution.
The debate over the validity of those numbers is worth noting, however. As long as the methods of the study are subject to debate, the oft-quoted 17 percent figure should probably be used with an asterisk behind it — because the matter doesn’t seem to be settled yet.
