Do open educational resources improve student learning?

Open educational resources (OER), such as free textbooks, have grown in popularity in recent years as institutions and faculty work to lower costs for students, but researchers have had difficulty determining if adopting OER impacts learning. A new study by Rice University researchers suggests previous OER research may have failed to turn up significant results because it has been looking at too wide a pool of students.

Photo courtesy of Jemel Agulto/OpenStax

Because OER will likely provide tangible learning benefits only to students who otherwise would go without a textbook, research needs to hone in on those students, the Rice study determined.

This problem came to the attention of Phillip Grimaldi, director of research at OpenStax, Rice University’s peer-reviewed OER publisher, at the 2017 Open Education conference. “I noticed a lot of research being conducted examining whether OER improved student learning, but I struggled to understand why OER should produce any difference in the first place,” he said. “If OER is comparable to the materials they are replacing, the learning outcomes should be comparable as well. When I asked researchers why they thought OER should make a difference, the standard reply was the ‘access hypothesis’ – the idea that access affects outcomes, since students can’t learn from a book they can’t afford to access.”

While Grimaldi found the hypothesis plausible, he worried that none of the research was designed to detect such effects. This led him to publish the paper, “Do open educational resources improve student learning? Implications of the access hypothesis,” which appears in the journal PLOS ONE.

Grimaldi and fellow researchers Richard Baraniuk, founder and director of OpenStax and Rice’s Victor E. Cameron Professor of Electrical and Computer Engineering; Drew Waters, machine learning scientist; and Debshila Basu Mallick, cognitive science postdoctoral research associate, conducted simulated experiments and found that even under ideal conditions, detecting an improvement to learning was nearly impossible using standard research designs. They found if a student can afford to buy the traditional textbook, changing to a comparable OER won’t produce a statistically significant change in outcome. The same goes for a student who won’t read the textbook either way. Both of those students might experience benefits outside the classroom – they may save money, for example – but those benefits won’t be represented in any data on course outcomes, the researchers wrote.

“Using OER could potentially make a very significant difference in course outcomes for a student who couldn’t afford the traditional textbook, and would try to make do without it,” Baraniuk said. “In short, the ‘access hypothesis’ could very well be accurate, but since it’s only relevant to a certain percentage of any class, those benefits are washed out when measuring outcomes of the entire class.”

“There’s been a lot of buzz about OER, and many instructors are requesting evidence of efficacy before adopting OER in their class,” Grimaldi said. “Our paper illustrates the complexity of this problem and shows that there’s still much to be done. I don’t think there is convincing evidence showing that adopting OER has positive effects or negative effects on learning outcomes. As far as the influence of OER on learning via access goes, our study shows that the current research simply is not well-suited for capturing such effects.”

Grimaldi is also concerned that the comparison to traditional materials ignores the fact that there isn’t much data on those, either. “There isn’t much in the way of prior research showing positive outcomes of having a textbook at all,” he said. “We seem to be judging the efficacy of OER by whether it outperforms a resource whose own efficacy hasn’t been measured. I think we as a community need to think deeply about how we view and measure the ‘effectiveness’ of an educational resource.”

The team of researchers has recommendations for measuring the impact of OER.

“It’s critical to realize that ‘OER’ itself does not directly affect learning – OER only describes the licensing of educational materials,” Grimaldi said. “However, having an open license can indirectly affect learning through other mechanisms, for example having access to read the materials due to lower prices or more flexible formats. Researchers wanting to examine the influence of OER should first start with a theory about what indirect mechanisms are plausible and then design their study and data collection accordingly.”