An extended version of this post will be published in the journal of ECNU Review of Education.

Judge a man by his questions rather than by his answers. ? Voltaire

The scientist is not a person who gives the right answers, he’s one who asks the right questions.? Claude Levi-Strauss

One of the most frequently and persistently asked questions about online education is “does it work” or “is it effective.” It is a legitimate question for it is only natural and reasonable for anyone faced with the decision to undertake online education to want to know if it works. But this seemingly reasonable question is not only meaningless but also dangerous.

The question is meaningless because there cannot be any definitive answer for a number of reasons. First, online education (and its variants such a online instruction, online teaching, distance education and distance learning) is a big umbrella that covers a wide array of different practices, which vary a great deal in terms of quality. Comparing the effectiveness of online education with face-to-face education has been the most common research approach to examine the effectiveness of online education. And the answer has been, for a long time, that there is no significant difference between the two. This answer, however, does not mean online is effective or not, it simply means there are plenty of effective and ineffective programs in both online and face-to-face education. In other words, the within variation is larger than the between variation (Zhao, Lei, Yan, Lai, & Tan, 2005). This is why after tens of thousands of research studies, the best answer one can get about whether online education works is that there is no significant difference from face-to-face education. Is “no difference” the same as effective? Does it mean it works or not?

Second, another reason that there cannot be a definitive answer to this question is the diversity of stakeholders in online education. While the typically assumed stakeholder is the student, hence most studies are about educational outcomes for students, but there are many other stakeholders. Teachers, online education providers, technology suppliers, families of students, tech support personnel, facility managers, real estate owners all have an interest in this question because the answer affects their welfare as well. And unfortunately what works for one stakeholder may not work for the others. For example, what works for technology suppliers simply means their revenue has increased because of online education, which can come at a cost of the real estate owners at a college town because with online education, students do not come to campus anymore. Thus, online education has its own losers and winners. It works for some while causing harm to some others. If online education brings tremendous profits to businesses, it certainly works for the business, but does it work for students?

Third, even within the same program and with only students as the stakeholder, there cannot be a definitive answer because no program can possibly have the same effects on all students equally. Students have differences in multiple domains such as personality, academic preparedness, experiences with technology, attitude towards online education, access to technology, physical location and settings etc. Their different characteristics create a different experience with the same course and thus result in different outcomes (Zhao, 2018). This is why there is no study that has shown that online education benefits all students and why there are so many studies about what kind of students are more suitable for online education. If a program works for 60% of the students, but has no benefit for the rest 40%, is it effective?

Fourth, yet another reason that the question cannot have a definitive answer is the multiplicity of outcomes. Education outcomes include more than what has been typically measured by grades or tests. Acquiring knowledge and skills is only part of the reasons for people to go to school or take a course. Other reasons can include making friends and learning to get along with others, living in a community, developing important personal qualities, and even having free meals, which is the case for poor children in public schools (Zhao, Wehmeyer, Basham, & Hansen, 2019). So it is far from sufficient to rely on test scores or grades to prove whether online education works. As has been discovered, while there is no significant difference between online and face-to-face education in terms of knowledge acquisition, there is significant difference in student satisfaction (Zhao et al., 2005). If online education works in one domain but causes harm in others, for example, motivation to learn or social-emotional development, does it work?

Fifth, the rapid changes in technology that can be used to deliver online education add to the elusiveness of a definitive answer to the question. While pedagogy, design, and human actors certainly paly a significant role in the experiences of online education, so does technology. Experiencing online education via text-only emails is definitely different from experiencing it via video conferencing and immersive VR/AR environments. Technology changes fast, so research done about online education five years ago may or may not be relevant today.

Thus, not surprisingly, decades of research about online or distance education are unable to give a definitive answer to the question: does it work? The best answer one can get is “it depends.” It depends on how the program is delivered; it depends on what outcomes are measured; it depends on whose interests is considered; it depends on the content, the context, the design, the delivery, the technology, the instructor, the student, and many other factors.

The lack of a definitive answer not only makes the question meaningless but also dangerous. It has sent and will continue to send researchers, policy makers, and others down the rabbit hole of effectiveness, perpetuating the futile mission to hunt for more evidence of effectiveness or lack thereof. This is a waste of precious time and resources, to say the least.

Worse, one can always find evidence to show both positive and negative effects (a treatment can be effective in bringing about desirable and undesirable effects) out of the mountains of evidence. But the battle over online education is rarely settled with empirical evidence of educational benefits but rather political and business interests. Thus the evidence is often selectively picked to support whatever decision one favors, leading to reckless decisions that could harm students and others.

A further danger is missing the opportunity to pursue answers to the right questions that can actually help improve online education. There is much complaint about how education as a field has not progressed as fast as other fields such as medicine and agriculture. One of the reasons for the lack of progress are the numerous wars such as the Math Wars and the Reading Wars that have been fought throughout education history, which contributed to the phenomenon of perpetual pendulum swings in educational practices and policies. The wars and the pendulum swings are at least partially caused by the obsession with collecting evidence of what works, without thinking that there is no definitive answer to that question (Zhao, 2017, 2018).

Online education could suffer the same fate without abandoning the question for more meaningful ones that can provide insights and guide efforts to make online education better for all children. After all, it is COVID-19, rather than overwhelming evidence of effectiveness, that converted face-to-face education to online for hundreds of millions of students around the world, virtually overnight!

In the next post, I will discuss what are the right questions to ask about online education.

References:

Zhao, Y. (2017). What Works Can Hurt: Side Effects in Education. Journal of Educational Change, 18(1), 1-19.

Zhao, Y. (2018). What Works May Hurt: Side Effects in Education. New York: Teachers College Press.

Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, S. (2005). What Makes the Difference: A Practical Analysis of Research on the Effectiveness of Distance Education. Teachers College Record, 107(8), 1836-1884.

Zhao, Y., Wehmeyer, M., Basham, J., & Hansen, D. (2019). Tackling the wicked problem of measuring what matters: Framing the questions. ECNU Review of Education, 2(3), 262-278.