Email This Post Email This Post
Home » Blogs

Does it Work? The Most Meaningless Question to Ask about Online Education

28 March 2020 7,365 4 Comments

An extended version of this post will be published in the journal of ECNU Review of Education.

Judge a man by his questions rather than by his answers. ? Voltaire

The scientist is not a person who gives the right answers, he’s one who asks the right questions.? Claude Levi-Strauss

One of the most frequently and persistently asked questions about online education is “does it work” or “is it effective.” It is a legitimate question for it is only natural and reasonable for anyone faced with the decision to undertake online education to want to know if it works. But this seemingly reasonable question is not only meaningless but also dangerous.

The question is meaningless because there cannot be any definitive answer for a number of reasons. First, online education (and its variants such a online instruction, online teaching, distance education and distance learning) is a big umbrella that covers a wide array of different practices, which vary a great deal in terms of quality. Comparing the effectiveness of online education with face-to-face education has been the most common research approach to examine the effectiveness of online education. And the answer has been, for a long time, that there is no significant difference between the two. This answer, however, does not mean online is effective or not, it simply means there are plenty of effective and ineffective programs in both online and face-to-face education. In other words, the within variation is larger than the between variation (Zhao, Lei, Yan, Lai, & Tan, 2005). This is why after tens of thousands of research studies, the best answer one can get about whether online education works is that there is no significant difference from face-to-face education. Is “no difference” the same as effective? Does it mean it works or not?

Second, another reason that there cannot be a definitive answer to this question is the diversity of stakeholders in online education. While the typically assumed stakeholder is the student, hence most studies are about educational outcomes for students, but there are many other stakeholders. Teachers, online education providers, technology suppliers, families of students, tech support personnel, facility managers, real estate owners all have an interest in this question because the answer affects their welfare as well. And unfortunately what works for one stakeholder may not work for the others. For example, what works for technology suppliers simply means their revenue has increased because of online education, which can come at a cost of the real estate owners at a college town because with online education, students do not come to campus anymore. Thus, online education has its own losers and winners. It works for some while causing harm to some others. If online education brings tremendous profits to businesses, it certainly works for the business, but does it work for students?

Third, even within the same program and with only students as the stakeholder, there cannot be a definitive answer because no program can possibly have the same effects on all students equally. Students have differences in multiple domains such as personality, academic preparedness, experiences with technology, attitude towards online education, access to technology, physical location and settings etc. Their different characteristics create a different experience with the same course and thus result in different outcomes (Zhao, 2018). This is why there is no study that has shown that online education benefits all students and why there are so many studies about what kind of students are more suitable for online education. If a program works for 60% of the students, but has no benefit for the rest 40%, is it effective?

Fourth, yet another reason that the question cannot have a definitive answer is the multiplicity of outcomes. Education outcomes include more than what has been typically measured by grades or tests. Acquiring knowledge and skills is only part of the reasons for people to go to school or take a course. Other reasons can include making friends and learning to get along with others, living in a community, developing important personal qualities, and even having free meals, which is the case for poor children in public schools (Zhao, Wehmeyer, Basham, & Hansen, 2019). So it is far from sufficient to rely on test scores or grades to prove whether online education works. As has been discovered, while there is no significant difference between online and face-to-face education in terms of knowledge acquisition, there is significant difference in student satisfaction (Zhao et al., 2005). If online education works in one domain but causes harm in others, for example, motivation to learn or social-emotional development, does it work?

Fifth, the rapid changes in technology that can be used to deliver online education add to the elusiveness of a definitive answer to the question. While pedagogy, design, and human actors certainly paly a significant role in the experiences of online education, so does technology. Experiencing online education via text-only emails is definitely different from experiencing it via video conferencing and immersive VR/AR environments. Technology changes fast, so research done about online education five years ago may or may not be relevant today.

Thus, not surprisingly, decades of research about online or distance education are unable to give a definitive answer to the question: does it work? The best answer one can get is “it depends.” It depends on how the program is delivered; it depends on what outcomes are measured; it depends on whose interests is considered; it depends on the content, the context, the design, the delivery, the technology, the instructor, the student, and many other factors.

The lack of a definitive answer not only makes the question meaningless but also dangerous. It has sent and will continue to send researchers, policy makers, and others down the rabbit hole of effectiveness, perpetuating the futile mission to hunt for more evidence of effectiveness or lack thereof. This is a waste of precious time and resources, to say the least.

Worse, one can always find evidence to show both positive and negative effects (a treatment can be effective in bringing about desirable and undesirable effects) out of the mountains of evidence. But the battle over online education is rarely settled with empirical evidence of educational benefits but rather political and business interests. Thus the evidence is often selectively picked to support whatever decision one favors, leading to reckless decisions that could harm students and others.

A further danger is missing the opportunity to pursue answers to the right questions that can actually help improve online education. There is much complaint about how education as a field has not progressed as fast as other fields such as medicine and agriculture. One of the reasons for the lack of progress are the numerous wars such as the Math Wars and the Reading Wars that have been fought throughout education history, which contributed to the phenomenon of perpetual pendulum swings in educational practices and policies. The wars and the pendulum swings are at least partially caused by the obsession with collecting evidence of what works, without thinking that there is no definitive answer to that question (Zhao, 2017, 2018).

Online education could suffer the same fate without abandoning the question for more meaningful ones that can provide insights and guide efforts to make online education better for all children. After all, it is COVID-19, rather than overwhelming evidence of effectiveness, that converted face-to-face education to online for hundreds of millions of students around the world, virtually overnight!

In the next post, I will discuss what are the right questions to ask about online education.

References:

Zhao, Y. (2017). What Works Can Hurt: Side Effects in Education. Journal of Educational Change, 18(1), 1-19.

Zhao, Y. (2018). What Works May Hurt: Side Effects in Education. New York: Teachers College Press.

Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, S. (2005). What Makes the Difference: A Practical Analysis of Research on the Effectiveness of Distance Education. Teachers College Record, 107(8), 1836-1884.

Zhao, Y., Wehmeyer, M., Basham, J., & Hansen, D. (2019). Tackling the wicked problem of measuring what matters: Framing the questions. ECNU Review of Education, 2(3), 262-278.

 

1 Star2 Stars3 Stars4 Stars5 Stars (4 votes, average: 3.75 out of 5)
Loading...

4 Comments »

  • Matt said:

    One size fits all? Never has and never will be the case in education, or anything creative in life. Virtual learning is allowing some students to bloom like never before. The 21st Century Skills we’ve been discussing for over two decades are now being reexamined and hopefully applied. Meaning, COVID-19 is allowing for (yes, many feel being “forced”) experimentation and innovation. Regardless, teachers are coming to grips with what is essential and that which is superfluous. More importantly, coming into question is context and how students might be able to learn differently. Break-out chats for collaboration on Zoom? Turning outdated methods of content delivery into more inquiry and possibly problem or project based learning.
    The jury may be out whether what we are doing is “effective” or not? But it always will be. Seems a better question might be, “What are we doing that is working?” Exciting days we are living in!

  • Jordan said:

    Good review of the “wrong” question to ask about online education. During today’s circumstances, my response to the question would be, “We have to make the most of it anyway.” Whether it “works” or not, this is currently our primary option for delivering instruction and pursuing educational outcomes. Let’s not focus on finding reasons why it wouldn’t work; let’s focus on making the most of this opportunity.

  • Jim Horn said:

    I work for an institution that has had open enrollment for most its programs since the college began in the early 1970s. It was founded with the mission to serve working adults whose prior educational experiences may have been thwarted by socioeconomic disadvantage and/or racism.

    The adult learning model that has been used up until now is based upon small classes, collaborative teaming for student projects, interactive classroom environments, and instructional adaptation based on student needs.

    Over the past several years, the college has been moving more and more of its courses online, without regard for the effects upon the quality of learning among its students.

    What does the research tell us about the effects of online learning on disadvantaged student populations?

  • Beware of science envy in designing the future of learning – Punya Mishra's Web said:

    […] not necessarily so in the world of education. Zhao makes a similar argument for the question “Does it work?” when it comes to online education, and calls it the most meaningless (and even dangerous) […]

Feel free to comment:

The views expressed on this site are entirely my own. They do not represent my employer or any other organization/institution. All comments are subject to approval.

Add your comment below, or trackback from your own site. You can also subscribe to these comments via RSS.

Be nice. Keep it clean. Stay on topic. No spam.

You can use these tags:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This is a Gravatar-enabled weblog. To get your own globally-recognized-avatar, please register at Gravatar.