Did the Shift from Paper to Computer Bring Down East Asia’s (China’s) PISA Performance?
I was surprised by China’s 2015 PISA performance, particularly in reading. I was confident that even the expansion beyond Shanghai would not cause a significant decline based on my understanding of the Chinese education system. And Beijing, Jiangsu, and Guangdong are traditionally strong performers in China, and among the most developed provinces, although behind Shanghai.
While I don’t believe PISA scores mean anything beyond the ability to perform on PISA tests, I wanted to see if I needed to change my thinking. Perhaps Chinese students are not as good at taking tests as I had thought? Perhaps students in Shanghai are drastically different from the rest of China? Perhaps China’s education system is changing very fast that they have truly moved away from test preparation? I am willing to consider any possibilities in the face of evidence.
I found a news article from Chinese Taipei (Taiwan), whose reading performance has declined significantly as well, although not as much as China (Shanghai to BSJG). Chinese Taipei’s reading ranking dropped from #8 in 2012 to #23 in 2015. According to the report from Chinese Taipei’s Central News Agency, a high-level official of its Ministry of Education attributed the decline to students’ unfamiliarity with reading on a computer. Because PISA changed from paper-based to computer-based tests. Although students in Taiwan may use smart devices for other purposes, they are not used to reading large amount of text, diagrams, and graphics on a computer screen. The official said in addition to improve reading in general, Taiwan would increase computers to support reading and adding computer-based tests to important exams in the future.
Seems a reasonable hypothesis. PISA shifted from paper-based tests to computer-based tests from 2012 to 2015 for most participants.
So I dug a little bit deeper. First I verified that indeed all students in China (BSJG) took the PISA on computers (looks like Windows-based laptops), which are very different from what most young students are using today (hand held devices), if they had access to digital devices. Second, I confirmed that the percentages of students categorized as rural are much higher in Beijing, Jiangsu, and Guangdong than Shanghai. That rural students have less access to technology is a safe assumption. So these students would be less familiar with computers. Finally, it is also common in China that many parents and teachers do not want their students to spend much time on the computer as they view it as a distraction from real study. It is thus unreasonable to assume the new student population representing China is not familiar with reading and taking tests on computers.
This may be true in all Asian systems. Although Korea, Japan, Hong Kong, and Singapore have all put much emphasis and invested heavily in educational technology (see my report Lessons that Matter: What Should We Learn from Asia’s School Systems). It is likely that students are still more used to doing school work on paper. While I am not able to find much evidence to confirm this hypothesis, I think it is likely based on my analysis of the changes in PISA scores across different cycles. All data are from NCES PISA Site (PISA Data Explorer).
First, all East Asian systems’ saw a decline in reading from 2012 to 2015, some are very significant, in contrast to the increase shared by almost all systems except Korea between 2009 and 2012. Previous cycles showed some increase and some loss (see Figures below).
Second, we see almost an identical pattern for math. Except for Macau, which had a modest increase, all other East Asian system saw a decline, most statistically significant, including Singapore (yes, the #1 Singapore) from 2012-2015. In contrast, the changes across 2009-2012 are all positive (Figures below).
Third, the situation in science is a bit more mixed, but still shows the majority of East Asian systems’ performance declined significant in 2015 from 2012. Singapore, Macau, and Chinese Taipei had some modest gains. However, all East Asian systems saw an increase in science scores, even the best performer Shanghai gained points from 2009 to 2012. (Figures below).
What could have caused such a uniform change in eight education systems in as short a time as three years? The only common factor I could find is the change of PISa delivery format: from paper to computer.
PISA did pay attention to the issue and make it possible to compare the 2015 computer-based results with paper-based tests in previous cycles, according to its report. However, whether the effort on tweaking items enough to offset the impact on populations of students who are not used to reading and doing tasks on a computer remains a question that PISA should answer.
Until we have the answer, I hope we do not jump to conclusions such as: After all, China’s education is not as great as we thought, based on Shanghai’s performance, or East Asian’s PISA performance is in decline, or East Asian students are not good at taking tests anymore, or perhaps East Asian countries education reforms are becoming successful so they are not as focused on test preparation as before (which I hope can indeed happen). Again, my advice: Don’t read too much into it.
If Jiangsu, Beijing and Guangdong had higher percentages of rural students than Shanghai, then why would you not expect a fall in “China’s” score to begin with. The rural/urban gap in PISA scores is pretty well-established across all countries, no?
Very good point. Certainly, the increase of rural students would affect the performance. But what caught my interest is: a. why Chinese students’ reading dropped much more significantly than math and science? It seems the impact is not evenly distributed and b. why other systems whose population did change also saw a decline. Thanks.
The reading scores for all of the chinese speakers seem depressed compared to science and math scores
Macau + taiwan + bsjg
This might be caused by bias in the reading section to chinese language speakers
[…] My own PISA 2015 forecast was way off-base — and taught me a lesson. After the recent TIMSS 2015 Mathematics results released in November 2016, an East Asian sweep, led by Singapore and Korea, seemed like a safe bet. How Finland performs also attracts far less attention than it did in its halcyon days back in 2003 and 2006. The significant OECD pivot away from excellence to equity caught me napping and I completely missed the significance of moving (2012 to 2015) from pencil-and-paper to computer-based tests. […]
[…] Did the Shift from Paper to Computer Bring Down East Asia’s (China’s) PISA Performance? Yong Zhao, Education in the Age of Globalization […]
[…] Did using computers lower China’s PISA score? What are the implications for the US? […]
A profound insight!
It is true that in most part of mainland China have not put much emphasis on educational technology, especially in the rural parts. And I believe that this transition of paper based test to computer based test did affect the performance of Chinese students, even the students from urban parts. Because as far as I know, most schools in mainland of China, both urban and rural schools, are still using paper based tests. So it is not hard to know that the sudden shift for the test mode will affect students’ performance to a certain extent.
As for the ranking, I agree with “Don’t read too much into it”. However, when it comes to the inspiration from this PISA results for our education system, I think there is a lot more for us to take a really good look at it.
I once read a very brilliant research from http://www.epi.org/publication/us-student-performance-testing/ called “What do international tests really show about U.S. student performance?”. This research made me think, the information that PISA provided us is a lot more than just a ranking. And now in our country, the People’s Republic of China, is having this important reform for our education system. For us, all the educators and researchers in China, we should make a really good use of the information from PISA to do thoroughly researches, so we can help to build a better education system for our next generation.
Feel free to comment:
The views expressed on this site are entirely my own. They do not represent my employer or any other organization/institution. All comments are subject to approval.Archive
07.29.24 AI Has Little to Contribute to Traditional Education: Problems and Possibilities
03.17.24 Focused: Understanding, Negotiating, and Maximizing Your Influence as a School Leader
02.25.24 What Happened to Global Competence?
08.05.23 Rethinking the time spent at school: Could flexibility improve engagement and performance for students and teachers?
01.17.23 Introduction to Improbable Probabilities: The Unlikely Journey of Yong Zhao
01.05.23 How Not to Kill Creativity?
08.19.22 Preface to Improbable Probabilities: The Unlikely Journey of Yong Zhao
02.05.22 Introduction to New Book: Learning for Uncertainty: Teaching Students How to Thrive in a Rapidly Evolving World
09.25.21 Side effects in education: Taxonomy of educational outcomes
07.13.21 Introduction to My New Book: Learners without Borders
03.09.21 New article: Build back better: Avoid the learning loss trap
02.18.21 New article: The changes we need: Education post COVID-19
09.15.20 Watch Ep4 Creativity in Crisis: How well is creativity understood? A Conversation with Barb Kerr, Haiying Long, Ron Beghetto, & Yong Zhao
08.15.20 Can Creativity be Taught? Ep 3 of Creativity in Crisis on August 28th 3:00-4:00pm Pacific Time
07.13.20 Speak a Different Language: Reimagine the Grammar of Schooling
Tag Cloud
Accountability achievement gap CCSSO China/Chinese Commissioner Common Core Standards education Educational Policy Education Reforms national standards New York NGA Singapore standardized testing Standards student performance
WP Cumulus Flash tag cloud by Roy Tanck and Luke Morton requires Flash Player 9 or better.
Silverliningforlearning
Most Commented
Most Viewed
Views expressed on this site are entirely personal. They do not necessarily represent the official positions or views of my employer
Powered by WordPress | Log in | Entries (RSS) | Comments (RSS) | Arthemia theme by Michael Hutagalung