How Does PISA Put the World at Risk (Part 3): Creating Illusory Models of Excellence

Statistik Penting	Angka
Bahagian Pasaran Mega888 di Malaysia	Melebihi 90%
Pemain dari Malaysia, Singapura, dan Thailand	80%
Jumlah Muat Turun Aplikasi Mega888	Lebih 1 juta

Statistik Mega888	Nilai
Bahagian Pasaran Permainan Slot di Malaysia	Lebih 90%
Pemain dari Malaysia, Singapura, dan Thailand	80%
Jumlah Muat Turun Aplikasi Mega888	Lebih 1 Juta

Jenis Bonus	Perincian	Syarat Tuntutan
Bonus Selamat Datang	Bonus sehingga RM1,000 untuk pemain baru	Daftar akaun baru dan buat deposit pertama
Bonus Deposit	Bonus sehingga 150% daripada jumlah deposit	Buat deposit ke dalam akaun dan penuhi syarat taruhan
Promosi Bermusim	Bonus dan ganjaran istimewa seperti Birthday Month Bonus	Ikuti syarat dan tempoh promosi yang ditetapkan

Ciri-Ciri	Touch 'n Go	GrabPay	Boost
Had Transaksi	Tinggi	Fleksibel	Munasabah
Kelajuan Pemprosesan	Pantas	Pantas	Pantas
Keselamatan	Baik	Baik	Baik
Populariti di Mega888	Tinggi	Sederhana	Rendah

Kaedah Sokongan	Waktu Operasi	Tindak Balas Purata
Telefon	24 jam sehari, 7 hari seminggu	20 saat
E-mel	24 jam sehari, 7 hari seminggu	6 jam
Borang Hubungi dalam talian	24 jam sehari, 7 hari seminggu	2 jam

Kaedah Pengeluaran	Tempoh Pemprosesan
E-wallet	1 hari bekerja
Perbankan dalam talian	1 hari bekerja

Creative, Entrepreneurial, and Global: 21st Century Education

Email This Post

Home » Blogs, China/Chinese, Education Reforms, Globalization

How Does PISA Put the World at Risk (Part 3): Creating Illusory Models of Excellence

23 March 2014 59,022 7 Comments

Few numbers command as much power as PISA scores, not even the number of Olympic medals or Nobel Prize winners in the world today. It is utterly shocking and embarrassing to see some otherwise rational and well-educated people (or at least they should be) in powerful positions believe that three test scores show the quality of their education systems, the effectiveness of their teachers, the ability of their students, and the future prosperity of their society.

PISA has become the star-maker in the education universe because of its bold claim to assess “the extent to which students near the end of compulsory education have acquired key knowledge and skills that are essential for full participation in modern societies” [1, p. 15]. Moreover, PISA claims to find educational stars by identifying which education systems better prepare their children for “full participation in modern societies”—as measured by PISA scores. The goal is for educational systems to learn from “the highest-performing and most rapidly improving school systems” [1, p. 15].

The claims are as bold as they are illusory. In a June 2013 article in the Times Education Supplement (TES) magazine, William Stewart questioned if PISA is fundamentally flawed:

what if there are “serious problems” with the Pisa data? What if the statistical techniques used to compile it are “utterly wrong” and based on a “profound conceptual error”? Suppose the whole idea of being able to accurately rank such diverse education systems is “meaningless”, “madness”?

What if you learned that Pisa’s comparisons are not based on a common test, but on different students answering different questions? And what if switching these questions around leads to huge variations in the all-important Pisa rankings, with the U.K. finishing anywhere between 14th and 30th and Denmark between fifth and 37th? What if these rankings—that so many reputations and billions of pounds depend on, that have so much impact on students and teachers around the world—are in fact “useless”?

The article’s findings are troublesome to PISA and should be extremely unsettling to its faithful, say scholars who have independently reached the same conclusions. “As far as they are concerned, the emperor has no clothes,” writes Stewart. Citing numerous publications and conversations with scholars in Denmark, Northern Ireland, and the U.K., as well as with OECD, he points out major technical flaws with PISA’s composition of the tests, administering of the tests, and use of statistical techniques to generate country rankings.

Svend Kreiner, a professor of biomedical statistics at the University of Copenhagen, presents a more serious challenge to PISA. He questions the appropriateness of the model PISA uses to produce the country rankings. PISA uses the Rasch model, a widely used psychometric model named after the late Danish mathematician and statistician Georg Rasch. For this model to work properly, certain requirements must be met. But according to Kreiner, who studied under Rasch and has worked with his model for 40 years, PISA’s application does not meet those requirements. In an article published in the academic journal Psychometrika, Kreiner and co-author Karl Bang Christensen show that the Rasch model does not fit the reading literacy data of PISA, and thus the resulting country rankings are not robust. As a result, rankings of countries can vary a great deal over different subsets. Denmark, for example, can rank anywhere between 5^th and 36^thout of 56 countries [2]. “That means that [PISA] comparisons between countries are meaningless,” Kreiner told TES.

Kreiner was not the first or only scholar to raise questions about PISA’s technical flaws. In 2007, a collection of nearly 20 researchers from multiple European countries presented their critical analyses in the book PISA According to PISA: Does PISA Keep What It Promises. Independent scholars from all over the world took apart PISA’s methodology, examining how it was designed; how it sampled, collected, and presented data; and what its outcomes were. Then the researchers compared the test’s real-life validity to its claims [3]. Almost all of them “raise[d] serious doubts concerning the theoretical and methodological standards applied within PISA, and particularly to its most prominent by-products, its national league tables or analyses of school systems” [3, p. 10]. Among their conclusions:

PISA is by design culturally biased and methodologically constrained to a degree which prohibits accurate representations of what actually is achieved in and by schools. Nor is there any proof that what it covers is a valid conceptualization of what every student should know.

The product of most public value, the national league tables, are based on so many weak links that they should be abandoned right away. If only a few of the methodological issues raised in this volume are on target, the league tables depend on assumptions about the validity and reliability which are unattainable.

The widely discussed by-products of PISA, such as the analyses of “good schools,” “good instruction” or differences between school systems … go far beyond what a cautious approach to these data allows for. They are more often than not speculative… [3, p. 12-13].

PISA did respond to some of the technical challenges. For example Andreas Schleicher, PISA’s face to the world, wrote a commentary[1] responding to Kreiner’s charges in TES.

While the dispute over PISA’s technical flaws continues, some argue that even if PISA did everything right technically, it still could not possibly claim to be measuring the quality of entire education systems, let alone their students’ ability to live in the modern world.

“There are very few things you can summarise with a number and yet Pisa claims to be able to capture a country’s entire education system in just three of them,” wrote Dr. Hugh Morrison of Queen’s University Belfast in Northern Ireland. “It can’t be possible. It is madness”[2]. Morrison, a mathematician, does not think the Rasch model should be used at all. He argues that “at the heart of Rasch, and other similar statistical models, lies a fundamental, insoluble mathematical error that renders Pisa rankings ‘valueless’ and means that the programme ‘will never work’” (Stewart, 2013). The problem of PISA, according to Morrison, violates a central principle of measurement drawn from physicist Niels Bohr’s work: the entity measured cannot be divorced from the measuring instrument. Morrison illustrates his point with an example. Suppose Einstein and a student both produced a perfect score on a test. “Surely to claim that the pupil has the same mathematical ability as Einstein is to communicate ambiguously?” The unambiguous communication would be “Einstein and the pupil have the same mathematical ability relative to this particular [test]… [M]athematical ability, indeed any ability, is not an intrinsic property of the individual; rather, it’s a joint property of the individual and the measuring instrument.”[3] In a nutshell, Morrison’s point is that PISA scores students’ ability to complete tasks included in the test, not their general ability to understand and succeed.

Even if PISA did measure cognitive abilities as accurately as it claims to, those abilities only span three domains: math, reading, and science. PISA makes the assumption that these skills are universally valuable. In other words, as Svein Sjøberg, a professor of science education at Norway’s University of Oslo, points out, PISA “assumes that the challenges of tomorrow’s world are more or less identical for young people across countries and cultures” and thus promotes “kind of universal, presumably culture-free, curriculum as decided by the OECD and its experts”. This assumption is mistaken. “Although life in many countries do [sic] have some similar traits, one can hardly assume that the 15-year olds in e.g. Japan, Greece, Mexico and Norway are preparing for the same challenges and need identical life skills and competencies” [4, p. 7].

Even if cognitive skills in math, science, and reading were the most important skills in the universe, they would not—could not—be the only skills an educational system should cultivate. Skills and knowledge in other domains, such as “the humanities, social sciences, foreign languages, history, geography, physical education etc.” [4, p. 3] play a crucial role if citizens of any country are to live a fulfilling life. So do non-cognitive skills: social-emotional skills, curiosity, creativity, resilience, engagement, passion, and a host of other personality traits. In fact, many would argue that talents, skills, knowledge, and creativity in domains outside math, science, and reading are at least as important, perhaps even more important to live successfully in the new world. Henry Levin, a professor in economics of education at Teachers College, Columbia University reviews empirical evidence that shows the essential value of non-cognitive skills to work and life in his article More Than Just Test Scores [5].

Thus, even if PISA were methodologically sound, conceptually correct, and properly administered, it is still a test. And as such, the results can only indicate the extent to which 15 year olds provided responses deemed correct by the test makers. The only unambiguous conclusion of Shanghai’s PISA ranking would be that 15-year-old students in Shanghai provided the most responses judged to be correct in math, reading, and science in 2009 and 2012. Leaping from the highest PISA score in three subjects to the best education system in the world is too big a jump for any logical person—unless the purpose of education is defined as doing well on the PISA.

Since no one, not the Chinese and not even the PISA team (I hope), would define the purpose of education as achieving good PISA scores, making China the world’s model of educational excellence just because some of its 15-year-olds received the highest PISA scores is not only inaccurate but misleading. The excellence is a simple illusion created by the PISA league tables.

Part 1: Romanticizing Misery

Part 2: Glorifying Educational Authoritarianism

Adapted from my book Who’s Afraid of the Big Bad Dragon: Why China has the Best (and worst) Education in the World.

References:

1. OECD, Ready to Learn: Students Engagement, Drive, and Self-beliefs. 2013, OECD: Paris.

2. Kreiner, S. and K.B. Christensen, Analyses of Model Fit and Robustness. A New Look at the PISA Scaling Model Underlying Ranking of Countries According to Reading Literacy. Psychometrika, 2013. June.

3. Hopmann, S.T., Gertrude Brinek, and M. Retzl, eds. PISA zufolge PISA – PISA According to PISA. 2007, Lit Verlag: Berlin.

4. Sjøberg, S. PISA: Politics, fundamental problems and intriguing results Recherches en Education, 2012. 14.

5. Levin, H.M., More Than Just Test Scores. Prospects: The Quarterly Review of Comparative Education, 2012. 42(3): p. 269-284.

[1] http://www.tes.co.uk/article.aspx?storycode=6345213

[2] https://paceni.wordpress.com/2013/12/01/pisa-2012-major-flaw-exposed/

[3] https://paceni.wordpress.com/2013/12/01/pisa-2012-major-flaw-exposed/

Share

(No Ratings Yet)

Loading...

7 Comments »

How Does PISA Put the World at Risk (Part 3): Creating Illusory Models of Excellence | Educational Policy Information said:

[…] YongZhao Few numbers in the world command as much power as PISA scores, not even the number of Olympic […]

# 23 March 2014 at 1:01 pm

Should UK pupils be assessed to international standards? | Asian Correspondent said:

[…] Kong and Shanghai, which do not provide a representative sample of the country’s students. Many agree Pisa’s flawed evaluation system makes it impossible to capture a country’s entire education, […]

# 7 April 2014 at 4:30 am

Yong Zhao: PISA and Illusory Models of Excellence | Diane Ravitch's blog said:

[…] Zhao, who was born and educated in China, is a sharp critic of standardized testing. This article is the third in a series in which he criticizes PISA for misleading the world and promoting standardization and […]

# 8 April 2014 at 4:00 am

Yong Zhao: PISA and Illusory Models of Excellence | Educational Policy Information said:

[…] Zhao, who was born and educated in China, is a sharp critic of standardized testing. This article is the third in a series in which he criticizes PISA for misleading the world and promoting standardization and […]

# 8 April 2014 at 8:00 am

Susan Lee Schwartz said:

This wonderful piece is linked/published at
http://www.opednews.com/Quicklink/How-Does-PISA-Put-the-Worl-in-Best_Web_OpEds-Education_Education_Excellence_Future-140408-897.html#comment482356
with a comment oat the end.
It is also on my Facebook page.

# 8 April 2014 at 8:10 am

Yong Zhao--U.S. needs to lead, not follow - I wonder if... said:

[…] How Does PISA Put the World at Risk (Part 3): Creating Illusory Models of Excellence. […]

# 9 April 2014 at 8:28 pm

Charlotte Sexauer » Photography » Should UK pupils be assessed to international standards? said:

[…] Kong and Shanghai, which do not provide a representative sample of the country’s students. Many agree Pisa’s flawed evaluation system makes it impossible to capture a country’s entire education, […]

# 19 September 2014 at 7:18 am

Feel free to comment:

The views expressed on this site are entirely my own. They do not represent my employer or any other organization/institution. All comments are subject to approval.

Silverliningforlearning

Most Commented

Most Viewed

A Discussion on China’s Recent Education Reform on CCTV 9 - 595,844 views
New article: Build back better: Avoid the learning loss trap - 415,904 views
You must be joking, Professor Chua: An open letter to the Chinese Tiger Mom - 297,449 views
Common Sense Vs. Common Core: How to Minimize the Damages of the Common Core - 283,025 views
Five Questions to Ask about the Common Core* - 266,388 views

Mega888 Online Casino Malaysia Duitnow Slot Ewallet Casino TNG

Intisari Utama

Pengenalan Kepada Platform Mega888 Malaysia

Sejarah Mega888 di Malaysia

Ciri-ciri Utama Platform

Lesen dan Keselamatan

Cara Mendaftar Akaun Mega888

Sistem Pembayaran DuitNow di Mega888

Kelebihan Menggunakan DuitNow

Langkah-langkah Transaksi DuitNow

slot ewallet mega888 TNG

Permainan Popular di Mega888

Slot Games Terbaik

Live Casino Games

Panduan Deposit Menggunakan E-wallet

Kelebihan Menggunakan E-wallet untuk Deposit

Langkah-langkah Deposit Menggunakan E-wallet

Keselamatan Transaksi di Mega888

Teknologi Enkripsi

Sistem Verifikasi

Bonus dan Promosi Mega888

Panduan Bermain Slot Online

Tips Menang

Strategi Pengurusan Modal

Aplikasi Mega888 Mobile

Ciri-ciri Utama Aplikasi Mega888 Mobile

Perbandingan E-wallet di Mega888

Touch 'n Go

GrabPay

Boost

Sokongan Pelanggan 24/7

Tips Keselamatan Akaun

Trend Terkini Casino Online Malaysia

Perkembangan Teknologi

Perubahan Industri

Panduan Pengeluaran Wang

Langkah-langkah Pengeluaran Wang

Had Pengeluaran

Tempoh Pemprosesan

Kesimpulan

FAQ

Apakah Mega888 dan bagaimana ia beroperasi di Malaysia?

Bagaimana saya boleh mendaftar akaun Mega888?

Apakah kelebihan menggunakan DuitNow sebagai kaedah pembayaran di Mega888?

Bagaimana saya boleh menggunakan e-wallet TNG untuk bermain slot di Mega888?

Apakah permainan slot dan kasino langsung yang paling popular di Mega888?

Bagaimana saya boleh membuat deposit di Mega888 menggunakan e-wallet?

Bagaimana Mega888 melindungi keselamatan transaksi pengguna?

Apakah bonus dan promosi yang ditawarkan oleh Mega888?

Bagaimana saya boleh bermain slot online dengan selamat dan berkesan di Mega888?

Apakah kelebihan menggunakan aplikasi Mega888 berbanding platform web?

How Does PISA Put the World at Risk (Part 3): Creating Illusory Models of Excellence

7 Comments »

Feel free to comment:

Archive

Tag Cloud

Silverliningforlearning

Most Commented

Most Viewed