PISA has many peculiar and surprising discoveries…

Having a growth mindset is negatively associated with academic performance for participating students from China (Beijing-Shanghai-Jiangsu-Zhejiang or B-S-J-Z China), according to the 2018 PISA results. That is, Chinese students who had a fixed mindset scored higher in PISA reading than those who had a growth mindset. Considering that Chinese students’ stunning scores that put them way above all other students in the world, this finding should be disconcerting to proponents of growth mindset, including the PISA team.

Does this invalidate the belief that growth mindset result in improved academic performance? Not according to PISA:

PISA findings support the idea that instilling a growth mindset in students could result in better academic performance (Blackwell, Trzesniewski and Dweck, 2007[9]; McCutchen et al., 2016[10]). On average across OECD countries, students who disagreed or strongly disagreed with the statement “Your intelligence is something about you that you can’t change very much” scored 41 points higher in reading than students who agreed or strongly agreed with the statement. The former group of students scored 32 points higher than the latter group after accounting for the socio-economic profile of students and schools. (OECD, 2019, p. 203).

The China case can be treated as noise in the data, an anomaly, and outlier. After all, China is one of the only two education systems in which the relationship between growth mindset and PISA reading scores is negative. In fact, only in four education systems (B-S-J-Z, Hong Kong, Lebanon and North Macedonia) out of the over 70 participants in 2018 was there no positive relationship between the two.

The overall relationship between percentage of students holding a growth mindset and reading scores is very significant and positive (R²=0.47) across different education systems as shown in PISA’s Figure III 14.3.

But Figure III 14.3 reveals more than a positive relationship between growth mindset and reading scores. It also shows that China is not alone in defying the belief of the power of growth mindset. The Figure puts the educational systems into four groups according to percentage of students holding a growth mindset and their reading PISA scores, using OECD averages as the cut scores. A number of intriguing observations jump out:

1. There are more education systems that are low on growth mindset AND reading scores than education systems that are high on both or one dimensions (below OECD averages).
2. Very few systems (13 out of 79 or 16%) are high on both (high growth mindset and high reading scores).
3. About 10% (7 out 79) education systems had high percentage of students holding a growth mindset but low PISA reading performance (below OECD average).
4. Over 10% (10 out of 79) systems had low growth mindset but high reading scores (both below OECD averages).
5. The 10 systems that had low growth mindset include the highest performers in reading—the top four (B-S-J-Z, Singapore, Macao, and Hong Kong), as well as #9 Korea, #10 Poland, #17 Chinese Taipei, #21 Slovenia, #23 France, and #25 Czech Republic.

What sense can we make of these observations? A few came to my mind (as of now):

1. The positive relationship between growth mindset and reading performance found in PISA 2018 is largely established by low-low systems. That is, the education systems that had low percentage of students holding a growth mindset and had low PISA scores contributed most to the positive relationship (over 60% of participating systems are in this group).
2. Education systems do not need to have high percentage of students having a growth mindset to score high on the PISA.
3. Education systems with high percentage of students holding a growth mindset are not guaranteed high scores on the PISA.

In other words, growth mindset does not have much to do with high performance of education systems. These words can be unsettling to mindset advocates and PISA lovers but they reflect the data PISA gathered. This conclusion may be surprising but makes sense because it is consistent with PISA’s fundamental problems that have been exposed by many.

Biased measurement

PISA has been criticized for imposing a uniform view of constructs (e.g, education quality, skills and knowledge for success in the future, student wellbeing etc.) on the diverse cultures and societies its participants represent. For example, a group of scholars in Japan exposed the bias of PISA’s conceptualization of “student well-being” in the article ‘Better policies for better lives’?: constructive critique of the OECD’s (mis)measure of student well-being published in the Journal of Education Policy (Rappleye, Komatsu, Uchida, Krys, & Markus, 2019). Based on careful theoretical and empirical analyses, they point out that PISA’s well-being measure is flawed because it uses a Western view of well-being on Eastern Asian students, who have different views of well-being/happiness.

Similarly, the conceptualization of “growth mindset vs. fixed mindset” may be very different in different cultures. The word “intelligence” is itself an invention of Western psychologist and cannot easily find cultural equivalent elsewhere. Thus when students respond to the question of whether they agree or disagree to the statement “Your intelligence is something about you that you can’t change very much,” they might be responding to different things.

Educational Colonization

This is another example of PISA’s attempt at colonization through education (Zhao, 2014, 2016, in press). PISA has the bad habit of deriving bold (but often misleading) lessons from half-truths or misunderstandings and promoting them as the policy/practice that everyone should adopt. For example, based on half (or perhaps less than a quarter of) truth, PISA (in collaboration with the global consulting firm McKinsey) came up with grand lesson that every education system should recruit top graduates (in terms of high academic performance) into the teaching force (Barber & Mourshed, 2007). This idea has been debunked (Zhao, 2018).

PISA latched on the idea of growth mindset, which has recently become extremely popular in the world of education and beyond. Growth mindset may be a meaningful concept and could indeed lead to better academic performance, as has been demonstrated by its proponents (Blackwell, Trzesniewski, & Dweck, 2007; Dweck, 2008). But does it apply to all students in all contexts in all cultures in the world? Perhaps yes, perhaps no. The answer is we don’t know.

However judging from the PISA data, the answer may be NO. The Chinese students (B-S-J-G) did not need it to be the best PISA performer, nor did the students from Hong Kong. It seems to work extremel positively for students in Australia, Brazil, Colombia, Iceland, New Zealand, the United Arab Emirates and the United States, but not so well for students in East Asian systems, as the PISA report writes:

While, on average across OECD countries, students with a growth mindset scored 32 points higher in reading than students with a fixed mindset, the difference in scores between the two groups of students was 22 points in Japan, 17 points in Korea and Macao (China) and 15 points in Chinese Taipei. (OECD, 2019, p. 203).

In the case of China (and Lebanon), promoting a growth mindset might even hurt their performance, if the negative relationship between growth mindset and PISA scores holds true.

Superficial and Convenient Interpretation

PISA has also a history of making superficial interpretations that feel convenient at the time. For example, Andreas Schleicher, in arguing against the charge that the PISA sample in Shanghai was flawed (constituting cheating), pointed to student self-reported data about what would make them succeed in learning as evidence of Shanghai’s success in 2013:

In many countries, students were quick to blame everyone but themselves: More than three-quarters of the students in France, an average performer on the PISA test, said the course material was simply too hard, two-thirds said the teacher did not get students interested in the material, and half said their teacher did not explain the concepts well or they were just unlucky. The results are very different for Shanghai. Students there believe they will succeed if they try hard and they trust their teachers to help them succeed. That tells us a lot about school education. And guess which of these two countries keeps improving and which is not? The fact that students in some countries consistently believe that achievement is mainly a product of hard work, rather than inherited intelligence [emphasis added], suggests that education and its social context can make a difference in instilling the values that foster success in education. (Schleicher, 2013).

It seemed that Chinese students did have a “growth mindset” as they believed their success did not come from inherited intelligence. Or perhaps, the Chinese students somehow did not connect intelligence to school success. Or perhaps the Chinese students believe that no matter what their intelligence was, they have to get good grades and test scores. Or perhaps the Chinese students do not care or are not allowed to care if their intelligence is fixed or malleable because they have to study hard regardless. Or perhaps the Chinese students did not think school success requires much intelligence. Or perhaps the Chinese students are too busy with schoolwork to figure out if they have growth or fixed mindset.

Whatever the reason or reasons, it is much more complex and localized than PISA portraits.

Bottom line:

Do not rush to teach growth mindset to every student in every country.

References:

Barber, M., & Mourshed, M. (2007). How the World’s Best-Performing School Systems Come out on Top. Retrieved from New York: https://www.mckinsey.com/industries/social-sector/our-insights/how-the-worlds-best-performing-school-systems-come-out-on-top

Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit Theories of Intelligence Predict Achievement Across an Adolescent Transition: A Longitudinal Study and an Intervention. Child Development, 78(1), 246 – 263.

Dweck, C. S. (2008). Mindset : the new psychology of success (Ballantine Books trade pbk. ed.). New York: Ballantine Books.

OECD. (2019). PISA 2018 Results (Volume III): What School Life Means for Students’ Lives. Retrieved from https://doi.org/10.1787/acd78851-en.

Rappleye, J., Komatsu, H., Uchida, Y., Krys, K., & Markus, H. (2019). ‘Better policies for better lives’?: constructive critique of the OECD’s (mis)measure of student well-being. Journal of Education Policy, 1-25. doi:10.1080/02680939.2019.1576923

Schleicher, A. (2013, December 10). Are the Chinese cheating in PISA or are we cheating ourselves? Retrieved from http://oecdeducationtoday.blogspot.com/2013/12/are-chinese-cheating-in-pisa-or-are-we.html

Zhao, Y. (2014). Who’s Afraid of the Big Bad Dragon: Why China has the Best (and Worst) Education System in the World. San Francisco: Jossey-Bass.

Zhao, Y. (2016). Who’s Afraid of PISA: The Fallacy of International Assessments of System Performance. In A. Harris & M. S. Jones (Eds.), Leading Futures (pp. 7-21). Thousand Oaks, CA: Sage.

Zhao, Y. (2018). What Works May Hurt: Side Effects in Education. New York: Teachers College Press.

Zhao, Y. (in press). Two Decades of Havoc: A Synthesis of Crticism Against PISA. Journal of Educational Change.