How Does PISA Put the World at Risk: Part 1 Part 2 Part 3

Part 4

China’s poorest beat our best pupils—The Telegraph (UK), 2-17-2014
Children of Shanghai cleaners better at math than kids of Israeli lawyers—Haaretz (Israel), 2-19-2014
Cleaners’ children in China beat kids of US, UK professionals at maths: study—NDTV (India), 2-18-2014
Children of Chinese janitors outscore wealthy Canadians in global exams—The Globe and Mail (Canada), 2-19- 2014

These are some of the most recent sensational headlines generated by PISA with a 4-page report entitled Do parents’ occupations have an impact on student performance released in February 2014. These headlines exemplify the secret of PISA’s great success as a masterful illusionist: effective misdirection of attention by exploiting human instinct for competition.

From the start, the entire PISA enterprise has been designed to capitalize on the intense nationalistic concern for global competitiveness by inducing strong emotional responses from the unsuspecting public, gullible politicians, and sensation-seeking media. Virtually all PISA products, particularly its signature product—the league tables, are intended to show winners and losers, in not only educational policies and practices of the past, but more important, in capacity for global competition in the future.  While this approach has made PISA an extremely successful global enterprise, it has misled the world down a path of self-destruction, resulting in irrational policies and practices that are more likely to squander precious resources and opportunities than enhancing capacity for future prosperity.

Take the report on the relationship between parents’ occupations and student achievement as an example, instead of highlighting its main finding—the well-known fact that family background has a strong impact on student achievement—to call attention to the issue of social-economic inequity in the world, the report chose to dramatize the finding that students of parents in “elementary occupations” in Shanghai and Singapore outperform children of professionals in the U.K. and U.S. in mathematics. In so doing, the report commits a basic logic fallacy called misleading vividness: “a small number of dramatic and vivid events are taken to outweigh a significant amount of statistical evidence.”[1] The result is the illogical and incoherent “bottom line” given by PISA:[2]

While there is a strong relationship between parents’ occupations and student performance, the fact that students in some education systems, regardless of what their parents do for a living, outperform children of professionals in other countries shows that it is possible to provide children of factory workers the same high-quality education opportunities that children of lawyers and doctors enjoy.

Unlike what the report claims, the fact that “students in some education systems, regardless of what their parents do for a living, outperform children of professionals in other countries” cannot be used to show “that it is possible to provide children of factory workers the same high-quality education opportunities that children of lawyers and doctors enjoy.” It only shows that the children of factory workers in some education systems received higher PISA scores than children of doctors and lawyers in other systems. Since higher PISA scores do not necessarily mean higher quality education opportunities, the causal link cannot be established. More important, even such a link were empirically verified, the fact does not say anything about whether children of factor workers receive the same high quality education as their peers in the same education system, belying the conclusion that “it is possible to provide children of factory workers the same high-quality education opportunities that children of lawyers and doctors enjoy.”

The data actually show the contrary. In both China and Singapore, children of manual labor workers scored way below children of professionals and managers. The gap in PISA scores between students whose parents in professional occupations and those in manual occupations is actually bigger in Shanghai than in the U.K. and U.S.. The gap in math scores between children of professionals and children of agricultural workers is well over 100 points in Shanghai but below 100 points in the U.K. and U.S., according to PISA’s own analysis[3]. The same is true for Singapore.

Worse yet, the data are not even reliable.

Over a quarter of students in Shanghai were excluded

Tom Loveless of Brookings Institute put forth a powerful challenge to the Shanghai sample, questioning the inclusion, or rather exclusion of children of migrant workers in the PISA sample. PISA, after initial denial, finally admitted that more than a quarter of Shanghai students were not covered by PISA. This is much higher than what PISA’s sampling standards permit, which is “up to a total of 5% of the relevant population.” As a result, Shanghai should be excluded from any analysis, as it did to several other countries such as Austria for sampling errors.

So many managers, so few cleaners

As a result of the sampling problem, the Shanghai sample is incredibly distorted. The distribution of parents’ professions of students who participated in PISA in no way represents the reality. According to data released by PISA, a total of 5177 students in Shanghai took the test. A total of 9031 parental jobs were reported, indicating that most students have two working parents. What immediately stands out is the contrast between the percentages of parents who are “managers” and parents who are in “elementary occupations:” 16% vs. 11% (Figure 1). In other words, more students have parents working in “manager” jobs than holding “elementary occupations” such as cleaners. By the way, only 165 out of the 9013 parents in Shanghai are cleaners and helpers. That is about 80 out of the over 5,000 students who took the PISA test.

If the sample reflects the reality, this would mean more people in Shanghai hold “manager” jobs than “elementary occupations” or managers produce more children than those in elementary occupations. Both are untrue.

According to China’s census data, managers only made up about 5% of all jobs in 2010 in Shanghai (Figure 2). The Census data did not use the same categorizations as PISA so did not have the “elementary occupation” group. But over 60% of people worked in manual labor jobs such as services/sales, plant/machine operators and drivers, and agricultural workers. Adding the 13% clerical workers, about 80% of people in Shanghai were in jobs not with high social and economical status. This group is significantly underrepresented in the PISA sample, which included a lot more students whose parents hold jobs with high social economic status.

Figure 1: Distribution of PISA Parents’ Occupations: Shanghai (Source: PISA)

Figure 2: Distribution of Occupations in Shanghai (Source: China 6th Census)

It is unlikely that managers and professionals produce more children either because of China’s one-child policy. If anything, people in manual labor jobs may be more likely to have more than one child because they could pay a fine to get away with it, while managers and professionals would lose their jobs for having an additional child.

Distorted samples produce distorted results. Another country where students with parents in “elementary occupations” beat children of professionals in the U.S. and the U.K., Singapore, also has a high proportion of managers, professionals, and technicians (48%) and low percentage (18%) in “elementary occupations” (Figure 3). The U.S. and U.K. samples have 30% managers and professionals and 20% (UK) and 24% (US) “elementary occupation” holders (Figures 4 & 5).

Figure 3. Distribution of PISA Parents’ Occupations: Singapore (Source: PISA)

Figure 4. Distribution of PISA Parents’ Occupations: U.K. (Source: PISA)

Figure 5. Distribution of PISA Parents’ Occupations: U.S. (Source: PISA)

The smart children of some cleaners

The small number of children of manual laborers included in the PISA sample are either lucky or good at taking tests. Given PISA’s two-staged stratified sampling design, which first selects 150 schools randomly from “a comprehensive national [in the case of Shanghai, municipal] list all PISA-eligible schools”[4] and then randomly selects 35 students, the small proportion of students with parents in manual labor occupations and the large percentage of managers suggest three possibilities. First, the sampled schools have a small number of students with parents in manual labor occupations and large number in managerial jobs, assuming all sampled schools have similar student population in terms of parental occupations. Second, the sampled schools included more schools that primarily serve children of managers, professionals, and technicians than schools with a high concentration of children with parents in manual labor occupations, assuming that schools serve different types of children in terms of their family background. The third possibility is a combination of both possibilities, which is perhaps what happened.

The PISA data show that the 11% of children whose parents with “elementary occupations” come from 152 of the 155 sampled schools and the 1% of students whose parents with agricultural jobs are from 44 schools. This means children of elementary occupation parents are more integrated in schools with children whose parents are professionals and managers, hence likely better schools. They could be in these schools in three ways. First, they happen to reside in the district (very likely). Second, their parents have the wealth or social connections to enroll them (quite unlikely). Third, they were very good at taking tests and had been recruited by the schools (quite possible). The third group would be the one to score very well on PISA and may raise the average scores of this category of students. It is of course, also possible, that because these schools are so excellent that they all got great education. But the data seem to show otherwise. The variation in math scores among the children of parents with “elementary occupations” is much bigger than those with professional and manager parents[5]. In other words, a portion of the children of parents with elementary occupations may have been selected for their academic achievement in the first place and thus boosting the PISA scores, while the others remain undereducated, even in the Chinese way.

So what if they scored higher

Even if PISA had done everything properly and indeed children of factory workers in Shanghai scored better than children of lawyers in the U.K. and the U.S., it does not necessarily mean they are better educated or prepared for the modern society, considering the limitation of PISA test scores as I discussed in Part 3 of this series. It could mean something entirely different: while PISA scores can be achieved with little resources and intense repetition of narrowly defined, uniformly prescribed content and skills, what truly matters—talent diversity, creativity, and entrepreneurialism—cannot. The multiplication table can be learned with a piece of paper, but it would be difficult to force anyone to play the piano without a piano. Everyone can be forced to memorize the Hamlet, but it is unlikely to force anyone to invent the iPhone.

As I wrote in The Difference between a $10,000 Education and a $10 Education three years ago:

…America’s rich kids’ poor performance in math and science and relatively good performance in reading is at least partially the result of choice—choice made by students, parents, schools, and communities. Rightly or wrongly, Americans may be more likely than others to believe that the academic abilities measured by PISA is not the only path to success in life. American schools also create more choices through a broader curriculum. America also has a more diverse set of higher education institutions that, unlike colleges in other countries, use a more diverse set of indicators beside test scores to admit students. There are also many Americans who believe they can succeed without going to college. In America there are also very successful entrepreneurs, entertainers, athletes, and other celebrities who did not necessarily have a great academic background, which can inspire many to pursue non-academic enterprises. And until very recently, there are no centralized national curriculum,standards, and high-stakes testing to push American schools, teachers, and students to narrow the options to what PISA measures.

…

The availability of choices in America is an envy of many countries. Having the option to choose different paths is not cheap or easily available. It is an advantage, a privilege, not a disadvantage or liability. A $10,000 education can and should buy much more than test scores while a $10 education can only buy test scores. Why do we want to turn a $10,000 education into one that can be achieved with $10?

The bottom line: Until OECD-PISA became the only employer in the world with PISA scores as the only qualification, I would not suggest lawyers and doctors in the U.S., U.K., or any nation to replace your children’s activities in music, arts, sports, dancing, debates, and field trips with math tutoring. For the same reason, it is not time yet for schools in developed countries to close your swimming pools, burn your musical instruments, end museums visits, or fire your art teachers.

Part 1: Romanticizing Misery

Part 2: Glorifying Educational Authoritarianism

Part 3: Illusory Models of Excellence

Adapted from my book Who’s Afraid of the Big Bad Dragon: Why China has the Best (and worst) Education in the World.

[2] http://www.oecd.org/pisa/pisaproducts/pisainfocus/PISA-in-Focus-N36-(eng)-FINAL.pdf

[3] http://beta.icm.edu.pl/PISAoccupations2012/OccupationsPISA2012.pdf

[4] http://www.oecd.org/pisa/pisaproducts/50036771.pdf, p. 58.

[5] http://beta.icm.edu.pl/PISAoccupations2012/OccupationsPISA2012.pdf