A Pretense of Science and Objectivity: Data and Race to the Top
Education has a new god: data. It is believed to have the power to save American education and thus everything in education must be about data—collect more data about our children, evaluate teachers and administrators based on data, and reward and punish schools using data.
The US federal government has been dangling $4.35 billion to recruit worshipers through the Race to the Top (RTT) program, and possibly a lot more through the upcoming reauthorization of the Elementary and Secondary Education Act (ESEA). Already 40 states have somehow made the conversion, although only two (Delaware and Tennessee) were deemed to have developed a system powerful enough to generate more converts in schools and thus rewarded with a total of $600 million. Other states will have a second chance and many are working to rewrite their pledges.
Both Delaware and Tennessee did an excellent job pledging to build an education system around data. The Delaware application for RTT lists three strengths that “few states can match:”
- Delaware’s state-of-the-art system (emphasis original) captures longitudinal information about both students and teachers, and links them together. Today, the State can quickly analyze the performance of any teacher’s students over time, can track how graduates perform in college, and can link teachers to teacher preparation programs, providing rich opportunities to use data to drive performance at the system, school, and classroom levels.
- Delaware’s rigorous statewide educator evaluation system is based on the most respected standards for teaching and leading. The system provides a multi-measure assessment of performance that incorporates student growth as one of five components…an educator can only be rated effective if they demonstrate satisfactory levels of student growth.
- Delaware’s newly-defined regulatory framework for school turnaround gives the State the authority to intervene directly in failing schools and requires schools to demonstrate results b achieving AYP within two years. (p. A4-A5)
All three are about data. With the RTT money, Delaware promises to become even more faithful—moving from data-rich to data-driven (watch the panel interview on Youtube). The State will “administer up to three formative and summative assessments per year per student in core subjects” (p. B-10), provide “immediate information that a teacher will use” (p. B-10), “provide data coaches to aid in the use of assessment data,” make tenure and licensure decisions for new teachers based on student performance (D-8), and include in teacher and administrator annual evaluations (p. D-16). With all these plans, Delaware will be “poised to promote data-driven instruction across all schools” (p. B-23). (http://www2.ed.gov/programs/racetothetop/phase1-applications/delaware.pdf)
Tennessee embraces data as enthusiastically as Delaware. Its RTT application begins with a poetic piece that asks the reviewers to:
Imagine, for a moment, a new day that is coming for Tennessee’s children and families, teachers and principals, and the State’s economic future….It is a day when middle school students will have the benefit of formative assessments, with immediate results so their teachers can measure their progress and make adjustments in time for the annual test. And then principals will use that student achievement data as a significant part of more rigorous teacher evaluations, and work with their teachers to help them succeed (p. 10).
I could go on, but you could read the applications and/or watch their presentations/panel Q&A on youTube (quite fun, actually).
I am not at all opposed to data but unfortunately data are not god, but rather man-made artifacts. Therefore they cannot be perfect. There are good data and there are bad data. A good example of bad data and bad decision driven by bad data is in fact the Race to the Top evaluation process.
William Peterson and Richard Rothstein published an analysis of the results of RTT Phase 1 review as a briefing paper of Economic Policy Institute this week. They conclude that “In short, the Race to the Top 500-point rating system presents a patina of scientific objectivity, but in truth masks a subjective and somewhat random process.” (p. 8)
They first raise some basic questions about this complex rating system invented by the Department of Education:
What was involved in the decision to use a scale of 500 total points rather than, say, 10, or 100, or 1,000? Is there scientific support for the “State Success Factors” being 90.6% as important as the “Great Teachers and Leaders” factor? Should the “Great Teachers” maximum points be 140, or maybe 163, instead of 138? And are there missing factors, such as “Developing Techniques to Promote Creativity,” or others?
They then conducted some experiments with the weights assigned to each category. By shifting the weights slightly (by increasing only 3% to four sub categories), they found:
If the weights were increased in this manner (and the weights of the other 25 indicators reduced by roughly half a point,5 so the total would remain 100%), Tennessee would no longer have won the competition—Georgia would have won instead. The official result can’t be justified by a claim that Tennessee is more reform-minded than Georgia. (p. 4)
They also found that Pennsylvania’s excellent proposal to focus on early childhood education, a priority of President Obama, was not recognized:
Pennsylvania, in short, has now been told by the Department of Education that if it wants to compete successfully in the next round of RTT competition, then the state should downplay its focus on early childhood and science education, and put its efforts instead into categories that get more points but which, in fact, have a much weaker research base. (p. 5)
Moreover, the rating system punished Massachusetts for its intention to take a bit more time to think whether it wants to lower its standards in order to be aligned with the Common Core standards.
In essence, the analysis shows that although “the process (or RTT reviewing) was presented as objective and scientific. However, further examination suggests that the selection of Delaware and Tennessee was subjective and arbitrary, more a matter of bias or chance than a result of these states’ superior compliance with reform policies.” (p. 1-2)
Read the complete paper: Let’s do the numbers: Department of Education’s “Race to the Top” Program Offers Only a Muddled Path to the Finish Line
If the RTT reviewing process—an elaborate apparatus developed by a data-driven Secretary of Education and Department of Education to evaluate efforts by 40 states that involved governors, state legislators, state school boards, superintendents(commissioners), school administrators, business partners, and teacher unions for their chances to receive many billions of dollars—can be so flawed, how much faith can we really have in the data that will be collected about students, used to determine their learning process, to evaluate their teachers and principals, and judge their schools?
Even if we could collect “good data” about students, there are still some questions we must ask:
- What is the cost, both financial and human, of all this data collection? Can the money for buying/developing/administering tests be used for truly educational activities? Can teachers and students better spend their time on actual teaching and learning rather than reporting and studying for tests?
- What are the implications of collecting so much data about individual student, from kindergarten to college or even post-graduate level? When we are so concerned about the misuse of health data, should we also consider about academic and school behavior data of our children?
- Will data-driven instruction truly work? This seems a very mechanical and arrogant—as if we knew exactly how children learn and develop and could prescribe their course of development as well as their purpose for life. But we know children are living organisms, with intentions, agency, emotions, interest, dreams, and hopes. They cannot and should not be deemed needing intervention just because they did do well to meet the prescribed “grade-level” expectations.
(4 votes, average: 5.00 out of 5)
Loading...
The over-emphasis on data in education is a reflection of the power of the bureaucracy in education. All decisions are to be data-driven. It is part of the rise of edu-babble (See http://educononline.com/2009/06/11/edu-babble-and-the-confusion-in-education/) that serves not to clarify but to confuse. The use of data in such fashion as you have indicated is also a reflection of the continuing dehumanizing of the educational process. Welcome to Singapore.
This is a really good post. It gets us to focus onthe validity of data and the fact that it can never really provide answers but only ever really raise questions. The value of any question lies in the reasoning behind its framing and the “values” of the framers. There is an obsession with data as scientific fact but even in science throughout the years there have been many questions about data and the fixing of data to present results that the enquirer wants…. the Governments in the U.S. and Britain (where I live) want to use data as a means to produce the kind of “product” student that they think the economy needs… but in the end they will produce the undereducated individuals taught by demoralised teachers that we have seen in both countries for too many years.
[…] Next, Jeffbinc, shows the consequence of the naive faith in numbers and evaluation that drives Obama’s education model. He quotes education expert Yong Zhao that the new “race to the top” makes a god out of data. […]
Shame on these educators at the top for pushing data. I believe they have lost their vision regarding education. SHAME ON THEM and their followers. They need to visit (and stay for a while) a classroom!!!
[…] Take a look at what Yong Zhao, whose TED lecture I recommended a while back, has to say about data and the new program, Race to the Top. […]
The problem with worshiping false gods is that they are always so seductive and their worshipers become emotional zealots who continue their worship practices without regard to reason or common sense.
Excessive use of data creates many practical difficulties beyond all of the issues discussed in Dr. Zhao’s comments. The one most often seen is that achieving improved data scores becomes the GOAL of education. People begin to forget that data is a measurement of progress, not the goal of it. In a perfect (or sane) world, high test scores and other data would be the RESULT of a quality educational program, not the GOAL of it.
Another problem with data is that so few practitioners (principals, teachers, district office administrators, etc) actually understand the data well enough to use it effectively. It is common to see practitioners use the test scores from the most recent year to guide their decision-making rather than looking at trend-data and other more useful indicators.
Over-reliance on data also presents a problem at the local level. Probably all of us have had the experience of going to a doctor who runs lots of tests and “can find nothing wrong,” while we continue to feel ill or have pain. Schools can achieve high test scores as they do in Texas and still not educate children.
Also, data is easily garnered about “curricular content” but not about processes like thinking skills or applications of learning. The grammar and language mechanics of an e.e.cummings poem would not pass any test of Language Arts, but I still believe he produces good poetry–in spite of what the data says.
Another great article about the misuse of data is this one from Educational Leadership, May 2010 “What the Research Says: Using Value-Added Measures to Evaluate Teachers.”
http://www.ascd.org/publications/educational_leadership/may10/vol67/num08/Using_Value-Added_Measures_to_Evaluate_Teachers.aspx
It does a nice job of reviewing what the data actually says about the validity of using test data to measure teacher effectiveness. The real science:
“The complexity and uncertainty of measuring student achievement growth and deciding how much responsibility for gains to attribute to the teacher argue against using such measures for high-stakes decisions about individuals.”
I find it interesting that so many people fear the word data. And in today’s politically driven educational world it is no wonder. As I read this post, what I guess I want to comment on is while I agree there is bad data, and worse, the misuse of data on a regular basis. We should be cautious about demonizing all data collection.
I see data as one of a set of tools for the educator to use to guide and revise instruction. With out collecting data in the form of informal and formal assessments, we can often assume too much based on simply observation. Revisions in how we teach can be discerned from the data we collect. But, it is that combined with student feedback, teacher reflection on lesson presentation, formal and informal assessments along with standardize tests scores that give us the information we need to make informed decisions. Not one of those alone can do it. And many times as any teachers will attest, they often are at odds with one another. How many of you have taught what you felt was the best unit of your life only to have students fail on the assessment. So what was the problem, the lesson or the assessment? If you were enthusiastic, students would probably agree it was a great lesson. So what went wrong? Without some data, and evaluation of several types you may never know.
I think the major point about using data as our “God” is that we are relying on a single assessment point to make huge decisions about curriculum and instruction. It in my humble opinion is merely one of a set of tools. Considering that the majority of educational reform in the US in my opinion is being driven by business, of course they want a single bottom line number. This is the real danger that I think Professor Zao is pointing out. Our observations of student learning are just a big a piece of the puzzle as standardized tests especially when those results are used inappropriately.
Since this is a human process of humans interacting with one another we need to know a lot of different information. I like Bernhardt’s four types of data: Demographic, Student Learning, Perception and School processes. The information gathered from these four areas can when used with common sense work together to tell us a lot. I agree we cannot make data God, neither too can we demonize it. Recognize it for what it is, one of a series of tools to help diagnose and adjust instruction as needed. Lastly we need to give educators time to use and training to actually make sense of all the types of data they collect correctly.
[…] of the pieces that caught my eye was A Pretense of Science and Objectivity: Data and Race to the Top, where he does not criticize data, but our worship of it’s collection and use as what’s […]
Thank you Dr. Zhao for this post. There are many unanswered questions about the value and utility of the types of data that we collect. As one of the other commenters said, I don’t think all data collection is bad but I do think we need to use this information to support student learning and not to punish teachers and schools.
I’ve been thinking a lot about our fascination with our ability to accumulate data. I’ve started to think there’s a continuum which spans a number of key points. They move from left to right or right to left from “data ignorant” to “data aware” to “data informed” (where I want to be) to “data driven’ to “data obsessed”.
i recently had a conversation with some colleagues who were working through the aggregated data for the school year who described themselves as being “data blind” from looking too long and too intensely at their spreadsheets and computer screens. Although unintentional I think that this is probably the other end of my continuum — data ignorant all the way to data blind. What an accomplishment.
I think we need to know our students as learners and as people and value our teachers for the incredibly challenging work that they do to help these learners find their life trajectories.
What are the implications? Among other things, I could bust any school in any of the districts in a 10-county radius for violations of the Buckley Act. Heck, I might be able to do that on-line for some of them.
As with health data, the stuff really worth knowing about someone doesn’t show in the statistics: Whether the student is trustworthy, or courteous, obedient when necessary, or able to function under stress.
Seriously, the Buckley Amendment should cover the privacy stuff. It is troubling to note that if you do a Google search for “Buckley Amendment,” nothing shows from the U.S. Department of Education until the 30th listing, and nothing of the first 29 is focused on elementary and/or secondary education. It’s not taken seriously, it appears to me.
Then there is the question about what it says for education generally. When I consulted with large corporations on quality, we often noted that organizations get what they measure. We measure test scores, not learning (there is some great debate about whether we can measure learning). We measure the ability to bubble-guess rather than maturity and skill in writing and reading.
What does it look like on the ground? Local schools here cancel recess and P.E. to add in time to study for the state tests. Arts programs wave at us from the history books. Much of the stuff that would otherwise mark an educated person is dropped from curriculum, and never taught.
My juniors, according to them and corroborated by what I see in the classroom, have never read “Paul Revere’s Ride,” nor the “Concord Hymn.” They might guess who Revere was on a multiple choice, or they might remember the phrase “the shot heard ’round the world, because those things are on the review sheets, and mentioned in the state standards. But they’ve never read, never heard, never heard about, the poems.
The data show progress, but that’s not education.
Regarding the importance of data in education: “Not everything that counts can be counted, and not everything that can be counted counts.” (Sign hanging in Einstein’s office at Princeton)
— Art Lader
Feel free to comment:
The views expressed on this site are entirely my own. They do not represent my employer or any other organization/institution. All comments are subject to approval.Archive
07.29.24 AI Has Little to Contribute to Traditional Education: Problems and Possibilities
03.17.24 Focused: Understanding, Negotiating, and Maximizing Your Influence as a School Leader
02.25.24 What Happened to Global Competence?
08.05.23 Rethinking the time spent at school: Could flexibility improve engagement and performance for students and teachers?
01.17.23 Introduction to Improbable Probabilities: The Unlikely Journey of Yong Zhao
01.05.23 How Not to Kill Creativity?
08.19.22 Preface to Improbable Probabilities: The Unlikely Journey of Yong Zhao
02.05.22 Introduction to New Book: Learning for Uncertainty: Teaching Students How to Thrive in a Rapidly Evolving World
09.25.21 Side effects in education: Taxonomy of educational outcomes
07.13.21 Introduction to My New Book: Learners without Borders
03.09.21 New article: Build back better: Avoid the learning loss trap
02.18.21 New article: The changes we need: Education post COVID-19
09.15.20 Watch Ep4 Creativity in Crisis: How well is creativity understood? A Conversation with Barb Kerr, Haiying Long, Ron Beghetto, & Yong Zhao
08.15.20 Can Creativity be Taught? Ep 3 of Creativity in Crisis on August 28th 3:00-4:00pm Pacific Time
07.13.20 Speak a Different Language: Reimagine the Grammar of Schooling
Tag Cloud
Accountability achievement gap CCSSO China/Chinese Commissioner Common Core Standards education Educational Policy Education Reforms national standards New York NGA Singapore standardized testing Standards student performance
WP Cumulus Flash tag cloud by Roy Tanck and Luke Morton requires Flash Player 9 or better.
Silverliningforlearning
Most Commented
Most Viewed
Views expressed on this site are entirely personal. They do not necessarily represent the official positions or views of my employer
Powered by WordPress | Log in | Entries (RSS) | Comments (RSS) | Arthemia theme by Michael Hutagalung