Shuffling the Deckchairs of Testing

I wrote this back in March but then … The Rona. I’m publishing it now since BPS just chose their vendor. I’ll likely dig into that vendor and the test first on twitter but eventually I’ll post something about it here.

After a very public breakup with test-maker Education Records Bureau (ERB), Boston Public Schools is seeking a “new test” that “accurately assesses a student’s knowledge of content they’re taught in class and has been rigorously reviewed to ensure it is free of bias.” This is disappointing since it shows that BPS seems to be set on continuing to ignore research and reiterating its faith in the cult of overtesting. 

Like many educational policy makers, BPS leaders seem unwilling to consider a world in which educational decisions are not highly influenced by the standardized tests that have dominated American education since the 1920s. The quest for a test that will somehow be free of the problems that plagued the ISEE seems naive and contrary to 100 years of research. In fact, in 1934, Carl Brigham, the SAT’s creator, wrote, “The test movement came to this country some twenty-five or thirty years ago accompanied by one of the most glorious fallacies in the history of science, namely that the test measured native intelligence purely and simply without regard to training or schooling. I hope nobody believes that now. The test scores very definitely are a composite including schooling, family background, familiarity with English, and everything else, relevant and irrelevant.”  

The notion that BPS will be able to solicit bids, verify appropriateness of the test, and organize logistics of a new test to replace in six months the broken system of testing that has existed for more than 25 years seems unlikely at best. The proposal to replace the ISEE by fall 2020 makes it almost impossible for any test to be created or adjusted to meet the needs of Boston schools. In fact, when contacted via email, Jinghua Liu, Chief Testing and Research Officer of Enrollment Management Association, ERB’s primary competitor, responded that “after examining the BPS requirement and framework, we are afraid that the alignment, especially on math, with the SSAT is weak.” She further added, “There are also a number of content specifications that are extremely focused on learning expectations that are not assessed at either the SSAT Middle Level nor the Upper Level. In some cases (e.g., box plots, interquartile range), our committee felt that they aren’t well known enough for the user population to know them and to know what to do with them. In other cases, topics like mean absolute deviation are assessed, which seems to be a topic implemented by common core, but which isn’t a topic that we assess at all.”

The problems BPS cited with the use of the ISEE and the statements made in issuing the RFP show the conflicting ideas that govern school districts’ relationships with standardized tests. On one hand BPS implies doubt in the relevance, fairness, and efficacy of the test they have used decades, while on the other hand the district demonstrates a dedication to the idea that a different standardized test can solve these problems. BPS seems to expect that the disparate outcomes along to racial and economic lines that are exacerbated by testing will be mitigated with a different test, despite all evidence to the contrary. BPS has only to look at other exam schools around the country to realize that despite each of these schools using different tests, the results are similar, when tests are emphasized low-income, African-American, and Latinx students are disadvantaged. 

	District	Test Used in Admission	Other Factors Considered	Percent Low income	Percent Black and Hispanic
Thomas Jefferson High School for Science & Technology	Alexandria, VA	Quant-Q, ACT Aspire Reading and Science	Yes	1.8% FRL	4.2%
Stuyvesant High School (2017 – 2018)	New York, New York	SHSAT	No	46%	4%
Walter Payton College Prep	Chicago Public Schools	NWEA MAP	Yes	29%	32%
Hunter College High School	New York, NY	HCHS Entrance Exam	Yes	9%	9%
Liberal Arts and Science Academy	Austin, TX	Cognitive Abilities Test	Yes	10%	23%
Maggie L. Walker Governor’s School for Government and International Studies	Richmond, VA	Unspecified	Yes
Oxford Academy	Cypress, CA	Unspecified	Yes
Academic Magnet High School (2016 – 2017)	Charleston, SC	MAP	Yes	1%	7%

Perhaps it’s time to consider that the problem might not be the specific test but instead the reliance on a test created outside of the school system that is highly susceptible to coaching and test prep. Perhaps it’s time to consider that the creation of admissions systems that ignores income will invariably advantage those with the means and opportunity to navigate or game that system. Perhaps it’s time to acknowledge that simple reliance on GPA and test scores give rise to grade inflation and test prep, which in turn gives advantages to private school students and wealthier students. 

These patterns were highlighted by a Brookings Institute report that shows economic and racial segregation of exam schools in several major cities.  As if this preponderance of evidence wasn’t enough to encourage BPS to make changes, a 2018 report from the Harvard Kennedy School advised BPS that using MCAS and GPA would help to address (though not completely solve) the problems BPS is looking to spend millions of dollars on a new test to address. 

If the evidence of the insufficiency and unnecessariness of test-focused admissions practices weren’t enough, BPS could look at the counter-examples provided by top schools that do not rely extensively on tests yet produce extraordinarily successful students, such as Basis Scottsdale and Advanced Technologies Academy. BPS might also look to test optional colleges as evidence that an educational institution can make good admissions decisions without relying on test scores. In fact the over-reliance on standardized tests has done little to improve education and much to increase income inequality. 

Maybe instead of reshuffling the deck chairs BPS should consider spending the money allocated for external testing and test prep into improving resources for under-funded schools. Let’s hope that those in control of these decisions give up their dogged dedication to a tool that research has demonstrated over and over to be of limited usefulness and one that fosters perverse educational habits and exacerbating inequity. Maybe the smarter and braver solution is to not to erect a barrier (an additional test) and then require families or the city to invest millions of dollars getting over that barrier (test prep).