Since the inception of the SAT in 1926, the admission world has debated (1976, 2001, 2008, 2015, 2018, 2019) the impact of and validity of the SAT (and later the ACT, CLT, CCTST, etc) on the pool of applicants and enrolled students at a university. Recently, more and more colleges have been asking themselves should they diminish the role of testing in their admission process and declare a test optional admissions policy. This debate has heated up recently with the release of Measuring Success: Testing, Grades, and the Future of College Admissions and the announcement of the University of Chicago’s test optional policy causing many institutions to look inward at their use of test scores. However, the question of whether colleges and universities should stop using the SAT and ACT might just be the wrong question. It’s certainly the wrong question at a particular set of schools. While there are some schools where the additional predictive information (generally .02 to .1) from tests lends support to difficult admissions decisions, I think there are as many institutions that should be test optional but are holding on to the tests. In requiring these tests are these schools doing more harm than good to themselves? By holding on to these tests are these schools abdicating their duties to fairly evaluate all candidates and relinquishing that authority to the SAT and ACT?
Since the 1980s, Educational Testing Service (ETS), which dominated educational admission testing from 1940 – 1980, has been hemorrhaging product lines. In its heyday (SAT word) ETS was the Sauron to US education’s Middle Earth, providing admissions tests for the vast majority of professional certification programs and higher ed admissions. Their services ranged from teacher certification exams to the SAT, GRE, GMAT, MCAT, and LSAT. In the last decade or so, ETS business strategy has changed and the organization has begun to aggressively market their most popular remaining assessment product, the Graduate Record Exam (commonly known by its initialism – GRE), as “the One Test to Assess Them All.” This strategic market grab, while an interesting business strategy, raises significant questions about all admission tests. Specifically, the expansion of the GRE into fields beyond its design should force responsible test users to reevaluate long-held assumptions about what information is being gained by requiring the GRE (and all its brethren) and at what cost.
After announcing changes in September 2016 and then teasing us with 10 sample questions and an FAQ in January 2017, the DOE finally released two full sample tests in May of 2017. The sample tests were included in the 2017 – 2018 Specialized High School Handbook and I spent the month of June perusing, categorizing, and quantifying the questions contained therein. With that work done (well it was finished in June but I didn’t get motivated to blog until this morning .. thanks Stacey H), I’m here to share all that I know about the changes to the SHSAT for 2017.
As the new school year begins, I am anxiously awaiting (read: dreading) the forthcoming SAT and ACT annual reports and with them the inevitable exaggerations, hand-wringings, misinterpretations, and statistical paralogisms that will follow. The College Board’s Total Group Reports and ACT’s Condition of College and Career Readiness Reports (or Profile Reports) will not only spark the annual “sky-is-falling because district scores have dropped .005 points” responses but will also likely lead to an uptick in the “SAT/ACT scores show students not ready to succeed in college, career, life, liberty or the pursuit of happiness.”
I’ve already bellowed into the void about the sky-is-fallingness of it all, so this time my windmill is the College and Career “Readiness” Benchmarks. More specifically, the lazy language surrounding and incomplete interpretation of those benchmarks that may be doing harm to the most vulnerable students in our schools. If you think I might be being a bit hyperbolic consider this particularly egregious example of “journalism” (which I will not link to):
For the last few years, the NYC DOE has been under pressure to address the demographic imbalances at the Specialized High Schools. While considered some of the city’s (and even the nation’s) top schools, these schools have not reflected the diversity and demographics of the city as a whole in decades. The De Blasio administration took steps to increase the diversity of these schools and signaled that they would actually attempt to address the fact that only 11% of specialized schools are black or hispanic while approximately 70% of other city schools are. The administration took another step later in the fall when it paved the way for changes to the Specialized High School Admission Test (SHSAT) , the sole means of entry to the Specialized High Schools.
Are New York City’s teachers as smart as their students? John Sexton, the ex-president of New York University, thinks not. During a talk he gave on the future of American universities at the Library of Congress last week, he claimed that in the past five years, New York City public schools have been hiring “teachers that have lower SAT scores than the students you are graduating. That’s a ticket for failure, because you’re hiring from the bottom half of the existing class.”
In a year with a mis-timed SAT (leading to 2 sections not counting), cheating scandals, delayed PSAT results, incessant marketing of a practice tool as the solution to societal inequity and educational injustice, and fights between SAT and ACT, it’s easy to understand why the one of the few positive changes that came with the revised SAT would have been largely overlooked and unmentioned. This past January, College Board completely revamped the way it delivered scores to schools. The organization that owns and is responsible for designing the SAT not only revamped its test (for the first time since 2005) but also revamped its reporting portal for school counselors for the first time since the dark ages. The new school reporting portal changes should provide significant benefit to schools and teachers around the country. To put these changes in context let me point out that until very recently, the College Board sent dot-matrix printed labels with PSAT scores on them to high school counseling offices and additional data schools had to be purchased and was mailed on CD to schools. Yes you read that right. In the 21st century the multi-billion dollar educational institution was sending dot-matrix labels and mailing CDs.
Welcome to the internet era College Board
In the fall of 2015, College Board went entirely online with their (singular their) reporting portal and has been steadily improving it ever since! The portal currently allows users to review and export almost all the data a school counselor or teacher could want. The College Board SAT educator’s portal is what I want from every standardized test administered in a public school. Timely, up-to-date access to my students’ test scores and performance data. While there is certainly room for improvement, this portal is miles ahead of anything I’ve seen from major national standardized tests. The College Board has created a tool that allows counselors to look either at the macro level (how did students do overall this year vs last year) of student performance or the micro (how did Malala answer on question 7 in the reading section). This is all available online from the comfort of your cubical at work. No waiting for a disc to be mailed, no asking a student to print and bring in their score report. Timely, detailed, and accessible. College Board has, with this one portal, potentially made the SAT a useful tool for teachers and schools. Approximately a month after a test is administered a school would have back complete data on all students, ready to be shared with the academic team and ready to be used to improve instruction. This is pretty close to exactly what testing (at least test result reporting) should be.
But I ramble, let me do more showing and less telling.
Question Analysis Report
- Performance comparison – the report compares the schools performance on the test to the district, state, and nation. Allowing a school to easily identify possible indicators of holes in the curriculum.
- Links to questions – the portal provide direct links to the questions from the PSAT (always) and the SAT (for released forms of the SAT) so that educators can easily access and see the questions.
- Answer selection percentiles – knowing the percentiles rates at which each answer choice (when combined with the actual question) is selected lets educators gain insight into the types of mistakes made.
Also of note are links to state standards (I’m told that its actually linked to the specific state standards), categorization of questions (yeah it’s still CB quasi-meaningful categories), sortable tables, filterable data sets, and ability to make sub-groups of students. Check out the screen shot to the right of the report options and below of the Question Analysis Report. The biggest surprise from the portal for me was that with the release of May SAT scores, the actual questions from two of the three forms given in May were available in the portal. Historically, CB made the Question and Answer Service available for purchase ($18) only by those who had taken that test administrations (thus creating a healthy grey market for QAS booklets that advantages students with tutors or a lot of info about dodgy internet corners). Schools were never given easy access to the questions from released tests and even those students who ordered the QAS service waited 2 – 3 months to receive a copy of the questions. Now the questions and answers are relatively quickly available in the school to the educators who can do something with it! Even if this is only available 3 of the 7 ( or 10 if you count the three SAT School Day dates or 17 if count Sunday test dates also) test dates. This is a boon to schools and students.
Most of the other reports in the dropdown (2nd screenshot in post) are just a variation on the screenshot below. They provide different ways to look at student scores, allowing you to look at the scores (versus district, state, nation) or the benchmarks (which kids reached CB’s benchmarks), or by fee waivers used. Most of these reports are mildly interesting to schools and teachers but not particularly actionable in terms of improving student performance. The benchmark report is probably the most interesting of the other reports. While, I’m not convinced that College Board’s benchmarks tell enough of a story to be independently useful for a student, I think they are useful as a comparison point for a school, district, and maybe even teacher. Being able to see where your school underperforms or out-performs schools in your district, state, or the nation can be used to inform revision of curriculum (when paired with analysis of the questions that feed info those benchmarks) or refocus instruction.
(Update: I found a few more gems after originally posting this that I wanted to point out) A few other reports that are great for counselors:
- View/Print Student Score Report
- View/Print Student Essay
- View/Print Student score sends (there is a whole post on the process and costs of sends that might be cooking).
How to make the SAT portal a game changer
If College Board really wants to make this product a game-changer here are a few “simple” things they could do:
1. Find a way to make this available to community based organizations
Given the sad state of the caseload of most school counselors a decent portion of the work of supporting students for the SAT (and college access) falls to community based organizations. Currently CBOs have no way to access group reports on the students they work with.
2. Add content tags to questions
When I ran my own test prep company, our database allowed me to tag question with up to 30 different content areas (not just Harts who love Algebra but also exponents, variable, and manipulating equations) and thus make it sortable and searchable by a wide range of characteristics. Adding this feature to the portal would allow teachers to search/sort questions by content areas by terms they know better.
3. Create an “content performance report”
The portal needs an easy way for the test coordinator/school counselor/principal to get the information out of the portal and put it in front of the teachers and academic department heads.
Despite my objection to categorizations of the SAT as an unbiased predictive assessment of college success, this new portal goes a long way toward making the SAT a useful assessment of certain academic performance. For me any assessment is valuable only when an educator or student can relatively quickly following the test review performance on that test and take steps to improve that performance. With the changes to its reporting portal College Board has done a lot to make the SAT such an assessment, so I say to David, Aaron, Steven, Shameek, Farhad, Teirney, and the team at CB:
On Tuesday April 19th, I woke to find the following press release in my inbox.
Las Vegas, NV (April 18th, 2016) – ACT, (the other test maker) has decided to partner with Kaplan Test Prep to offer free standardized test prep, in a new effort to emphasize the ineffectuality of standardized tests. This new state of the art online testing program will be at least partially live (Take that KHAN ACADEMY with your pre-recorded doodles). Though it’s not entirely free to all students, Kaplan promises to provide lots of free test prep to low-income students to help dial down the reality that it is a for-profit company making bank on this new partnership. With this new program, Kaplan promises to utilize the same top-notch online portal used for classes at Kaplan University. (Ranked 137th Best Online College and ranked equally with the notable Oral Roberts University.)
This new program entitled: Kaplan Online Program Outreach for Underserved Tutorials (or simply KOPOUT) features lessons from seasoned Master Ninjas who no longer need silly things like textbooks to cover the material that isn’t showing up in high school this year.
“We know that helping kids help understand the help they need should not go unhelped,” says ACT COO, Kyle Ren. “After all, we think it’s beneficial to work with test prep companies, as they’ve made it their business to recognize the flaws we’ve created. It’s like, you know when like the Terminator came back and he was like, I’m not here to kill you like I’m here to help save your kid from the shiny new guy who wants to kill you. And he can like mold himself into anything at all, and that’s like, useful and stuff. So we’ve got the Terminator on our side. Or are we the Terminator? I don’t know. I guess one of us is the Terminator.”
KOPOUT also includes the following features:
- An online platform with prerecorded lessons over the backdrop of a 1980’s yearbook setting.
- Sciencey science for the Science section discussing the charts and graphs and other things ACT takes for college level Science.
- Real ACT questions which are totally different from the ones you’d find in the book they’re about to publish for more money.
- A social network of like-minded kids using the service as an excuse for learning instead of snap chatting nasty comments to their ex-girlfriend.