Are New York City’s teachers as smart as their students? John Sexton, the ex-president of New York University, thinks not. During a talk he gave on the future of American universities at the Library of Congress last week, he claimed that in the past five years, New York City public schools have been hiring “teachers that have lower SAT scores than the students you are graduating. That’s a ticket for failure, because you’re hiring from the bottom half of the existing class.”
Earlier this August, the College Board released raw score to scaled score conversion charts for the redesigned SAT, giving the world the final piece of the puzzle to the structure, format, and scoring of this, the newest version of the SAT. With this last nugget of information, we in the test prep world can begin to comprehensively understand how (whether) the redesign will impact student performance. If you, like me, have been following the trail of breadcrumbs released by the College Board since the March 5th of 2014 announcement, finally having a conversion chart is like opening your last gift on Christmas morning (and yeah I mean both that feeling of disappointment when you get a sweater or socks and that feeling of excitement when you get your brand new Atari).
So with the release of the conversion chart (ostensibly after exhaustive beta testing, research, calibrating, and equating by the College Board), we finally have official word on how raw scores will be converted to scaled scores and how many questions you’ll need to answer correctly in order to get the score you want. Let’s look at what we know so far and how the new SAT and old SAT compare. Specifically, let’s investigate how “difficult” it is to get a 500 math score on either test. Why 500? Because 500 is the approximate median score on the current SAT and very likely the median score on the redesigned SAT. Additionally 500 is the much ballyhooed college readiness benchmark which College Board seems hell bent on trying to convince us there is causation with success in college. Anyway let’s dive right in.
To get a 500 on the redesigned SAT a test-taker will have to achieve a raw score of 24 out of a maximum possible 58 raw score points. Since the new SAT uses a simple scoring system awarding 1 raw point for each question correctly answer and with no wrong answer penalty, getting a raw score of 24 simply means answering 24 questions correctly (or answering fewer than 24 and guessing the remaining questions correctly, but in this comparison we won’t factor in random guessing and luck). So on the redesigned test to get a 500, the likely median score, a test-taker will have to answer correctly approximately 41% of the questions.
Conversely on the currently, to get a 500 a test-taker has to achieve a raw score of 25 (out of the maximum possible raw score of 54). This means that a test-taker will have to a get between 46% and 57% of the questions correct. Because the current SAT takes off a quarter point for each incorrect response the scoring is more complex and there are 23 different ways a test-taker could get a 25.
The verdict is…
When you take into account the greater accuracy level required and the complexity of the scoring system, getting an average score on the current form of the SAT seems to be harder than it will be on the redesigned SAT. So when we look at the scoring system one has to say that the new SAT is going to make it easier for a student to get a score that indicates “college readiness.” Now, if only the content is easier on the redesigned test as well… but that’s another vacation’s writing entirely.
Over the last few days, I’ve been texted, tweeted, gchatted, emailed and called about the release of College Board’s/Khan Academy’s SAT prep resources. I’ve been forwarded article, after article, after article, about the playing field leveling that College Board is touting its partnership with Khan Academy will bring. I’ve been asked for my opinion and thoughts on Khan’s resources and the implications for my job and industry. So here it is, my unfiltered (mostly) thoughts on Khan Academy “Official SAT Practice.”
The partnering of Khan Academy and College Board is certainly a significant step for low income students. The key benefits of Khan’s SAT prep are:
- Greater opportunity and access for free – This relationship creates the opportunity for low income students to have reliable free practice, which until now has not been easily found. Students, parents, teachers, and counselors now know immediately where they can send a student to find practice tools for the SAT.
- High quality practice questions – Khan’s practice questions will be high quality because their relationship with the College Board will give KA a resource to verify that accuracy and appropriateness of their questions. Prior to this relationship the validity of free online SAT practice materials was questionable at best. (I’ve seen some terrible practice SAT materials both in stores and online, in fact most of the free SAT resources online are terrible.)
- Ease of use and access – Setting up and using KA’s site is relatively painless and in this day and age most students have probably already accessed it at some point. This means by adding SAT practice tools, they are simply improving an already useful tool. This is great for students.
- Integration with College Board results - In the fall there are plans to integrate further with College Board test results. PSAT and SAT test-takers will be able to add their test results to their KA accounts and get analysis and feedback. This again is great since it allows a one-stop shopping for information and analysis.
- Boys and Girls Clubs of America - The most interesting and least clear part of the College Board’s venture into test prep is the partnership not with Khan Academy but with the Boys and Girls Club. Reports are spotty but have indicated everything from College Board providing support setting up computer labs to College Board putting tutors in the Boys and Girls Clubs to provide actual teaching. If there is large scale free instruction supported by the College Board that will be truly interesting and helpful for low income students.
- Ongoing improvement - It seems that KA has an active team of professionals working to improve the product. This is amazing and bodes well because I’m anticipating that the College Board hasn’t yet finished tinkering with the SAT (and won’t finish until May 2016).
So while clearly there are great potential benefits to the advent of KA’s SAT tools, it’s also important to be aware of the limitations. Most of the articles I’ve seen about KA have ignored completely or paid scant attention to the potential problems with KA. These articles are touting Khan as the grand equalizer of economically and racially aligned score discrepancies on the SAT. It’s not. Khan is a tool. It’s a nice, well-designed free tool. And like any tool it will only be as good as those using it. I don’t object to the existence of KA SAT tools (in fact I’m excited by them), my concern is about the impact of touting it as a solution to inequalities. Let’s explore some of the key limitations with KA:
- Access is not effectiveness – Khan provides OPPORTUNITY to practice. It’s ACCESS to materials. But just because you are provided access and opportunity that does not mean it will be used and if it’s not used then no matter how good it is there will be no effect or leveling of the playing field. One of my concerns about Khan’s effectiveness is about access (since there are lots of studies saying low income students do not have the same internet access or solely access the web via mobile).
- Access is not engagement- Another of my concerns is engagement. Logging on to KA periodically when you have a sticky math problem in homework is very different from the consistent practice generally necessary to improve SAT scores. Will students be engaged enough to use the site? Historically, College Board’s prep tools have only been mildly used (I’ve been told by districts that usage of College Board’s SAT Online Course which comes free with most SAT School Day contracts is less than 10% as is use of the My College Quickstart site that is included for every PSAT test taker). Are these videos enough to keep student engaged?
- Free access is free for everyone – No matter how great Khan is at providing resources for low income students, high income students will also be able to access those resources to supplement their high priced tutoring programs. Any claims that this tool will minimize the score differentials overlooks that KA tools are available to all regardless of how much the family makes.
- Academic preparation is not test preparation - There is a big difference between academic learning and preparing specifically for a test. College Board and Khan provide more of the academic learning (they’ve actually said so). They are focused on more academic approaches. Here is an interesting comparison by Stacey Howe-Lott of Stellar Scores on how a test prep person might do a question vs how an academic might do the same question.
- Testing is not the same – All the great practice in the world is generally not sufficient to replicate the experience of take a proctored exam in a crowded room with other kids sniffling and tapping and stressing. Khan will never be able to truly simulate the experience of taking the test.
So what’s the upshot of all of this?
While Khan is shaping up to be a great resource it’s important to not get too enamored with the potential of the shiny new toy. Khan will help those who have had no access to quality free resources, but it will probably not level a playing field that is slanted at every level of education starting in utero and culminating in the workplace. Additionally Khan has been around for years delivering lessons for everything from algebra to physics and yet somehow the teaching industry has not been disrupted, it’s unlikely this will upend the test prep industry.
The keys for taking advantage of Khan will be to start using it early and over a sustained period of time to build academic skills and gain comfort with the material on the SAT. If you don’t make the gains you want or have very little time, then it might make sense to look into actual test preparation options.
What’s your thoughts ? What did I miss? What articles or research do I need to read or link to? Please put it in the comments!
A couple of articles that get it right
As we approach the launch of the latest iteration of the SAT, more and more students, counselors, and parents are becoming worried about the “new SAT” and how these changes are going to impact scores and college admissions. These concerns have ranged from reasonable (is there some different that is unlike other tests that we know of) to illogical (will the new SAT test the “new math” and thus unknown and unknowable) to irrational (will colleges not accept the new SAT and taking it keep my child from the college of their dreams).
In hopes alleviating some of the fear and panic around this test I’ve put together this handy list of who should and who shouldn’t be concerned about the upcoming changes to the SAT. And to be clear, I’m not saying that the SAT doesn’t matter and won’t continue to matter, because it will. I’m advocating thinking carefully about how the fact that there are changes to the content of the SAT will have limited impact on most of the people who are concerned about those changes.
Let’s start with those who should ignore the “changes” hype (and just pay attention to preparing well for the test you choose to take… since you still have a choice between SAT and ACT).
- Classes of 2018 and onward
Changes only matter if you have prior context or experience. Having very likely never taken any version of the SAT or PSAT, the classes of 2018 and after should completely ignore all the talk of the changes to the SAT. The version of the SAT they take will be the first and likely only version of the SAT they have ever seen. They should prepare for it much like students have been preparing for the SAT for years. These students should currently be planning their college application timelines and determining when they are going to take exams and focusing on the academic skills they are learning in school that will have lasting value. Any discussion of the old SAT versus the new SAT are completely irrelevant and distracting noise. Students in the class of 2018 and later should no more be worried about the new SAT and its changes than they are about the changes college professors are making to their grading criteria.
- Students who are planning to take the redesigned SAT
Any student who has already determined that they will be taking the redesigned SAT (whether it’s because they are seeking the National Merit Scholarship, schedule doesn’t permit them to take the SAT before the new test launches in March, their school is providing the new SAT for free, etc) should also ignore all the chatter about the changes. Once you’ve reached a decision about which test you are taking it no longer matters that that test is different than it used to be, it only matters what it will be when you take it. The only reason to be aware that it has changed is to ensure that you aren’t practicing with outdated material or information.
As a parent your concern shouldn’t be with the newness or changedness of the test but rather with how to make sure your child is best prepared for the test. That preparation might mean choosing to take a different test (since there is the option of SAT or ACT) so as to best display her academic abilities. For most parents your role will remain what it has always been and will continue to be for a long time: the planner, the payer, and the helper. This was true for the 1926 SAT, the 1974 version, as well as the 1994 and 2005 versions (Want to see the history of SAT changes? click here) and will be true when they revise the SAT again in 6 – 10 years. Until then you just need to worry about figuring out how to guide and support your child in taking the test they choose to take, not worrying about the details of the changes.
And now who should be concerned about the fact that the SAT has changed.
- College Board
College Board should and is really concerned about the new SAT. They are concerned that the new test might drive more test-takers to their competition (the ACT) in the short-term and exacerbate the market shift that’s been happening over the last few years. The College Board also has the additional concern that the new test won’t perform as advertised, hoped, or projected. What if, instead of scores being distributed on a nice happy normally distributed bell curve, the results for the redesigned SAT are skewed toward an average score of 650 per section (anyone else remember recentering aka tweeking the curve)? What will happen if the results are completely out of wack with all beta testing? Will colleges still require or accept these scores? Will the changes to the SAT to “align it with the work of schools” (read: align with Common Core) exacerbate the Test Optional movement? The College Board has lots of reasons to be concerned.
The team at the Iowa-based newly crowned king of college testing has lots of reasons to worry about the changes to the SAT. Since many of the changes to the SAT address criticisms of the old SAT, ACT is rightly concerned that test-takers may return to ETS’s warm bosom and the comfort of the “known” college admission exam. In redesigning its test, College Board not only address many criticisms of its exam but also took direct shots at perceived areas of superiority that the ACT possessed (and often touted). ACT has lots to be concerned about as it must wonder if this newest SAT (or doppelganger ACT) will retake its position at the top of the psychometric food chain.
- Test prep companies and tutors
Test prep professionals are the ones who should be most concerned about the changes to the SAT. These changes mean that if you’re a test prep person doing your job right you’ve got to learn new stuff and you’ve got to create or find new materials. Changes to the test, while usually causing a spike in business, cause a great deal of work and research for test prep folk. It will take test prep community a while to gather and analyze sample questions and tests in order to learn the ins and out of the new test and the quarks and nuances that can most easily be turned into points. Test prep folks are and should be concerned about the changes, burning the midnight oil, and clicking refresh thousands of times on release days for new samples from College Board.
- College counselors of students in the class of 2017
College counselors should be mildly concerned about the changes to the SAT, especially for the next 12 – 18 months where there are still 3 viable options for college admissions tests. While choice is often good, too many choices just create a big confusing mess, this will probably be true for counselors and students in the class of 2017 until March 2016. Counselors are going to struggle to figure out which of the three college admission tests to recommend, how to interpret the results of the new SAT and PSAT, and how colleges are going to handle comparing students taking different tests. Yes, counselors working with current sophomores will have a bit to be concerned about relative to the changes.
- Class of 2017
Students in the class of 2017 are right to be a little concerned about changes. This class took the old format PSAT, and if they liked it, they may be rightly concerned that the new test might not be as easy for them to get a good score (anytime there is change, how an individual student will handle that change is nigh-impossible to predict). The introduction of the new test will create confusion and thus concern for students. These students who’ve taken the PSAT formatted after the current SAT had some understanding of what they were soon to face, however the introduction of the new SAT could make that 2.5 hour preview entirely moot.
To make a long blog short, not many students or their families should be worried about the newness or changedness of the test until they determine that it actually impacts them. If you’ve determined that you’re going to take the new SAT than its not the newness or changedness of the test that matters so much as making sure you properly prepare (which means ensuring that you have up-to-date practice materials and resources. Unsurprisingly these same preparation concerns have been true since the invention of testing and thus the real concern is not that changes but proper preparation.