Monday, February 29, 2016

What The NFL Isn't Learning From College Admissions

If you've been paying attention to college admissions lately, you know that the last year or so has seen quite a lot of interest in the changes to the SAT, which started with the PSAT last October and will fully take effect with the March test date. I have previously written about the efforts the College Board has made to teach admissions professionals on both sides of the desk about the new test, and also about my concerns that, based on PSAT scores, the revised test seems to be yielding higher than expected scores, which makes it hard for people to use them to help identify target schools for students.  Also, as my colleague Evan Wessler has pointed out, it will be impossible for colleges to "superscore" the old and new SATs, because they are so completely different. Evan has also observed that the new SAT is harder than the old one, and so students with equivalent scores will probably have fewer correct answers on the revised assessment. 

Google "changes to SAT", and you will find thousands of posts that try to explain the changes and help users (students, parents, teachers, school counselors, independent consultants, and college admissions officers) understand what they mean. College Board president David Coleman was quoted in a CNN report in 2014 as saying that the test needed to change because " [a]dmissions officers and counselors have said they find the data from admissions exams useful, but are concerned that these exams have become disconnected from the work of high school classrooms..." Despite that admirable motive,  in a New York Times article from Feb. 8, 2016, Eric Furda, the dean of admissions at the University of Pennsylvania seems worried about how to interpret student results. "We're going to need to see how they did, which test is going to be better, how can we weigh it," before being able to fully embrace the revised SAT. 

A related quote can be found in a USA Today article from Feb. 21, 2016
, where an official muses "Our first focus is to look at what we do currently and making sure that that's relevant. And if it is, great, we'll continue to do it, because historical comparison is really important to the evaluation process. But if we believe that there's something that's not relevant, then what can we replace it with that will help us evaluate"? While this quote seems to be quite similar to what we've seen above, it actually comes from Jeff Foster, president of National Football Scouting, Inc., which runs the annual "NFL Scouting Combine". The Scouting Combine is a very popular staple of ESPN and the NFL Network broadcasting, and hundreds of thousands of people tune in to watch former college pigskin stars try to achieve their best results in the following events:

  • 40 yard dash
  • Bench Press-maximum reps at 225 pounds
  • Vertical jump
  • Broad jump
  • 3 cone drill
  • Shuttle run

The USA Today article notes that a committee "of league executives, scouts, coaches, athletic trainers, team physicians and others" is being formed to "review all phases of the annual event". 
No, a quarterback's throwing session on the field won't be swapped for one in a virtual reality environment anytime soon. But the days of players training for months to score high in tests such as the 40-yard dash, vertical leap and bench press--sometimes derided as the "underwear Olympics" --could be numbered.
 "We're continuing to explore everything in an effort to improve," NFL spokesman Brian McCarthy said. "So, if there are ways to tweak, improve, modify anything we do, we'll explore that...The mantra is, how can we get better?"
Wanting to improve is an admirable goal, but there is already backlash against the proposal. The Monday Morning Quarterback column in Sports Illustrated wrote about the proposed changes on Feb. 25, 2016. While reporter Robert Klemko (who has an interesting past) notes that making offensive linemen and interior defensive linemen risk injury trying to run a sub five seconds 40-yard dash is pointless, considering that they will rarely if ever have to sprint 120 feet in the course of a game, he notes that other changes have stakeholders nervous:
"You know you don't want to get away from the traditional drills that you've done because you have so much information stored over the years for a comparison," says Pittsburgh Steelers general manager Kevin Colbert, "but you don't want to grow stale either."
Such will be the main argument against any proposal to scrap or alter drills like the 40-yard dash and the vertical jump: A vast ocean of data sets from previous combines help evaluators compare and contrast prospects from various years.
"The one thing about the combine traditionally over the years is you have the ability to compare, year after year, the same drills," says Los Angeles Rams coach Jeff Fisher. "So we have information where we can go back 10, 15, 20 years and compare players to players. In the information age right now, it's becoming much easier to do that." Says one AFC scout: "In order for new drills to be used, it will take years for them to have as much meaning."  
The Scouting Combine may very well need to change. It has been derided as the "underwear Olympics" due to the skimpy outfits the players wear, it's also been compared to an antebellum slave market, and I haven't even mentioned the controversial Wonderlic intelligence test, which can tarnish someone's reputation forever, especially if they neglected to do any test prep (I'm looking at you, Vince Young). 

The changes to the SAT will probably be net positives in the long run, but in the short term they have caused a lot of worry for a great many people, and they also feed the fire of doubt about the validity of the tests as meaningful ways to compare applicants from year to year. The proposed changes to the NFL Scouting Combine seem to threaten to have similar results. It's not too late for the NFL to try to learn from the turmoil and upheaval going on in college admissions. Of course, football has never really paid attention to the SAT, so perhaps my hope is misplaced.

Friday, February 12, 2016

If I Can't Understand The Score, What's The Point Of The Test?

2016 is shaping up to be an exciting year in the field of standardized tests for college admissions. The list of "test optional" colleges and universities is bound to grow, President Obama has joined the 
clamor deriding "test prep" in schools, and both national admissions testing organizations--the ACT and the SAT-- have instituted changes to their flagship assessments that have resulted in confusion and anxiety on the part of students, parents and schools. Having recently become part of the test prep industry, I can't help but think of the famous curse "May you live in interesting times".

As I mentioned previously, I now work for Method Test Prep, a company that serves around 1,000 schools, community organizations and independent college counselors to help democratize access to high quality test prep for high schoolers. Earlier articles on this blog have critiqued the helpfulness of the SAT's "college readiness" assessment and the ACT and SAT's obsession about security; this one is going to be a little different. It occurs to me that one of the sources of uncertainty felt by college applicants is that, while they know the tests are important, it is increasingly difficult to  determine just what a score on these tests really means, which makes it hard for students to feel confident that their $50+ and 3+ hours are well spent.

I suppose that it is important to state at the outset that, despite the popularity of colleges' stated goal to evaluate applicants "holistically", standardized test scores play a major role in application decisions. In the 2014 "State of College Admission", the National Association for College Admission Counseling (NACAC) observed that 58% of colleges attributed "considerable importance" to test scores, third out of 16 variables, and only slightly behind the students' strength of curriculum. 69.7% of public colleges gave considerable importance to test scores, which was ranked second only behind the applicant's high school grades. Test scores are also increasingly important as selectivity increases. Only 55% of colleges which accept 85% or more of applicants consider admission test scores to be considerably important, but 62% of colleges which accept 50%-70% of applicants rate test scores highly. 

So these tests are important, and it is desirable to get a higher score. But depending on the institution(s) to which students are applying, it can be hard to tell what score to aim for. Websites like CollegeSimply can point you in the right direction, and the ACT and the College Board have published multiple documents to help interpret test scores. Unfortunately, both of the tests have experienced revisions in the last year that have only served to increase the anxiety and uncertainty among students and educators.

In the Fall of 2015, the ACT made a change in how it scored its optional essay. (Note: Even though the writing section is "optional" on the ACT and SAT, it is still a good idea to do it.) According to the ACT, the new writing section gives students 40 minutes (instead of 30) to "analyze multiple perspectives on a given issue in relation to their own perspectives". The new writing prompt calls on a "broader range of subject matter" related to "contemporary issues beyond school experience". This certainly sounds straightforward. Unfortunately, it turned out to be quite the opposite, and the ACT had to issue a 13 page document explaining how to interpret the writing scores, and justifying its rigor and value as an assessment. 

The confusion was based on the new scoring scale. Since 2005 the writing section had been scored on a scale from 2-12. The new test grades the writing on four "domain scores", each of which is on a 2-12 scale, and then the sum of the four domain scores is "converted to a scaled score on a 1-36 scale". Since the rest of the ACT's sections (English, Math, Science, Reading) are scaled on a 1-36 basis, it only makes sense that students and educators thought that the score would be equivalent to the other sections. In other words, a student who scored a 32 on English and a 32 on Reading should probably get a 32 or thereabouts on the essay. Unfortunately, that has not been the case in most situations, and people are quite concerned about the ramifications of what seems like a "bad" essay score.

ACT admits that the writing scores are "generally lower than other scores--on average...3-4 points lower than the ACT Composite or English scores." This is mainly due to the inherent variability of grading a single task, compared to the other sections which have up to 75 questions. The ACT urges that "scores across subjects are not strictly interchangeable and should not be compared". But that seems naive since they made it so easy to make a (faulty) comparison when they rescaled the essay. This seems like a blunder that needlessly muddied the water.

The SAT has also changed its test, but in a much more comprehensive way. The SAT has been overhauled from the ground up and beginning in March, millions of students will take a test that covers different topics, asks different types of questions, and is scored on a different scale from its predecessor. Method Test Prep has written extensively about the new test, and our prep materials for the new SAT have been used by tens of thousands of students since last summer. Many students used our program to prepare for the PSAT/NMSQT which was administered in October and which mirrored the structure and scoring scale of the new SAT. While the College Board proclaims that "students will not be guinea pigs" for the new test, there sure do seem to have been some glitches associated with the new assessment. For instance, the release of PSAT scores was seriously delayed--in previous years they were sent to schools in early December, but this year they were not released until mid-January, and there have been numerous reports of errors and poor user interface to the online data in the weeks since. 

Once the scores came out, schools were (pleasantly) surprised to see that, in many cases, the scores seemed to be much higher than they had expected. Considering that the new test includes more difficult math and lengthier, more complicated reading passages, this was quite unexpected. I have spoken with numerous guidance counselors all over the country who have shared similar anecdotes with me. It seems that most students are ranked in a much higher percentile than they had been on previous College Board assessments. A Google search on "PSAT percentiles higher than expected" yields over 24,000 results. While this might be good for students' self-esteem, it makes it even harder to decide whether the PSAT should be considered as a practice for the SAT or as a scholarship test

The problem with score reporting is troubling for several reasons, but most notably, because for the next few years, for anyone to be able to make sense of the new SAT (and PSAT), they will have to compare new scores to the old test's scale. This is one reason why the College Board will be delaying the release of the March SAT scores for so long, and only after the March and May test scores are released will they publish a concordance between the new and old tests. In other words, students, teachers and admissions offices will not really be able to interpret the new scores in a meaningful way without the concordance, and to compare the new SAT to the ACT will require TWO concordances.  Current high school juniors and their counselors who should be busy trying to identify appropriate colleges are operating in the dark, since they have no way of knowing what scores colleges will be looking for. The only thing they can do is look at their percentile to see how they compare to other students. But if that number is not accurate, then what?

Major media sources such as the New York Times have been devoting more attention to worries about the new SAT. In recent articles, Tamar Lewin wrote "A New SAT Aims to Realign With Schoolwork", and Anemona Hartocollis wrote "A New, Reading-Heavy SAT has Students Worried". Lewin's article gave a nice synopsis of the changes, and extolled College Board president David Coleman for his goal to make the SAT more closely related to what kids should be learning in school (Coleman's background with creating the controversial Common Core standards may influence him here). The piece definitely struck a nerve with readers, because in just a few days over 550 comments were posted on the online version of the article. The Times summarized the comments in a later article, and readers seemed generally unhappy with the announced changes. 

One of the biggest points that the College Board cites in favor of their new test is that free test preparation will be available through Khan Academy. One Times reader took issue with that:

A commenter under the handle R-son from Glen Allen, VA said his stepson, who is better in math than reading, would soon be taking the test. "The new SAT will be hard for him, but he has an advantage over other students--an $800 Kaplan prep course. So it boils down to this--he'll score better on the SAT than a lower income student with the same abilities whose family can't afford to fork out close to 1K to prep for and take this test. So how is this test, in any form, fair?"

R-son is onto something here. While students could take a prep course for much less than he paid for his child, I have no doubt that parents and educators will be seeking out more test prep in the near future than they had previously. In my daily interactions with high schools all over the country I hear over and over that the school boards and administrations are placing greater emphasis on standardized test scores, and that they are eager to find a tool that can help prepare kids for both tests

Ultimately, the ACT and SAT are important to colleges as a way of comparing students from vastly different educational backgrounds. That said, the built in cultural and economic biases to the tests are troubling, as is the relationship between test scores and college rankings. My alma mater, Hampshire College recently became the first "test blind" institution in America and they had several significant results: 
  1. They were excluded from the US News and World Reports college rankings list
  2. The number of applications decreased, but the quality of the applications increased due to the increased emphasis on essays
  3. Diversity increased to 31% students of color, and first generation college students increased to 18% of the class that entered in 2015.
It's unlikely that colleges will relax their emphasis on tests in the near future, and the complicated nature of the tests, and of interpreting test scores will only make things harder for applicants, their families, and their high school teachers and counselors. Instead, the likelihood is that the proliferation of test prep options will continue as people grasp at options to improve the admissions chances of the students with whom they interact.