Friday, September 22, 2017

What I Saw At NACAC 2017: A Summary Of "Testing Achievement and the Future of College Admission"

I attended the 2017 National Association for College Admission Counseling conference in Boston in mid-September. This was the third straight year I've been able to attend, thanks to my day job with Method Test Prep, a leading ACT/SAT preparation company. I spoke to lots of people (old friends, customers, colleagues) and the consensus seems to be that the college visits, the exhibits and the conference sessions were all very valuable.

I was able to attend two sessions:
  1. What Admission Deans Think-Results of an Inside Higher Ed and Gallup Survey
  2. Testing Achievement and the Future of College Admission 

The presenters for the session "Testing Achievement and the Future of College Admission" were:
  • Ben Wildavsky, College Board Senior Fellow & Executive Director
  • Lynn Letukas: College Board Associate Research Scientist
  • Edgar Sanchez: ACT Research Scientist in Statistical & Applied Research
  • Jerome Lucido: Professor at USC Center for Enrollment Research; Associate Dean of Strategic Enrollment Services
  • Jack Buckley: American Institutes for Research Vice President
Working as I do for a company that prepares thousands of students for the ACT and SAT while simultaneously recognizing the inherent flaws of standardized testing, I was quite excited for this session. The description of the talk was promising, though in the end I think the actual presentation did not live up to its potential. According to the brochure:
Despite widespread media coverage, underlying claims about the benefits of “going test-optional” have largely escaped empirical scrutiny, and support for these claims tends to be of limited generalizability and/or fail to adequately control for student selectivity and other factors. Hear a rigorous and balanced approach to the contemporary debate on standardized testing and the test-optional movement. Explore how test-optional practices emerged and expanded, how widespread grade inflation has made it increasingly difficult for institutions to solely rely on students' prior achievement, and how standardized admission tests can be used in predicting student retention, achievement, and graduation.
Looks good, right? Unfortunately, I found the session to be a disappointment. Instead of a debate, the members of the panel are contributors or editors to a forthcoming book called "Measuring Success" from Johns Hopkins University Press, and they spent the next hour giving lengthy summaries of a few of the chapters from the book. It didn't take long for the suspicious part of my nature (which is never far from the surface) to begin to wonder if this book would be "rigorous and balanced" the same way that Fox News is "fair and balanced" and would actually be a vehicle for the College Board and ACT to cast doubt on "test optional" policies to preserve their core businesses.

Ben Wildavsky introduced the panelists and then described the book. I'm amused when I look at my notes from the session; at first I wrote "PREORDER THIS BOOK!!!!", but as the talk went on I crossed that out and wrote "this book is going to be VERY dry and boring with lots of numbers. To help clarify? Or to obfuscate?" (there's that suspicious nature again). Wildavsky said that the premise of the book is that "arguments for test optional policies have not been studied rigorously; that they are based on isolated, non-reproducible case studies, and that they may have a downside, especially in a time of widespread grade inflation."

This led to the first presentation. Jack Buckley (whose biography notes that he "helped lead the redesign of the SAT") began by saying that he had eight minutes to discuss two topics: 

  1. To what extent should tests play a role in admissions?
  2. How should an institution best assess students in a manner that promotes fairness?

Interesting questions to be sure, but he didn't really talk about them. Instead, Buckley spent the next seven minutes giving a general overview of the forthcoming book, and then dove into a series of graphs describing what he saw as rising grade inflation at American high schools. According to his data, students' self-reported grades on SAT forms rose over time, while SAT scores decreased. Buckley says this has been confirmed by reviewing actual high school transcripts. Further, he said that research showed that "wealthier, less diverse schools have faster grade inflation than otherwise". More than once Buckley mentioned an article that had appeared in that morning's Inside Higher Ed--he seemed somewhat unhappy with the article and if you read it you will see why. I will be interested to read his research, but to me there are quite a few, if not "flaws", then "problematic areas" of this data. Specifically, he seems to use falling SAT scores as evidence of a negative trend, as opposed to seeing rising high school grades as a cause for celebration. Especially since the SAT has been completely revised (by Buckley!) to more closely hew to typical high school curricula since the data was recorded, doesn't that indicate that he might be viewing this through a distorted filter?

The next speaker was Edgar Sanchez of the ACT. He definitely spoke for longer than eight minutes and attempted to prove the following assumptions:

  1. There has been little research to show the impact of test prep and coaching.
  2. "Many studies" highlight the predictive power of the ACT. High School GPA is the strongest predictor of college success, but tests are valuable.
  3. Many institutions choose to go test optional out of concerns about diversity, but it's impossible to tell if test optional policies result in increased diversity because "other factors such as recruitment and awareness are present."

Sanchez told us that 25%-30% of the population had "discrepant ACT and High School GPAs" and that the only way to solve this "mixed message" was to consider both. Tests, he said, focus on cognitive skills while grades shed light on non-cognitive factors such as engagement, self-regulation, discipline and habits of inquiry. In other words, test optional schools lose the full picture. That might be true, but if he's right I could give up insights into "cognitive skills" if I still had all the other stuff he described. 

Part of the argument Sanchez made was that students with "high or standard GPAs and low ACT scores are more likely to suppress test scores and are less likely to be admitted. Not only that, but the ones who choose test optional are more likely to be women, poor or minorities and are less likely to come back for year two of college." I'll have to see the data, but this seems somewhat dubious to me. First of all, what's the proof that students with high grades who don't send test scores are less likely to be admitted? And secondly, he's basically saying that students with low ACTs are more likely to suppress those scores. But of course they are! And if the scores really are low, how would sending them help? It beggars the imagination to think that if high grades/no ACT is more likely to be denied that high grades/ low ACT are more likely to be accepted.

Sanchez summed up his point by concluding that "We stand behind the belief that more information is better than less information." Sure, that makes sense. But have the researchers at the ACT actually ever worked in college admissions?

Probably not, but the next speaker has. Jerry Lucido is both a scholar and a dean of college admissions, so I started to hope that he might engage with the topic in a more practical, less numbers-heavy way. Apparently his chapter in the book deals with "practitioners", but his presentation was rather problematic. First of all, he spent most of his time giving a long description of what test optional means and why institutions choose it, which only summarized what everyone in the room already knew. Lucido's point was that he "never doubted the sincerity of motivation to choose test optional", but that he was skeptical of the value of the policy. He concluded by saying that "the test optional practitioner is a pragmatic idealist" and that test optional admissions was "neither a panacea or the only way". In other words, he was the third straight presenter who told us that colleges should decide what's best for them, but they should decide on test optional policies after careful study. 

During the first two speakers I began to be aware of a certain, shall we say, restiveness on the part of the audience. Murmuring, people getting up and leaving, and lots of gesturing at friends in the audience increased steadily. Lucido's presentation didn't help with this, in part because of his constant stammering and repetition. He looked uncomfortable at the podium and this seemed to effect his speaking skills. The final speaker was Lynn Letukas of the College Board. Her presentation was marred by an inability to speak into the microphone and by the content. Basically she spent ten minutes describing every chapter of the book. That would have been fine if the purpose of the session was a book launch, but it purported to be a chance to engage with an important topic.

Just as I was becoming convinced that Letukas' role in the session was to eat up time and prevent questions, Wildavsky came back to the podium. He noted that there was 20 minutes left and that he wanted to leave "at least 15" for questions from the audience. By now, if you've been catching the tone of this summary you won't be shocked to learn that he only left seven minutes and there were already six people lined up at the microphones. Before asking for audience questions Wildavsky asked a couple of questions of the panel; the answer that seemed most emblematic of the whole session was when Buckley said that "case studies are not good ways to evaluate whether there is causality between test optional policies and student outcomes". 

When the seven minutes of audience questions began, I began to wonder even more if the fix was in. The first speaker was sloppily dressed and after asking if any research has been done to see why low GPA/high ACT scores exist (answer:no) wouldn't leave the microphone. At this point one member of the audience angrily shouted at Wildavsky (I didn't catch what he said) and stalked out of the room. 

The next question was from Jon Reider of University High in San Francisco. He asked "if the argument that test prep helps isn't rigorously proven, then why do ACT and SAT now urge test prep (from their preferred providers)?" He asserted that "research by anyone not from a test prep company shows that the impact of test prep is next to zero."  Ben Wildavsky responded with a short promotional statement about Khan Academy (the chosen partner of the College Board). Reider responded by saying that he thinks that "test prep is harmful, because it's a lost opportunity to study other things". Sanchez chimed in by agreeing that "any class time spent on test prep is inappropriate and has the chance to harm students." He said that people should do "some test prep, to avoid stress" but not more. 

That exchange was quite interesting to me. I work with hundreds of schools around the country, a large number of which use my company's ACT/SAT prep program. In many cases they are in states which require ALL juniors to take the ACT or SAT and reach a particular score--often the results of these tests are a big part of the school's official rating from the state Education Department and can impact the funding they receive. While this is almost certainly a misuse of the tests, it has also become an incredibly lucrative part of the test companies' business models. It's pretty bald-faced for an employee of the ACT to complain about declining test scores and then also complain about schools trying to raise them.

The next question came from an independent consultant from Brooklyn who asked if the book will be "promoted in a fair and balanced way". There was no clear answer, but since the question was so vague it didn't really deserve one. The final questioner was Wayne Camara who is a VP at the ACT. But guess what? Wildavsky interrupted to say that time was up and we had to clear the room right away to make space for the next session.  The other people waiting at the microphone were obviously disappointed. Jon Boeckenstedt, VP of Enrollment Management at DePaul (whose own writings are longtime favorite sources of mine) was next in line behind Camara and he made his way to the microphone to ask "How do you advise readers to consider research when writers and presenters have vested economic interest in the outcome?" As Wildavsky chided him for the tone of his question, Boeckenstedt smiled seraphically and walked away. 

As the speakers began packing up to leave, Camara and Boeckenstedt had a lengthy discussion at the back of the room. As this was going on, Jay Rosner of the Princeton Review Foundation told me that some of the back story of the tension in the room came from the fact that the editors had sought (largely in vain) for admissions professionals to write for the book but that it was evident that the articles would skew heavily against test optional policies. Hearing that made me even more willing to believe that the entire presentation was carefully structured to prevent any audience participation or, in fact, any argument in favor of test optional policies. In an era where people are increasingly sensitive to, and prejudiced against perceived bias, I can't help but worry that this book will be like lots of "fake news" and generate lots of heat, but shed little light on an important subject.

Thursday, September 21, 2017

What I Saw At NACAC 2017: A Summary Of "What Admission Deans Think"

I attended the 2017 National Association for College Admission Counseling conference in Boston in mid-September. This was the third straight year I've been able to attend, thanks to my day job with Method Test Prep, a leading ACT/SAT preparation company. I spoke to lots of people (old friends, customers, colleagues) and the consensus seems to be that the college visits, the exhibits and the conference sessions were all very valuable.

I was able to attend two sessions:
  1. What Admission Deans Think-Results of an Inside Higher Ed and Gallup Survey
  2. Testing Achievement and the Future of College Admission 

The presenters for the session on "What Admission Deans Think" were:

  • Scott Jaschik, the editor of Inside Higher Ed
  • David Hawkins, Executive Director for Educational Content and Policy for NACAC
  • Mary Ann Willis, the Director of College Counseling at Bayside Academy (Alabama)
I have to admit that I didn't read the full description of this session, and thought it would be more along the lines of "what kind of students do colleges look for", but I was pleasantly surprised to find out what it was really about. The purpose of the session was to reveal the findings of Inside Higher Ed's annual survey (in conjunction with Gallup) of admission deans. You can download the full report at their website. The focus of the survey and the presentation were on the difficulty of meeting enrollment goals, the "Trump Effect" on international recruitment and retention, and efforts to increase racial and ethnic diversity on campus. There was also brief discussion of challenges facing liberal arts institutions, rising student debt and the move toward "free" tuition at public institutions. The survey goes into depth on these and other topics and is worth the read.

The survey was sent to 3,547 admissions directors and enrollment managers, of whom 13% responded. The respondents came from 200 public institutions, 245 private ones, and 8 for-profit schools. 13% sounds kind of low, but if you've ever given a survey you know that it can be hard to get people to participate fully. 

One of the first findings discussed was the difficulty of meeting recruitment goals by the national decision day of May 1st. According to the survey, the number of colleges reporting that they met their target by May 1st is small, and shrinking fast:

This was quite surprising, and led to an interesting discussion. Scott observed that "if you didn't meet your goal by May 1 you're the norm in higher ed". He noted that the joke in the field is that "June 1 is the new May 1" and that some have argued that SEPTEMBER 1 is the new May 1. The panelists tried to discuss this development and the conclusion seemed to be that the enrollment goals that are being set (sometimes by offices higher up the chain from admissions) are unrealistically high. Plus families seem to be waiting longer to send in deposits (perhaps to try to play institutions off each other). Regardless of why, Scott noted that he feared that "some institutions will be pressured to do unwise things" to meet their goals by May 1. In response to an audience question, Scott noted that they will ask more questions next year to try to elicit just how "realistic" admission goals are.

The only sector that overwhelmingly met their enrollment goals was public doctoral institutions. Reasons could range from the large number of in-state students for whom these are the "first-choice" schools to the (perceived) prestige of these universities by international students and their families. Scott noted that even as states have eroded their support for universities, these institutions have great advantages in fundraising, donations, applicant pool and federal grants for research.

One question in the poll was "which group(s) will be the target of increased recruiting?" and the results were quite interesting:

I was very surprised to see the increased attention that private colleges plan to pay to transfer students; I mean, the pool of college age students may not be growing as rapidly as it was a few years ago, but overall enrollment in colleges continues to grow year over year. This effort to harvest transfer students represents almost a Malthusian world view that seems to reflect a zero-sum competition that I'm not sure exists. Also telling is the comparative de-emphasis on the part of private colleges and universities on veterans and minority students. It almost seems that public colleges and universities are more comfortable with diverse student populations compared to the privates. 

 At this point I want to point out that I was impressed by Scott's willingness to inject politics into the discussion. I personally believe that every topic is a political topic, but I can imagine that some people might have been nonplussed by this. Scott passionately addressed the data on the preceding chart by saying "We should think about these priorities. If private college's emphasis is non-need based merit aid, are they really going after the population not currently represented in American higher ed? Colleges measure demonstrated interest by favoring those who visit. But this has an obvious, unstated economic meaning--wealthier kids can afford college visits more." Especially since the topic of access to higher ed was a major theme of the conference, I welcomed Scott's focus on this issue.

You'll notice that the last item in the table shows a remarkable divergence on expected focus on international students.  They went into more detail later on:

This was quite interesting to me. I was a college counselor for eight years at a boarding school with a substantial international population and in many cases their default was to consider large public universities. But the leaders in the field clearly seem to be ceding international students to private institutions who in turn are preparing to pay heavily to increase international enrollment. Could this be due to a backlash against international kids at publics? Or could it be a "Trump Effect" that predicts a generally more hostile environment for immigrants? It was noted that international enrollment was flat this year, and expectations are for it to fall this year; apparently some schools are down 20-50%. Considering how many places depend on full-pay international students to help finance their discounts to American students, this could be a looming financial disaster for some schools.

Related to this is another side of the "Trump Effect": a concern that too many colleges (and college towns) are "excessively liberal". Apparently, independent consultants have reported that more parents are vetoing college choices due to a worry about the political views of the campus and area. Scott asked us to consider what it means when people perceive higher education this way. Personally, I don't think this is new. Deriding someone as a "college boy" (implying an effete elitism out of touch with common people) goes back for at least a century. That said, it is no secret that college populations (and university towns) are often isolated blue islands in the overwhelmingly red American landscape; this of course explains Republican efforts to disfranchise college students.

The twin topics of liberal arts colleges and student debt were addressed in an interesting way. Apparently, only 8% of students and 9% of parents "understand the value of liberal arts". As a result, there is overwhelming pressure to focus on college as a "pre-professional" education. Tied with this is the idea of getting one's money's worth and fear of graduating with debts no honest man can pay (as Bruce Springsteen would say). 36% of public institutions and a startling 78% of privates believe that "public discussion of student debt has discouraged applications to my college". Wow.

Taking a deeper dive into the topic of student debt, there is a growing divide between what admissions people think is "reasonable debt" for students:

What a difference! 36% of admission leaders at private colleges and universities seem comfortable with students graduating with at least $30,000 in debt, while only 10% of public institution admissions heads would find that reasonable. And 63% of publics think a maximum of $20,000 in debt is reasonable compared to 3% at privates. Quite a chasm between those two! Obviously the fact that privates cost less than publics might have something to do with this, but it's clear to me that going forward, the focus on containing costs will be at the public level. In fact, the survey took a look at that and the fear on the part of private institutions was palpable:

The only conclusion I have is that these limousine liberals running private colleges are probably secretly happy that Bernie Sanders' "free public college for all" idea didn't go anywhere!


After the data was presented, Mary Ann Willis and David Hawkins spoke briefly. Mary Ann agreed that "the general public doesn't understand liberal arts" and think college is for career preparation. She urged that admissions officers work harder to recruit students at smaller, rural schools like hers. She counseled that they shouldn't make assumptions about who goes to these schools: "I have students with their family name on buildings, and I have some who are Pell eligible". She also noted that at her school, kids apply to an average of 3.5 schools.

David Hawkins continued the focus on some political issues. He noted that America "confront(s) a real crisis with race, with 'have vs. have not'" He said that NACAC will be stepping up its effort to do a better job to communicate the value of higher education, not just economic, but personal and intellectual value. He also raised an interesting point that I would have liked to hear more about: he said that in 2002 the average yield rate for colleges was 48% but is now 36%. Why? I wonder if it's reliance on "write once, send many" tools like the Common Application, the Coalition Application and the Universal College Application. He closed with the frightening image of a "harsh new equilibrium in college admissions where the admission cycle is so long it reaches around and touches itself on the other end."

There were a number of questions from the audience which elicited more elaboration on these topics. The final question came from a secondary school counselor from Wisconsin who asked what the survey will look like in five years. Scott's response was glum: "I fear that five years from now we will have fewer small private colleges, more over-crowded regional public universities and flagship public universities will be more like elite private colleges. I'm worried about accessibility going forward."

Heavy stuff for sure! What have you or your students seen in your college admission experiences? Share your thoughts in the comments space below.