Prologue: I once worked with someone who held an MBA from Kellogg at Northwestern University. He could not calculate percentage change.

This was a smart person. He had gone to an august undergraduate institution (I no longer use the word “elite”), worked in business for a while, took the GRE, nailed it, and graduated with that MBA in marketing, a discipline that requires some considerable level of higher order math skills. I’m sure if he had been reviewing for a freshman (in high school, not college) course, he’d have been able to remember how you calculate percentage change via a memorized formula, but he was complete unable to parse the phrase “percentage change” and figure out how to do it.

This is no joke. It actually happened.

All this got me thinking about the news lately, and I’m sure lots of people are going to hate this post, even before they get past the headline. So be it. And, fair warning: This takes a long time to wind up where I want it to head. There are some detours and re-directions that make it best to read when you have some time.

You have probably heard that almost 1,000 faculty in the University of California system are recommending that the UCs return to requiring the SAT for admission. That’s not surprising: Berkeley, among others, adopted the SAT as a requirement because it wanted to be seen as an equal to the prestigious east-coast institutions in the 1950s who required it, and of course, imitation is the sincerest form of flattery, especially in higher education. (It should be noted that ETS opened an office in Berkeley to encourage this lunacy.) The recommendation follows other august (I refuse to use the “e” word) institutions who have returned to requiring the SAT for admission, after the COVID pandemic made the test optional almost everywhere. It’s not entirely surprising.

The curious thing about the letter, though, especially coming from mathematicians and scientists who got to where they are by conducting research, is the rapid movement from problem statement to “solution.” In a nutshell, it’s “Students are struggling in math. So the problem is the admissions office. Bring back the SAT for STEM majors.” It seems like a conclusion a freshman might leap to, and it seems like overkill, given first-to-second year retention at the UCs does not appear to be affected, which would be the first place you’d expect to see serious academic deficiencies show up (we’ll have to wait a while for four-year graduation rate data to see how the first test-free class looks). It is possible, of course, that students are switching out of STEM majors into business or humanities or social sciences once they fail Calculus, but that’s hardly a new trend in higher education. It’s so common, it’s cliché. (click images to view larger).

I am not going to argue with the lived experiences of the faculty who teach mathematics in the UC System, although I do like to point out the research of Saul Geiser at Berkeley who has consistently shown big problems with the results from (and our assumptions about) the SAT in predicting performance, even before you look at issues of inequity, using the College Board’s own data. And famously, of course, Geiser responded with an “Oops, you forgot something” when the UC Faculty said tests were better predictors of performance than GPA. I do wonder, however, how this report leads anyone to think the SAT is the answer.

Even I, someone with little respect for the value of the SAT, and even less for the company that owns it, The College Board, have to admit the test is not worthless. It is really good at figuring out who can spot the right or the best answer quickly. Ceteris paribus, that is a skill you’d rather have than not, I guess. I have just never thought it was a necessary skill in places like philosophy class, where you don’t even know the questions, or in a history or English class, where even experts disagree on what things mean, or in an engineering classroom, where some of the math measured on the SAT is necessary, but not the kind where someone gives you the right answer hidden among a few others.

The SAT, it might be noted, was meant to measure something in the individual student although curiously, no one can really even define what that something is. But US News and World Report, and George W. Bush’s No Child Left Behind started using test scores (including the SAT) to measure schools: Both High Schools and Colleges were subject to this scrutiny by people who were both loud and influential, but not experts on educational measurement or the meaning of the results. Having bureaucrats screaming about “Accountability by the Numbers” is the dream, of course, of standardized test makers, who always–and I mean always–fail to mention that norm-referenced tests don’t measure achievement the way a geometry test for high school sophomores does, and can always give you test results as a number: A nice, easy number that even non-mathematicians can understand (unless those numbers are actually string variables and have no intuitive relationship to percentiles, of course. Ahem.)

No, a standardized test is designed to sort people, and will always have an almost perfectly shaped bell curve in the outcome because the test is made up of questions of varying levels of difficulty, based on research from prior test administrations where they are tested for validity: Lots of people end up in the middle, and fewer at the tails. Anthony Carnevale, a former executive at ETS, even admitted in the film “The Test and The Art of Thinking” that in order to design a standardized test that was effective, you had to try to trick people into choosing the wrong answer.

Compare this to a traditional A through F grade in classroom, where if everyone studies hard for the geometry test and meets all the class objectives, there should be no problem giving everyone a top grade. Are you listening, Harvard?

Here, for instance, is the distribution of Composite test scores on the ACT in 2008 (orange), 2013 (aqua), and 2018 (purple).

Do you notice a pattern? That’s a feature, not a bug. (And for all the test fans and faculty in the report who cite “grade inflation, well, this chart should be interesting, even though state-mandated testing ebbs and flows probably explain some of it. You never hear about “test score inflation, do you?)

Elementary, secondary, and higher education in the US has turned into a sorting machine for industry, assisted by the standardized testing industry, and the backlash against liberal arts, the campaign fighting the boogeyman of “college for all” (which never meant everyone should go to college, it might be mentioned), and the defunding of public education are all part of the same equation that started with Ronald Reagan in the mid-1960s. That’s just one of the reasons why tests are so popular: They are easy to administer and grade, and damn it, we get a number. And numbers don’t lie.

In fact, the whole assumption among the critics of “grade inflation” is that the purpose of secondary education is to sort students for the Highly Rejectives. Read that again, and ask yourself if that idea ever occurred to you, or if you believe it’s a valid purpose of public education.

It is my personal opinion that standardized tests are popular for college admission at the Highly Rejectives because they are in the business of rationing slots, and want to be as certain as they can that these slots are allocated to students who can take advantage of the educational environment, notwithstanding the absurdist proposition that somehow the tests are academic qualifications. The SAT and ACT have very low rates of false positives: That is, if you score a 1580 or a 35, it’s extraordinarily unlikely you lucked out. You must have something going on upstairs, even if it’s just a good test prep program. It’s also relevant to the discussion, I think, that Robert Sternberg, with degrees from Yale and Stanford that certainly suggest he was adept at standardized tests, says some faculty feel they owe something to the SAT, and their blind allegiance to it is payback.

I won’t even touch the research of Joseph Soares, who uncovered a memo in the Yale archives from the 1960’s that suggested, sure, the SAT didn’t add much to the prediction equation, but it did a fine job of keeping institutional financial aid expenses down by helping identify the wealthiest students in the applicant pool.

I went to what was considered the college prep high school in my city, and no one in our class of 460 graduates in 1977 had taken calculus in high school (or maybe it’s safer to say calculus was not a part of the course offerings in my high school.) It was not expected of high school seniors applying to college, and in fact, I’ve met many math professors who say they wish students spent more time building foundational math skills in high school so they could learn–actually learn–calculus and mathematics in college. Many of those same professors bemoan the “preparation” of students who take Calculus in high school, suggesting the depth of their understanding is too shallow to prepare them for additional classes. This sounds a lot like what the UC faculty are saying (although Calculus and Freshman composition are two courses that seem to be way down on the list of attractive courses, if we’re all being honest here.)

Alas, no one I know would suggest you have a good shot at admission to the Highly Rejectives these days without AP or some other Calculus course. So we end up insisting on things we wish didn’t exist in the first place.

The funniest thing in all this seems to be this news story from Berkeley. The version that’s current on the web is different than the original one, which you can read here, via the Wayback Machine of the Internet Archive.

I’ll save you the time: Whereas the current article says, “Yet, in 2016, campuswide data showed that a number of students starting Calculus 1 tested below entry level for the course. Some even needed refreshers on fractions and exponents,” the original said, “In 2016, the Department of Mathematics decided to assess incoming students’ readiness. They found that around 40 percent of students starting Calculus 1 tested below entry level for the course. Some even needed refreshers on fractions and exponents.” The emphasis in both is mine, and the folks at Berkeley are apparently double-checking the numbers; they could not provide a report citation for me when I checked.

But that original statistic is interesting, because, of course, in 2016 (and for at least 50 years before and several years after), the UCs required the SAT for admission. Here is a summary of the IPEDS-reported standardized test scores at Berkeley for the the years in the middle of that decade, showing the 75th (orange) and 25th (blue) scores.

For those who aren’t used to looking at numbers like this, the short answer is you had to smoke the SAT math section to get into Berkeley at this time. And yet, “some number” or maybe “40 percent” of students placed into Calculus had challenges with 6th, 7th, and 8th grade math concepts. It raises several questions for me:

• Did anyone consider any other possible explanations for this current iteration of the problem? Like, I don’t know, the fact that these students were in middle school when COVID hit and everyone was learning online even though no one was prepared for it? Are experienced elementary and high school math teachers leaving the profession? Are class sizes in California getting larger? Is state funding for primary and secondary education falling? Or is the lack of the SAT just a facile, post hoc, ergo propter hoc solution?
• Why isn’t the placement test catching this, allowing students to be placed in a course they’re not ready for? Is this a matter of faculty thinking that throwing students into the deep end of the pool is good for them? The report says, but only near the end, “We see that the pool of students who did not pass Calculus I is overwhelmingly dominated by those who entered with severe deficits.” That needs some explanation and some accountability, I think: Why were students with “severe deficits” allowed to go into a class they were not ready for? Would you put these students into Differential Equations if they weren’t ready? This is really confusing.
• Why weren’t the UC System and the other Highly Rejectives telling College Board to fix the SAT? Other than the time Richard Atkinson did just that, and found out about the College Board PR Machine, I mean.
• What happened to the UC-specific test that some members of the faculty suggested during the debate about going test-free? Maybe that would be better, especially if it’s not multiple choice?
• Finally, is a multiple choice test really a good way to measure achievement in mathematics?

To that last point, Frederick Kelly, a doctoral student at Kansas State Teachers College in 1914 who is credited with inventing the multiple choice test, created the assessment mostly to find people at the bottom of the bell curve in an America that was slowly becoming more industrialized; that is, those who could not read or follow the most simple of instructions. If you were told to circle the most valuable animal to have on a farm and you underlined “cow,” for instance, you got no credit. He famously later called multiple choice tests tools to “measure lower order thinking skills,” and, while president at the University of Idaho, called for them to be abandoned because they were “too crude to be used.” (image below taken from this page)

This being America, of course, the multiple choice test (related to, but not the same as a norm-referenced test) caught on because it was quick to grade and evaluate, and, people thought, “precise.” Sternberg, in the article above, also mentions the “illusion of precision” that the tests generate.

I once asked a test prep person about the one thing he’d tell someone planning to sit for the SAT, and without hesitating, he said, “Plug and Play,” the idea that it’s a waste of time to try to solve the some problems on the test because it’s often faster and easier to just plug the answers into the equation to see which one is right. So again, this type of behavior can make you look highly accomplished; it probably doesn’t necessarily mean you are good at mathematics. Learning how to beat the test is probably unlikely to make you better at the type of math faculty expect you to do, even if it does make you better at the skill The College Board tests. (And, of course, it’s important to remember that students who have access to this advice probably come from families with more resources. Talent is everywhere; opportunity is not.)

Just like all poodles are dogs, but not all dogs are poodles, all highly accomplished math students are high testers, but not all high testers are highly accomplished math students.

I’ve talked to enough math professors to extrapolate that what they hope for is students who understand mathematics rather than students who have been trained just to get the “right” answer. If I can put words in their mouths, they want students who solve the problem and get the solution, not those who back-solve to get the right answer. Unfortunately, the SAT won’t tell you who’s who. And, frankly, they don’t care, even though about a third of the new SAT is not multiple choice.

To that end, excellent teachers like Ralph Pantozzi in New Jersey teach students about mathematics, not just about “getting the right answer.” In a world where students are pressured to accelerate their math education and perform better on standardized tests to get into a highly rejective, Ralph’s style is a bit different. But I suspect students in his class who can think about math don’t have problems with fractions, even if it’s not a topic they covered. And they might be able to school that Northwestern MBA, too.

Again, I believe the problem is real, but we cannot get a sense of the scope, because as far as I can tell, we don’t know what percentage of first-year students came through AP Calculus (either AB or the more rigorous BC). I’m assuming many of them may be exempt from taking Calculus in college in the first place, so the problem lies with those who have to start it in college, which may be a relatively small subset. (Whether I’m right or wrong, this would be an interesting statistic to see.) We just don’t know.

The challenge with math in high school and college is not new, and I don’t see that the faculty have tied these challenges to a lack of SAT scores. In fact, I think the proliferation of multiple choice math tests, and our (forced) focus on standardized tests actually cause student to think about math in ways that keep students from succeeding.

And as history shows us, even having the SAT, and only admitting students with high scores, doesn’t eliminate the challenge. It just makes the solution even more complex.

And there is nothing a mathematician loves more than an elegant solution to a complex problem.