One of the very first posts I put on my other blog–the one focused on higher education data–was about the Claremont McKenna test score reporting scandal.  You can take a look at it here if you’d like a summary of the data.  At the time, I thought the difference between the actual scores (which many colleges would love to be able to report) and the reported scores (which even more would want to report) was pretty tiny.  Hardly worth it.

But I think one of the reasons people obsess over things like average test scores and admission rates is precisely because they have something Robert Sternberg has called, “The illusion of precision.”  This gets exacerbated when, in the case of CMC, for instance, the perception that tiny changes in the numbers can cause you to fall out of the top 10 into the god-forsaken land of 11 or 12.

It’s just one of the things that adds to confusion and, probably, stress, among everyone associated with college admissions.  That includes parents, students, admissions officers, and high school and independent counselors.

What’s really interesting, though, is that we’re doing it all wrong, at least in the case of test scores.  This is not a post about the value of standardized testing in admissions; I’ve already expressed my opinion about that.  Instead, this is a little bit about numbers, and the types of numbers used in research as variables. You may remember these as nominal, ordinal, interval, and ratio.

A nominal variable is not really a number: It just looks like one.  Mickey Mantle’s 7, or ZIP Code 90210.  You can’t really do anything with these “numbers.”  For instance, if the Cubs infielders word 10, 11, 18, and 14 (Santo, Kessinger, Beckert, and Banks), you can’t really say the average infielder wore number 13.25. And if you had a million records in a census file, trying to average ZIP codes might give you a number, but it wouldn’t mean anything: ZIP codes are just labels that look like numbers.

Then, there are ordinal numbers, used to rank things.  “The Cubs finished 4th in the Division.”  “Mary was the (1st) tallest girl in the class.”

 

People get into trouble all the time using ordinal numbers, because there is some sense to them.  A team that finishes first is better than one who finishes third.  In a room of 19 men, the tallest man in the room is taller than the fourth-tallest man in the room.  But if you try to average 1st, 2nd, 3rd….19th, you’ll always get 10.  And it doesn’t matter if you have the Chicago Bulls in one room and the Wizard of Oz Munchkins in another.  The average of ranks will always be 10 in a room of 19.

This also happens with survey data.  Suppose you ask two questions, and ask people to respond on a scale of 1-to-5, where 1 means “not at all,” and 5 means “a whole lot.” :

• How much do you like cupcakes?
• How much do you like sardines?

You might find that the average response is 4.8 for cupcakes and 2.4 for sardines.  But despite those results, it does not mean that people like cupcakes twice as much as sardines. The numbers are just ordinal, essentially meaningless for precise comparisons (it is safe to say, however, that people would, in this example, like cupcakes more than sardines.  I know I do.)

Interval variables and ratio variables are more like the numbers you think of all the time.  Getting four hits in a baseball game means you had four times the number of hits of the guy who got one; it also means you have three more hits than he got.  Buying six bananas means you bought twice as many as the person who bought three.

 

Here’s the shocker, though: People average test scores all the time, even though they’re ordinal values.

Go to this link and look at the table.  These are percentile scores for each possible composite score.  You’ll see that a 30 represents a score in the 95th percentile, which means 95% of all test takers scored at or below 30.  And that a 20 represents a score in the 49th percentile, and so on.  If you average the 30 and the 20, you get 25, which is the 79th percentile, not the average of the 95th and the 49th percentile (which would be 72.5).

The point, of course, is that a 30 has a higher score than a 20, but that it’s meaningless to say it’s “10 better.”  And moving your score from a 19 to a 29 is a much bigger percentile gain than from a 25 to a 35.  They’re numbers we expect to make sense as numbers, but they don’t.

What does all this mean? In short, colleges and students are focused on numbers that are not nearly as meaningful as we might think they are.  The same can be said of admission rates, which can be manipulated in a variety of ways, so much so that people obsess over differences that are essentially meaningless.

Could we have a simple solution? Maybe (assuming you can’t wave a magic wand to make the crazy go away.)

What if test score ranges and admit rates were renamed and grouped into categories?  We could name the categories by letter, animal, color, or anything, even a number if we wanted to.  So Harvard is now a Green on Test Scores and an A on selectivity; DePaul is now a Blue on test scores and a G on selectivity; or Lafayette is a Purple D; Columbia College (there are a lot of Columbia Colleges, so no one’s going to get mad at me here) is a Yellow M.

We’d still have a hierarchy, of course, because that isn’t going away any time soon.  And colleges who are right on the cusp of moving up or down are probably still going to focus on attempting to move up or avoiding moving down.  Of course, the obsessive will always be with us, and will want to know whether your admission rate was 9.8% or 10.1%.  But possibly, some of the bad stuff will go away, or least begin to be less important in the larger educational context.

But if we change some of our language, if we admit that these numbers are not as precise as they seem, we might make a small step toward a more rational, reasonable, discussion of ranking colleges and universities.

What do you think?