I had a boss who once talked about “sending ideas out to the universe.” It’s poppycock, of course, if for no other reason than the universe doesn’t care. But that doesn’t mean it won’t work sometimes, I suppose. Yesterday, I tweeted this:
And this morning, the Twitter-verse unknowingly responded with this:
Which gave me–as only Twitter can–the ability to legally eavesdrop on three colleagues: John Lawlor, Deb Maue, and Chris Lydon. I know exactly what the conversation means: Measuring the quality of the educational products of a college by measuring the freshman class is like measuring how good a basketball team is by evaluating the average height of its players. It might work, on occasion, but that doesn’t mean it’s right. However, I responded to all three, suggesting that in fact graduation rates were inputs, in a weird sort of way.
That probably would have been the end of it, but it got connected to something I frequently see on my walk between Union Station and my office, about a mile each way. And it is especially common as it gets colder and people are eager to get indoors.
Here’s what happens: You’re standing at a corner with twenty other people waiting for the light to change, and someone notices no traffic coming, so they start to cross against the light. Others follow, even though the sign is clearly “Do Not Walk.” People come from the other direction, and, seeing the throngs of people crossing, assume the light says “Walk” and proceed to do so. But by this time, a car or truck or two has come barreling down the street, and I almost witness a pedestrian hit by a vehicle. What happened? It’s clear that the pedestrians have equated large groups of people crossing the street with a “Walk” sign. And usually, that signal is right; occasionally it’s not, and trouble ensues.
Thus, a blog post. And the end of Idea Block. And the head knocking resumes.
If we attempt to measure the value of an institution by its outputs (graduation rates) we are really just confusing signals, like pedestrians on Jackson Boulevard in Chicago on a cold morning. We’ve become so used to the mix up of inputs and outputs that we forget to look at the real signals; because in a way, inputs and outputs are the same things. Don’t believe me?
Take a look at this screen shot of IPEDS Data visualized. It shows test scores on the x-axis, and graduation rates on the y-axis. Note how they line up?
That alone would be enough to make the point, I think. But play with this visualization by clicking here. See if you can find any type of institution where it doesn’t hold: Urban, rural; any region; any religious affiliation; public or private. And notice two more things: The color of the dot (which represents a single institution) shows the percentage of freshmen with a Pell Grant. And the size of the dot shows the rejection rate: More selective institutions have bigger dots. Then, just for fun, filter to institutions that don’t’ rely on student tuition to manage the budget. Just pull the top slider down to 50%. See who’s left.
Then ask yourself: Is a high graduation rate a function of what goes on inside the institution? Or could it be a function of selectivity, test scores and family income (which are pretty much the same thing) and resources?
Would love to hear what you think.
Jon Boeckenstedt's Admissions Weblog 
John:
Since I am writing in a “school zone,” the light is blinking and I am proceeding with caution.
Love your visualization graphs and the points you make. I don’t know if there is a simple conclusion or rather to your point, a generalization, because so much depends on a variety of variables.
All in all, I do believe in two key ideas: First of all, finding the right college is about FIT. I once met a young woman who was accepted to Harvard and MIT, but she chose New Mexico Tech. She told me that she wouldn’t have fit in and probably wouldn’t have graduated from either of those fine schools. She graduated from NMT.
So my second point is that I believe outcomes (grad rates, post-grad success) are meaningful metrics BUT they must also take into account FIT. If a student finds a group of schools that represent a great fit, then I would strongly encourage that student to look at outcomes as a variable in making a value judgement and selection decision.
Green light!
LikeLike
Jon,
Let’s reframe the discussion using a different analogy.
We would think it silly if the medical industry employed a single metric–say rate of survival–across all its offerings and then evaluated all providers against that single metric (as some hasten to do with colleges and graduation rates). Say I need a heart replacement or I’m seeking treatment for stage-3 cancer. I’d be a fool to use only a single, generic survival rate metric to compare the efficacy of treatment at my neighborhood walk-in family clinic to that of the critical cardiac unit at the Cleveland Clinic or that of a specialized cancer center. As a patient making a decision about where to receive treatment–or an evaluator of the efficacy of treatment–I would want to use survival rate (or some other metric or set of metrics) to compare 1) hospitals or providers of similar purpose that 2) handle patients with a condition similar to mine.
Of course, in a medical industry analogy, we’re perfectly fine with differentiating among “severity” of condition as related to “likelihood” to survive. In higher education, while we “insiders” know the factor constructs that go into models of retention, persistence, or graduation, we’re never going to publicly label a student as having low likelihood to survive (we will go as far as using terminology like “first-generation”). And a college is sure not going to promote itself as a place for “late-stage” survivors–not a good brand platform.
So, is a high graduation rate (you ask) a function of what goes on inside the institution or a function of other inputs? Just like medical treatment, it’s a combination of: 1) the whole condition of the patient entering treatment; 2) the skillfulness of diagnosis, intervention, and resources applied to treatment; and 3) the patient’s cooperation with treatment. (And, in both higher education and medical care, there is the matter of the degree of access of the student/patient to good and proper education/treatment.)
Maybe a better metric would be rate of intellectual/behavioral change over time attended rather than graduation rate. I’ve always thought that colleges that take on some of the biggest transformational challenges in students should get some recognition for their work, in relationship to the schools that preserve an intellectual status quo. Still a progressive after all these years.
At any rate, I thought this post would give me a chance to reconnect rather than just lurk on your blog. Cheers.
LikeLike
Hi Ron. It’s nice to hear from you!
And, as I’ve become accustomed to saying, you are, of course, absolutely right. My point in writing was to talk about the tendency to make easy connections between cause and effect. Someone else I know–another person who is almost always absolutely right–is fond of saying that making a silk purse out of silk is not easy, but it’s also not as difficult as the alternative.
LikeLike