academia | advice | alcohol | American Indians | architecture | art | artificial intelligence | Barnard | best | biography | bitcoin | blogging | broken umbrellas | candide | censorship | children's books | Columbia | comics | consciousness | cooking | crime | criticism | dance | data analysis | design | dishonesty | economics | education | energy | epistemology | error correction | essays | family | fashion | finance | food | foreign policy | futurism | games | gender | Georgia | health | history | inspiration | intellectual property | Israel | journalism | Judaism | labor | language | law | leadership | letters | literature | management | marketing | memoir | movies | music | mystery | mythology | New Mexico | New York | parenting | philosophy | photography | podcast | poetry | politics | prediction | product | productivity | programming | psychology | public transportation | publishing | puzzles | race | reading | recommendation | religion | reputation | review | RSI | Russia | sci-fi | science | sex | short stories | social justice | social media | sports | startups | statistics | teaching | technology | Texas | theater | translation | travel | trivia | tv | typography | unreliable narrators | video | video games | violence | war | weather | wordplay | writing

Monday, March 19, 2007

Ranking professors and everything in sight; or, the fourth-best thing I've written this month

On Gary Becker and Richard Posner's blog, they have been discussing rankings and anonymous ratings. This pricked my ear because of Culpa, Columbia's course review site, which I built with my friend Ashran while an undergrad. When we launched our version of Culpa (it was then a defunct review site that had few reviews but lots of attitude), we weren't sure if students would be fair and honest in their reviews. More than 10,000 reviews later, we still aren't sure, and we've had to pull some offensive reviews, but on the whole I think it adds more than it takes away.

But there's one troubling side-effect: many reviews applaud teachers for being easy graders and, worse, for being entertaining rather than substantial. Not all review do this, and there are much-loved profs with rave reviews who are notorious stiffs with the curve. But there are few reviews that fault profs for demanding of students too little learning. And it hurts to know that at least one professor friend of mine is irked by the site, though she is reviewed favorably.

Becker writes of this type of unintended side effect:

Perhaps the most serious problem with rankings is that institutions "game the measure". So if the ratio of admissions to acceptances were used, then as Posner indicates, schools might tend to admit applicants who do not have good alternatives. If hospitals are ranked partly by the death rate among patients, then hospitals have an incentive to shy away from admitting terminally ill patients, or those with difficult-to cure conditions.
And if professors are rated by how pleased students are with their grades at the end of the semester, they have an incentive not only to be lenient graders, not only to demand less work of their students, but also to be less intellectually challenging, less confrontative and less eccentric, all of which can translate into negative reviews.

Life is full of initially off-putting experiences that we come to appreciate. A by-product of giving students information about professors is that it gets in the way of surprising and serendipitous encounters. On the other hand, it allows students who crave inspiring teachers to find ones they might never otherwise come across.

And there's always the possibility that a professor might improve after reading the feedback. Becker:
Yet schools and other organizations respond to their ranking position not only by gaming the measure, but also by improving what they provide. In this way, some business schools and colleges ranked low in the amenities and other characteristics of the learning experience provided students have responded by improving physical facilities and the guidance offered to students, reducing class size, and increasing networking.
Becker, in his concern about gaming the rankings systems, doesn't realize what a bigger problem it is that rankings systems so often measure the wrong thing. On Culpa, the satisfaction of the mouthiest student must stand in for the quality of a lecturer; I wonder sometimes if these elements are not negatively correlated.

Meanwhile, the Wealth Project blog, by a clandestine friend of mine, asks regarding college rankings, is the value of an Ivy League education overrated?
...a fair number of the Ivy Leaguers also point out that the undergraduate experience at an Ivy League school is not all about intense intellectual debate and curiosity, and you may be surprised by what goes on within the student body - everything from stereotypical StateU-type binge drinking and partying to rampant cheating and plagiarism to a more than uncommon intellectual non-curiosity. So, while certain synergies come from putting a bunch of smart people together, does a driven high achiever excel because of the environment or is that same person going to excel no matter where he or she goes to school?
The author isn't overstating the case. I majored in history and computer science, and while I seldom heard of plagiarism in history (Alice? Do you hear scuttlebutt among the English professors about this at Columbia?), cheating was rampant in engineering. Friends joked that the major "industrial engineering and operations research", or IEOR, was actually a major in cheating. Intellectual non-curiosity ruled even a large number of ostensible intellectuals, including quite a few at the Columbia Spectator, where Alice and I met (she edited me and many others, and she'd have demanded heavy cuts in this piece). And while Ivy academics is harder for some and easier for others, I think that even if you never read the books, it would be hard to major in history, attend the lectures and turn in gramatically-correct assignments on time and earn less than a B average.

Meanwhile, it's clearly possible to be smart and successful without going to a top school. Look at the heads of most businesses and organizations you care about; few went to Ivy League schools. Even most Ivy League professors didn't go to Ivies as undergraduates. So that's another problem with rankings: ordering imposes a sense of relative value that may not apply. Surely a Culpa reader can find a few professors who sound right up their alley, but I wonder if graduating students would rate their professors higher today, under the Culpa regime, than they would have ten years ago.

It might be that ratings and rankings are more about our instinct to organize--like John Cusack in High Fidelity, I would happily make top ten lists of things in a corner all day--than they are about the underlying quality of the ordered things themselves. I'm not kidding about the title of this post; the three better entries are The bells and barbarus of Seville, Internet comes to town, Internet leaves town, and What I'd pay for a mind free of DC subway minutiae, and I loved choosing them. Is my desire to select, rank and order proto-autistic? Or is it as powerful in most people?

Labels: , , , , , ,

Blogger Alice on Fri Mar 23, 05:28:00 PM:
Jaime pointed out the other day that the posters on review sites such as CULPA tend to be writing to impress one another, not to add to a general ethic of recommendation or appraisal. He also sent this parody of what Socrates' course evaluations would have looked like:

I guess I'm more optimistic about student work than those who decry the current state of intellectual debate on college campuses. I have two reactions to those claims: 1) certainly there are anti-intellectual people out there and instances of bad behavior or inflated senses of entitlement are disheartening--and familiar. 2) Nevertheless, what if, for every article about student anti-intellectualism--which tend to sound very similar and always hearken back to some unspecified past when everyone was alert and engaged and brilliant--someone wrote another article about something inspiring that happened in the classroom? I guess that's what the Gold Stars were for on CULPA. I don't mean this recommendation in a Pollyanna-ish way, or even in a Heidi Julavits anti-snark book review kind of way: people do good work in college classes sometimes, and we should come up with new ways to generate excitement so we can see what works. I'm much more interested to hear someone be excited about their work than someone give a litany of complaints I've heard before. It's something like that essay you like so much, Ben, about making your own projects when you're bored in school.
Blogger Ben on Sat May 19, 05:13:00 PM:
There's truth to Jaime's observation, but I think the reality of the matter is more mundane, as is so often the case with pronouncements of doom (or at least, valuelessness), and as is the case, as you point out, Alice, with my own worries about the dearth of intellectualism on campuses.

(Whew! If being an intellectual means writing sentences like that, no wonder it's not a more popular pastime.)

Thankfully, the median review on CULPA is pretty straightforward -- prof X was pretty good/bad for reason Y. But I am proud of the overall tone of the site, which has stayed surprisingly close to the tone that Alex Feerst and I set with our 25 or so reviews each when CULPA started. I like to think it's irreverent but not arrogant, and informative but not neutral.

Ironically, I think the joking reviews of Socrates are actually pretty useful. There's a lot of information you can draw from the contrasting reviews. Take, for example, "I was amazed at how he could take just about any argument and prove it wrong" and "He makes students feel bad by criticizing them all the time. He pretends like he's teaching them, but he's really ramming his ideas down student's throtes." It's pretty clear to me that I would more agree with the first reviewer than the second.

I think many folks are so enamored of sharing their opinions that they don't notice how often their less attractive qualities show. The reviewer who says "I spend serious money for my education and I need something I can use in the real world" may be petty, but that's useful information to be able to draw from a review. Building a model of the author's intention while you read their text--that's pomo thinking in action!