I won­der some days what we think we’re test­ing in acad­e­mia. This is not merely the usual harp­ing about giv­ing or get­ting grades, nor merely another thing about whether test­ing instru­ments are valid, and nor what sorts of gen­er­al­iza­tion they can inform. Not even revis­it­ing my hare-​​brained ideas on multi-​​objective grading.

Just what it is we actu­ally test, and what we think we’re testing.

I’ve had an impor­tant les­son slammed painfully home this week. I’m one year into a entirely new dis­ci­pline (indus­trial engi­neer­ing), after 20+ years’ com­fort­able suc­cess in another cou­ple (the­o­ret­i­cal biol­ogy, and what in hind­sight was appar­ently “not-​​industrial” engi­neer­ing): all my plan­ning esti­mates are off by a fac­tor of 4x or 5x. I suck at deadlines.

But I’ve sucked at dead­lines, like, for­ever. A lot of peo­ple in the Uni­ver­sity do. Look at the aca­d­e­mic blogs, and much of what you’ll read is com­plaints about “Oh, my, I have so many meet­ings!” or, “Can you believe my stu­dents can’t do every­thing I tell them to?!” I’m still con­vinced that the demo­graphic who sur­vive to become pro­fes­sors are exactly those who’re best at post­pon­ing things to the knife-​​edge of the last minute, with­out push­ing them too often or too far over the edge into reg­u­lar tar­di­ness. And like those folks, I’ve never been very tardy. Well, as such. Sure there are some papers I’ve wished I could give another edit, some thoughts left for fol­lowup papers, some deeply abbre­vi­ated dis­cus­sions. But gen­er­ally I get the job done.

Because I have a strong sense of the effort required. I know the stuff: domain knowl­edge takes lit­tle or no time for me to dredge up. Regard­less of the domain. I was, after all, employed as a Know-​​it-​​all. And along with “all”, I also know how to write. So I’ve got­ten pretty good through the years at weigh­ing these strengths against the rel­a­tive com­plex­ity of the tasks I’m assigned. Yeah sure, that’s a kind of lazi­ness. This is the point where I can remind you that all the best engi­neers are intrin­si­cally lazy, that it’s the dri­ving force in the field.

But this year I suck. All my esti­mates are off, and all my work suf­fers because of it. Not because of this new “yes, really math­e­mat­ics means indus­trial” engi­neer­ing domain knowl­edge I need to learn; maybe it takes me twice the time to dredge up some long-​​ignored fact about lin­ear alge­bra or matrix mul­ti­pli­ca­tion, but what’s an extra ten min­utes? No, what’s off is my abil­ity to communicate.

I spend all my time edit­ing. I spend extra time,not my own, edit­ing. And re-​​editing. Not just “Did that sen­tence make sense?” or “Do I under­stand that word?” but also “If I use this com­mon­place term, will the reader know what I’m talk­ing about?” or “To me this is the first obvi­ous step. If I were them, would it be?”

The words make sense, but maybe not to the reader: my use of “fit­ness land­scape” and “auto­cor­re­la­tion struc­ture” come to mind as recent exam­ples that have proven I’m from Mars. Or I under­stand the word, but not in the same way as the reader: “com­plex sys­tems” and “data min­ing” come to mind. Obvi­ous rules of thumb and tools I always use, unthink­ingly, in every case—like ran­dom sam­pling the search set of an opti­miza­tion prob­lem, or sav­ing a record of every solu­tion tested so I can watch progress of an ongo­ing search.… well, I may as well have invoked Marx or done a lit­tle inter­pre­tive dance. And at the same time, the con­sumers of my prod­ucts are watch­ing for cer­tain sig­nif­i­cant pass­words, and they’re just not com­ing. Because I’m not see­ing the cues, the raised eye­brows, the head tilted, the mut­tered “…and—?”

So every­thing I do needs to be edited. Four times, five times. I’ve learned the hard way that it may as well be blank, as handed in unin­ter­pretable. Indeed, it’s because I’ve so deeply mis­un­der­es­ti­mated the amount of social edit­ing required, that yes­ter­day I was forced to turn in a paper—one that I was orig­i­nally very proud of—mostly blank.

Crip­pled. What am I going to do, fac­ing a strict dead­line and already hav­ing spent two weeks and 30 hours straight work­ing on it, leave a skele­ton of head­ers for all the beau­ti­ful things I have in my head but left myself no time to com­mu­ni­cate? Even though I’d eas­ily pass them along in a face-​​to-​​face con­ver­sa­tion, or writ­ing to an old col­league? All it would take is a few moments and a few choice turns of famil­iar phrase. Which I can’t use; when I do use them, when I write a com­plete and cogent account of some inter­est­ing things I’ve done, still the cues are not there, and what I get is another “…and–?”

So my final report for this class was a piece of embar­rass­ing crap. If I were given it, I’d throw it away half-​​way through. But then, I didn’t even get halfway through writ­ing it; who knows, maybe that could play out in my favor.

As I heard Dr. Dan Got­tlieb say on the radio, there’s noth­ing more painful than see­ing your­self help­less to aid some­body you love. I know all about that. It’s also true, though to a lesser extent, about something you love doing.

Because, I sup­pose, the stuff is you.

Which-​​all sends me down the dispir­it­ing spi­ral, and starts me think­ing about the nature of grad­u­ate pedagogy.

When a stu­dent is being tested, to suc­ceed they need to (1) rec­og­nize the explicit and tacit expec­ta­tions of the ques­tioner, (2) obtain (recon­struct or recall) the domain knowl­edge and raw facts nec­es­sary to answer the explicit ques­tion, (3) con­struct a plan for a cogent response, and then (4) exe­cute and con­vey that response with the appro­pri­ate for­mal­ism and idioms. This is true in edu­ca­tion from the ele­men­tary level to the pro­fes­so­r­ial job talk: Some­body poses a chal­lenge to you, and your response will be judged not merely on its fac­tual con­tent but also on the degree to which it com­mu­ni­cates your mem­ber­ship in a joint com­mu­nity of practice.

In my new domain with its weird idioms and cul­tural assump­tions, my first response to hear­ing many ped­a­gogic utter­ances is typ­i­cally a mix of

1. “Oh, cool, I’ve never thought of that!”
2. “Uh huh, sure. Get to the point.”
3. “Why would any­body in their right minds con­sider that reasonable?”

Think­ing back to my days as an instruc­tor, and talk­ing after the fact with my pro­fes­sors here, surely they have the same set of responses to my (and other stu­dents’) work prod­uct. What dif­fers from case to case is the pro­por­tion of the time spent in the first and last ones. We’re all brim­ming over with domain knowl­edge; what dif­fer­en­ti­ates us is dis­ci­pli­nary frame­work, cul­ture, confidence.

On the sub­ject of con­fi­dence, briefly: it gets far too much respect in acad­e­mia. Unlike knowl­edge or expe­ri­ence, con­fi­dence has a dan­ger­ous Goldilocks-​​curve type of non­lin­ear­ity. Nei­ther too lit­tle nor too much is a good thing. Treat con­fi­dence warily.

The goal of higher-​​level edu­ca­tion is to build con­fi­dence in the instruc­tor that the stu­dent is spend­ing increas­ingly more of the time in the sec­ond phase of “Uh huh, sure. Get to the point,” and less in “Oh, cool, I’ve never thought of that!” The last one is always unde­sir­able, both in the stu­dent and in the instruc­tor, except in the rare occa­sions when somebody’s out­side their discipline.

Indeed, I sus­pect the proper way of think­ing of a dis­ci­pline is like a bio­log­i­cal species, which is dif­fer­en­ti­ated by repro­duc­tive bar­ri­ers. In acad­e­mia, it’s not about the fuck­ing, but about think­ing the other peo­ple are fools. If they’re time-​​wasting fools, they’re in another dis­ci­pline. (And if they’re in another discipline.…)

Yet I find I often say that bit about, “Why would any­body in their right minds con­sider that rea­son­able?” My instruc­tors report they say it a lot about me. Not sur­pris­ingly, this leads to increased pres­sure in test­ing situations.

The point being: We often seem to for­get how impor­tant issues of prag­mat­ics and cul­ture are in the ped­a­gogic cycle. Instruc­tors know the damned answer. The older the stu­dent is, the more con­fi­dence the instruc­tor should have that the stu­dent also knows the damned answer. What an advancedin­struc­tor should be doing is look­ing for cues that some­where in that stranger’s head is a cul­tural frame­work and body of knowl­edge that together will suf­fice in the future. Maybe (for some weird rea­son we shouldn’t dwell upon) that grad­u­ate stu­dent won’t end up being a pro­fes­sor… I sup­pose… but in the mean­time, a men­tor should be see­ing numer­ous cues that the stu­dent will soon be able to talk the talk, even out­side this pro­tected, nur­tur­ing creche that is grad­u­ate school.

So, two objec­tives: knowl­edge and communication.

I’m think­ing back to years spent grad­ing intro biol­ogy stuff—and for that mat­ter some of the man­u­scripts I’m sent for peer review—and reflect­ing on the “mem­ory dump” pat­tern that we all know and love. The canon­i­cal “mem­ory dump” appears on the face of it as a blind list­ing of facts, and we’re tempted to see it as a clear demon­stra­tion of fail­ure to syn­the­size knowl­edge. I won­der, though, if the author of a “mem­ory dump” might be feel­ing chal­lenged in the com­mu­nica­tive aspects of the process. In con­ver­sa­tions, where they can read cues and adjust the flow of talk on the fly, they never get a chance to spew ran­dom facts. [Well, most don’t.] They come over as “smarter”. Thus it is con­fus­ing when we encounter a stupid-​​sounding list, an essay that dances around the sub­ject and never gets to the point.

When we test knowl­edge, we are also test­ing accul­tur­a­tion. When we embark on an assigned task, to suc­ceed we need not merely know, but demon­strate the knowl­edge, and for that mat­ter show we have the abil­ity to do so with facility.

And when we are given a test, an essay, a report, a man­u­script to review, and it’s full of gaps and holes, what then should we infer?

What do we test, exactly?