I wonder some days what we think we’re testing in academia. This is not merely the usual harping about giving or getting grades, nor merely another thing about whether testing instruments are valid, and nor what sorts of generalization they can inform. Not even revisiting my hare-brained ideas on multi-objective grading.

Just what it is we actually test, and what we think we’re testing.

I’ve had an important lesson slammed painfully home this week. I’m one year into a entirely new discipline (industrial engineering), after 20+ years’ comfortable success in another couple (theoretical biology, and what in hindsight was apparently “not-industrial” engineering): all my planning estimates are off by a factor of 4x or 5x. I suck at deadlines.

But I’ve sucked at deadlines, like, forever. A lot of people in the University do. Look at the academic blogs, and much of what you’ll read is complaints about “Oh, my, I have so many meetings!” or, “Can you believe my students can’t do everything I tell them to?!” I’m still convinced that the demographic who survive to become professors are exactly those who’re best at postponing things to the knife-edge of the last minute, without pushing them too often or too far over the edge into regular tardiness. And like those folks, I’ve never been very tardy. Well, as such. Sure there are some papers I’ve wished I could give another edit, some thoughts left for followup papers, some deeply abbreviated discussions. But generally I get the job done.

Because I have a strong sense of the effort required. I know the stuff: domain knowledge takes little or no time for me to dredge up. Regardless of the domain. I was, after all, employed as a Know-it-all. And along with “all”, I also know how to write. So I’ve gotten pretty good through the years at weighing these strengths against the relative complexity of the tasks I’m assigned. Yeah sure, that’s a kind of laziness. This is the point where I can remind you that all the best engineers are intrinsically lazy, that it’s the driving force in the field.

But this year I suck. All my estimates are off, and all my work suffers because of it. Not because of this new “yes, really mathematics means industrial” engineering domain knowledge I need to learn; maybe it takes me twice the time to dredge up some long-ignored fact about linear algebra or matrix multiplication, but what’s an extra ten minutes? No, what’s off is my ability to communicate.

I spend all my time editing. I spend extra time,not my own, editing. And re-editing. Not just “Did that sentence make sense?” or “Do I understand that word?” but also “If I use this commonplace term, will the reader know what I’m talking about?” or “To me this is the first obvious step. If I were them, would it be?”

The words make sense, but maybe not to the reader: my use of “fitness landscape” and “autocorrelation structure” come to mind as recent examples that have proven I’m from Mars. Or I understand the word, but not in the same way as the reader: “complex systems” and “data mining” come to mind. Obvious rules of thumb and tools I always use, unthinkingly, in every case—like random sampling the search set of an optimization problem, or saving a record of every solution tested so I can watch progress of an ongoing search…. well, I may as well have invoked Marx or done a little interpretive dance. And at the same time, the consumers of my products are watching for certain significant passwords, and they’re just not coming. Because I’m not seeing the cues, the raised eyebrows, the head tilted, the muttered “…and—?”

So everything I do needs to be edited. Four times, five times. I’ve learned the hard way that it may as well be blank, as handed in uninterpretable. Indeed, it’s because I’ve so deeply misunderestimated the amount of social editing required, that yesterday I was forced to turn in a paper—one that I was originally very proud of—mostly blank.

Crippled. What am I going to do, facing a strict deadline and already having spent two weeks and 30 hours straight working on it, leave a skeleton of headers for all the beautiful things I have in my head but left myself no time to communicate? Even though I’d easily pass them along in a face-to-face conversation, or writing to an old colleague? All it would take is a few moments and a few choice turns of familiar phrase. Which I can’t use; when I do use them, when I write a complete and cogent account of some interesting things I’ve done, still the cues are not there, and what I get is another “…and–?”

So my final report for this class was a piece of embarrassing crap. If I were given it, I’d throw it away half-way through. But then, I didn’t even get halfway through writing it; who knows, maybe that could play out in my favor.

As I heard Dr. Dan Gottlieb say on the radio, there’s nothing more painful than seeing yourself helpless to aid somebody you love. I know all about that. It’s also true, though to a lesser extent, about something you love doing.

Because, I suppose, the stuff is you.

Which-all sends me down the dispiriting spiral, and starts me thinking about the nature of graduate pedagogy.

When a student is being tested, to succeed they need to (1) recognize the explicit and tacit expectations of the questioner, (2) obtain (reconstruct or recall) the domain knowledge and raw facts necessary to answer the explicit question, (3) construct a plan for a cogent response, and then (4) execute and convey that response with the appropriate formalism and idioms. This is true in education from the elementary level to the professorial job talk: Somebody poses a challenge to you, and your response will be judged not merely on its factual content but also on the degree to which it communicates your membership in a joint community of practice.

In my new domain with its weird idioms and cultural assumptions, my first response to hearing many pedagogic utterances is typically a mix of

1. “Oh, cool, I’ve never thought of that!”
2. “Uh huh, sure. Get to the point.”
3. “Why would anybody in their right minds consider that reasonable?”

Thinking back to my days as an instructor, and talking after the fact with my professors here, surely they have the same set of responses to my (and other students’) work product. What differs from case to case is the proportion of the time spent in the first and last ones. We’re all brimming over with domain knowledge; what differentiates us is disciplinary framework, culture, confidence.

On the subject of confidence, briefly: it gets far too much respect in academia. Unlike knowledge or experience, confidence has a dangerous Goldilocks-curve type of nonlinearity. Neither too little nor too much is a good thing. Treat confidence warily.

The goal of higher-level education is to build confidence in the instructor that the student is spending increasingly more of the time in the second phase of “Uh huh, sure. Get to the point,” and less in “Oh, cool, I’ve never thought of that!” The last one is always undesirable, both in the student and in the instructor, except in the rare occasions when somebody’s outside their discipline.

Indeed, I suspect the proper way of thinking of a discipline is like a biological species, which is differentiated by reproductive barriers. In academia, it’s not about the fucking, but about thinking the other people are fools. If they’re time-wasting fools, they’re in another discipline. (And if they’re in another discipline….)

Yet I find I often say that bit about, “Why would anybody in their right minds consider that reasonable?” My instructors report they say it a lot about me. Not surprisingly, this leads to increased pressure in testing situations.

The point being: We often seem to forget how important issues of pragmatics and culture are in the pedagogic cycle. Instructors know the damned answer. The older the student is, the more confidence the instructor should have that the student also knows the damned answer. What an advancedinstructor should be doing is looking for cues that somewhere in that stranger’s head is a cultural framework and body of knowledge that together will suffice in the future. Maybe (for some weird reason we shouldn’t dwell upon) that graduate student won’t end up being a professor… I suppose… but in the meantime, a mentor should be seeing numerous cues that the student will soon be able to talk the talk, even outside this protected, nurturing creche that is graduate school.

So, two objectives: knowledge and communication.

I’m thinking back to years spent grading intro biology stuff—and for that matter some of the manuscripts I’m sent for peer review—and reflecting on the “memory dump” pattern that we all know and love. The canonical “memory dump” appears on the face of it as a blind listing of facts, and we’re tempted to see it as a clear demonstration of failure to synthesize knowledge. I wonder, though, if the author of a “memory dump” might be feeling challenged in the communicative aspects of the process. In conversations, where they can read cues and adjust the flow of talk on the fly, they never get a chance to spew random facts. [Well, most don't.] They come over as “smarter”. Thus it is confusing when we encounter a stupid-sounding list, an essay that dances around the subject and never gets to the point.

When we test knowledge, we are also testing acculturation. When we embark on an assigned task, to succeed we need not merely know, but demonstrate the knowledge, and for that matter show we have the ability to do so with facility.

And when we are given a test, an essay, a report, a manuscript to review, and it’s full of gaps and holes, what then should we infer?

What do we test, exactly?