“We’re not unreasonable, I mean, no one’s gonna eat your eyes”
Jonathan Coulton’s Re Your Brains. Be sure to listen. Consider buying some songs.
[via Scalzi's blog]
Archive for April, 2006
Notes towards a long-awaited project
I’m finishing up (and significantly revising) some code I wrote for a research project this semester. More of the details of the project and results over the next few weeks, but in the meantime, some very neat tools and packages you may find interesting:
pliẓ kal stɛla
Could a really, really, really fluent speaker of Old English, or maybe Klingon, or Esperanto or Volapük, affect an authentic accent? As in: an accent when speaking English?
If so, I would give a lot to hear a recording of the elicitation paragraph read by one of those folks who wouldn’t normally be expected to appear on the list.
What it about not being the
I wonder some days what we think we’re testing in academia. This is not merely the usual harping about giving or getting grades, nor merely another thing about whether testing instruments are valid, and nor what sorts of generalization they can inform. Not even revisiting my hare-brained ideas on multi-objective grading.
Just what it is we actually test, and what we think we’re testing.
I’ve had an important lesson slammed painfully home this week. I’m one year into a entirely new discipline (industrial engineering), after 20+ years’ comfortable success in another couple (theoretical biology, and what in hindsight was apparently “not-industrial” engineering): all my planning estimates are off by a factor of 4x or 5x. I suck at deadlines.
But I’ve sucked at deadlines, like, forever. A lot of people in the University do. Look at the academic blogs, and much of what you’ll read is complaints about “Oh, my, I have so many meetings!” or, “Can you believe my students can’t do everything I tell them to?!” I’m still convinced that the demographic who survive to become professors are exactly those who’re best at postponing things to the knife-edge of the last minute, without pushing them too often or too far over the edge into regular tardiness. And like those folks, I’ve never been very tardy. Well, as such. Sure there are some papers I’ve wished I could give another edit, some thoughts left for followup papers, some deeply abbreviated discussions. But generally I get the job done.
Because I have a strong sense of the effort required. I know the stuff: domain knowledge takes little or no time for me to dredge up. Regardless of the domain. I was, after all, employed as a Know-it-all. And along with “all”, I also know how to write. So I’ve gotten pretty good through the years at weighing these strengths against the relative complexity of the tasks I’m assigned. Yeah sure, that’s a kind of laziness. This is the point where I can remind you that all the best engineers are intrinsically lazy, that it’s the driving force in the field.
But this year I suck. All my estimates are off, and all my work suffers because of it. Not because of this new “yes, really mathematics means industrial” engineering domain knowledge I need to learn; maybe it takes me twice the time to dredge up some long-ignored fact about linear algebra or matrix multiplication, but what’s an extra ten minutes? No, what’s off is my ability to communicate.
I spend all my time editing. I spend extra time,not my own, editing. And re-editing. Not just “Did that sentence make sense?” or “Do I understand that word?” but also “If I use this commonplace term, will the reader know what I’m talking about?” or “To me this is the first obvious step. If I were them, would it be?”
The words make sense, but maybe not to the reader: my use of “fitness landscape” and “autocorrelation structure” come to mind as recent examples that have proven I’m from Mars. Or I understand the word, but not in the same way as the reader: “complex systems” and “data mining” come to mind. Obvious rules of thumb and tools I always use, unthinkingly, in every case—like random sampling the search set of an optimization problem, or saving a record of every solution tested so I can watch progress of an ongoing search…. well, I may as well have invoked Marx or done a little interpretive dance. And at the same time, the consumers of my products are watching for certain significant passwords, and they’re just not coming. Because I’m not seeing the cues, the raised eyebrows, the head tilted, the muttered “…and—?”
So everything I do needs to be edited. Four times, five times. I’ve learned the hard way that it may as well be blank, as handed in uninterpretable. Indeed, it’s because I’ve so deeply misunderestimated the amount of social editing required, that yesterday I was forced to turn in a paper—one that I was originally very proud of—mostly blank.
Crippled. What am I going to do, facing a strict deadline and already having spent two weeks and 30 hours straight working on it, leave a skeleton of headers for all the beautiful things I have in my head but left myself no time to communicate? Even though I’d easily pass them along in a face-to-face conversation, or writing to an old colleague? All it would take is a few moments and a few choice turns of familiar phrase. Which I can’t use; when I do use them, when I write a complete and cogent account of some interesting things I’ve done, still the cues are not there, and what I get is another “…and–?”
So my final report for this class was a piece of embarrassing crap. If I were given it, I’d throw it away half-way through. But then, I didn’t even get halfway through writing it; who knows, maybe that could play out in my favor.
As I heard Dr. Dan Gottlieb say on the radio, there’s nothing more painful than seeing yourself helpless to aid somebody you love. I know all about that. It’s also true, though to a lesser extent, about something you love doing.
Because, I suppose, the stuff is you.
Which-all sends me down the dispiriting spiral, and starts me thinking about the nature of graduate pedagogy.
When a student is being tested, to succeed they need to (1) recognize the explicit and tacit expectations of the questioner, (2) obtain (reconstruct or recall) the domain knowledge and raw facts necessary to answer the explicit question, (3) construct a plan for a cogent response, and then (4) execute and convey that response with the appropriate formalism and idioms. This is true in education from the elementary level to the professorial job talk: Somebody poses a challenge to you, and your response will be judged not merely on its factual content but also on the degree to which it communicates your membership in a joint community of practice.
In my new domain with its weird idioms and cultural assumptions, my first response to hearing many pedagogic utterances is typically a mix of
- “Oh, cool, I’ve never thought of that!”
- “Uh huh, sure. Get to the point.”
- “Why would anybody in their right minds consider that reasonable?”
Thinking back to my days as an instructor, and talking after the fact with my professors here, surely they have the same set of responses to my (and other students’) work product. What differs from case to case is the proportion of the time spent in the first and last ones. We’re all brimming over with domain knowledge; what differentiates us is disciplinary framework, culture, confidence.
On the subject of confidence, briefly: it gets far too much respect in academia. Unlike knowledge or experience, confidence has a dangerous Goldilocks-curve type of nonlinearity. Neither too little nor too much is a good thing. Treat confidence warily.
The goal of higher-level education is to build confidence in the instructor that the student is spending increasingly more of the time in the second phase of “Uh huh, sure. Get to the point,” and less in “Oh, cool, I’ve never thought of that!” The last one is always undesirable, both in the student and in the instructor, except in the rare occasions when somebody’s outside their discipline.
Indeed, I suspect the proper way of thinking of a discipline is like a biological species, which is differentiated by reproductive barriers. In academia, it’s not about the fucking, but about thinking the other people are fools. If they’re time-wasting fools, they’re in another discipline. (And if they’re in another discipline….)
Yet I find I often say that bit about, “Why would anybody in their right minds consider that reasonable?” My instructors report they say it a lot about me. Not surprisingly, this leads to increased pressure in testing situations.
The point being: We often seem to forget how important issues of pragmatics and culture are in the pedagogic cycle. Instructors know the damned answer. The older the student is, the more confidence the instructor should have that the student also knows the damned answer. What an advancedinstructor should be doing is looking for cues that somewhere in that stranger’s head is a cultural framework and body of knowledge that together will suffice in the future. Maybe (for some weird reason we shouldn’t dwell upon) that graduate student won’t end up being a professor… I suppose… but in the meantime, a mentor should be seeing numerous cues that the student will soon be able to talk the talk, even outside this protected, nurturing creche that is graduate school.
So, two objectives: knowledge and communication.
I’m thinking back to years spent grading intro biology stuff—and for that matter some of the manuscripts I’m sent for peer review—and reflecting on the “memory dump” pattern that we all know and love. The canonical “memory dump” appears on the face of it as a blind listing of facts, and we’re tempted to see it as a clear demonstration of failure to synthesize knowledge. I wonder, though, if the author of a “memory dump” might be feeling challenged in the communicative aspects of the process. In conversations, where they can read cues and adjust the flow of talk on the fly, they never get a chance to spew random facts. [Well, most don't.] They come over as “smarter”. Thus it is confusing when we encounter a stupid-sounding list, an essay that dances around the subject and never gets to the point.
When we test knowledge, we are also testing acculturation. When we embark on an assigned task, to succeed we need not merely know, but demonstrate the knowledge, and for that matter show we have the ability to do so with facility.
And when we are given a test, an essay, a report, a manuscript to review, and it’s full of gaps and holes, what then should we infer?
What do we test, exactly?
You sick people
- My Mom has sleep apnea. I do too. My Dad did, too, worse than anybody I’ve ever heard. A lot of people do, it seems.
The University Health Services have a fancy clinic now, just down the street from our house, where they will fit you with a fancy CPAP machine that you will wear every night for the rest of your life and it will possibly make you better, so your risk of heart disease is lower because you don’t wake up so much in the night.
The sleep apnea thing is a huge growth industry, clearly. It’s a simply-described condition with a broad range of possibly serious detrimental medical effects, and it’s often treatable. Made for windfall profits: Soon it will be clear that you’re being selfish and hogging medical effort if you don’t get treated. Think of the savings that could be passed on to treat real diseases, if only we mouth-breathers didn’t clog up the healthcare profession with our trivial problems.
- An over-produced glossy six-color flyer shoved in every grad student’s mailbox (what? maybe 5000 of them?) the other day pointed out that over 30% of graduate students seek help for depression or stress-related syndromes. That makes sense to me; seems a little scant, frankly.
When I was a Biology grad student at U Penn, there came a point where my cohort suddenly looked across the lunchtime bitch-table at one another, and something in folks’ eyes made me call out, “Anybody not taking antidepressants here, now, raise your hand.” And we all sat there raising our eyebrows at each other. Not our hands.
I’m not depressed but merely stressed these days. I find I don’t get depressed much any more, perhaps because now I am old and like many old people I have much more rational first-hand experience of how much worse it can be. The inimical evil of the world, the relentless march towards the grave, &c &c. Sure there’s some stress, but… stress, well, that’s just an extra dose of cortisol and some insomnia. Everybody knows older people sleep less.
At any rate, I suspect the rate of pharmaceutically treated depression among the populace is relatively high. Every week get between three and five emails announcing “Take Time Off From Your Ph.D.!” and “Winding Down the Stress Levels” and “No, Really, Delayed Gratification is On Its Way To You Someday Real Soon Now, We Promise” meetings from the grad school HQ.
I am led to conclude that smart people probably get depressed more than dumb ones. Which makes a kind of sense, in hindsight. Right? Think of the savings that would result if we all sought treatment, and the accompanying productivity gains if smarties just stopped being depressed and stressed, stopped malingering. It would foster a new era of… well, something.
Medical breakthroughs, surely. Because smart people, they helped make medicine the revolutionary necessity it is today.
- Obesity is rampant in America (including Canada and some US Protectorates, I ’spect). We’re all gonna get vascular disease and diabetes. You’re a bad person if you don’t diligently exercise, eat reasonable amounts of bran and pomegranate and spelt wheat, and end up using valuable resources that could better be applied to help people who need it.
I am led to believe it’s a worse social sin for smart people to be overweight. Which makes a kind of sense. Because poor people are fat, and smart people aren’t poor. Right? (I’m still working on this one). But think of the savings if we all just took three hours a day and walked around in bright white shoes and officewear, and ate salad at the cafeteria—savings that would benefit the really sick.
That’s the feeling I got from my recent Health Maintenance Exam, at least. “When are you going to get in shape?” Because, my young man, we’re not going to wait for you much longer.
- I am told that my generation (”13ers”, also known as Early-onset GenX Syndrome Sufferers) and those afterwards are prone to getting allergies and asthma and autoimmune diseases because we didn’t eat enough dirt when we were kids, and now it’s too late to eat dirt because our immune systems have learned incorrectly that (a) many important molecules in our bodies look something like the surfaces of Bad Microbes, and (b) the surfaces of many ubiquitous molecules on the surface of Regular Household Stuff (dogs, books, flowers) also bear a striking similarity to the surfaces of Bad Microbes, so (c) our noses are always running and we complain frequently of serious headaches, and therefore (d) we’ll need to take prophylactic antihistamines for the rest of our lives.
I bet the smart kids are more prone to allergies. Think of the stereotype: asthmatic, myopic dweebs. No dirt-eating at all as a kid = sickly and sniffly. Stereotypes, as it happens, don’t actually lie that often. Right? So as a parent you’re shirking if you don’t take your kids outside now and then and force-feed them some dirt.
And then spray them quickly with antibacterial soap.
Think of the savings &c &c.
- And there’s statins. Wow. Biggest-selling pharmaceuticals in the world. We’re all much better for taking them. Think of the savings in the medical community that can be passed on to treat other, more serious pathological conditions. ‘Nuff said.
- New-generation NSAIDs. Aspirin is dangerous; it might give you tummy troubles. Never take aspirin! We should all switch to newer NSAIDs, that are much better because they’re newer. Savings. Passed on. Think!
Am I saying I think we’re a bunch of whiners? No. These are all real problems, especially the just plain constant pain one—my Mom suffers from terrible osteoarthritis, for instance, and NSAIDs give her the ability to walk around a bit. Depression sucks, is dangerous, is more terrible in a way than all the others, because you watch yourself being sick and unable to do anything about it. Heart disease, diabetes, sleeplessness, are all actual Bad Things. I’ve watched several people die badly from diabetes and its complications.
Nobody should suffer any of these problems or risks. Nobody anywhere. We are not—the sick ones—whiners.
Alas, we all do suffer these problems. If not these, then some other ones. Don’t get me started on cancer.
An underreported statistic I have not read lately, presented here as a reminder: nearly 100% of patients seeking medical treatment eventually die.
How can we stomach that? Why doesn’t anybody do something about it?!
What I’m prodding with my Short Stick here is something about our collective attitudes. Culture. Tacit assumptions, implicit understandings. I don’t know whose in particular. Doctors? Sure. They are, in general, ignorant overworked habitualists who have so little time to listen to details or think about consequences that they often as not end up mistreating their patients. That is, prescribing anything just to get the patients off their backs.
And the patients, well, they’re stuffed so full of risk aversion that they immediately seek advice and treatment and solace for any complaint, including (as noted in (2) above) over-complaining and Generalized Life Difficulty. They want surety, and feel betrayed when something goes wrong that isn’t immediately replaced or refunded under their implicit Warranty Service Agreement.
And then it hits me: This is how religions get started.
Look at it. Think about it.
We have the overwhelming desire for simple, effective solutions to ubiquitous problems. We have a harried special class, who by dint of historical contingency and a few early successes have become the class explicitly charged with providing solutions for those problems. But inevitably they’re unable to cope with the world’s diversity, entropy’s insidious creativity. So these people, they focus on low-hanging fruit, and increasingly point out how much more work they could get done if people just went through a few simple ritual moves: statins, CPAP machines, salad not fat, just plain aspirin is now Bad For You. And the observant populace, and the majority of the specialist priesthood, don’t have the time or inclination to actually think about what happens in the end. What happens to every solace-seeker, no matter what they try. They shunt old and sick people off to special temples, for preparatory purification ceremonies involving respiratory support.
Those sick folks, they’ve gone to A Better Place. Some lucky few come back, briefly, and tell thankful tales of the wonders.
But they always go back, in the end. Back to the place we don’t see them.
Religion is not a symptom of irrationality. Irrationality takes hold of all of us, no matter what. We are not rational beings. Indeed, the modern scientific atheist’s tendency to decry irrational religiosity so flies in the face of facts as to be reduced to an artistic act of Grand Irony. Prescriptivist Philosophy as Performance Art.
Religion is a robust cultural system, a cultural attractor that sucks in societies wholesale. Look at how far it’s progressed, in just the last 100 years or so. In the case of Medicine, I mean.
So many converts. So many ads in all the magazines. “Ask Your Doctor.”
It thrives because any priesthood inevitably fails when asked to work on the Really Hard Stuff. So it adapts by playing to our inherent pain-avoidance, trotting out the old favorite solutions of sequestration and denunciation of problematic facts, elision of complexity, denial of diversity.
Religion thrives because we are built to classify, but made to over-generalize.
Of course, it’s more complicated than that. But we don’t have time to go into that now.
Four nice search and optimization thesis projects I will not be doing, principally because a surprising number of search and optimization practitioners make me very tired
Where “make me very tired” can be understood as a euphemism for: “Drive me to a sketchily-explained mysterious ‘club meeting’ at an undisclosed bunker supposedly located along the disused country road connecting Despair and Rue, playing a really annoying Classic Hits of the 70s cassette over and over so loud that I am forced to fling myself from the moving car before we get much farther than the edge of Despair—but look! they drove off with my license in the frickin’ car, so I am forced to walk all the god. damned. rest. of. the. way. to. Rue.”
Roughly. That’s been my graduate school newcomer’s experience of Operations Research to date. Your mileage may vary.
Like many crackpots, I write stuff on 3×5 cards. Here are four cards I found while going through my notes for this other paper I’m supposed to write, and all four of them bear simple statements of nice little research projects, any one of which would be suitable for somebody’s thesis work. Not mine. Because I’m just a first-year student. A badly-trained one, being a half-assed biologist raised by wolves.
Me, I’m in this Advanced Class where we’re learning that Mathematical Programming is not the be-all end-all problem solver it’s cracked up to be. Hunh. Go fig. Been learning that all semester. Lucky thing too, because what with my complete and utter lack of meeting anybody in more than 15 years (until this last year or so) who thought it was worth considering for a minute, I might’ve slipped and fallen into the kind of unquestioning reliance of Mighty Oracular CPLEX that my peers in this new discipline are prone to exhibit. I picked that up, back in Despair.
Anyway, I need to write a paper on “How Mathematical Programming Is Sometimes Hard, and This May Excuse the Occasional Lapse into Heuristics. Briefly.” And I need to do it without seeming for a moment to be… well, uppity. That’s a word. I will use “Made tired”, and “uppity”, and leave it at that.
Raised by wolves. Biologist. Ignorance out the wazoo. ‘Nuff said.
So I will not be writing about:
- What is the VC dimension of a search algorithm? Think of it as a classifier that finds the examples in a set of test problems either easy or hard. Consider how this classifier shatters the set of possible problems of that type. Pick your favorite search algorithm and class of problems: consider using Traveling Salesman Problems (everybody else does), and maybe simulated annealing? Your call. Heck, try Linear Programming with simplex moves, and arbitrary diet problems.
- What formal representation of the set of all search and optimization problems would be useful in building a classifier that let you automatically recognize the canonical types: knapsack, assignment, traveling salesman, bin-packing, stock-cutting, knee-biting? If you could create an automated classifier, what would it mean if it told you that a problem was “93% knapsack”?
- Surely—surely—there is a No Free Lunch Theorem for branch-and-bound strategies. I mean, otherwise when you pick a variable (or some combination of variables) to branch on, you’re just guessing, and that’s so… dumb… well, I don’t even want to think it of you. So tell me you know there is an NFL Theorem there.
- Discuss a standard optimization problem, with your choice of objective function. Now restate it to find the second-best answer. Is that harder? Why? Sure seems harder than finding the worst. What do you have to do to the representation to make it easy again?
You, who are not in my discipline, who are not a first-year graduate student with a Very Poor Understanding of the finer details of Linear Programming and the Appalling Fractionality of Linear Relaxations…. you go have fun. Send me a cite, if you get something done. I can’t stand the thought of doing them, except that they sound quite interesting.
But I would have to explain them to my colleagues. That would be hard. I can tell already. If you need to get in touch about them, just drop me a line.
I’ll be somewhere between here and Rue.
He kiplinged in his joy
As reported in the “Most amusing (or astonishing) text you’ve come across” thread at the Distributed Proofreaders forum. To find this and more such treasure, consider volunteering a few minutes of your time to proofread a page or two. It’s simple, it’s free, it helps preserve the great works of the past, and above all else brings to light the unremarked gems among the lesser-known works.
THE JABBERWOCKY OF AUTHORS
‘Twas gilbert. The kchesterton
Did locke and bennett in the reed.
All meredith was the nicholson,
And harrison outqueed.Beware the see-enn-william, son,
The londonjack with call that’s wild.
Beware the gertroo datherton
And richardwashburnchild.He took his brady blade in hand;
Long time the partridge foe he sought.
Then stood a time by the oppenheim
In deep mcnaughton thought.In warwick deeping thought he stood–
He poised on edithwharton brink;
He cried, “Ohbernardshaw! I could
If basilking would kink.”Rexbeach! rexbeach!–and each on each
O. Henry’s mantles ferber fell.
It was the same’sif henryjames
Had wally eaton well.“And hast thou writ the greatest book?
Come to thy birmingham, my boy!
Oh, beresford way! Oh, holman day!”
He kiplinged in his joy.‘Twas gilbert. The kchesterton
Did locke and bennett in the reed.
All meredith was the nicholson,
And harrison outqueed.Harry Persons Taber.
As seen in The Book of Humorous Verse, edited by Carolyn Wells, 1920. Doran.
A fine way with words
The first — and only — sentence of the preface from European Glimpses and Glances by J. M. Emerson. 1889.
The ground covered by the contents of the following pages having produced an almost unending series of more or less successful fruitages, the author can hardly expect that the modest result which he has brought to market may be thought worthy of attention; and if perchance it shall be found agreeable to the tastes of some, he will be gratified to know that they have realized something of the enjoyment in consuming that he has experienced in producing it.
Measuring differential attention in a distributed volunteer community
Distributed Proofreaders (DP) is an online community of several hundred volunteers who scan books and magazines that are the public domain, and then collectively correct character recognition and formatting errors made by automated OCR software, producing high-quality text and HTML files that are released into the public domain — for free — at Project Gutenberg.
A “project” at DP is typically a single work: a book, a magazine number, a small portion of an encyclopedia, a monograph, a song. The complex DP workflow moves these projects through a series of well-defined stages, with tasks ranging from the scanning of the physical work by individual content-providers, through two separate rounds of (text) proofreading of the individual pages of the work, two further rounds of formatting to capture details of typographic and mid-scale semantic structure, and finally a number of rounds of post-processing to re-integrate the separated pages into a single coherent work. Only after they’ve passed through the “hands” of dozens of volunteers are projects considered complete and sent to the archives for public distribution.
In the proofreading and formatting stages of the workflow, there are several hundred works in progress at a given time, each of which may have several hundred pages. Volunteers can choose whichever active project they prefer to work on, one page at a time. Thus, at any given moment there is a wide variety of works in process, in many languages, of many styles and in numerous domains and categories, aimed at many different audiences and age levels. Each project poses unique challenges and rewards to the project managers, the volunteers, and the eventual consumers.
In the administrative sense, the flow dynamics of projects in DP is controlled by a suite of complex parallel and sequential queues governed by numerous rules, and projects are pushed and pulled in various ways through the different stages. But at its core, the flow rate of projects through DP is driven by the collective preferences of the volunteers who pick pages — each according to her own criteria and ability — to work on. Through the interaction of the driving force of collective interest with the underlying structure of administrative rules, some projects languish for months or years without progressing, while others race through the stages of DP in a matter of hours.
Not surprisingly, the reasons for these differences are not simple.
Projects in DP differ widely on the basis of many structural features — the number of pages, the length of the pages, the number of illustrations and tables. And quality features — the clarity of the scans and the accuracy of the OCR, the presence of automatically-produced HTML markup. And matters of content — language, fiction vs. nonfiction, technical vs. narrative work, religious content, &c. And there are the ill-defined “marketing” factors that most affect their relative interest to the volunteers, that might drive somebody to choose one over another to proofread. Thus, a novel that moves through the system quickly could be doing so because the pages are short, or because the OCR is especially clean, or because the story is engaging, or because the author is famous. An essay on linguistics might take weeks to complete a stage simply because it concerns an uncomfortable topic, or the pages are inordinately messy, or there is a single difficult table “blocking” the volunteers’ progress.
The average speed at which work passes through the DP system has been remarkably constant over the last year, at least in terms of projects completed. But there is a growing problem with the workflow: the queue of projects waiting to enter the second stage of proofreading is already dangerously large, and there is no evidence that the community will be able to compensate without a radical structural solution or gross change in policy. While a number of discussions are underway about how best the alleviate the backlog, experience conspires to undermine the collective confidence in any one of them: the current system was put in place nearly one year ago to address other aspects of unbalanced workflow. More importantly the system of controls and queues is so complex that it defies understanding and simple control.
To that end, my wife Barbara and I are planning to build a detailed model of the DP workflow, at a sufficiently fine scale to capture every important aspect of the site’s dynamics. With those models, we’ll be able to really explore and understand “what if” scenarios: changes in the dynamics of queues, changes in the balance and even the quality of work. Over the next few months, Barbara and I will report our progress on this overarching project.
That said, we’re left with an important job to do beforehand. To compare one model or policy with another, we first need to understand not just what’s happening but also what’s wanted. We need to establish a set of benchmark measurements.
“Attention is being paid”
We’re not concerned with the relative contribution or performance of any volunteer or project manager, nor should we be. The diversity of volunteer interests, abilities and intentions is what makes the Distributed Proofreaders community so effective and strong.
Rather, the family of Project Managers and administrators — and also the eventual consumers — is much more concerned with the flow of projects through the system. Any Project Manager prefers to see their work move ahead quickly. Any administrator prefers to see backlogs emptied as much as possible. Any interested reader wants to read the book as soon as possible, and they can’t until it’s released to the public.
Throughout the DP community there’s clearly already a broad appreciation of this. “Special Days” have been set up to ease the release of works that would otherwise wait in queues. The first project of any new Project Manager is expedited. Ad hoc teams of volunteers have arisen that specialize in completing works that have spent the “longest time” in the system. More crucial, the current backlog of projects waiting for the second round of proofreading is in itself no problem, except insofar as it threatens to delay ongoing projects too long.
In a way, all these are adaptations to bring extra “attention” to projects that are otherwise being missed. Work only gets done by voluntary attention from the community members. While every individual wants most to do what’s fun and entertaining, at the same time there’s a real sense that the emergent progress over all projects should be equitable and in some sense balanced.
That it takes all kinds, in other words.
From an administrative standpoint, any comparison between alternative policies and structural changes should take into account the effects they have on both the individual and overall progress of projects.
Current practice for estimating the progress of projects tends to focus on the number of projects being released into and out of each stage. But as I mentioned above, this ignores the diversity of work underway. Is it preferable to release many small novels at the expense of important — but difficult — larger reference works? Anecdotally, the “fastest” projects are the smallest and simplest ones. Because there are more of them, and they get read faster, they often bypass larger more scholarly works in the system… and thus they tend to form a large fraction of the end product released.
In the spirit of preserving diversity, is this the most desired outcome?
Quantifying the degree of attention
What I’m proposing here is a method of measuring the relative attention paid to projects. One that takes into consideration the many diverse attributes of those projects: the language, clarity, difficulty, scale, and all the other qualities that must be balanced to compare them equitably. And equity is the important point, of course. There is a tacit but strong sense among the volunteers is that everything in process should move on, no one at the expense of any other.
The overall DP workflow is surprisingly complex. Because there is a pressing need to address it right now, I’ll focus in this work on projects passing through the second proofreading (P2) stage. It’s hoped that by developing this approach there, we can later derive a more general approach for balancing flow across the entire DP system.
Briefly, I’m proposing to apply a mathematical technique called Data Envelopment Analysis (DEA) to compare projects that have completed the P2 stage. DEA allows the analyst to understand especially complex problems without over-simplifying effects or throwing away data that gets lost in regression and many other statistical methods.
In framing a DEA model of P2 projects at DP, I’ll collect measurements that are roughly divided into a set of “inputs” and categories, and a set of “outputs”. “Inputs” and categorical variables for this analysis should include traits of the projects (not just the works, but the DP projects) themselves: the number of pages in the work, the number of characters on each page, the language, the number of different proofreaders working in the two proofreading phases P1 and P2, the amount of poetry or mathematics involved, &c. “Outputs” in this case should reflect our standardized measurement of “productivity”. I’ll propose we look at the number of days needed per page completed in the P2 stage.
DEA in this context is an exploratory, not a prescriptive technique. While the technical details aren’t important here, the gist is this: DEA is first used to identify the subset of projects that are processed fastest compared to others that “use” the same amount of the “inputs”; we’ll call these the ones receiving “maximum attention” from the volunteers, but in DEA parlance they’re often referred to as “efficient”. Moreover, DEA provides a measure of how “inefficient” the other projects are; here we can think of this as a measure of how much less attention they’re getting, compared with other projects with comparable characteristics.
I’m being careful not to talk about “efficiency” as such, mainly because it’s a term of art in this context. This isn’t some sort of hare-brained proposal to “improve efficiency at DP”. Rather, it’s a way of measuring what’s actually getting done. My hope is that by measuring it, we (as analysts, administrators and volunteers) can look at the process and learn more about what actually happens in DP. While the biggest benefit of distributing work among many individuals is the “wisdom of crowds” effect we read about in the scientific press these days, the negative side is that crowds as such aren’t that good at communicating the finer details of self-knowledge.
Useful outcomes
This project offers at least four benefits I can see.
First, it can be the rational framework we use to compare alternative policies. This will be especially useful in a few weeks a modeling framework comes along that allows simultaneous exploration of numerous alternatives. DP is extraordinarily complex, and understanding the shifts and flows and their effects on productivity and quality is difficult even for small segments, let alone the whole system.
Second, labeling each active project with a number called “attention being paid to this project” could provide important information for project managers and administrators, even within the current system. My analysis may highlight ways individual PMs can manage and fine-tune the progress of their own projects (even at the expense of others’), or it could spark the invention of marketing and incentive systems to help drive volunteer attention one way or another.
Third, this might focus the collective attention of the volunteer community as a whole. We’ve all heard to old saw: “What gets measured gets done.” As I’ve pointed out above, I think there’s a consensus that equity is good. If only the collective knew what progress was being made, they might be able to better collectively balance it.
Finally, say it works. If a robust automated system can be built that helps with this sort of “balance”, it might save DP. There are going to be more volunteers all the time, and more content and more projects waiting to move ahead. The present setup strains the collective management skills of the current community; witness the backlog of P2 projects. If we incorporate this sort of self-awareness into the automated infrastructure to better steer projects more transparently and efficiently, we’ll surely benefit over the situation with the current suite of ad hoc rules that are frequently overridden or “gamed”.
I’m sending requests for the necessary data (and a duplicate of this post) to the DP website itself, but will be happy to discuss the progress of the analysis here or there.
There aren’t a lot of collective collaborative production-oriented communities yet. I’d be interested in hearing anything you might know about other examples.
There goes that bandwagon again.
- “Light Mass Prayers” — Porcupine Tree — Signify
- “Piya Re Piya Re” — Ustad Nusrat Fateh Ali Khan — The Rough Guide To Bhangra
- “China” — Tori Amos — Little Earthquakes
- “Intensify (Fearless Remix)” — Pygmy Children — Industrial Revolution: Third Edition
- “Good Thing Going” — Sugar Minott — Steal This Disc
- “Roots and Wings (Original Madras Mix)” — Sheila Chandra — Roots and Wings
- “Everything Changes” — Staind — Chapter V
- “Uptight” — Ruth Ruth — Kink-FM And Watt Present: Alle 17 Goed
- “Bornlivedie” — Porcupine Tree — Signify
- “Hello Skinny” — The Residents — Steal This Disc
Not that random, as it happens. Mainly because our library disk crashed and I’m having to re-rip every goddamned CD in our collection.
How many eggs are you supposed to put in the basket again?
Subscribe to posts feed