I’m work­ing on a pre­sen­ta­tion and a chap­ter for the forth­com­ing GPTP Work­shop, and try­ing to cap­ture some­thing that’s both­ered me for… well, as long as I’ve been writ­ing com­puter sim­u­la­tions and doing algo­rith­mic search and opti­miza­tion, which is (jesus) like ³⁄₄ of my life. And moreso recently, when I went back to grad­u­ate school in Indus­trial & Oper­a­tions Engi­neer­ing, and was exposed to a suite of cul­tural norms I had only expe­ri­enced indi­rectly when I was a biologist.

And I’m not sure how best to put my fin­ger on it or sum it up, so let me just dump a lit­tle pile here to fes­ter while I try to think more: A core myth of “mod­ern” com­puter sci­ence and applied mathematics—a foun­da­tional one it seems—is that algo­rithms are autonomous and atomic.

And yes, this prob­a­bly seems like a “yeah, so?” real­iza­tion. But I sit here work­ing on the Nudge sys­tem and design­ing it to be used inter­ac­tively in exploratory set­tings (unlike, as far as I know, any other GP sys­tem). And I found myself rolling my eyes (again) at the sense­less folderol a com­puter sci­ence grad­u­ate was say­ing about soft­ware devel­op­ment the other day at lunch, about how any­thing that “answers the ques­tion as fast as pos­si­ble” is the best pro­gram­ming solu­tion, QED. And so on.

I can’t think of a sin­gle exam­ple of a search, opti­miza­tion, machine learn­ing, neural net train­ing, agent-​​based sim­u­la­tion, AMPL opti­miza­tion or other pro­gram­ming project and “run”, in a 25 year span, where I didn’t watch what was hap­pen­ing, see a prob­lem, stop the “run”, make changes, and re-​​start it. Not one. I’ve fid­dled with training/​test data break­downs, seen symp­toms of bugs and model defi­cien­cies and sta­tis­ti­cal anom­alies that lead me to inter­vene, or seen slow­ness (or over-​​eagerness) to con­verge that led me to improve my code, or seen tran­sient pat­terns that were more use­ful or inter­est­ing than the “real” pro­gram paid atten­tion to.

Well, OK: Maybe I’m not a very good pro­gram­mer. This is a thing I would agree with.

I note that I haven’t writ­ten a paper or even an email with­out revi­sion. I haven’t had an earnest con­ver­sa­tion on a tech­ni­cal topic with­out some minor argu­ment and restate­ment and analy­sis. I haven’t will­ingly pro­grammed in maybe a decade with­out unit tests and a dynamic notion of “require­ments” and “goals”. And I haven’t been in a sem­i­nar with­out ques­tion­ing the direc­tion of the research, ask­ing about tan­gents and par­al­lel tracks and the roads untraveled.

It’s what peo­ple do.

Yet in AI research, and in not just the lit­tle byway that’s genetic pro­gram­ming but also that broader world of com­puter sci­ence and oper­a­tions research and machine learn­ing and dat­a­min­ing and so on, peo­ple still act as if analy­sis, mod­el­ing, design and pro­gram­ming were some­thing utterly, dis­tantly sep­a­rate from exe­cu­tion of code. As if there were a “right” algo­rithm in a gen­eral case, as if faster was always bet­ter, as if it is not the job of an engi­neer to know any­thing about domain, or to adapt in any way to “externalities”.

As if you could spec­ify a prob­lem up front, spell out every­thing in a nice three-​​ring binder, and “hand” this spec­i­fi­ca­tion to some plug-​​compatible mech­a­nis­tic “solver” or “pro­gram­mer” that was opti­mally fast and prov­ably con­ver­gent and cor­rect in the limit, and the lights would flash and the bell would go “ding” and a lit­tle punch card would poke out at you like the pert tongue of Athena her­self with the answer.

This is a prob­lem for me.

Quite lit­er­ally, since I gladly walked away from my last Ph.D. pro­gram (which was an excel­lent one in its field) for essen­tially this core dif­fer­ence. There’s some­thing wrong, and I increas­ingly believe dan­ger­ous, about… well, some­thing I can’t quite name. Call it “hubris” or “cow­boy cul­ture” or “objec­tivism” if you really want to get nasty. That suite of traits that includes finan­cial engineering’s unques­tion­ing reliance on stu­pid “sim­pli­fy­ing” assump­tions; and com­puter science’s inter­est in algo­rith­mic com­plex­ity at the expense of find­ing answers to ques­tions; and almost all of oper­a­tions research, where “be wise, lin­earize” is a mantra; and my own tech­ni­cal spe­cialty of meta­heuris­tics, where even today peo­ple hand me charts labeled “aver­age per­for­mance vs. time” no mat­ter how many times I reject their papers and yell at them in print because I have never cared about aver­age per­for­mance.

There’s a stink of mind:body dual­ity in there. A kind of biased religio-​​mathematization that imag­ines there is a best, an ideal, a way of delim­it­ing a ide­al­ized set of prob­lems that is bet­ter and more tractable and more ele­gant than any sin­gle instance.

Than the real world, for example.

And increas­ingly, I think Herb Simon is the antichrist because of it.

When I’m design­ing a genetic pro­gram­ming sys­tem, or a mul­ti­a­gent sim­u­la­tion, or a soft­ware devel­op­ment (not com­puter sci­ence) project, or a meet­ing or a story for that mat­ter, I’m not look­ing for autonomy.

The basis of my inter­est in genetic pro­gram­ming (and machine learn­ing and sta­tis­tics more gen­er­ally) is how it aids peo­ple. The C pro­gram­ming lan­guage, as far as I’m con­cerned, is not auto­mat­i­cally “faster” than Python, because I count the time it takes to think and write and debug and under­stand a C pro­gram and a Python pro­gram. If the same algo­rithm will take ten times longer to code in C than Python, and may hide secret bugs behind stu­pid pointer errors or strange type han­dling, and which blocks my abil­ity to use test-​​driven devel­op­ment and emer­gent soft­ware design… that’s worse, not better.

And that same short­com­ing is true, I real­ize, about the way aca­d­e­mics approach non­lin­ear pro­gram­ming and bioin­for­mat­ics and swarm-​​based com­put­ing and stuff, too. Papers are writ­ten, projects under­taken, grant monies spent, and grad­u­ates pooped out into the work­place as if peo­ple who haven’t even met me could deter­mine what I wanted in a given situation.

They piss me off like the worst mar­keters do, in other words. [Iron­i­cally, the most beloved of my aca­d­e­mic friends never watch TV, and the most beloved of my mar­ket­ing friends never pay atten­tion to the math.…]

Here: No mat­ter what your pro­fes­sor tells you, peo­ple still have to ana­lyze and model a prob­lem; spend time typ­ing C or Python or AMPL code some­where; debug semi­colons or mem­ory man­age­ment or matrix def­i­n­i­tions or recur­sion stacks; spend hours star­ing at results try­ing to con­coct rules from their intu­itions for accept­abil­ity (or risk re-​​running their exper­i­ments ten­fold with dif­fer­ent para­me­ters in an attempt to “get bet­ter results”).

I count the con­ver­sa­tions, the lab meet­ings, the code review and unit test writ­ing, the peer review and the con­fer­ences and the late nights spent work­ing wait­ing to see—like Kekule—the dev­ils dance in a cir­cle before we under­stand benzene’s struc­ture. I count how hard it is to talk about some­thing, how long it takes to see a way of solv­ing a prob­lem, how hard it is to under­stand what you have in the end, to tell whether you’re “done” or not. And how hard it is to do it again, to re-​​use what you’ve learned. I count that as wall-​​clock time, as my own mea­sure of “net com­pu­ta­tional complexity”.

I sup­pose my men­tal model is much more a kind of heuris­tic con­ver­sa­tion, a part­ner­ship between math­e­mat­ics, man and machine. Where soft­ware and math­e­mat­ics are a sim­ply ways of fram­ing spe­cial parts of a conversation.

Value does not auto­mat­i­cally come with speed, or even with rigor. I do not value rigor in my con­ver­sa­tions; I find it cloy­ing. I pre­fer explo­ration (of ideas and errors) and exploita­tion (of good ideas and cliches) in bal­ance, not just one or the other.

Why do you think I blame Herb? Hint: prag­ma­tism. And if not Herb, who should I blame?

update: Part of why I want to blame Herb Simon comes from con­ver­sa­tions with Michael Cohen, some years ago. See, for exam­ple, his “Read­ing Dewey: Reflec­tions on the Study of Rou­tine” in Orga­ni­za­tion Stud­ies (2007) vol 28 pg 773.