The Mirror Dojo: Genetic Programming for Agile Teams

This is the cur­rent iter­a­tion of the Genetic Pro­gram­ming work­shop née Agile Team Dojo I’ve been work­ing on over the last few years.

I’m look­ing at the Michigan/​Ohio/​Indiana region for an inter­est­ing place to run it. If you’re inter­ested in sched­ul­ing me for a two– or three-​​day work­shop, feel free to con­tact me online.

You know how to do that.

The Mir­ror Dojo: Genetic Pro­gram­ming for Agile Teams

Genetic Pro­gram­ming has been actively researched and pro­moted for more than a quar­ter cen­tury. It’s a broad col­lec­tion of design prac­tices and mod­el­ing tech­niques for the “auto­matic” dis­cov­ery of abstract pat­terns and struc­tures.

And that means full-​​fledged pat­terns and struc­tures: algo­rithms, pre­dic­tive mod­els, com­plete mechan­i­cal and opti­cal and elec­tronic designs, and even blue-​​sky arti­fi­cial intel­li­gence systems.

Some of the field’s big hits include:

Sexy stuff! Nerds like us love it.

Bet­ter yet: I can describe all the basic design prin­ci­ples of Genetic Pro­gram­ming in four sen­tences. It’s so sim­ple to describe that I’m com­pletely con­fi­dent that I can help you you—a com­pe­tent soft­ware devel­oper work­ing on an agile team—write a work­ing GP sys­tem in an hour or so!

And that’s just what we’ll do in this two-​​day workshop.

But there’s one more thing.

You’ve prob­a­bly noticed that I always put extra-​​scary scare-​​quotes around “auto­matic” when­ever it comes up.

Dur­ing this dojo we’ll be approach­ing this mate­r­ial as an agile team. We’ll build at least two full-​​featured genetic pro­gram­ming sys­tems, and we’ll bump very quickly into those scare-​​quotes.

And that’s what this work­shop is about.

See, I’ve been work­ing in this field for most of 20 years. It turns out that even after all that time, there’s a large and trou­bling gap between the tuto­ri­als and demos of genetic pro­gram­ming, and suc­cess­ful problem-​​solving with GP. You can mea­sure that gap in terms of time, or com­pu­ta­tional resources, or expected qual­ity of results.

Sound famil­iar?

Much of the advanced aca­d­e­mic research being done today in Genetic Pro­gram­ming focuses on ways to increase the com­pu­ta­tional power, to bring more proces­sors and faster code to bear so that “auto­matic” problem-​​solving has a bet­ter “chance of suc­cess” on a com­plex problem.

Ah, look; more scare quotes.

See, in this work­shop we’re not advanced aca­d­e­mic researchers in Genetic Pro­gram­ming. We’re much bet­ter pre­pared than they are: we’re an agile team.

In the work­shop we’re going to be explor­ing how to tell our lit­tle arti­fi­cial “team” of “auto­matic devel­op­ers” what it is we want, and how they should go about mak­ing it for us, and (because GP just works) they’ll be “releas­ing soft­ware” for us. What we’ll be doing is design­ing the rules by which they solve our prob­lem: espe­cially the ones that spell out how we want them to inter­act with one another.

Which should explain the name of this dojo. And maybe even why it will take lit­tle bit longer and a bit more effort than most oth­ers you’ll run into.

Scope: This is a two– or three-​​day work­shop, for three to eight soft­ware devel­op­ers, engi­neers, coaches, design­ers, sci­en­tists, and other nerds.

The major­ity of par­tic­i­pants should be famil­iar with com­mon platform-​​agnostic pro­gram­ming lan­guages (Ruby, Python, Java, Smalltalk). They should be com­fort­able work­ing in an agile team: we’ll col­lec­tively work in one shared pro­gram­ming lan­guage, and rely on auto­mated unit and accep­tance tests, rapid release sched­ules, agile plan­ning, and pair pro­gram­ming. There should be enough lap­tops for every pair to code, and net­work con­nec­tiv­ity enough to use github for ver­sion con­trol and coordination.

On the first day of the work­shop (6 hours plus lunch) we’ll estab­lish the social infra­struc­ture, and imple­ment a sim­ple but full-​​scale genetic pro­gram­ming sys­tem for sym­bolic regres­sion. At the end of the day, we’ll choose an advanced project for the next day.

Because we’re all nerds, you and I both know you can’t “stop work­ing” after just five or six hours—and that’s fine. But the work day for the project is six hours plus lunch. So no com­mits overnight!

On the sec­ond day (6 hours plus lunch) , we’ll use genetic pro­gram­ming to address a tech­ni­cal prob­lem where results are obvi­ously practical—and prob­a­bly pub­lish­able.

In three-​​day work­shops, the final day can be used (at the team’s dis­cre­tion) for either refine­ment and pub­lic release of tan­gi­ble prod­uct from the prior days, or for a third project using dif­fer­ent GP design patterns.

Why?: The dojo is just what it says: an expo­sure for agile soft­ware devel­op­ers to a sexy but poorly-​​understood tech­ni­cal prac­tice with great eco­nomic poten­tial in the com­ing years. At the same time they’re learn­ing about the tech, they’ll be sur­fac­ing aspects of their own work, and the way agile prac­tices mold project man­age­ment in the “real world”: require­ments, goal-​​settings, information-​​sharing, met­rics, col­lab­o­ra­tion pat­terns, infra­struc­ture, deliv­ery sched­ules, and even the juris­dic­tion of man­age­ment vs. developers.

Cost: The most impor­tant cost for this exer­cise is the par­tic­i­pants’ inter­est and atten­tion. If those have been made avail­able, the only finan­cial costs are for the venue, travel, food and board (where needed) for the participants.

Entrepreneurship as Social Evil

[cross-​​posted from non­trapre­neur]

Little-​​e entre­pre­neur­ship is the charm­ing eccen­tric­ity that dri­ves busi­ness inno­va­tion in our cul­ture and economy.

It’s a will­ing­ness to accept risks that oth­ers would shy away from, in exchange for even­tual rewards nobody else can see.

It’s the Ear­li­est Adopter’s enthu­si­asm for a fad that doesn’t yet exist.

It’s the heady taste of hubris that helps you move step past think­ing I could do that, and actu­ally give it a try.

It’s an inor­di­nate will­ing­ness to ignore risks, to forge ahead, to plot a course into the unknown. On a promise.

Big-​​E Entre­pre­neur­ship is the cul­tural fetishiza­tion of that risk-​​seeking behav­ior, mag­i­cal think­ing and obses­sion. It’s taught in busi­ness schools. It’s the sole focus of some eco­nomic devel­op­ment insti­tu­tions, it gets investors’ hearts rac­ing, it’s the stated core of our government’s hope for the national future.

This car­toon “Entre­pre­neur­ship” has become a per­va­sive eco­nomic fetish.

Why is that a prob­lem? Look:

Some young women are nat­u­rally beau­ti­ful, and also nat­u­rally thin. Our culture’s fetishiza­tion of Thin Beauty has fos­tered deadly anorexia, poor self-​​images among nor­mal women, the sex­u­al­iza­tion of chil­dren, drug abuse, and more.

A real cot­tage in the coun­try is unusual, and can also be pretty and rest­ful. Our culture’s fetishiza­tion of Sub­ur­ban Life has fos­tered an indus­try of chem­i­cal lawn treat­ments, greige devel­op­ments at the edge of every city where the win­dows never open, social iso­la­tion, mort­gage debt, finan­cial cri­sis, the neces­sity of dri­ving every­where, and more.

It’s reward­ing and healthy to play sports. Our culture’s fetishiza­tion of Pro­fes­sional Sports has built media empires and lob­by­ing com­pa­nies, offered false promise to dis­ad­van­taged youth, encour­aged drug abuse by even school-​​age ath­letes, glossed over the effects on city cen­ters, and more.

We’ve fetishized com­merce and craft into shop­ping mall sprawl. We’ve fetishized the com­plex consensus-​​bulding of pol­i­tics into talk­ing points and intran­si­gent argu­ment. We’ve fetishized com­bat and national defense into gun sports.

In the same way these other unusual but nat­ural extremes have given birth to social evils, the notion of big-​​E Entre­pre­neur­ship depends on over-​​exaggeration and over-​​generalization of nat­ural but unusual extremes: the little-​​e entrepreneur’s eccen­tric­i­ties of risk-​​seeking, and mag­i­cal think­ing and obsession.

We’re told we can be “entre­pre­neur­ial” church mem­bers, “entre­pre­neur­ial” social activists, “entre­pre­neur­ial” artists, “entre­pre­neur­ial” employees.

Think about that. What does that really mean?

You don’t need Angels or VC to change the world. They need you. They need you to rush ahead. They need lots of you in their port­fo­lios; your rare returns are their sole resource. You are their crop. You are their slot machines.

You don’t need to mon­e­tize every­thing, or promise ten-​​fold returns. Finan­cial cap­i­tal is not the only kind. A project can make you rich in social cap­i­tal, intel­lec­tual cap­i­tal, indi­vid­ual capital.

You don’t need to grow for­ever, or to burn down to bank­ruptcy. Maybe what you’ve done so far is enough. Even if you dis­ap­point busi­ness cul­ture because you’ve started a “lifestyle busi­ness”, at least you still have a life to live.

You don’t need to think of peo­ple as tools and resources. Peo­ple are peo­ple. This insti­tu­tion you’ve started must be for the peo­ple who com­prise it, more than they are expected to work for it. Never lose sight of the fact that it is an it.

You never have just one goal. Your ven­ture is not your world. Even the most obses­sive investor will admit that reduc­ing risk is as much a goal of any ven­ture as increas­ing returns. When you begin to believe some sub­set of “win­ning” is the only goal, when your investors drive you to forge ahead at all costs, when your instinct is to cut away the parts of your life that other peo­ple think are impor­tant just to make it to launch? That’s when you’ve become a dan­ger to your­self, and to society.

Big-​​E Entre­pre­neur­ship is just like Hol­ly­wood and the NBA and the Bill­board charts and the bridal mag­a­zines. You are not going to make next Google or Face­book. Your idea isn’t as orig­i­nal as you imag­ine, your skills aren’t all you need, your beau­ti­ful office in a fash­ion­able ZIP code won’t make your prod­uct any better.

And those suc­cess­ful, rich peo­ple you find egging you on, “advis­ing” you and “sup­port­ing” you and “con­nect­ing” you?

They’re just as caught up in the illu­sion as you are. Pity them. It was their luck that got them through the maze. Not their skill, not their men­tors, not their investors, not their “best peo­ple”, and cer­tainly not The Sys­tem as a whole.

The cul­ture rein­forces them at every turn. Is it any coin­ci­dence they’re sur­rounded by all the evi­dence they need to keep believ­ing that their illu­sion is uni­ver­sal and valid? They’re swim­ming in suc­cess. They see evi­dence of the Sys­tem of Accepted Busi­ness Prac­tices and Rit­u­als work­ing around them, all the time.

Because they have arranged life so they never see it fail. They’re not allowed to see any­thing else as success.

Where are the Big-​​E Entre­pre­neurs whose ven­tures didn’t grow? Didn’t hit it big? They were torn down for parts and raw mate­ri­als, skillsets and cap­i­tal, and dumped right back into hop­per to be fed into the machine.

Who are you? If you define your­self by your project, I don’t think you’ve answered the question.

What do you want? If you only men­tion your project, you’re a liar.

What are the risks? If you don’t know, I can start your list with this one: “I don’t know the risks”.

What will be enough? If you don’t have any idea, I’ll guar­an­tee that “more” isn’t the only answer.

What will you sac­ri­fice? If you didn’t say “myself”, then take a moment to con­sider the Big-​​E Entre­pre­neur­ship com­plex out there, wait­ing and ready, yearn­ing to drop you into the hopper.

You’re a pile of raw materials.

Port­fo­lio filler for investors.

Pro­mo­tional mate­r­ial for your city.

Future donat­ing alumni of your University.

The cover of unsold magazines.

Oh yeah, and you did some stuff once. What was that thing, that com­pany you did back when?

That was your vision? Huh. Who knew?

Grasping at golden straws

Yes­ter­day Bar­bara and I attended a panel dis­cus­sion at the Ker­ry­town Book­Fest called “The Future of Print Jour­nal­ism”. I’ll leave the details to oth­ers; what I found of par­tic­u­lar inter­est was the thrust of the dis­cus­sion among the pan­elists, who were all edi­tors of one sort or another who’ve sur­vived in tran­si­tion from being old-​​fashioned newspaperfolk.

On the face of it, the nar­ra­tive was about the future of print jour­nal­ism in a world where the busi­ness model has been under­mined by free online con­tent. There was talk of aggre­ga­tion by Yahoo! (and Google, though nobody men­tioned them by name once) and how it under­mines the author­ity of news­pa­pers. There was a stern com­ment from the audi­ence about how blog­gers steal­ing con­tent from papers with­out cit­ing it should be sued. There was a lot of realistic-​​sounding explo­ration of pay­wall pro­tec­tion of con­tent and the appar­ent fail­ure of news­pa­pers to fathom micro­pay­ment approaches. A lot of dis­cus­sion of “free mod­els”, and what came across as antag­o­nism from the folks still at the big plop-​​on-​​your-​​steps papers at the notion of free content.

I started being bemused half-​​way through, though. Because four of the five pan­elists explic­itly described the eco­nom­ics of their busi­ness, talked about it wor­riedly, and then wan­dered away again into how cru­cial good writ­ing is, and how expen­sive pro­fes­sional jour­nal­ism can be, and all the other stuff that jus­ti­fies their spe­cial cre­den­tialled sociopo­lit­i­cal role in whichever Estate they used to be.

I’m sure the fifth pan­elist would have acknowl­edged the busi­ness facts in an instant… if it only been brought up explic­itly: Mod­ern news­pa­pers don’t sell jour­nal­ism. They sell adver­tis­ing. Dur­ing the 20th Cen­tury, news­pa­per rev­enue has come pri­mar­ily from advertisers.

And from about 1900 to about a decade ago, news­pa­pers sold print adver­tis­ing at monop­o­lis­tic prices. They were essen­tially a car­tel. Ads in books never took off; ads in mag­a­zines reached only widely-​​distributed sub­scriber demo­graph­ics. Only the local news­pa­per reached the walk-​​in traf­fic that retail­ers sought; coupons really don’t work well in tele­phone cam­paigns; TV is ephemeral, leaves no record.

Yet nobody says of the Inter­net, “Those unqual­i­fied online adver­tis­ers are under­min­ing our professionally-​​trained crack adver­tis­ing team,” or “Do you real­ize what it costs to pick an ad to run next to an arti­cle on a for­eign war?” or “Pho­tographs of ham can’t just be down­loaded from some web­site you know; you need pro­fes­sion­als on staff 24–7 to get the qual­ity our cus­tomers deserve.”

No, the dis­cus­sion was about “the econ­omy being bad” and “read­ers out there expect con­tent to be free” and blog­gers and cus­tomer bases and the threats and uses of aggregation.

I’m sure if there had been time to drive the con­ver­sa­tion my way, some­body would have jumped in and said, “Yes, of course we know print adver­tis­ing pays the bills, but nobody would buy the adver­tis­ing and get the bills paid if it weren’t for the high-​​quality report­ing we gen­er­ate using all that rev­enue.

But: Maybe peo­ple are still buy­ing adver­tis­ing. Just not from you.

Here. We’re friends. I’m just as pre­dictable as any­body else: I’m going to talk about his­tory now.

Pick up an actual print news­pa­per from 1820, from 1840, from 1860, from 1880, from 1900, from 1920, from 1940, from 1960. From 1980. Count the ads. Think care­fully and look at the books (if you have access to them) and esti­mate the pro­por­tion of the income of each news­pa­per that came from adver­tis­ing rev­enue. Yes, I know in the early days they were small, local affairs with maybe a thou­sand sub­scribers each.

But they got their bills paid. What pro­por­tion of those bills were paid by monies com­ing from the sale of print ads?

I’ll bet you a Get Out of Dis­in­ter­me­di­a­tion Free Pass right now that the ear­lier papers had almost no adver­tis­ing (includ­ing the money from arti­cles some­body was paid off to print), that the pro­por­tion bloomed into a major­ity in the days of Hearst and Pulitzer and the Great Syn­di­ca­tors, that it became a cash crop pay­ing 80% or more of the bills in the latter-​​day cull that killed all sec­ond papers in cities.

Print adver­tis­ing was a monop­oly. Still is, one supposes.

You can’t buy ubiq­ui­tous home-​​delivered print adver­tis­ing any­where else. Sure, you can pay sub-​​minimum wage peo­ple to wan­der neigh­bor­hoods and rubber-​​band fly­ers to front doors, or wait a few days and send out coupons in the Clip­per thingie.

And yet. And yet. Every­body knows (and for once I mean it uniron­i­cally) we all love the vis­ceral qual­ity of print, the solid­ity, the abil­ity to page back and check, the clip­ping, the pass­ing it around, the cross­words, the comics. The biggest fuss when a news­pa­per shuts down comes not from the adver­tis­ers (who are already gone by then), but from the sub­scribers. The peo­ple with the blue paper­boxes lin­ing the coun­try roads. The ones will­ing to trudge out to the road­side in win­ter, before break­fast, and take in the paper and sit and read it in their homes.

Phys­i­cal paper. Peo­ple love print. Peo­ple live print. If they get sad enough at the dimin­ish­ment of print jour­nal­ism, do you think they will let it die?

Don’t be a fool. They’ll pay some­body good money to pass it out to them.

Are peo­ple buy­ing ads? Shut your stu­pid mar­ket­ing department’s yam­mer­ing up and look. Peo­ple don’t want ads, they want printed infor­ma­tion. Even the peo­ple who clip coupons would be just as happy to pay you if you just listed the prices of every item at every store in town. They don’t want the coupon, they want the infor­ma­tion about pricing.

And so what’s the future of print journalism?

In many cities in this coun­try, the one news­pa­per is fac­ing finan­cial cri­sis. In smaller towns and wannabe cities (like ours), “the one news­pa­per” is dying. Yes of course in all those places there is prob­a­bly also a super­local paper about high school lunches and church meet­ings, and an edgy coun­ter­cul­ture free monthly, and a free coupon col­lec­tor, and a free real estate list­ing in the super­mar­ket foyer.…

Like I said, The One News­pa­per is dying.

You might think this is what it will be like: Like 1882. Or 1860 or 1900 or 1930, even. The Empire of news is dying, not news itself. Not jour­nal­ism itself.

The adver­tis­ing monop­oly is dying. The eco­log­i­cal niche occu­pied by The One Paper is a goner, not papers them­selves. Specif­i­cally, the One Paper’s national-​​scale ad rev­enues are a goner.

Printed news­pa­pers will have to start rely­ing, again, on the rev­enue streams they enjoyed in the 19th century.

And because it’s how I always do it, let me jink sud­denly from his­tor­i­cal anal­ogy over into bio­log­i­cal metaphor:

Big ani­mals get big not because they are spe­cial­ists in what they eat, but to take advan­tage of economies of scale in their eat­ing. The biggest cats are oblig­ate hunter-​​carnivores just like some shrews, but have very spe­cial char­ac­ter­is­tics of gigan­tism and com­plex lifestyles to keep from wan­der­ing around all day burn­ing calo­ries hunt­ing. Big whales eat very spe­cial meals (giant squid, krill), like many other marine species do, but are huge so they can avoid flash­ing around in big schools all over the place. Big dinosaurs prob­a­bly got big so they could reach or man­han­dle their very spe­cial meals, but lit­tler species could as eas­ily have climbed trees or ganged up. And mar­su­pial lions and wolves? Giant car­niv­o­rous birds, or moas? Giant sloths and mam­moths? Spe­cial­ists, but big because of economies of scale in their diets.

In the big picture—in the course of evo­lu­tion­ary history—megafauna come and go. As a type, fol­low­ing a par­tic­u­lar spe­cial­ized strat­egy that depends on being gigan­tic, they’re often dri­ven to extremes by the pres­ence of a small fruit­ful slice of resources in their envi­ron­ment. Unlike their smaller cousins, they go out on a limb and opti­mize their energy use and lifestyles so they can spend as lit­tle as pos­si­ble to get as much food as pos­si­ble as eas­ily as possible.

But even­tu­ally the limb is gone.

And there you are, you big pile of yummy meat. Sur­rounded by other kinds of spe­cial­ists, who didn’t invest in becom­ing huge.

The future of print jour­nal­ism is a feast, not a famine. The One City News­pa­per, the national news­pa­per, the Inher­ited News­pa­per Empire: that is the main course.

A decade ago I would have pre­dicted we’d see the indus­try roll back all that expen­sive infra­struc­ture the One City News­pa­pers have devel­oped, in set­ting them­selves up as megafau­nal ad-​​eaters, and we’d end up back in a sit­u­a­tion about like 1880. A dozen papers, each with a slice of the sub­scriber pie, with a lit­tle adver­tis­ing rev­enue each to keep them afloat.

Now I’m older and not so sure. Now I see a lovely chaos, a bloom of strate­gies, a roil of use­ful col­lab­o­ra­tion and competition.

What I won­der though, is what was never asked yes­ter­day: who will be the first to fire the mar­ket­ing depart­ment and keep the writ­ers and edi­tors?

That’s the next wave. That’s the imme­di­ate future of print journalism.

Working out the details of a real options framework

Sup­pose a prospec­tive client approaches you to do work for hire. In many con­tracts for tech­ni­cal work, there will also be a req­ui­site nondis­clo­sure agree­ment (NDA), which may be uni­lat­eral or bilat­eral, but which in either case spec­i­fies that you (the con­trac­tor) will keep secret cer­tain infor­ma­tion regard­ing the client and contract.

Lack­ing such a nondis­clo­sure agree­ment, it’s com­monly under­stood that you could when­ever you wish dis­close what­ever infor­ma­tion you like about the name of the client, the nature of the work pro­posed or done, or even trade secrets you learned in the course of the con­ver­sa­tion. Depend­ing on the char­ac­ter and val­ues of the client, one or more of those facts will prob­a­bly be sub­ject to nondis­clo­sure clauses in the con­tract they want you to sign. And those clauses may (if you’re not thought­ful or care­ful) have no expi­ra­tion date.

I’m think­ing of one for­mer client—a large Mid­west­ern corn hybridizer—who made us sign an indef­i­nite nondis­clo­sure agree­ment that promised we would never state the name of the com­pany. That’s it; just the name.

At any rate, let’s look for a moment at what you’re signing.

As we under­stand the law, until you sign that con­tract you pos­sess the option to dis­close what­ever you want, at any time you want. Within cer­tain extreme lim­its (libel, slan­der, state secrets, and so forth).

Let’s give you the ben­e­fit of the doubt as a con­trac­tor, and assume you’re not inter­ested in reveal­ing the pre-​​existing trade secrets of your client. Let’s just say you shouldn’t ever do that, and that they have a clear right to keep you from doing that before any are revealed to you, and that there­fore the deal is bro­ken if you demand the abil­ity to tell any­body anything.

But com­mon nondis­clo­sure lan­guage also cov­ers the broad range of knowl­edge and infor­ma­tion that arise dur­ing the course of the project: not just the stuff you make together, but the name of the client, the terms of the con­tract, the out­come of the project… all kinds of new infor­ma­tion that many clients would like to keep you from pass­ing along.

What is the value of that option to speak about your col­lab­o­ra­tion with one another? Finan­cially, I mean?

Well, the value to you might be sub­stan­tial, espe­cially in the cur­rent busi­ness cul­ture: Assum­ing you’re a con­sul­tant, con­trac­tor, advi­sor, or other non­em­ployer firm, the rel­a­tive mar­ket­ing value of adding this infor­ma­tion to your pub­lic port­fo­lio of work may be huge, depend­ing on the nature of the client and work. If you had com­pleted con­tract work with­out the bur­den of the NDA, you would have the option to brag about the name of the client, the nature of the work, the details of the tools and cun­ning solu­tions you brought to bear, the amount you were paid… all inar­guably use­ful infor­ma­tion to trot out the next time you’re speak­ing with a sim­i­lar client.

Lack­ing the abil­ity to share any of that infor­ma­tion, as an indi­vid­ual con­trac­tor or con­sul­tant, you have inar­guably lim­ited your abil­ity to mar­ket your­self. Who did you work with? Can’t say. What did you do? Can’t say.… And (again, give the cur­rent busi­ness cul­ture) that mar­ket­ing value is not dimin­ished even if the project was a total tech­ni­cal fail­ure. So from the side of the con­sul­tant, it’s clear the abil­ity to pro­mote one’s own exper­tise has pos­i­tive value.

Now sup­pose you do sign an NDA that restricts your abil­ity to pass along this infor­ma­tion for one year. In real options terms, it seems that you’re post­pon­ing the exer­cise of your implicit right to mar­ket your busi­ness. In exchange for com­pen­sa­tion, of course: the pay­ment you receive from the client, and what­ever gen­eral knowl­edge and expe­ri­ence might be accrued dur­ing the course of the project.

So at this point it seems that the amount you should charge the client just to sign an NDA depends on the expected loss of rev­enue you will expe­ri­ence from lim­i­ta­tions of your abil­ity to mar­ket your work. If we pare away the deci­sion to work on the project together, we can­not get looped into ridicu­lous conun­drums like, “Well, if you don’t sign the NDA we aren’t going to have a con­tract.” The value of the NDA is not the value of the entire con­tract; the work you do for the client is what causes them to pay you.

The NDA’s value must be separable.

But there’s where I get hung up, some­how, so I keep mulling it over look­ing for a way to model the trans­ac­tion that cap­tures the risks and ben­e­fits of dis­clo­sure and nondis­clo­sure so they can be made more explicit. Maybe because in this com­bined deal (contract-​​plus-​​NDA) there is also a set of com­plex options being cre­ated, sold and exer­cised by the client, I admit I get tied up.

I’m encour­aged, though. Con­sider that a well-​​formed con­tract for work is above all an aid to the plan­ning processes for both par­tic­i­pants, in that it reduces the uncer­tainty regard­ing pos­si­ble out­comes. As a con­tracted worker, you have more assur­ance of income in the near future; as a con­tracted client, you have more assur­ance that the project will pro­ceed, and you have a bet­ter han­dle on the costs.

Still, the value of nondis­clo­sure within one of these con­tracts feels com­pli­cated, though not nec­es­sar­ily from the stand­point of the con­trac­tor. What are the sources of value and uncer­tainty on the client’s side of this plan­ning process?

Surely the client believes that by engag­ing you and apply­ing your exper­tise and effort there will be pos­i­tive busi­ness value com­pared to what they would achieve with­out your par­tic­i­pa­tion. Or per­haps your pres­ence reduces the risk of fail­ure by a detectable amount. In any case, let’s limit the scope of the analy­sis by assum­ing there is a clear-​​cut case in terms of risk and return for them to engage you.

But they clearly also believe—whether or not it’s true—that pub­lic dis­clo­sure of cer­tain infor­ma­tion will put them at a com­pet­i­tive dis­ad­van­tage. As if you didn’t know it already, this is the assump­tion I’m most prone to chal­lenge. It’s clearly the rea­son cur­rent prac­tice so often makes nondis­clo­sure a deal­breaker: it’s com­mon knowl­edge that the rev­e­la­tion of trade secrets is expensive.

Now I con­fess there is a ten­dency among those of us who have been entre­pre­neurs or ana­lysts or mod­el­ers or IT pro­fes­sion­als or experts of any sort who type and draw on white­boards a lot to imag­ine that the sort of trade secrets that a client might want to pro­tect are the same kind of sim­ple inno­va­tion that we cre­ate almost every day: bet­ter soft­ware, work­ing ana­lyt­ics, cun­ning and insight­ful reports, graphic designs, improve­ments in insti­tu­tional struc­ture. Insights, call ‘em.

These “secrets” are the kind of thing we joke about around here by say­ing (quite accu­rately), “A good idea is born worth minus $25000.” Because ideas are cheap to for­mu­late, but each one has real costs to imple­ment. Over the course of a decade one inevitably hears the same idea pitched a dozen times in whis­pered tones as if it were made of gold: a real estate aggre­ga­tor, a stock pre­dic­tion sys­tem, a social site for book lovers, a killer app on the iPhone.…

These are, in my expe­ri­ence, the most com­mon kind of client projects: the sort any mod­er­ately smart pro­fes­sor or middle-​​manager or grad­u­ate stu­dent stum­bles across in the course of their “real work”, sees unbounded upside poten­tial of, and (with­out explor­ing the prac­ti­cal­i­ties) pur­sues opti­misti­cally. And thus tends inevitably to over­value.

In the case of such triv­ial secrets, let’s assume that the client’s model of the risks from dis­clo­sure of their “secret” greatly over­es­ti­mates the chances or the losses, or both. Your model, or per­haps “the market’s” model, would pro­duce a much lower risk for the client, and there­fore a lower price for [non]disclosure.

But as an expert con­tribut­ing skills to com­plet­ing the project, the abil­ity to pro­mote the sort of work you are brought in to do is no less valu­able to you—inde­pen­dent of its valid­ity as a “secret”. You write, you type, you answer ques­tions, you con­tribute insights whether they are build­ing a hugely inno­v­a­tive first-​​mover, or a bog-​​standard also-​​ran.

So it strikes me that the prob­lem in these cases lies with the qual­ity of the client’s mod­els of their intel­lec­tual prop­erty and com­pet­i­tive land­scape. They over­es­ti­mate the recov­er­able value (or under­es­ti­mate risks) asso­ci­ated with the project, and as a result the real­iz­able long-​​term value to them of keep­ing the secret appears to be greater than the imme­di­ate value to you—and to them—of pro­mot­ing the work.

Because we shouldn’t dis­re­gard a qual­i­ta­tively dif­fer­ent model of the con­tract: Sup­pose instead of being client and cus­tomer you are part­ners, and you are faced with the deci­sion whether to pro­mote your project or keep it secret together. There is mar­ket­ing value to both of you, but also risk from com­pe­ti­tion to both of you upon dis­clo­sure. And dis­clo­sure is irre­versible, don’t forget.

So from a real options per­spec­tive if you can post­pone the deci­sion to dis­close until the ben­e­fits of pro­mo­tion def­i­nitely out­weigh the risks of com­pe­ti­tion, you both win. Whether you’re part­ners, or con­sul­tant and client.

Hope­fully you can see the same real options struc­ture I do. At some point, if they’re pay­ing atten­tion, the client will even­tu­ally improve their model of the real value of their “secret infor­ma­tion.” We just don’t know when that will be, exter­nal­i­ties and uncer­tain­ties of life being what they are.

So sup­pose you enter into a suite of sim­ple options con­tracts regard­ing dis­clo­sure in which (a) you cede your right to dis­close the infor­ma­tion for a fixed length of time (say a year) in exchange for a cer­tain sum of money to off­set your lost mar­ket­ing value; (b) your client is granted an option to renew that con­tract for another year at its end; and © your client is granted an option to aban­don the entire nondis­clo­sure struc­ture (includ­ing sched­uled pay­ments) at any time. They should exer­cise this option, obvi­ously, when they’re out of the money: when the costs they will be pay­ing in future out­weigh the real­iz­able ben­e­fits given new information.

What is the price for nondis­clo­sure, here? It can be esti­mated as the loss of rev­enue you as con­trac­tor will expe­ri­ence from fail­ure to mar­ket your­self. If your client receives new infor­ma­tion at any time that reduces the per­ceived value of secrecy to the point it no longer seems to be worth pay­ing you for it, they can aban­don the agree­ment and your right to irre­versibly dis­close the infor­ma­tion reverts to you. If at the end of a con­tract period they still per­ceive pos­i­tive value in secrecy, they may renew (per­haps at a new price).

Now it’s been pointed out to me that there’s more than just this sort of “naive secrecy” I’ve sketched. While it’s com­mon in star­tups and small busi­nesses, a larger or more capa­ble client prob­a­bly has bet­ter mod­els of the risks and val­ues of dis­clo­sure. If noth­ing else, larger firms are more likely to be aware of real com­pet­i­tive land­scapes and best prac­tices, and tend to out­source devel­op­ment as opposed to research projects.

The secrets in these cases are not so much inno­va­tions as they are well-​​defined func­tional prac­tices and infor­ma­tion that’s been tried and tested. In many cases there are smart account­ing mod­els of exactly how much they’re worth.

But I don’t see how this neg­a­tively affects the cal­cu­la­tion of the cost of secrecy. Indeed, it should improve mat­ters and sim­plify for all involved if the com­po­nents of the con­tract regard­ing secrecy are sep­a­rate from those regard­ing work-​​for-​​hire. Give the cus­tomer the ben­e­fit of the doubt here, and assume we’re now at the oppo­site extreme from “naive secrecy”: now the least accu­rate pre­dic­tive model is prob­a­bly the contractor’s, in that it over­es­ti­mates the value of mar­ket­ing (disclosure).

What we do in this sit­u­a­tion? I’m not sure.

And uncer­tainty is the key: that’s what real options pric­ing is all about. So maybe (after I think about it for a while) we can work the rest of the model out, and maybe slap some prob­a­bil­i­ties and prices on there.

In gen­eral, here’s where I feel like I am: The pres­ence or absence of an NDA clause in a con­tract should not mate­ri­ally affect the expected cost of the actual work per­formed, and there­fore it can be sep­a­rated away from the work-​​for-​​hire clauses. Fur­ther, the mat­ter of dis­clo­sure of pre-​​existing trade secrets (in either direc­tion) is not what I’m think­ing about here, and that should be sep­a­rated as well; I’m talk­ing about novel infor­ma­tion mate­r­ial to one par­tic­u­lar project, rang­ing from the exis­tence of the project, to state­ments of the goals of the project, to descrip­tions of the par­tic­u­lar tech­niques applied, to news of the even­tual outcome.

This infor­ma­tion would be of value to the con­trac­tor (and arguably the client, but we’ll ignore that) for mar­ket­ing pur­poses, who there­fore expects a finan­cial advan­tage when it is dis­closed. But the infor­ma­tion is also (arguably) of value to com­peti­tors of the client, who there­fore expects a finan­cial cost should it be disclosed.

There is uncer­tainty asso­ci­ated with all these val­u­a­tions, and with the prob­a­bil­i­ties of the events occur­ring. How do we model that in such a way as to make it sim­pler to sep­a­rate agree­ments for work from agree­ments regard­ing nondisclosure?

It’s sim­ple refac­tor­ing, really: The mod­ules have very dif­fer­ent func­tions, and yet they’re too often interconnected.