Spent a few cycles yesterday thinking about Laurent Bossavit’s provocation, and while I’m having fun gettin’ all complexological after so many years away from the Santa Fe Style, I’m increasingly confident it’s not quite what he was asking for.
After all, complex systems models offer no predictive value, except in the sense that they provide insights into models’ intrinsic consequences. As with Genetic Programming vs Statistics, agent-based complex systems models are useful only insofar as they surprise and inspire further models, not as tools interacting with the “real world”.
At any rate, I’m finding it a useful exercise.
Here’s where I am now, as of this writing (link to old commit).
Personally I’m confident that the toolkit of complexological modeling is eminently practical; Nk models, Boolean networks, agent-based simulation, all the rest is in my blood, so the design patterns are so obvious to me that I can slap something together in a few minutes. But it’s interesting as I talk with folks (Laurent and Ron Jeffries, among others) about this little sketch, how often my correspondent wants to drop my modeling toolkit back down to the familiar level of aggregative statistics, or stock-and-flow continuous-valued systems models.
Subjectively, when folks ask for something so intrinsically focused on story-telling to be “simplified down to cleaner math”, it fees a lot like when old-school programmers ask to use Integer values for Boolean values (“-1 is false!”), or when I see simple structs of primitives used instead of full-fledged objects in an object-oriented language. Something about habits and familiarity, surely, but also an utterly different sense of elegance and purpose, of craft, creeping in.
(Which is interesting, and worth watching over time. Something very similar comes up when I read this.)
At any rate, a Boolean Network feels like a nice, tunable model of a complex world with hidden internal structure. In addition to tunable internal complexity, one can always add externalities, noise, structural dynamics, all sorts of things that model the way we treat error and uncertainty in the real world much more authentically than merely adding some normally-distributed epsilon to a number.
But there’s an interesting modeling choice before me, now, and I’m thinking aloud about it. As you’ve read, I’ve sketched a world with useful tunable parameters; I’ve sketched a comparison (“waterfall” vs “agile” as variations in timing and order of staged work). But one does want one’s simulated “Team” to be able to do that work in a reasonable way.
So what is “requirements gathering”? Or design, or coding, or testing? As happens now and then when building complex systems models, the question is not “What do people really do?”, but rather, What is the simplest convincing mimic of what is implied by a story in which we say, ‘Five requirements have been gathered’? Or, Sixteen designs have been considered, or Eleven tests have been created and run?
I have some notions, as you can see in my notes at GitHub. But there’s also this feeling that people brought up in the classical mathematical modeling world (the one that perfuses computer science as well) will probably raise their eyebrows. One is brought up to think of variables as primitives, and of interactions not as algorithms but as equations. One is trained by decades of expensive computing and resource limitation to assume that aggregation takes place properly within a model, using stocks and flows and aggregative measures and statistics, derivatives and averages all over—and not to see each instance or run of a model as being contingent.
This is a kind of premature optimization that I haven’t heard remarked before. But I think it’s close to the core of what makes “complexology” abhorrent unto the journalist. And what’s interesting is that it’s coming from nerds who have almost certainly dabbled a bit with these alien, thrilling, complexological models: implemented a Game of Life, for example.
And yet I think most haven’t realized that contingency and subjectivity are baked right in. Protip: On average, over long time scales, all Game of Life runs are empty.
Maybe nobody’s pointed that out before. I think they probably ought to have done so. 
Bill -
The story telling aspect of this is the interesting part, because inevitably things get simplified down when things are complex.
The world I live in at the moment has failures from time to time within a System, and so there’s a set of narratives that get constructed about each of these. Inevitably the user perspective on the System is somewhat abstract, so you don’t want your story about what went wrong with the System to be so full of sharp pointy edges that people lose confidence in its abstract beauty. Yet the developers of the System know all too well, or need to be reminded from time to time, that there are indeed sharp pointy edges and that you can’t always do everything in a way that looks like the obvious way to do it.
The perpetual challenge is to tell a story that is simplified enough to make sense to the user, but not so oversimplified that the system builder doesn’t recognize it.
I’m sure that your correspondents simply want you to tell them a story that doesn’t require that you invoke magic to explain it; or, if there really is magic going on deep inside, that it gets hidden enough with soothing words that sound like you are describing something non-magical (at least from the outside).
This is interesting, and already serving one of the purposes you’ve set for it: inspiring further modeling efforts.
For instance I immediately started wondering how one might transpose the two Agile practices that are currently topmost in my mind at the moment (for contingent reasons) into this model’s terms: Simple Design, aka YAGNI, and signing up for tasks.
The latter is interesting, insofar as it relies on “the knowledge that one kind of agent has about other agents”, at least if you want to model the pre-Agile approach to project management, where the PM is the one expected to parcel out tasks to developers. (In many of the places where I worked pre-Agile this was in fact one of the PM’s major prerogatives, and to question it would have been politically unthinkable.)
As for YAGNI, the properties of NK networks are obviously applicable directly — many local optima and short hill-climbs, directly leading to the often-discussed problem of “planning ahead” in software design.
I’m also kind of surprised at where you’ve placed the emphasis in your modeling effort: The Market, a large aggregate of people external to the small aggregate of people who are actually directly involved in writing the software. I might have started with The Customer. Then again, perhaps we’d end up in the same place, since The Customer can often be modeled as an aggregate of conflicting desires, not totally unlike an aggregate of many people.
One of the differences of interest in Agile vs. “plan driven” (for lack of a better term, but I’d rather use a phrase more constructive than “non-Agile”) is the differing focus on specialization and boundaries.
In plan-driven work, more of the “requirements” work consists of having Analysts talking to Customers, where Analysts are agents whose toolkit does not include “Coding”, but does include “Questioning”. You could model “Questioning” as an activity that yields some amount of information about The Market (or about The Customer’s understanding of The Market). Only when the Questioning starts to converge on a stable set of answers do these projects move on to the implementation phase, which is taken over by Developers who have Coding in their toolkit but not Questioning (or have it but have a high cost imposed on them for using it, etc.).
In Agile work, both Questioning and Coding yield information, Developers are expected to have both in their toolkit, and to engage in both. Coding additionally yields information about the “internal” fitness landscape of the product, *and* is a noisy generator of that same internal fitness landscape. What kind of Developer works on what kind of technical task is part of the sources of noise. A potential downside is that the Customer is now playing an active role throughout, with Coding having an impact on how they respond to Questioning.
To me, the kind of model you seem to be proposing are valuable insofar as they provide a way to “mimic” the *differences* between these approaches. Agent-based seems like a potential good fit because I can express policies like YAGNI or incremental construction, vs. speculative design or sequential phases, in terms of algorithms, of explicit policies (a.k.a. “process”).
The downside is that the model must, up-front, provide a rich enough ontology that these algorithms will say something meaningful, and that was the barrier I hit.
I’m curious to see where you’re going; I promise to jump in and start writing code (“you had me at .rb”) as soon as I see a way to express the distinctions I’m interested in.
I feel fairly comfortable with contingency — the idea that just setting the right policies may not deterministically lead to the outcomes I expect, that things like butterfly effects might come into play. I feel fairly comfortable with subjectivity — the idea that the model by itself *proves nothing* about Agile one way or another, and that after seeing how the models turn out I still have to do all of the hard work of thinking about my own situation and taking decisions I will be accountable for.
Interesting thoughts, Laurent. In particular, as I think about there being two “agents”, an Analyst and a Programmer, with the Analyst doing the questioning in one case, the Programmer doing it in another … and HEY! how about if they sat together during the Questioning …
… it seems to me that an important issue is how many of the Answers wind up in the Programmer’s head, because only the Programmer builds the program, and it will have nothing in it that was not in the Programmer’s head.
So if the Analyst gets X% of the Answers when he works alone, and the Programmer gets Y%, Y probably < X, and then the Analyst communicates the Answers to the Programmer, who then gets f(X) answers, where f is some function with range 0 <= f <= 1, we can model and experiment with the notion of hand-off. And if we put the Programmer in the room while the Analyst Questions, the Programmer will hear both the Questions and the Answers. She may get them better having heard them first hand, which we can model. And there’s more! She may even ask Questions of her own, since she is now in a conversation that has removed much of the discomfort, and we can model her different (and in some ways better for the implementation) Questions and her better apprehension of Answers she cares about.
Perhaps that makes sense. I know it interests me.
Useful pushes. You’ve made me think a bit more about my old friend Scott’s work regarding diversity of “mental toolkits” and problem-solving.
I’ll spend a bit of time this afternoon and fill in the “tbd” parts of my sketchy précis. In particular, what “requirements”, “design”, “implementation”, “verification” and “maintenance” might be.
We’ll see if we can still come to an agreement about what’s of interest, since as you say Laurent, I was starting with the premise that we’re talking about a difference between institutions, and that all externalities (the Market) are simply received information about some abstract process outside. The comparison I’m heading towards comes out of a chat with Ron the other day, where he reminded me that of course “analysis”, “design”, “coding”, “testing” and that other one are all present in both institutional models; just that the criteria and responsibility for moving from one to another are different.
As a preview-ish thing, what I was considering was that a “unit” of “requirements/analysis” might involve looking at the historical data for one feature in The Market, and building a statistical regression model, say a simple random forest or CART tree—something that would predict the future of that feature, but not in a useful format (since our Team isn’t building statistical models, it’s building Boolean Networks).
“Design” would involve translating the model (of one or a few features) into a Boolean network template; “implementation” would be a time-consuming matter of “simply” filling out the values in the tables; “testing” would be comparing the more recent Market behavior to the output of the code, and “maintenance” would red-flag features as “re-work” if the errors passed a certain threshold.
Now I can see a couple of places where your interest in the localization of skills (in “analysts” or “coders”) might slot in there.
My goal, though, is simply to deliver a model that if you squint is able to perform as a “waterfall” or an “agile” team, and which has plenty of places for errors, risks and such to be explored after a “perfect” version is written.
> of course “analysis”, “design”, “coding”, “testing” and that other one are all present in both institutional models
I’m not totally sure there’s an “of course” about that — I tend to treat these activities more as sense-making devices than as the One True way to carve the nature of software development at its joints. But that shouldn’t be a problem for this conversation.
“of course” had secret eyebrow-raising markup on it.
Laurent, point of order: an Nk model is different from a Random Boolean Network. An Nk model is a fixed function of N inputs mapped to a single “performance” or “fitness” measure. A Boolean Network is a dynamical system, with no intrinsic “fitness” attribute.
Pingback: Five Blogs – 12 March 2012 « 5blogs
What I’m hearing in your discussions is a focus on the role of internal communication and learning to the success of a project.
I had originally intended to model these functionalities as essentially optimal to begin with, and permit perfect transfer of information between them in both project structures (one-cycle, many-cycle). And then later add in some realistic inefficiencies, like bounded rationality, inertia in learning or just-plain-errors.
But it sounds as if you both really want the actions of Analysis (“requirements-gathering”) and Coding (“implementation”) to start off as the subject of our storytelling, rather than being a mere control parameter as I’d considered. Can do.