Sapiens and Collective Fictions
Recently I read Sapiens: A Brief History of Humankind
by Yuval Harari. The basic thesis of the book is that humans require ‘collective fictions’ so that we can collaborate in larger numbers than the 150 or so our brains are big enough to cope with by default. Collective fictions are things that don’t describe solid objects in the real world we can see and touch. Things like religions, nationalism, liberal democracy, or Popperian falsifiability in science. Things that don’t exist, but when we act like they do, we easily forget that they don’t.
Collective Fictions in IT – Waterfall
This got me thinking about some of the things that bother me today about the world of software engineering. When I started in software 20 years ago, God was waterfall. I joined a consultancy (ca. 400 people) that wrote very long specs which were honed to within an inch of their life, down to the individual Java classes and attributes. These specs were submitted to the customer (God knows what they made of it), who signed it off. This was then built, delivered, and monies were received soon after. Life was simpler then and everyone was happy. Except there were gaps in the story – customers complained that the spec didn’t match the delivery, and often the product delivered would not match the spec, as ‘things’ changed while the project went on. In other words, the waterfall process was a ‘collective fiction’ that gave us enough stability and coherence to collaborate, get something out of the door, and get paid. This consultancy went out of business soon after I joined. No conclusions can be drawn from this.Collective Fictions in IT – Startups ca. 2000
I got a job at another software development company that had a niche with lots of work in the pipe. I was employee #39. There was no waterfall. In fact, there was nothing in the way of methodology I could see at all. Specs were agreed with a phone call. Design, prototype and build were indistinguishable. In fact it felt like total chaos; it was against all of the precepts of my training. There was more work than we could handle, and we got on with it. The fact was, we were small enough not to need a collective fiction we had to name. Relationships and facts could be kept in our heads, and if you needed help, you literally called out to the room. The tone was like this, basically:
Of course there were collective fictions, we just didn’t name them:
- We will never have a mission statement
- We don’t need HR or corporate communications, we have the pub (tough luck if you have a family)
- We only hire the best
Collective Fictions in IT – Agile
I started hearing the word ‘Agile’ about 2003. Again, I don’t think I properly read up on it… ever, actually. I got snippets here and there from various websites I visited and occasionally from customers or evangelists that talked about it. When I quizzed people who claimed to know about it their explanations almost invariably lost coherence quickly. The few that really had read up on it seemed incapable of actually dealing with the very real pressures we faced when delivering software to non-sprint-friendly customers, timescales, and blockers. So we carried on delivering software with our specs, and some sprinkling of agile terminology. Meetings were called ‘scrums’ now, but otherwise it felt very similar to what went on before. As a collective fiction it worked, because it kept customers and project managers off our backs while we wrote the software. Since then I’ve worked in a company that grew to 700 people, and now work in a corporation of 100K+ employees, but the pattern is essentially the same: which incantation of the liturgy will satisfy this congregation before me?Don’t You Believe?
I’m not going to beat up on any of these paradigms, because what’s the point? If software methodologies didn’t exist we’d have to invent them, because how else would we work together effectively? You need these fictions in order to function at scale. It’s no coincidence that the Agile paradigm has such a quasi-religious hold over a workforce that is immensely fluid and mobile. (If you want to know what I really think about software development methodologies, read this because it lays it out much better than I ever could.) One of many interesting arguments in Sapiens is that because these collective fictions can’t adequately explain the world, and often conflict with each other, the interesting parts of a culture are those where these tensions are felt. Often, humour derives from these tensions.‘The test of a first-rate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function.’ F. Scott Fitzgerald
I don’t know about you, but I often feel this tension when discussion of Agile goes beyond a small team. When I’m told in a motivational poster written by someone I’ve never met and who knows nothing about my job that I should ‘obliterate my blockers’, and those blockers are both external and non-negotiable, what else can I do but laugh at it?
How can you be agile when there are blockers outside your control at every turn? Infrastructure, audit, security, financial planning, financial structures all militate against the ability to quickly deliver meaningful iterations of products. And who is the customer here, anyway? We’re talking about the square of despair:
When within a smaller and well-functioning functioning team, the totems of Agile often fly out of the window and what you’re left with (when it’s good) is a team that trusts each other, is open about its trials, and has a clear structure (formal or informal) in which agreement and solutions can be found and co-operation is productive. Google recently articulated this (reported briefly here, and more in-depth here).
So Why Not Tell It Like It Is?
You might think the answer is to come up with a new methodology that’s better. It’s not like we haven’t tried:
It’s just not that easy, like the book says:
‘Telling effective stories is not easy. The difficulty lies not in telling the story, but in convincing everyone else to believe it. Much of history revolves around this question: how does one convince millions of people to believe particular stories about gods, or nations, or limited liability companies? Yet when it succeeds, it gives Sapiens immense power, because it enables millions of strangers to cooperate and work towards common goals. Just try to imagine how difficult it would have been to create states, or churches, or legal systems if we could speak only about things that really exist, such as rivers, trees and lions.’
Let’s rephrase that:
‘Coming up with useful software methodologies is not easy. The difficulty lies not in defining them, but in convincing others to follow it. Much of the history of software development revolves around this question: how does one convince engineers to believe particular stories about the effectiveness of requirements gathering, story points, burndown charts or backlog grooming? Yet when adopted, it gives organisations immense power, because it enables distributed teams to cooperate and work towards delivery. Just try to images how difficult it would have been to create Microsoft, Google, or IBM if we could only speak about specific technical challenges.’
Anyway, does the world need more methodologies? It’s not like some very smart people haven’t already thought about this.
Acceptance
So I’m cool with it. Lean, Agile, Waterfall, whatever, the fact is we need some kind of common ideology to co-operate in large numbers. None of them are evil, so it’s not like you’re picking racism over socialism or something. Whichever one you pick is not going to reflect the reality, but if you expect perfection you will be disappointed. And watch yourself for unspoken or unarticulated collective fictions. Your life is full of them. Like that your opinion is important. I can’t resist quoting this passage from Sapiens about our relationship with wheat: ‘The body of Homo sapiens had not evolved for [farming wheat]. It was adapted to climbing apple trees and running after gazelles, not to clearing rocks and carrying water buckets. Human spines, knees, necks and arches paid the price. Studies of ancient skeletons indicate that the transition to agriculture brought about a plethora of ailments, such as slipped discs, arthritis and hernias. Moreover, the new agricultural tasks demanded so much time that people were forced to settle permanently next to their wheat fields. This completely changed their way of life. We did not domesticate wheat. It domesticated us. The word ‘domesticate’ comes from the Latin domus, which means ‘house’. Who’s the one living in a house? Not the wheat. It’s the Sapiens.’ Maybe we’re not here to direct the code, but the code is directing us. Who’s the one compromising reason and logic to grow code? Not the code. It’s the Sapiens.If you like this, you might like one of my books: Learn Bash the Hard Way Learn Git the Hard Way Learn Terraform the Hard Way
Buy in a bundle here
If you enjoyed this, then please consider buying me a coffee to encourage me to do more.
zwischenzugs 
Great article, matches my experience. And thanks for the link and compliment on my article.
Wow, that was yours? Have toted that article around for years. Pleasure to finally meet you!
“And watch yourself for unspoken or unarticulated collective fictions. Your life is full of them.”
Agree completely.
As for software development methodologies, I personally think that with a few tweaks the waterfall methodology could work quite well. The key changes I’d suggest would help is to introduce developer guidance at the planning stage, including timeboxed explorations of the feasibility of the proposals, as well as aiming for specs to outline business requirements rather than dictating how they should be implemented.
And then there were 16 competing standards.
wonderful
A very entertaining article! I have as similar experience and outlook. I’ve not tried LEAN. I once heard a senior developer say that methodologies were just a stick with which to beat developers. This was largely in the case of clients who agree to engage in whatever process when amongst business people and then are absent at grooming, demos, releases, feedback meetings and so on. When the software is delivered at progressively short notice, it’s always the developer that has to carry the burden of ensuring quality, feeling keenly responsible for the work they do (the conscientious ones anyway). Then non-technical management hide behind the process and failing to have the client fully engaged is quickly forgotten.
It reminds me (I’m rambling now, sorry) of factory workers in the 80s complaining about working conditions and the management nodding and smiling while doing nothing to rectify the situation and doomed to repeat the same error. Except now the workers are intelligent and will walk, taking their business knowledge and skill set with them.
Very enjoyable. I had a stab at the small sub-trail of ‘syntonicity’ here: http://www.scidata.ca/?p=895
Syntonicity is Stuart Watt’s term which he probably got from Seymour Papert.
Of course, this may all become moot soon as our robot overlords take their place at the keyboard.
Reblogged this on josephdung.
A great article! I was very much inspired by Yuval’s book myself. So much that I wrote a post about DevOps being a collective fiction : http://otomato.link/devops-is-a-myth/
Basically same ideas as yours but from a different angle.
Fantastic article – I wonder what the next fashionable methodology will be?
A great article. Thanks a lot for writing.
This site truly has all the information I needed about this subject and didn’t
know who to ask.
Brilliant – well said Ian!
I think part of the “need” for methodology is the desire for a common terminology. However, if everyone has their own view of what these terms mean, then it all starts to go horribly wrong. The focus quickly becomes adhering to the methodology rather than getting the work done.
A very well-written article. I retired from corporate development in 2014 but am still developing my own projects. I have written on this very subject and these pieces have been published as well.
The idea that the Waterfall technique for development was the only one in use as we go back towards the earlier years is a myth that has been built up by the folks who have been promoting the Agile technique, which for seniors like me has been just another word for what we used to call “guerrilla programming”. In fact, if one were to review that standards of design in software engineering there are 13 types of design techniques, all of which have been used at one time or another by many different companies successfully. Waterfall was just one of them and was only recommended for very large projects.
The author is correct to conclude by implication that the best technique for design and implementation is the RAD technique promoted by Stephen McConnell of Construx and a team that can work well with other. His book, still in its first edition since 1996, is considered the Bible for software development and describes every aspect of software engineering one could require. His point. However, his book is only suggested as a guide where engineers can pick what they really need for the development of their projects; not hard standards. Nonetheless, McConnell stresses the need for good specifications and risk management, the latter if not used always causes a project to fail or result in less than satisfactory results. His work is proven by over 35 years of research…
Hilarious and oh so true. Remember the first time you were being taught Agile and they told you that the stakeholders would take responsibility for their role and decisions. What a hoot! Seriously, I guess they did used to write detailed specs, but in my twenty some years, I’ve just been thrilled if I had a business analyst that knew about what they wanted
OK, here’s a collective fiction for you. “Methodologies don’t work. They don’t reflect reality. They are just something we tell customers because they are appalled when we admit that our software is developed in a chaotic and unprofessional manner.” This fiction serves those people who already don’t like process, and gives them excuses.
We do things the same way over and over for a reason. We have traffic lights because it reduces congestion and reduces traffic fatalities. We make cakes using a recipe because we like it when the result is consistently pleasing. So too with software methodologies.
Like cake recipes, not all software methodologies are equally good at producing a consistently good result. This fact alone should tell you that there is something of value in the best ones. While there may be a very few software chefs who can whip up a perfect result every time, the vast bulk of developers need a recipe to follow or the results are predictably bad.
Your diatribe against process does the community a disservice.
I have arrived at the conclusion that any and all methodologies would work – IF (and it’s a big one), everyone managed to arrive at a place where they considered the benefit of others before themselves. And, perhaps, they all used the same approach.
For me, it comes down to character rather than anything else. I can learn the skills or trade a chore with someone else.
Software developers; the ones who create “new stuff”, by definition, have no roadmap. They have experience, good judgment, the ability to ‘survive in the wild’, are always wanting to “see what is over there” and trust, as was noted is key. And there are varying levels of developer. Some want to build the roads; others use the roads built for them and some want to survey for the road yet to be built. None of these are wrong – or right.
The various methodology fights are like arguing over what side of the road to drive on, how to spell colour and color. Just pick one, get over yourself and help your partner(s) become successful.
Ah, right… Where do the various methodologies resolve greed, envy, distrust, selfishness, stepping on others for personal gain, and all of the other REAL killers of success again?
I have seen great teams succeed and far too many fail. Those that have failed more often than not did so for character-related issues rather than technical ones.
One of the best articles I have read in a long time about – well everything :-).
Before there exists any success, a methodology must freeze a definition for roles, as well as process. Unless there exist sufficient numbers and specifications of roles, and appropriate numbers of sapiens to hold those roles, then the one on the end becomes overburdened and triggers systemic failure.
There has never been a sufficiently-complex methodology that could encompass every field, duty, and responsibility in a software development task. (This is one of the reasons “chaos” is successful. At least it accepts the natural order of things, and works within the interstitial spaces of a thousand objects moving at once.)
We even lie to ourselves when we name what we’re doing: Methodology. It sounds so official, so logical, so orderly. That’s a myth. It’s just a way of pushing the responsibility down from the most powerful to the least powerful — every time.
For every “methodology,” who is the caboose on the end of this authority train? The “coder.”
The tighter the role definitions become in any methodology, the more actual responsibilities cascade down to the “coder.” If the specs conflict, who raises his hand and asks the question? If a deadline is unreasonable, who complains? If a technique is unusable in a situation, who brings that up?
The person is obviously the “coder.” And what happens when the coder asks this question?
In one methodology the “coder” is told to stop production and raise the issue with the manager who will talk to the analyst who will talk to the client who will complain that his instructions were clear and it all falls back to the “coder” who, obviously, was too dim to understand the 1,200 pages of specifications the analyst handed him.
In another, the “coder” is told, “you just work it out.” And the concomitant chaos renders the project unstable.
In another, the “coder” is told “just do what you’re told.” And the result is incompatible with the rest of the project.
I’ve stopped “coding” for these reasons and… because everybody is happy with the myth of programming process because they aren’t the caboose.
I was going to make fun of this post for being whiney and defeatust. But the more I thought about it, the more I realized it contained a big nugget of truth. A lot of methodologies, as practiced, have the purpose of putting off risk onto the developers, of fixing responsibility on developers so the managers aren’t responsible for any of the things that can go wrong with projects.
A pleasure to read. Gödel incompleteness in software? Development environments are nothing if not formalisms. :-)
Really enjoyable and matches my 20+ years in the industry. Thank you.
Reblogged this on High output engineering.
Great article! I have experienced the same regarding software methodologies. And at a greater level, thank you for introducing me to the concept of collective fictions; it makes so much sense. I will be reading Sapiens.
Actually, come to think of it there are two types of Software Engineers who take process very seriously. One who is acutely aware of software entropy and wants to pro -actively fight against it because they want to engineer to a high standard and don’t like working the weekend. So they wants things organised. Then there’s another type who can come across as being a bit dogmatic. Maybe your links with collective delusions help explain some of the human psychology here.
Great Article. Here is one I did about about Agile Management Systems, which are trying to lay the managerial foundations for “Agile” . Or should I say to liberate Organisations? None of the systems help if a full is using this tool, though.
https://managementdigital.net/2017/06/30/holacracy-liberation-and-management-3-0/
Mumtaz, i like this so much
really good site
First of all this is a great article, very well written. A couple of remarks. Early in waterfall, the large business requirements documents didn’t work for two reasons: There was no new business process, it was the same business process that should be applied within a new technology (from mainframes to open unix systems, from ascii to RAD tools and 4-GL languages). . Second many consultancy companies (mostly the big 4) there were using “copy&paste” methods to fill these documents, submit the time and material forms for the consultants, increasing the revenue and move on. Things have change with the adoption of the smartphones use etc …
To reflect the author idea, to my humble opinion the collective fictions is the embedded quality of work into the whole life cycle development
Thanks
Kostas
Sorry, did you forget to finish the article? I don’t see the conclusion providing the one true programming methodology that works in all occasions. What is the magic procedure? Thanks in advance.
thanks for sharing the post. Keep up the good post