Software innovation has reached a level of necessary sophistication where purely technical skills are necessary, but not sufficient. “Computer science” now significantly overlaps social science, incorporating the insights and methods of psychology, sociology, economics, and anthropology. This evolution of the profession has profound implications for skills, roles, organizational structures, product and project decisions, innovation timelines, and a whole host of other considerations that go to the heart of how software professionals work.
This topic is far too large to encompass in a single blog post, so I will address it across several. Eventually, we will identify the reasons behind the merging of computer and social science, and look ahead at the implications for organizations who trying to keep pace with this development. But first, let’s identify the signs that this convergence is happening. The most conspicuous symptoms exist in the Agile community.
One of the great engines of change in the software profession has been Agile. Not only has it transformed the way that development teams work, but it has had profound ripple effects across the entire software value stream. Agile is far more than a difference in batch size. The challenges Agile posed to traditional assumptions about planning in the face of uncertainty, the centrality of the team, the delivery of value, and other fundamental issues have affected everything from the inception of an idea to its eventual retirement.
Testing, requirements gathering, rapid and continuous delivery, governance rules, customer collaboration, marketing, change management — all of these activities within the value stream, and more, have had to adjust to Agile. While technical improvements have played a major part in this transformation, they are hardly the whole story.
The DevOps movement provides a prime example. The original confusion between DevOps and continuous delivery was, and frequently still is, a mistaken conflation of technical practices (for example, the automation of delivery pipelines) with the resolution of a larger internal political problem (the high level of distrust and friction between Dev and Ops). As DevOps advocates and practitioners are quick to point out, without a substantial increase in trust between the two groups, technical improvements that speed delivery into production are moot, since Ops won’t let Dev put these improvements into practice.
These organizational challenges are not easy to resolve. The broad canvas of Agile adoptions has many successes, but also many failures where corporate culture, unresolved team dysfunctions, and other organizational anti-patterns crushed the hopes of would-be Agilists. Since the remedies to these anti-patterns are not obvious, and frequently not universally applicable, this situation has led to a hunger for advice from people with deep insights into organizational issues.
Many of the best and brightest in the Agile community express this hunger in highly eclectic tastes in reading matter. Talk to any good Agile thinker long enough, and you’ll get a recommendation for a book from some discipline outside of computer science. Walker Royce has incorporated the work of economists into the understanding of the business value of Agile disciplines. Alistair Cockburn can tell you about the value of information theory, and point you to some good references about information flow within organizations. Murray Cantor has incorporated the work of mathematicians specializing in forecasting into a set of methods for assessing the likely outcome of different release and sprint planning decisions. The entire Lean-Agile movement is, of course, based on conclusions from industrial sociologists, not just some anecdotes about Toyota. James Coplien’s excellent work on Lean architecture rests on the work of systems theorists. And, of course, practically everyone will tell you to read Drive, a book that summarizes the important conclusions of motivational psychologists over the last couple of decades.
I once joked with Jean Tabaka, one of the best Agile coaches around, that we needed to formalize the Agile Book Club. Otherwise, how can we keep up with the rapid-fire book recommendations among Agilists?
These are not people who are just intellectually showing off, or decorating the Agile tree with a few shiny gewgaws purchases from the social sciences. Agile would not have advanced to the level of success it currently enjoys without borrowing frequently and heavily from the social sciences. (Given that I have a background in social science myself, I can say safely that social scientists can learn a lot from Agilists about how their ideas have worked out in practice, after many experiments applying them.) If Agile had depended purely on technical issues, such as the choice of source control systems, or best practices for code refactoring, it would not be nearly as widely adopted as it is today. Nor would it have inspired the adoption of complementary techniques, such as user experience design, Kanban, DevOps, Lean, complexity theory, and the like.
In the next blog post in this series, I’ll talk about how team dynamics, the original and primary organizational problem in Agile adoption, depends to a great extent on social science. Not only do teams benefit from applying insights that social scientists have generated, but the best Agile teams adopt an experimental frame of mind, including questions about hypothesis testing and research design, that would look very familiar to social scientists.