Oct 212008

Last week, we celebrated the 40th anniversary of software engineering. Between 7 and 11 October 1968, the NATO Science Committee hosted 62 leading academics and professionals of the young computer industry in Garmisch, a beautiful place in Bavaria, Germany, at the foot of the north face of the highest German mountain. During this conference, the term “software engineering” became popular and started its journey through our domain.

Not too long ago, I understood agile development as an oppositional concept to software engineering. I had identified software engineering with the heavy document-driven processes widely in use in the 1980s and 1990s. This is a popular conception; especially one the proponents of traditional methodologies like to support. The term “software engineering” carries the flavor of a sound, professional discipline and connecting it to “capability” models or processes definitely helps in the debate.

So is traditional software engineering really the sound, disciplined approach, while agile is the free-roaming, chaotic (and thus unprofessional) hacker style? I don’t think so. I’d like to take you on a short journey 40 years back in time. Admittedly, I was more concerned about my sandbox at that time than about software development, but the conference reports available today give a good impression of the discussions that took place in Garmisch.

One statement caught my attention at the first reading. Douglas T. Ross, member of the Massachusetts Insititute of Technology (MIT) at that time and inventor of the CAD paradigm, said about the design of software: “The most deadly thing in software is the concept, which almost universally seems to be followed, that you are going to specify what you are going to do, and then do it. And that is where most of our troubles come from. The projects that are called successful have met their specifications. But those specifications were based upon the designers’ ignorance before they started the job” [1].

Does this sound familiar to you? It is exactly the same reasoning we use today to explain the success of agile projects. The agile seed was already sown 40 years ago, but for some reason the waterfall plants grew faster. They proliferated the industry to an extent that they were identified with the term software engineering. And only a few professionals — mainly from the Smalltalk camp — still cared for the iterative plants and developed them further until they became what we know as agile today.

Agile development uses many elements of traditional software engineering in a much more rigorous way, such as automated testing, version control, and code reviews (at least if you do pair programming). It also celebrates the quality of design.

My conclusion is: rather than being an opposition to software engineering, agile development is an alternative draft of software engineering. It avoids many of the traditional misconceptions, such as “you have to draw your plans first, before you build,” or “building software is like constructing a house.” Instead it adds the focus on business value, like any other mature engineering discipline. It is a full-grown descendant of the early ideas of software engineering.

Agile development may add some new misconceptions we don’t know about yet. Probably we will know 40 years from now, but then most of us will not be concerned about software development too much anymore.


Jens Coldewey

Jens Coldewey, based in Munich, Germany, is a Senior Consultant with Cutter's Agile Product & Project Management Practice. He specializes in deploying agile development and object-oriented techniques in large organizations.


  7 Responses to “An Agile View of Software Engineering”

  1. Jens,

    It is interesting to read your post, and others like it that come from the Agile community these days. I agree with the principles of Agile, that it can be considered an alternative draft of software engineering. I was at the PNSQC last week, and listened to Ron Jeffries and Chet Hendrickson talk about Quality Dynamics of Agile Software Development. To summarize, a lot of very strong software engineering concepts, sprinkled with a number of suggestions that these are ideas that are new and that make agile unique and different from ‘those other approaches’ (there is often the suggestion that if the project is bad, it can’t be agile). I strongly disagreed with that that implication, and fear that it overpowered their more important message that personal skills matter. It is not that agile is new, and your blog agrees with that by quoting challenges from 40 years ago that still exist today, it is rather, as you suggest, a re-packaging of strong ideas that have been around a while.

    The parallel I see between software engineering from 40 years ago and agile today is the huge gulf that remains between the state of knowledge and the state of practice. Fred Brooks said that “The gap between the best software engineering practice and the average practice is very wide – perhaps wider than in any other engineering discipline.” Compare that to something Peter Coffee said a couple of years ago: “Auto mechanics, it has been observed, invest more of their time and money than the average IT pro puts into maintaining professional competence, and it shows.”

    I agree with both of these quotes, regardless of when they were originally spoken.

    The challenge I see both with Software Engineering and agile is not the collection of ideas, but the difficulties in reasonably disseminating these ideas into the masses. In both cases, what we often see is superficial implementation of the specific practices, rather than a deep understanding of the principles behind these practices. The educational institutions must bear some of the blame, but much of the responsibility for this deeper dissemination of these ideas lies with the evangelists.

    Back in July 2004, around the time that Jim Highsmith released his Agile Project Management book, I wrote an article for the Cutter IT Journal titled Beyond the Hype of a New Approach. What I saw then (and still see today, as agile has, if nothing else, set records for remaining high on the hype curve) is that these great ideas have been challenged in making it across the chasm, in truly making inroads beyond the early adopters. This remains true for agile today. The evangelists need to step up and be responsible for fostering a deeper knowledge in the community, rather than the pithy comments that agile is better, suggesting that there were no project successes before a bunch of people got together in Utah a few years back. I do not see this happening quickly enough.

    My experience with a wide range of companies is twofold. When working with the large, established companies in the private sector, or large public sector IT shops, agile is either not on the radar whatsoever, or is just generating interest in some early-adopter groups, indicating that the movement is only now approaching this chasm. This is contrasted with my experience with small startups, the early adopters. In most of these shops where companies suggest they are using agile techniques, the state of practice is no better than before. Indeed in several companies, the business value being generated by the technology teams has diminished significantly. There are more instance of this in my experience than there are of success using agile techniques. I do not market myself as an agilist (and don’t expect to), but I end up correcting far too many ‘agile misconceptions’ in the field. The practice rarely meshes with the theory, particularly at the principle level.

    The core ideas behind agility: strong visibility into progress, wide open communication, early delivery of value, adaptation to current knowledge, personal skills matter, are being lost in the cacophony of things that the developers want to hear: index cards over specifications, refactoring over good design. “Don’t worry, we’ll get you something” becomes a mantra that developers like, but business owners dread. The core ideas are being stated, but they are not being heard. This has been true for 40 years or more.

    If we can understand and address this communication gap (by taking ownership for closing this gap), then there is the possibility that software engineering (or agility in its current guise), may actually make a difference in our lifetimes.

    If we continue to simply rant from the pulpit, though, the important message will still miss the masses, and agility will fail to cross that same chasm that software engineering has failed to cross, even after 40 years.



  2. I think the biggest misconception about (non-software) engineering is that it is complete specification up-front. The more I’ve talked with or read about “real” engineers, the more I’ve learned that they follow a very incremental approach–when they can and when it’s not cost prohibitive to do so. Make a small hypothesis, do a small experiment, reflect on the results and generate a new hypothesis. I believe agile is far closer to this true notion of engineering than “software engineering” ever was.

    I like the points Jim brings out in his reply. It’s not the practices, it’s the core ideas, and those aren’t really getting across. But that’s not unique to agile. In reading more about lean manufacturing and about classic management truisms (Deming & others), it’s interesting how they say many of the same core things (e.g. quality decreases cost, improvement needs to be continuous, fear increases cost) as agile.

    Yet we find that most modern organizations still don’t want to listen. We see application of management practices without holistic understanding, and misery as a result.

    Unfortunately, I think the basic premise of the Law of Raspberry Jam can’t be refuted.

  3. I think, Jim is addressing a very important issue here: The best ideas are of no use if not communicated good enough. And they may cause even more harm than good if used imporperly. For example, just adopting an agile planning style using fast iterations and prioritized backlogs may lead into very unpleasant situations, if they are not supplemented with software engineering techniques that are able to support this agility over the long term; practices such as automated testing and regular refactoring.

    Jim is also right in my opinion, that the communication of these issues still needs to be improved, especially towards management and executives. The popularity (call it hype if you want to) of Scrum is partly because it is communicated in a way that appeals to management. Any feedback that helps to improves this is certainly wellcome in the agile community.

    And we also seem to fail persistenly communicating some of the more complex concepts of agility. Jim writes: “The core ideas behind agility: strong visibility into progress, wide open communication, early delivery of value, adaptation to current knowledge, personal skills matter…”. This is a great collection of core ideas. But then he states “It is not that agile is new … It is rather … a repackaging of strong ideas that have been around a while.” What’s wrong with that?

    The theory of complex adaptive systems shows us, that these systems develop their behaviour not from the single elements but from their packaging. Consider a single ant: An ant is quite a simple creature showing pretty predictable behaviour. However, a state of ants, having no other mechanisms than some million individual ants interacting, shows extremely complex and adaptive behaviour that is unpredictable. So agile is not just a repackaging of of practices, it is the packaging of practises that leads to the desired behaviour – if done well.

    By the way, there also new practices, such as refactoring. It was given birth by the PhD thesis of Bill Opdyke in 1992 and is still heavily researched today.

  4. avatar

    The notion that we software dev/eng teams are ignorant when we start an effort is not a great generalization. Three major categories can be carved out here:
    – Experienced team saavy with the domain in question, known challenge, known system design pattern — Recipe for success. Team is not ignorant
    – Inexperienced team in the domain – stop there. Team may have a lot of learning. Success is less likely, but worth a try
    – Inexperienced team, relatively new challenge, pioneering new system ideas. Sure that’s risky.

    This may be a question of Risk, not ignorance. If an organization is willing to take on a new challenge lead by inexperienced developers or lacking sufficient domain expertise, that is their deal — they need to assess risk ahead of time. Maybe software projects today still have a high failure rate because we’ve had a few decades of risk takers and constant evolution.

    I do not see this as much an engineering/software team failure as it is leadership choices. Hence Agile methods try to deal with incremental near term successes (to avoid longer term political, managerial complications)…. They also help with reducing level of expertise required for success.

    Summary — in past decades, failure of big projects is not an issue of developer ignorance across the board. Agile methods help, certainly.

    just a thought.

  5. Marc wrote: “The notion that we software dev/eng teams are ignorant when we start an effort is not a great generalization.”

    Well, I agree completely. Unfortunatley I failed in trying to find out, where you saw this notion expressed. Can you help me?


  6. Thanks for this useful post…

  7. Super share it is without doubt. We have been awaiting for this information.

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>