We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
— The Agile Manifesto.
Computer science is a young field that has evolved at an amazing speed. It’s easy to forget that the first general-purpose electronic computer, the ENIAC, was turned on for the first time in 1946. Today we have computers even in our pockets.
Due to that quick progress computer scientists and technicians knowledge and abilities were outrun by the computers capabilities. This widening gap brought humanity to what was called the software crisis:
The major cause of the software crisis is that the machines have become several orders of magnitude more powerful!. To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem.
— Edsger W. Dijkstra, The Humble Programmer
The crisis manifested itself in several ways:
- Projects running over-budget.
- Projects running over-time.
- Software was very inefficient.
- Software was of low quality.
- Software often did not meet requirements.
- Projects were unmanageable and code difficult to maintain.
- Software was never delivered.
It had become such a great problem that in 1968, NATO held the Software Engineering Conference to address it, and was at that conference where the term Software Engineering appeared for the first time.
As the programming problems being tackled became larger and more complex, the ad-hoc processes that had been useful for small research teams became unfeasible and project managers turned their heads to other technical fields, such as industrial engineering or architecture, and their methodologies and processes where took as a reference.
But in these industries risk cannot be tolerated due safety and economical reasons that make corrections after the design phase impossible or extremely expensive to implement. So their processes and methodologies are strict and averse to change. Every phase must be completed and validated before moving to the next one.
This led to a methodology that mimicked those constraints, even though in most software projects they are not present. This methodology was first, informally, introduced in 1956 in the Production of Large Computer Programs presentation by Herbert D. Benington, where he described the software production for a defense system.
But it wasn’t until 1970 when Dr. Winston W. Royce published the paper Managing the Development of Large Software Systems in which the methodology, that became known as Waterfall, was introduced. Dr. Royce wrote the paper after his experience managing large software teams developing software for spacecraft mission planning, commanding and post-flight analysis, a field where error is not an option.
In this model the progress goes from one phase to the next downwards, like a waterfall.
The Waterfall Model
The problem of this model, as stated by Dr. Royce, is that the implementation is risky and invites to failures if the requirements aren’t perfectly defined and static. The worst thing is that when there’re failures, usually noticed at the testing phase, they’re costly to fix as a major redesign might be required and those changes are likely to be so disruptive that a major redesign or even new requirements are required, creating up to a 100% overrun in time and/or costs.
The solution, also stated in the paper, is feedback and iteration. But the tools they had at the moment made it basically impossible, developers didn’t have access to much computer time.
Fortunately as computers became cheaper and more powerful and the development tools improved, the Iterative and Incremental Development process and derivates became possible. In these methodologies the programs are developed through repeated cycles (iterative), like small waterfalls, adding functionality on each repetition (incremental).
Iterative and Incremental Development
The first documented case of Incremental and Iterative Development is NASA’s Project Mercury in the 60s. In 1986 Barry Boehm published A Spiral Model of Software Development and Enhancement in which he presented the Spiral Model, an iterative and incremental model with very long loops, from six months to two years. It was the first time that the risk-driven-iterations and the need to do risk assessment in each cycle was formalized and highlighted.
It wasn’t until the 90s that a number of methodologies were developed based on the Iterative and Incremental Development model with shorter cycles. Methodologies such as Scrum (1995), Extreme Programming (1999) or Test-driven development, that nowadays are known as Agile Methodologies after the 2001 Manifesto for Agile Software Development.
The Agile Methodologies share some core concepts but differ on how they implement them. Those concepts are:
- Do not involve long term planning.
- Features are broken in small pieces and iterated.
- Iterations are short, usually from one to four weeks.
- Each iteration ends with a deliverable, which may be released or not. So it includes a full software development cycle including planning, requirements analysis, design, implementation and testing.
- Teams are small, multidisciplinary and self-organizing. Efforts that require more people can be obtained by using several teams working on different features of the product.
- Focus on the code instead of the documentation.
- The customer is a stakeholder and should be present during the whole process.
At emtrics we are Agile, we use a mix of Scrum and Kanban which I will talk about in another post, but we always have present that there are no silver bullets.
