Exposing Agile Software Development Myths

Agile development is a movement that started 11 years ago and now is a commonplace discussion for software development teams. Prior to the signing of the Agile Manifesto in Utah in 2001 [endnote 1], the world was using a phased methodology called “Waterfall,” introduced (but not named) by Winston W. Royce [2], ironically trying to explain why that methodology was a bad idea. The methodology forced the completion of a phase in order to continue with the next one. That caused some problems since, no matter how hard the developers tried to gather all requirements and create a full system design, there was always the need in the following phases to go back and change things. Those regressions had a big impact on cost.

The Agile principles tried to eliminate all the rigidity of Waterfall by focusing on more productive tasks, like working code over comprehensive documentation, adding more customer collaboration, enforcing individual interactions instead of tools and being agile by responding to change instead of following a static plan [1]. Those ideas took the world by storm and suddenly many software development teams were adopting one of the different methodologies that claimed to be Agile in order to deliver better products at a lower cost, on time. But that did not always happen.

Some public information began to appear trying to document Agile failures [3] [4]. Why was Agile failing? The common answer was that people did not truly understand Agile and were doing it incorrectly. Although that may sound more an excuse with the methodologies not truly acting in an agile way, with static procedures the team must follow without missing a single step, the truth is people were adopting Agile without fully learning and understanding it. Due to that misunderstanding, a set of myths emerged, misconceptions that further made people fail to achieve the promised business value. Let’s review some of those myths.

Myth 1: Agile means no phased development

Even worse: phased development means Waterfall which, in turn, means having design up front, and anything up front is not Agile. It is true that sometimes you can go on following Agile methods and still have Waterfall underneath [5], but there is also the underlying thinking of phased development as a synonym for Waterfall. That is a misinterpretation of the Agile principles and Royce’s original paper.

Royce explained the problem of large development by means of an example where he divided the development tasks (gathering requirements, designing, coding, testing) in strict sequential phases. The problem, he explains, is related to the distance from where the requirement is defined to where the actual achievement is verified (testing). He said a jump was needed to go back to the requirements phase, and the larger the jump, the higher the cost. But he didn’t say the phasing of work or even separating the task was the problem.

Still, the phases became a forbidden word in the growing Agile world, and teams started to try to do all the tasks at the same time or execute smaller phased cycles. Reducing the time for tasks causes an accompanying reduction in the result quality. Having too little or no time at all for designing (just jump into coding) generates a product that will probably need refactoring later; or even more, it may not integrate well or will not deliver the needed business value. The same will happen for poorly analyzed requirements or inadequate testing. Also, this leads to the next myth.

Myth 2: Agile needs no architect or project management

This is an interesting myth, based on the principles of favoring the response to changes over following a plan and favoring working code over comprehensive documentation. To many, project managers are directors that only dictate to software developers, draining precious time from development with useless meetings, reports and control. The architects are seen as bureaucrats, large document generators focused on up-front modeling.

So, in order to avoid following a plan and generating documents, Agile teams removed the architect and project manager roles. The problem is much of the project management and architecture tasks are required for successful development, especially in large systems. Many of the actual problems in development are due to the freedom the developers experience in the Agile process. While freedom is good, having no one to watch over in order to assure everything is working correctly can lead to a software product, large-scale Web, or mobile app that is not able to integrate correctly. The result can be time lost in refactoring and feature fixing.

Being able to take a look at the whole system in higher views (architect) and control and help with resources (project manager) is necessary for some more challenging and large projects to succeed. And of course, having these two roles in the team, working following solid Agile principles, will not break the Agile outcome desired.

Agile Architecture [6] and Agile Project Management [7] proposals are being created, this last one even presented in a book by Jim Highsmith, one of the original signers of the Agile Manifesto. Not to mention, Scrum Masters are often cited as proof of the need for project management in the Agile methodology.

Myth 3: Documentation is bad

Despite several attempts to make documentation fit the Agile world [5] [6], the ideal in the Agile movement is to have documents whose cost of production is lower than the benefit. That is, at least someone has to read the documents and use that knowledge. Documentation should then be created when necessary and at the correct time, not up front.

The documentation should be replaced by face to face communication per Agile principles. The problem here arises when there are multiple teams, separated by distance, or changing over time in a long-lasting software development project. In any case, some knowledge has to be captured in documents for others to read in case there is no chance of face-to-face communication.

Documents are also commonly associated with models, diagrams and manuals. There are other types of information that can be forgotten. The lack of documentation about decisions and the rationale behind them generates rework when team members start repeating the same mistakes or going the same path that was proven unsuccessful in previous projects. The lack of models also condemns the project managers and product owners to read code or ask developers directly if they want to know what is going on. This generates costs and there is a risk of poor quality information institutionalized to support future decisions.

Myth 4: Agile means no planning

This myth somehow relates to Myth 2 (we need no project manager controlling us with a static plan). The sins of a predictive plan are the risk of the actual prediction (requirements and time estimates) and the cost of changing the tasks in the plan to incorporate requirement changes.

But in Agile we still need to estimate work, albeit not to define the project length and cost, but at least to accommodate the work in a sprint and see if all the backlog of tasks are still executable under the timeframe and budget. Also, we still need to know what is going to be done in the near future to assign resources and check dependencies. So, we cannot get rid of planning that easy.

The actual meaning of “responding to change over following a plan” is related to awareness and feedback. Static plans are made using predictions and not allowing space for changes and decisions introduced by late-arriving information. In that case, a plan should be the one that defines the development principles and provides a guide for decisions along the path.

It must be structured to stop development at precise moments to incorporate all the knowledge gathered up to that point (feedback) and make the changes to the plan accordingly (no predictability, but adjustability). So, plans are important to provide a guide, but they should always be open to change.

Myth 5: Follow the methodology and you have Agile

The common answer to why Agile projects fail is simply because you are not doing Agile the correct way. Isn’t following Agile methodologies good enough to be agile? Actually no.

Agility principles try to improve the development cycle by making developers more collaborative, and including clients in that collaboration. It also tries to improve productivity by reducing non-beneficial tasks. Still, there are a lot of decisions that should be taken using information gathered from all stakeholders and coming from traditional development tasks like requirement analysis and system design.

.

The common outcome is, for large companies with large software systems using traditional processes, the change of methodology only happens at software team level, meaning the client stakeholders are not actively included on the team. With that we have lost most of the information that should come from the client. Then, by following the methodology only, there are other tasks that are not performed, hurting the second part of the information (peer reviews is an example).

Agile methodologies are a guide, but they are not the whole solution, and should be adjusted to the project characteristics. This also introduces our last myth.

Myth 6: Agile is good for any project

This myth may not be totally wrong but should be taken with caution. Agile has been proven as a good paradigm for small teams working on small projects or even large projects in small stages. It was clear right after the Agile Manifesto was signed, with the XP methodology stating it was good for small teams, and clearly explained in the first book about the topic in 2004 by Jutta Eckstein [10].

The main point here is there may be ways to apply the Agile principles to large projects and large/distributed teams, but all require modifications and special care. We cannot take the methodologies as they are and expect to have successful results without taking care of information volume, team and sub-teaming, colocation, etc.

All these myths can be found in discussions, blog posts, even books. Being aware of them helps managers and developers to adopt strategies in order to avoid them and also helps to understand more about Agile and how it can help deliver the desired outcomes.

William Martinez, a software architect and R&D manager at the nearshore software engineering services company, Avantica Technologies, works with venture-backed startups and the Software 500 in development, testing, high-end productivity evaluations, SOA technologies and mobile platforms. He is a REST evangelist and is currently writing a book on the subject from the architecture perspective. He also leads the IASA Costa Rica Chapter (International Association of Software Architects) and is a professor at the University of Costa Rica. He has been a technical reviewer of popular books including Fast SOA and Rest in Practice, and co-wrote the Planning and Evaluation of IT Projects book for UNED.