agile sucks brings up an impressive 3.2million pages. Viewing agile (failures) through the feedback loop lens can give interesting insights that helps us to make agile work better
Projects fail, and they do so often. Some statistics claim that 70% of all projects fail1. Especially "agile" faced criticism from many sides. For example, googling for agile sucks brings up an impressive 3.2million pages.
Viewing Agile Through The Feedback Loop Lens
It is difficult to make predictions, especially about the future.
– someone smart
Feedback loops are an important (the most important?) way to align development with new knowledge and changed business. Feedback loops allow to react to opportunities and misconceptions. They are an2 important lens through which to analyze software development.
The traditional (think Waterfall) development models often have very slow feedback loops and are not only prone to not meet the specification, worse, they are prone to not meet the needs of the business.
Agile accelerates the feedback loops in order to ensure that specification and project results meet the needs of the business as good, as possible.
To achieve this, agile models implement certain measures, such as
- making business an integral part of the team
- aligning the goals and the plan with business more often
- deciding locally (govern globally, decide locally)
Specifically the "classical" agile approaches focus on the feedback loop between business and developer in order to increase the elusive3 business value.
Yet, many agile projects fail to meet the expectation of their sponsors and investors (the business). By using then lens of feedback loops, some insights into reasons and measures can be gained.
Making business an integral part of the team
Knowing business and predicting what would improve business is an expert skill that requires intimate knowledge of the business. This knowledge is often present in domain experts (or subject matter experts as they are also called) that have worked in the business for many years. Domain experts are good at identifying and understanding processes, business rules, and they many unnamed things that make doing a specific job difficult. When looking at improving internal processes it is also very helpful to have a good network throughout the company; this is also something that domain experts often shine in.
Writing software is also an expert skill and it is very fortunate to have a software engineer that is also a business expert. In the vast majority of cases though, the software engineer and the domain expert are different people.
Getting the domain expert and the software engineer together is what making business an integral part of the team is all about. When ideas flow freely between the domain expert and the developer, then there is a good chance that the resulting software has the desired business value and is technologically sound. There is also a good chance that novel and disruptive ideas come up when the developer challenges assumptions of "undoable" of the business expert or vice versa.
Aligning the goals and the plan with business more often
Having great ideas and great development speed in a project is great. But sometimes circumstances change. A new business opportunity came up and resources need to be shifted. New technologies would make the in-progress project obsolete in the foreseeable future.
Agile projects accept these changes as a reality and embrace them.
Agile means to change the plan as often as necessary. But in order to change a plan, one must have a plan. Stakeholder management with customers and management, resource planing with HR and finance, and cooperation with other teams are necessary for a company to thrive and grow, and they all can greatly benefit from a plan.
It is a common misconception that agile projects do not need a plan. Slogans like YAGNI (You aren't gonna need it) are at times understood as do not plan for the future. But YAGNI is better understood as to do things when they are actually providing benefit for the current problem at hand4. Having a plan supports company wide communication, cooperation, and coordination. Having an overly detailed plan is arguably something you aren't gonna need.
Amazon with their working backwords philosophy very vividly shows, that having a plan can be a key differentiator for a business.
Deciding locally (govern globally, decide locally)
A feedback loop is only as fast as its slowest decision. Certain decisions, for example by a steering board, can take weeks to prepare and get. It is not unheard of, that getting a decision is more work than implementing it.
The mathematical truth is that each added stakeholder increases communication cost quadratically at worst5. Decision contention can turn work that could be parallelized into sequential work, with severe effects6.
TL;DR -or- But What Does It Cost?
To summarize: Faster feedback loops are good for business. Agile accelerates the feedback loops by implementing measures:
- Agile makes business an integral part of the team, so the right things are done, and they are done right
- Agile aligns the goals & plan with business more often, so business agility is supported by embracing and communicating change
- Agile projects decide locally to relieve the organization and speed up development
These things are not independent of each other. For example, local decisions are only possible with a domain expert on the team. Creating a sensible plan is also much more difficult without a business expert on the team. A business expert in the team that can only bring in ideas coming from the management is also not functional.
Business has to support the agile way of working. Agile does not work without business support for local decisions and agile planing. Although not always a reality, this is a widely accepted prerequisite for successful agile projects.
The elephant in the room is: How much time will the domain expert need?
It is common wisdom that project management adds 8-20% of developer time. In other words, one full-time project manager for every 5 to 13 developers.
Given the prevalent management acceptance of substantial project management efforts, it is a curious observation that this does not seem to hold true for domain experts. In my last 27 years of professional work in the software industry, this observation has been confirmed again and again.
For example, a widely adopted technique for agile in-house projects is that the domain expert supports the agile team in parallel to his or her line duty tasks. In product development I have often seen that the product manager spends substantial time with external obligations such as preparing for fares, customer sales contacts, or press releases. The sad reality is that line duty and external obligations almost always win over being the domain expert of the team.
For almost all projects, what remains is not enough. And the consequences are severe, as the following 10 steps to project failure illustrate.
- domain experts are overloaded and overwhelmed
- developers are stuck in endless meetings trying to get insight from their domain expert
- the domain expert feels compelled to just tell them anything to do without rigor thought or planning
- the feature scope is no longer aligned with business needs
- the domain expert gets swamped with feature wishes that he or she can no longer assess if they make sense or not and if they fit into the overall product vision at all
- developers struggle to find clear abstractions and architectural artifacts
- more and more waste is generated because requirements and goals are not clear and implementation is difficult because of missing abstractions and consistence
- developers get frustrated because they cannot invest time into technical quality measures because the backlog is full
- management loses trust because the project fails to deliver value
- the project fails
It is very important to observe that this is not the fault of the domain expert. It is caused by the domain expert not having the time needed.
What is the solution then?
- Plan the domain expert with at least 15% of the developers in the team (one domain expert every 6-7 developers)
- Plan the domain expert fulltime with no other obligations. It is better to have a domain expert that has time to look for ways on how to further support the team than to have one that is not available
- Plan the project manager in addition to the domain expert
Footnotes
Statistics about project failure by Teamstage:
- 70% of all projects fail
- […]
- 55% of project managers cite budget overrun as a reason for project failure.
- […]
Teamstage further claims that 44% of projects fail due to lack of alignment with business:
44% of projects fail due to a lack of alignment between business and project objectives.
As software project failure statistics show, the lack of alignment of projects with the business objectives is the reason why nearly half of the strategic initiatives fail. The lack of alignment results in less organizational agility which leaves businesses unable to reach strategic shifts in customer demands, leverage the expected economic growth, and mitigate losses.
Also compare with The DevOps Handbook by Kim et al. and The DORA report
I reccomend the excellent "The Art of Business Value" by Mark Schwartz
Compare to Ron Jefferies quote "Always implement things when you actually need them, never when you just foresee that you will need them" https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it
Coda Hale has some good thoughts on that in "Work is Work" (https://codahale.com/work-is-work/)
Amdahl's law states that all work is dominated by the sequential efforts