I was listening to this SE Radio podcast on why big IT projects fail and it struck a nerve. I’ve been a part of my fair share of big IT projects that have failed. The common occurrence of big IT projects failing is not a new development. As technical leaders, we have to be mindful of the perceptions of our executive peers and work to manage projects to achieve the necessary outcome for the business.

The discussion on this podcast was wide ranging from a business perspective on the risks built in large IT projects and how to mitigate those risks. I recommend as a technical leader listening to this discussion because it provides an objective point of view on how the business perceives our large technical projects. In particular, it confronts the optimism that often comes from technical managers that results in cost or time overruns.

This McKinsey article authored by one of the guests highlights four ways to improve project performance. In particular, there are four key practices the article discusses. To paraphrase:

Manage strategy and stakeholders instead of just budget and schedule
Get the right talent to master the technology
Align the team incentives with the goals of the project
Excel at core project management practice, including short delivery cycles

If you are like me and a have found your professional career evolving through the development of agile and scrum, your mind may immediately start to wonder if agile is the solution. It isn’t surprising these sound a lot like the Agile Manifesto. In particular:

Individuals and interactions over process and tools.
Customer collaboration over contract negotiation.
Working software over comprehensive documentation.

The guests address whether agile as a methodology can mitigate these risks effectively (it would be one of my tactics). The conclusion was that most agile projects are smaller than the projects they assessed and therefore were difficult to compare apples-to-apples. In the “agile at scale” projects they did research, they reflected that the mission of agile (deliver the right thing eventually) created a different kind of incentive from large-scale waterfall projects to deliver on-time or on-budget.

Another interesting point from the guests was that organizations that use agile to manage projects have a fundamentally different orientation. An organization that uses agile processes does not take a baseline of a project for the full life of the work, and therefore it isn’t possible to determine if a project has gone over budget or off schedule. Instead the focus is on hitting the right end-product deliverable exclusively. I found that conclusion compelling in understanding how outside executives view agile development - you must fundamentally let go of making a fixed budget for a deliverable. This principal is very uncomfortable for many organizations.

I found it interesting that one of the observations was that the risk of failure wasn’t significantly dependent on the process the project used. Meaning, in their opinion, agile isn’t a silver bullet to solve the risks of large project failure. I would conclude that many of the practices promoted in this discussion are exactly why agile works. Focus on the stakeholder and communication with aligned incentives for your project. Manage your project tightly. Like so many lessons in life, there is more than one way to implement these values.

One of the key lessons from the research discussed in the podcast was that a large portion of the projects that are canceled are due to changing requirements. Many long-running IT projects are canceled not because of poor performance of the project, instead the outside environment changes enough to eliminate the value of the project. This conclusion is an area where the agile development process shines, using short iterations and adjusting what the right end-product outcome that will serve the business value.

In the end, this discussion was valuable in the objective assessment of process where sometimes as technical leaders we are easily blinded by a process we believe is “right”. There is more than one way to implement the important values required to keep a project from failing. Simply implementing agile is not sufficient to mitigate the risk of failure, and as a technical leader we must be prepared to discuss the practices we see as critical to success inside any methodology.