The Big Lie Inside Project Management

Why your projects struggle and often disappoint.

The way you manage projects is probably broken, though you may not know it. There’s a big lie embedded in our traditional view of project management, one that creates excessive optimism, followed by deep despair and then loads of pain as teams struggle to succeed.

The lie is a complex one that involves how optimistic managers and leaders can be; how blind we can be to how poorly we communicate; a mistaken notion that the problems we have on projects are because of our teams, the people doing the work; and a completely mistaken notion of how and when scope becomes defined.

In the more than 120 organizations we’ve worked with, the Big Lie — evident in what we call the Ignorance Gap — emerges as the single greatest source of project challenges and failure, spreading misery throughout the organization and even your clients and stakeholders. That’s a lot, right?

Project Management Fails Most of the Time

Most people don’t realize that modern project management is pretty much a failure. One of the most credible sources of data on this failure is the Standish Group and their biannual project success survey, the Chaos Report. In their three categories of project completion (Successful, Challenged and Failed), the data across the last 10-plus years has been pretty stable, matching what the 2015 Chaos Report showed (see below): only about 30% of all projects are actually successful. Almost twice that many do finish, but they struggle with one or more issues of scope, schedule, budget, quality, satisfaction, and more. These are the Challenged ones.

Consider this: We are in year 25 of the practice of modern project management. You could call this moment the zenith of the craft, as there are a massive number of educational programs, including degreed programs, and a wealth of tools and techniques available. But a 30% success rate is a clear indictment of the discipline.

The Big Lie of Project Management is that having a plan and a project manager creates the understanding within the team doing the actual work.

It seems that what we call project management does little to ensure project success but is good at tracking and documenting struggles and failure. That’s what the Chaos Report confirms.

Understanding Ignorance and Project Management

Here’s a really useful way to think of managing a project and its scope: at the start of the project, we are at the point of maximum ignorance, and when we’re done, we’re near the point of maximum understanding.

It is safe to say that the two points shown above are immutable — they will always exist regardless of what we do — unless we fail, of course, in which case the second point may be lower on the chart. Want to write a great project plan? Wait until the project is over.

The real measure of how well we are doing in project management is how quickly we remove ignorance from the project. That would show up as the shape of the path between the two points. So let’s redefine the art of project management:

Project management is the art of guiding a project from ignorance to understanding as effectively as possible.

The sooner we remove ignorance, as we will see, the better things will go.

Our Project Blindness: A Very Human Tragedy

Let’s take a look at how we think project management works — a quick review of our behaviors. I’ll use a term, Apparent Understanding, and a score on a 1–10 scale to reflect how well we think we understand the project (scope) at that point.

1.A project is identified by some managers and they talk about it a bit. If this is a sales-driven organization (like an agency), then this is the sales and client-services people. Apparent Understanding level: 3.

2.We’ll conduct a few meetings, kicking around some alternatives and choosing an approach that we think might be the best. At an agency, this would mean the top specialists look at it and give their opinions on it and shape it a bit. Apparent Understanding level: 7.

3.Someone actually writes all this up and maybe even makes a plan in Microsoft Project with a schedule or GANTT chart or some other fancy thing that implies that this is really well thought-out. As humans, we respond to such shows of knowledge and responsibility by naively assuming that everything is taken care of. In an agency, this looks like three pages of descriptions, assumptions and schedule attached to the standard contract template. In other organizations it may not even be that, but just a discussion during a meeting where heads nodded, and then the next topic came up. Once our stakeholder or client gives the go-ahead, we’re all set. Apparent Understanding level: 10.

4.We hand it off to the team that will actually do the work, briefing them, sharing whatever documents we have with them. They will find a few mistakes or misunderstandings, we’re sure, but really not more than 5% or something that will not really impact our success. Apparent Understanding level: dips a bit during execution, but the team figures it out and gets it back up to a 10.

5.We’re done! It works, and we’re all happy, especially the project manager, who is wearing the “I told you so” smile.

But is that true? Managers feel that way, for sure, but teams probably feel differently. As we’ve discovered over years of studying this, and as shown in the image below, the Actual Understanding of Scope is much lower.

We are very optimistic creatures, and optimism may be what it takes to get out of bed each morning, especially if you’re a team member. The real path from ignorance to understanding usually takes a far more dismal route:

1.The project is identified by some managers and they talk about it a bit. If this is a sales-driven organization (like an agency), then this is the sales and client services people; otherwise it is your internal clients or your organization’s leadership. Actual Understanding level: 2.

2.We are very optimistic and excited about what a great project this could be, trading between those involved a set of phrases such as: “Don’t worry, the client or stakeholder gets it.” Though in the next breath they will often point out that that same person “doesn’t get it.” Or: “It is just like that other one that we did.” Although that was last year and those people are gone, and everybody has conveniently forgotten how bad that project was. Or: “It should be simple to do quickly,” or, “How hard could it be?” Even though it is a new technology or something that we have never done before. (A study that we did for Digiday showed that project failure rates increase with the newness of technology and technique.) As a result, nobody is asking the hard questions. We call this the Plateau of Optimism. Actual Understanding level: 3.

3. At the far edge of the Plateau, while people are high fiving, the hand-off happens, the plan is locked in, the client or stakeholder signs off, and the unlucky team gets assigned. The crows come home to roost, darkening the sky as the team slides downhill into the Chasm of Despair. In the Chasm the hard questions are finally being asked and few answers are available. The team struggles with unexpected choices and unforeseen obstacles, reaching up in desperation to those who originated the project, pulling them into the chasm as well. Managers will blame the team for its lack of skill, vision, or basic competence. There is good news though: the Chasm itself is the crucible within which true Understanding will emerge. Actual Understanding level 2–4 (and the only direction from here is up).

4. And up they go, the team they do. With a Seussian tenacity, the team pulls itself up that mountain, aptly named Mount Death March. The height of the mountain comes from the lack of Understanding, but the steepness of the mountain comes from the optimism in the project plan, the belief that existed on that flower-bedecked Plateau of Optimism, where things that go wrong are never spoken of. That plateau was a long time ago — or so it can feel to that team climbing the Mount. Actual Understanding level: climbing, eventually reaching a 10… or close.

Project managers (as well as most people involved in projects) will find the above narrative all too familiar. That moment when we are deep inside the Chasm is, in fact, the moment that we experience the Ignorance Gap and the effect of the Big Lie.

The Big Lie of Project Management is that having a plan and a project manager creates the understanding within the team doing the actual work. In reality, most plans are not read, nor are they accurate, and the act of creating a role (project manager) that will “own scope” merely means that we are again delaying the team’s ascendance to Understanding. That project management makes projects successful is a lie; it is teams and their understanding that make projects successful.

Bridge the Chasm with Conversation

The project manager (and project management as a whole) can be thought of as a way to bridge the gap between fast-moving, idea-driven managers and clients and the slow, thoughtful and thorough world of the team. But in reality, substituting a project manager for a thorough conversation allows managers to avoid the details and the discussion needed to create knowledgeable teams and chase away Ignorance.

We can improve project success rates dramatically and defeat Despair.

If we remove the project manager and just fix the conversation instead, then things get a lot better, and quickly. It is not an easy conversation — neither intuitive nor comfortable — and it took us years to figure out what that looks like, but it creates some amazing results. Our clients typically report a 50- 80% reduction in late or over-budget projects, and also happier teams and clients that spend their days far from Mount Death March.

It’s ironic, but getting projects done quickly requires moving more slowly, avoiding optimistic blindness, and slowing down the speed of conversation while increasing its depth. You can think of it as a Bridge of Enlightenment, a different way of bridging between managers and teams, a shared understanding of scope.

The Hidden Costs of the Ignorance Gap

There are many direct costs from allowing the Gap to live on, including the health of the business (who wouldn’t want more project success?) and the happiness of the organization.

There are more subtle effects of the Gap as well, including Project Contagion, misaligned roles, and clients who frequently change their minds on scope — I’ll discuss these in subsequent articles.

Many shops feel that they just need better project managers or need to manage “harder.” Neither works; flogging the team does nothing to boost productivity, quality, or morale. Instead, teams just slog onward, doing the best they can. The big lie of project management casts a dark cloud over these organizations; even project managers can feel powerless as teams struggle to feel that they’re doing a good job or ever getting enough done.

The whole story in this article, its insights, and the understanding I hope you gained, are from one of the most popular class sessions in our AgencyAgile trainings. Lately I’ve realized that there is a pretty high level of misery in business (not just agencies) because of project management failure and the managerial conundrum that it presents.

Jack Skeels is a former RAND senior analyst, recovering client services executive, and the founder of the Agile transformation and coaching firm that helps agencies, marketers, and other complex service organizations, AgencyAgile.