The Biggest Challenge in Estimating and Planning Knowledge Work

In knowledge work, the biggest challenge in estimation and planning is not incorrect estimation or weak execution—it's the inherent variability of the work itself.

Frameworks like Cynefin and Stacey's Matrix remind us that complex work (like software development) has a non-linear and unpredictable relationship between cause and effect. Unlike complicated work, where expert analysis can narrow down scope and cost with precision, complex work doesn't follow a predictable formula.

Take this example: fixing a car or laptop is complicated, expert knowledge can help understand the problem and find a precise solution with a high confidence. Preparing for the PMP, SAT or GMAT, on the other hand, is complex—you may know the syllabus, but estimating the time, effort, and strategy required is far less certain. And sometimes, we might need more than one attempt to get the desired result.

Software development falls into this complex domain. Variability in requirements, technical choices, and team dynamics makes accurate prediction extremely difficult.

The Common Mistake Leaders Make

The real issue isn’t the unpredictability—it’s how we respond to it.

Many leaders continue to treat complex work as if it were complicated. They try to eliminate variability through upfront analysis, expecting teams to “just follow the plan.” When actuals deviate from estimates (which they often do), the blame often falls on teams – “They did not estimate correctly” or “They did not work hard enough.”

Instead of adapting their approach, leaders double down—tracking the difference between estimates and actuals, and tightening control.

The frameworks change, the practices change, the projects change – yet leaders continue to use complicated-work approach for estimating and planning complex work, hoping they will achieve the desired predictability very soon.

This approach has led to shallow Agile adoption in many organizations. Despite widespread use of Agile frameworks, few have truly embraced Agile planning and estimation practices designed for complexity.

So, What Works Instead?

Below is a summary of a more effective approach to estimation and planning in complex work—drawn from my Udemy course Succeeding with Agile: Estimation and Sprint Planning:

1. Acknowledge complexity. Understand that software development is knowledge work with non-linear cause-effect dynamics.

2. Effort is a range (a probability distribution), not a precise number. Every work item has inherent variability due to individual and environmental factors.

   

   
   

3. Collections of tasks amplify variability. If one work item has effort variability, a collection of work items (Epic, Feature or Sprint) would have a wider effort variability. It is difficult to pre-determine exactly how much effort will be required to complete a collection of work items.

   

   
   

4. We must adopt a flexible and adaptive approach to planning and execution. It is not practical to freeze both scope and schedule. We should keep at least one of them flexible during planning, and then use feedback loops to learn and adjust during execution.

5. Fix the timebox, flex the scope. Since the Sprint length is fixed, we should consider allowing some flexibility in what exactly gets done during Sprint.

6. Prioritize value over volume (and direction over pace). To ensure the scope flexibility has minimal impact on delivery, we must focus on maximizing value instead of maximizing the number of work items completed during Sprint (whatever order).

7. Plan for adaptability during execution. While some flexibility of scope can be planned during Sprint planning, we would need adaptability during Sprint execution as well.

Learn More and Go Deeper

These are just the highlights. If you're a product leader, Scrum Master, or team member wanting to improve how you plan and estimate in Agile, I invite you to explore the full course on Udemy [limited time discount link]:

Succeeding with Agile: Estimation and Sprint Planning

I’d love to hear your feedback, and your own experiences with estimation and planning challenges.