Digital PM Metrics: Outcomes vs. Outputs

If you are a Project Manager, I’m sure you are very familiar with at least some of the popular metrics such as SPI, CPI, SLOC, Defect rate, etc. I am also confident that you are often asked to report many metrics to senior management that may or may not make sense to you personally because they don’t seem to add much value. Assuming that this is true, I would like to share a few thoughts on this topic with you and see what you think.

As someone who has managed more projects than I had planned on in my career, I have spent countless hours gathering and organizing data in order to meet the standards of my employer at the time. I often find myself wondering “What’s the point of doing all this? Who looks at this and do they really care about this information?”. I know this sounds a bit cynical and negative, but as I worked on more and more projects for different companies, I began to see a pattern: most organizations don’t know how to measure success very well. Having worked in many PMOs, I noticed that most PMOs stick to the basic metrics for assessing project or program health – data such as schedule performance (SPI) and cost performance (CPI) are standard for most project-based organizations because that is what companies are used to doing for decades. While these metrics offer insights into how a project is progressing, I argue that they do not necessarily measure success. Even if a project is completed on-schedule, on-scope, and within budget, which seldom happens based on numerous studies and surveys that have been conducted over the past several decades, these basic metrics fail to provide an indication of how the outputs of the projects actually affect the organization.

So what am I talking about here? What I’m seeing is that most organization have been going about metrics incorrectly, and more any number of reasons, continue to fail to understand how the activities of their employees actually impact the business. Even the most technologically advanced companies will struggle to understand how their projects impact the strategic goals of the organization if they do not focus on outcomes rather than output.

Here’s my hypothesis (which some of you may not agree with, which is perfectly understandable): companies need to learn to measure outcomes instead of outputs in they want to gain a true understanding of the value of the projects they undertake.

The next obvious question is: What’s the difference between output and outcomes? I will provide a few examples below.

Outputs relate to attributes such as volume (how much work did we do?) and quality (how “good” is the product or service that we produced?). Metrics such as CPI measure how well we did in terms of managing project cost, whereas SPI provides an indication on how we did we follow the original plan in terms of timeliness of activities and deliveries. Volume tells us how many “things” we produced, how many features did we build/ship, how many lines of software code did we write/test, etc. Also, quality tells us how many issues did we encounter, how many defects did we create/resolve, etc. While all these data points are important to understand, they do not always contribute directly to fulfillment of business objectives.

How do outcomes fit into this? A few examples of outcomes are as follows:

1. Increase number of active users/customers by 10%
2. Improve employee engagement survey scores by 20%
3. Decrease time to repair/restore by 10% each quarter
4. Reduce defect rate by 5% each release
5. Increase number of product shipments by 10% each quarter

Outcomes provide a different focus than outputs because they are directly related to the organization’s key objectives. The concept of OKR (Objectives and Key Results) is very similar to KPIs, and it enables an organization to focus their energy towards the desired end state. Hence, the goal is not simply to complete projects successfully (i.e. on-schedule, on-budget), but to ensure that the project delivers tangible benefits to the organization that is quantifiable and meaningful.

All this may sound like common sense, so why do companies not put this into practice? I believe that there are many potential causes for this. Some companies do not understand how to define a direct relationship between activities (projects) to the outcomes due to lack of experience. Benefits realization, in my experience, is something that sets the leaders apart from the rest of the field because it requires a concerted effort to master. I feel that a good place to start is with measurable goals (i.e. “SMART” goals). By making goals Specific, Measurable, Achievable, Realistic and Time-bound, teams and organizations can take a step towards having meaningful measures.

Eugene Lai