Counting story point is not a good practice
If you are leading a software engineering team, you are responsible for presenting a clear picture of team performance to the business. However, this can be challenging as commonly advised Key Performance Indicators (KPIs) are often problematic.
Here are some examples of misleading KPIs:
- A high volume of code produced may seem impressive, but it could indicate inefficient practices leading to increased bugs, accumulated technical debt, and degraded performance.
- High test coverage percentages might appear impressive, but if the tests are poorly written or don’t cover critical scenarios, they provide a false sense of security.
- A low number of reported bugs might seem positive, but it could suggest that users aren’t actively using the product or aren’t reporting issues they encounter.
- Consistently meeting all sprint commitments might appear productive, but if the team is under-committing to avoid failure, it doesn’t reflect true productivity or potential.
- Counting fixed bugs without considering their severity or recurrence doesn’t accurately reflect software quality.
- Delivering many features looks good, but if they don’t provide value to customers, it’s misleading.
With these examples in mind, the concept of output versus outcome comes into focus.
Let’s define output and outcome:
Addy Osmani, in his book “Leading Effective Engineering Teams” explains the concept clearly:
An output is a deliverable resulting from engineering tasks. It proves that some action was taken or the team did some work.
An outcome, on the other hand, is the actual result of the work done. It proves the team’s effort brought about a positive and valuable change.
Focusing solely on outputs can lead to the “watermelon effect.” This term describes a situation where outputs meet defined targets and metrics appear “green” on the surface, but underneath, it’s “red” because the desired outcomes or results haven’t been achieved.
A tangible example could be defining test thresholds in a pipeline based solely on overall coverage without specific statements on Function coverage, Statement coverage or Branches coverage.
Watermelon metrics can cause significant problems in project management and performance evaluation. These metrics are deceptive because they present a positive surface appearance while hiding underlying issues — like a watermelon: green outside, red inside.
This false picture of success can lead teams and companies to believe everything is fine when it’s not. Consequently, they might neglect to address actual problems. This can create a cycle where things seem to improve, but critical issues remain unresolved.
Ultimately, this can negatively impact project outcomes and overall organisational effectiveness. It’s crucial to look beyond these surface-level metrics to understand and address real challenges.
But the million-dollar question is: how do we measure outcomes?
Measuring outcomes is more challenging than measuring outputs. To do it effectively, you need information about the business and user needs. As an engineering team leader, this means maintaining close connections with the product team or business partners who possess this information. A communication gap can lead to project issues.
Therefore, the key to measuring outcomes doesn’t lie solely with the engineering team but in overall business performance, spanning multiple pillars of the business from Customer Care to Product and beyond.
Leave a Reply