All About Agile | Agile Development Made Easy!
Metrics
Hi all. One area I have always struggled with over the years is metrics. Since doing agile development, Velocity has given me an indicative measure of output, but it's not really possible to use as a benchmark between teams, so it gives me no real sense of whether the output is good or bad, just whether it's better or worse than before, which admittedly is still very useful. We've also been using a metric we call 'reliability'. This is the number of story points delivered as a percentage of the number of points committed to in sprint planning. This gives us a reasonable gauge of how good the team is at delivering on their commitments. This is a major factor in stakeholder satisfaction because it measures how well we meet expectations, at least in terms of volume of work delivered. However it doesn't help at all to measure quality, either in terms of how well the software works or whether it's what the users really expected. This kind of qualitative measure has always been very tricky in software development I think and I've never quite managed to crack it.
I would be very interested to understand what other people have done with metrics and what you have found to be effective or ineffective, whether it relates to quality or any other aspect of software development?
Kelly.
I would be very interested to understand what other people have done with metrics and what you have found to be effective or ineffective, whether it relates to quality or any other aspect of software development?
Kelly.
Views: 47
Replies to This Discussion
-
Permalink Reply by Jeff McKenna on -
I like the term reliability. You can compare teams on this. Once a standard of reliability is reached then raising the quality bar is desirable. I think the best leading indicator is the time from when a bug is introduced into the system until a fix is delivered. For many places that I coach just getting the bug rate (new bugs / sprint) is enough to get going. This of course means that some regression/acceptance testing regime is in place.
We have to walk before we run.
-
Permalink Reply by Kelly Waters on -
Thanks for the reply Jeff, that's interesting. Why not add a photo to your profile - it's always good to put a face to a name? I suppose another idea might be to ask the product owner to rate their satisfaction on completion of each user story. Although it would still be subjective, I think it might still provide a useful measure of quality - at least the product owner's perception of quality.
I'd still be interested in hearing what others have done to measure quality, even if they're just ideas?
Kelly.
-
Permalink Reply by Manfred Lange on -
I use several metrics and I change metrics over time as I think that you get what you measure. And sometimes that's precisely what you don't want. Also, it seems that depending on the maturity of the development team some metrics are more helpful than others. For example when I start working with a new team I might use very basic metrics to highlight areas that I believe are should be addressed first. Later as the team matures I drop certain metrics and them with different ones. There is no hard and fast rule, though. All teams are (slightly) different. In my view the most important thing is: The metric has to work and the whole set should not take away too much of your teams time to track them.
Also not all metrics consist of just a numbers and a unit. Some might be statements such as being satisfied or dissatisfied (although there are certainly ways of turning this into numbers as well).
There are, however, metrics that I have used for many years now because they have been proven to be valuable in a lot of cases. Examples include:
- the defect arrival rate (when a bug is reported),
- the defect fix rate (how many can you fix or resolve within a given time frame),
- how long does it take to fix a bug (duration), and
- how much effort was required to fix a bug.
The latter two certainly are not just the testing and the coding but the entire process from reporting the bug to the customer having the fix in production. As Jeff said and I agree with him these numbers make a lot more sense if you have a continuous improvement process in place that also includes acceptance and developer tests that are used for regression testing.
Manfred.
---
My Blog on Agile Leadership
-
Permalink Reply by Ilja Preuß on -
I'd actually argue that when introducing a metric, we should do everything we can to *not* "get what we measure". That is, we should make sure that those who are measured don't see it as their goal to optimize the metric.
-
Permalink Reply by Robert Sfeir on -
100% agree with this. If we start using metrics, and even velocity, to judge the performance of the team, or a team member, you're likely to get yourself in trouble. Why not simply judge if your team's performance is good based on customer satisfaction?
-
Permalink Reply by Kurt Häusler on -
For quality I like "Story Points of Technical Debt".
For productivity I like reporting ROI, which can be done in a variety of ways. I like a simple ratio of profit / amount invested, both on a story basis and a project basis.
-
Permalink Reply by Kelly Waters on
-
They sound interesting Kurt. Please can you explain how you track and calculate them?
Kelly.
-
Permalink Reply by Ilja Preuß on
-
What do you want to use the metrics for? Why are you interested in comparing teams to each other?
Note that a change in velocity doesn't mean at all that the team got better or worse. Velocity is *not* a measure of productivity.
Also, I think if your reliability metric measures how well expectations are met, you should probably change your expectation management. It sounds unhealthy to me to expect a team to always know exactly how much it can deliver in a sprint, and it likely leads to dysfunction: the best way I know to optimize that metric is to undercommit and then to make sure that I don't overdeliver.
As an aside, according to the Scrum guide, a team doesn't commit to story points, but to a more abstract sprint goal. Sounds more healthy to me.
Alistair Cockburn uses two major metrics in a simulation during his CSM course: he simply asks the PO to express both his level of satisfaction and his level of surprise on a scale of 0 to 100 at the end of each sprint. The primary goal is to get the level of surprise down to zero, the secondary to improve the level of satisfaction. (He also asks to the team members to guess what the level of satisfaction and the level of surprise will be. It was interesting - and sligthly disturbing - to see how big a disconnect there often is between the perception of the PO and the perception of the rest of the team.)
-
Permalink Reply by Kelly Waters on
-
Thanks for the reply Ilja.
I don't especially want to compare teams, but I would like a better way of measuring whether or not a team is as productive as it could be. I realise Velocity doesn't do this, which was one of the reasons I was curious about what other people have done, if anything?
I agree you can fiddle the Reliability metric in the way you describe, although doing so would mean Velocity would probably go down. The two at least provide some counter-balance to alleviate this. The other factor in this is just a subjective view of the work being delivered, and whether the technical team leader and managers feel it is appropriate for the size of the team and the complexity of the work.
The teams generally are committing to a set of user stories which effectively comprise the sprint goal. It's not the points they commit to as such, but the fact is the committed stories do add up to a number of points, which can be used as a gauge. I don't think we ever feel the team has to know exactly how many points they can deliver, but it's a useful gauge of how well we can rely on Velocity to inform our planning and whether or not we're delivering all the stories we expected to.
I like the satisfaction/surprise thing you've mentioned from Alistair Cockburn. I wonder if anyone is using this in practice and whether it's helping to keep the team aligned with their stakeholders?
Kelly.
-
Permalink Reply by Ilja Preuß on
-
Hi Kelly,
a team is never as productive as it could be. That's why we continuously inspect and adapt. And that should probably even include our notion of what comprises "productivity".
I think already the notion of trying to establish whether a team is "productive enough" is dysfunctional, and not at all helpful. The two things that are helpful are:
- knowing how much the team can currently deliver, so that we can plan accordingly, and
- continuously improving the way we work together
Regarding the sprint goal: according to the scrum guide, the team commits to an abstract sprint goal, and then determines what stories should be implemented to reach that goal. If later they find out that they can't implement all those stories, they negotiate with the PO how to change what gets implemented so that the sprint goal still can be reached.
-
Permalink Reply by Kelly Waters on
-
Hi Ilja. Yes, I think I agree with you. When I say 'as productive as they could be', it's definitely better expressed as 'improving the way the team works together'; I agree and I think that is the point I was trying to make even though I prefer your language. The purpose of that drive for continuous improvement may be for many reasons, one of which is often to deliver more value. More value could be more functionality in the time or with the current costs, or it could be better quality. Subjectively we can get a feel for these things, but they are very hard to measure. Hence my original question.
I like your views about sprint goals. I guess it depends how specific the user stories are. If they are expressed as pure business requirements rather than specific features, they probably already allow a lot of flexibility. If they're expressed as micro level features then you're right, there probably won't be enough flexibility to still meet the goals if scope needs to change. Personally I like user stories expressed as requirements.
Kelly.
-
Permalink Reply by Robert Sfeir on
-
Amen again!
© 2012 Created by Kelly Waters.
