A Model and Metrics for Tracking Product Success


by Saeed Khan

There’s an adage that states, you can’t manage what you can’t measure.

And if that is the case, then you have to wonder how much actual management of products, Product Management departments are doing?

Why? Because when you talk to Product Managers about product metrics, it’s usually a pretty short conversation. :-(

Unfortunately the metrics a lot of PMs are most closely following relate to bug counts or velocity in an iteration, or simple high level business metrics such as quarterly sales or average deal size (ADS).

Not that these are bad metrics, but they shouldn’t be the ones that dominate focus to the exclusion of a more holistic view of the product.

There is some discussion on this topic happening in the blogging community.

Cindy Alvarez had a post recently entitled 57 Questions About Metrics. There are a lot (57 in fact!) of good questions listed there, with categories such as:

  • Acquisition
  • Activation
  • Retention
  • Referral
  • Revenue

But that list of questions is heavily skewed towards funnel metrics for SaaS/Web applications.  It’s also not clear which of the 57 questions are more important or less important given your situation. But, it’s a good list nonetheless.

Brainmates have a document on their site entitled How do you manage the financial health of your product?

This is also a good document, but again focuses on SaaS applications with metrics like Cost per Subscriber, Monthly Disconnect Rate, Total Operating Costs etc.

It’s important for Product Managers — particularly those who are more focused on the business side of product (i.e. product MANAGERS) — to look at the state of a product more holistically, dare I say, “cross-functionally”,  to manage it effectively.

Managing Products Holistically

This topic is something that I’ve thought about quite a bit. I wrote several articles last year on the topic of Product Management Metrics. But I didn’t get into specific metrics in those posts. I’m not sure why exactly, but the fact that I didn’t — even though that was my intent — is likely an indication of the difficulty of nailing this topic. I think I’ve found a good general model that can be applied in a wide variety of cases regardless of product type, deployment type etc.

Before I continue, I’d like to thank Tom Grant, Sue Raisty-Egami, Jim Holland and Alan Armstrong for helping me find some more clarity on this subject and giving me feedback on my ideas.

To manage products holistically, there are 4 categories that Product Management needs to understand. These are:

  1. Business
  2. Go-to-Market
  3. Organization
  4. Product

Business Objectives

I’ve listed this first, because it is the most important. Product Management has to be focused on business success of the product and a clear picture of the state of the business for any product is critical.This could include many things, but metrics could include Revenue (weekly, monthly, quarterly etc.), Deal Pipeline, Win/Loss numbers, Average Deal Size, Renewal Rates, Pricing etc.

Go-to-Market Objectives

This includes major areas of go-to-market readiness such as proper positioning, competitive analysis, customer references, status (and plan) of awareness activities, collateral etc. These and other GTM activities need to be assessed to ensure objectives are clear and focus is being placed in the right areas. Note that these are not all activities that Product Management (or Product Marketing) will explicitly complete (other marketing teams will likely do much of this work), but Product Management must have a clear picture into the state and effectiveness of these activities.

Organizational Objectives

This category focuses on cross-functional readiness across the organization. While the Go-to-Market objectives cover a lot of Marketing activities, other groups including Sales, Sales Consulting, Technical Support, Professional Services, Customer Education etc. all need to be assessed to ensure they are enabled and have the knowledge and tools they need to do their jobs. Any gaps found in this assessment can be acted upon to ensure those teams get enabled and are supplied with the necessary information or tools as needed.

Product Objectives

I’ve listed this last simply because, it’s the one most people focus on (first!) and I wanted to emphasize the first 3. A great product is DOA if the rest of company cannot market, sell and support it. Product objectives include Product Strategy, Capabilities, Gaps, Quality, Performance, Stability etc. i.e. high level aspects of the product that could impact sales or customer satisfaction.

These 4 categories, and the specific metrics measured in them can provide a very clear picture of both the state of a product and any areas of strength/weakness. Weak areas should be acted up to address the underlying issues causing the weakness.


Objectives change across the Product Lifecycle

There is one other aspect to fold into this model. Depending on the stage of the Product Lifecycle a product is in, there will be a different emphasis on the specifics of the 4 categories of Objectives listed above. For example, Organizational Readiness is absolutely critical when a product is in growth stage, but far less critical towards end-of-life.  Or at a more granular level, product stability is less important in the early develop/launch stage (when there are very few customers) but critical once the product enters a growth or maturity stage.

In short, the OBJECTIVES of the stage of the Product Lifecycle a product is in, determine the focus of the objectives when measuring the product.

Back in my post Product Management Metrics (part 3), I talked about the stages of the product lifecycle and how each stage had different objectives. I’ve summarized them in the following table:

If you take the Product Lifecycle stages/objectives and tie them into the 4 categories for a holistic product picture, you get a grid that looks something like this.

This diagram shows some of the differences in Product Management FOCUS for the objectives, across the stages of the lifecycle. i.e. The focus for these categories will NOT be constant across the lifecycle. i.e Organizational Readiness is less important early on, but critical once the product reaches growth stage and beyond.

NOTE: This diagram won’t be 100% the same for all products in all companies, but I’ve created it based on my own experience managing products across the lifecycle stages. But please modify these as you see fit for your company and products.

Now, for each of these boxes, there will be a number of specific metrics to measure. These are really up to you to define. I’ve listed a number of them for each objective, earlier in this article. In some cases, you could use some of the metrics listed in Cindy Alvarez’s article, or in the Brainmates article. There is NO silver bullet here. It completely depends on your company, product strategy and goals.

But, you should be able to come up with a standard set (say 5-7 maximum) of key metrics for each objective (i.e. row) that are meaningful across all stages of the lifecycle. In some stages they won’t be as important (e.g. # of customer references) as in others, but once you come up with those metrics and define a clear way to measure them, they just become part of the overall picture that you look at when measuring product success. Keep in mind that for any given product, you only have to worry about 1 column at a time. i.e. it will only be in one stage of the lifecycle and so that’s all you have to measure.

So, there you have it, a general model to use. What do you think? Make sense? Useful? Yes? No? Have questions?  Regardless,  I’d like to hear from you.

Saeed

Additional Articles: The follow articles drill down into this topic in more detail.

  1. Business-oriented metrics for Product Success
  2. Organization and Go-to-Market Metrics for Product Success
  3. Product Metrics for Product Success

Tweet this: @saeedwkhan A Model and Metrics for Tracking Product Success http://wp.me/pXBON-2oL #prodmgmt #prodmktg #innovation

If you enjoyed this post, please consider leaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.
24 Responses to A Model and Metrics for Tracking Product Success
  1. David Inniss says:

    Excellent article. Well done.


  2. The Ecosystem of a Product @saeedwkhan A Model and Metrics for Tracking Product Success http://t.co/dHEQptXe #prodmgmt #prodmktg #innovation


  3. @StewartRogers @jonwhite123 A Model and Metrics for Tracking Product Success http://t.co/GP4EYk5G #prodmgmt #prodmktg


  4. RT @gharpreet @saeedwkhan A Model and Metrics for Tracking Product Success http://t.co/kGWCv4B #prodmgmt #prodmktg #innovation

  5. Brian Ahart says:

    RT @TopsyRT: A Model and Metrics for Tracking Product Success — On Product Management http://t.co/5AjWT60 #AWDPM


  6. RT @gharpreet @saeedwkhan A Model and Metrics for Tracking Product Success http://t.co/kGWCv4B #prodmgmt #prodmktg #innovation


  7. RT @gharpreet @saeedwkhan A Model and Metrics for Tracking Product Success http://t.co/kGWCv4B #prodmgmt #prodmktg #innovation


  8. @saeedwkhan A Model and Metrics for Tracking Product Success http://t.co/kGWCv4B #prodmgmt #prodmktg #innovation

  9. GregCott says:

    RT @TopsyRT: A Model and Metrics for Tracking Product Success — On Product Management http://t.co/p2PoC7S


  10. RT @onpm: A Model and Metrics for Tracking Product Success — On Product Management http://bit.ly/ek12pl


  11. RT @Fibol: RT @onpm: A Model and Metrics for Tracking Product Success — On Product Management http://bit.ly/ek12pl ) Check links at the …


  12. RT @onpm: A Model and Metrics for Tracking Product Success — On Product Management http://bit.ly/ek12pl ) Check links at the bottom of page


  13. […] Last week, I wrote about a Model and Metrics for Track Product Success. […]

  14. Susan Zheng says:

    RT @onpm: A Model and Metrics for Tracking Product Success — On Product Management http://bit.ly/ek12pl #prodmgmt

  15. Raj Raman says:

    Absolutely. For startups, a great book to read about this topic is Getting to Plan B

    Randy Komisar employs the concept of a dashboard with 3-5 metrics that are relevant to you right now. Measure them, review them, replace them with new ones.

    In my experience, if you’re able to assess Organizational readiness and effect change then you’ve broken out of the box of a “Product” Manager into the world of the “Business” Manager. E.g. Are your sales people incentivized to sell your product? This could involve working with Sales Management to change the incentive structure. Are customers not even considering you because the proof-of-concept was not effective? These are examples of issues that are outside the “Product” box, outside of feature/function.

    Thanks for writing this.

    Raj | Flekscale.com

    • Saeed says:

      Raj

      Thanks for the comment and the book reference. I’ll definitely check it out.

      WRT to being a “business” manager, that’s really what “product” management is about — business management at the product/product line/portfolio level. Unfortunately because of the name (i.e. product manager) and the initial necessarily heavy focus on the actual product, a lot of people never get beyond that, and into the other issues that are necessary to make the product successful.

      I hope the diagram and discussion help address that issue.


  16. A Model and Metrics for Tracking Product Success by @onpm http://bit.ly/faENcQ

  17. Pranav Desai says:

    RT @saeedwkhan: New Post: A Model and Metrics for Tracking Product Success http://wp.me/pXBON-2oL #prodmgmt #prodmktg #innovation

  18. Wendy_B says:

    Great article! @saeedwkhan A Model and Metrics for Tracking Product Success http://wp.me/pXBON-2oL #prodmgmt #prodmktg #innovation


  19. This is great Saeed, I especially love the green Product Management FOCUS diagram. It is a good reminder for ourselves as well a great tool on onboard slightly inexperienced product managers/ marketers.

    Organization readiness is always the tricky one for me. If you are a 500 person organisation or larger, it is hard to see the gaps. And once you see them, it can be difficult to change attitudes and remove inefficiences. Something i believe management should always have an eye on. Otherwise your growth phase will always be 2 steps behind no matter how superior your product or your product management team.

    Cheers.

    • Saeed says:

      Amrita

      Thanks for the comment. About the green PM focus chart — I think that’s really a key takeaway from the post. That objectives and focus change over the course of the product’s life. As PMs and PMMs, we may naturally take that into stride, but a lot of organizations don’t so they don’t hire and staff properly. As the product ramps up, so should the team managing it so that full coverage can be given across all 4 categories, otherwise, as you correctly say, things will be “2 steps behind” where they should be. Imagine all the lost revenue and opportunity that occurs in companies because of this fact alone.

  20. lisaw1 says:

    RT @saeedwkhan: New Post: A Model and Metrics for Tracking Product Success http://wp.me/pXBON-2oL #prodmgmt #prodmktg #innovation

  21. saeedwkhan says:

    New Post: A Model and Metrics for Tracking Product Success http://wp.me/pXBON-2oL #prodmgmt #prodmktg #innovation

We really want to hear your thoughts...