Creating Metrics that matter

“When a measure becomes a target, it ceases to be a good measure,” Goodhart’s Law. As Product managers we love to create metrics, they are after all supposed to be checkpoints to get to the desired business goals. However, in my experience most teams still get these wrong, which results in their engineering teams spin a lot of cycles, only to realize that there was no meaningful improvement to Customer experience and Business outputs.

There is so much written on different type of metrics (Acquisition, Engagement, Adoption, Satisfaction, Retention, Pirate framework, etc) which I wont get into. Instead, I will spend time on common pit-falls I have seen , how to identify ineffective metrics and simple framework to use for your North-star metrics.

“So, why do we end up with so many metrics that just don’t cut it? Well, creating effective metrics is no walk in the park — it takes a deep understanding of both customers and business goals.” It is much easier to create a new vanity metric which looks cool and sexy and gives a superficial sense of progress.

Here are a few common pitfalls I’ve noticed in ineffective metrics:

1. Output Metrics as North-star:

Without A/B capabilities to identify causation, these metrics are susceptible to fluctuations influenced by seasonality, customer growth, and external factors beyond your control. We should definitely track these metrics but don’t take Goals on these or try to establish causation without the necessary A/B Infrastructure.

2. Metrics without Guardrails:

When one metric goes up, there’s usually another one that takes a dip. If you’re only keeping tabs on one side without looking at the other, it ends up hurting the overall business results. Take, for instance, a metric tracking the number of new products, but ignoring the impact of product quality — it will lead to a poor customer experience. Or consider a metric focused on customer acquisition but neglecting engagement and retention — it’s a surefire way to miss the bigger picture.

3. Metrics that can be gamified:

These metrics are prone to manipulation by external factors. For instance, if you’re tracking Monthly Active Users, it’s tempting for someone to create a bot that visits the site every few days, artificially inflating the numbers. Similarly, if the metric is the number of sellers acquired, it’s tempting to have friends and family create new accounts just to boost the count.

4. Complicated and Insufficient:

These metrics can be tough to grasp, often involve complex logic and machine learning for creation, making it challenging to figure out how reducing this metric actually enhances the customer experience. Frequently, you’ll notice that the customer experience stays the same, regardless of whether this metric goes up or down.”

So, how do we tackle this and make sure the metrics we come up with actually work? In my experience, it’s crucial to invest time in deeply understanding what you want to measure and why it’s significant. Creating meaningful metrics is no walk in the park — it requires hard work and a profound understanding of both your customers and the business value you aim to achieve. While crafting good metrics may take time, once developed, they will be actionable, explainable, and impactful.

Here is a framework that I have used

First and foremost, establish an experimentation infrastructure to gauge the impact of your feature launches. Without this setup, attributing causation from your efforts to business value becomes a daunting task. Surprisingly, many teams still operate without this foundation, leading to substantial time and energy spent on correlation and hypothesizing.

Next, dive into metrics that genuinely drive business impact. If your company is public, the Shareholders’ letter is a treasure trove for uncovering these vital metrics. Consider a few examples Amazon highlighted in the last earnings:

• Sales soared from $15.7 million in 1996 to an impressive $147.8 million.
• Cumulative customer accounts surged from 180,000 to 1,510,000, marking a remarkable 738% increase.
• The percentage of orders from repeat customers jumped from over 46% in Q4 1996 to over 58% in the same period in 1997.

Finally, scrutinize your product’s role in the flywheel and formulate Input metrics that are Actionable, Explainable, and Impactful while keeping an eye on the pitfalls we discussed above.

Actionable: It should be easy to find the reasons behind metric changes from internal or external events. It should be easy to break down the metric into components and understand the root-cause why the metric fluctuated and easy to identify actionable insights to fix the issues identified.

Explainable: The metric should be easy to explain and should not require deep expertise to explain how this metric was calculated. It should also be reproducible making it is easy to build trust on this metric with stakeholders and leadership.

Impactful: Improving the metric should directly result in higher downstream impact. It should be clear how improving this metric is directly improving the Customer experience.

Good luck to you on your journey to to discover your North-star metrics !

These insights are solely my personal opinions and experiences, not reflective of Amazon, shared with the intention of offering a perspective that may be helpful to others.