Hello experiMENTAList, it’s Sundar 👋

Thank you for being a valuable part of this growing community of 2.6K+ marketing leaders from Uber, DoorDash, Google, Spotify, and many more.

If you know a colleague or friend that might benefit from this newsletter, send them here.

Now let’s get experiMENTAL!

ChatGPT is a superpower if you know how to use it correctly.

Discover how HubSpot's guide to AI can elevate both your productivity and creativity to get more things done.

Learn to automate tasks, enhance decision-making, and foster innovation with the power of AI.

Download the free guide.

Have you ever experienced the frustration of wanting to measure something, but not being able to because you either couldn’t wait that long or didn’t have the data?

Well, look no more! I have the solution for you!

Enter proxy metrics.

Proxy metrics, as their name suggests, are metrics that act as directional indicators for the metric you would have wanted to measure.

Below I'll share:

The two main times to use proxy metrics are:

Example 1:

"You don't have the data" often shows up when you're trying to measure brand or offline marketing. The proxy metrics helps you understand "Is my marketing moving the signal that I want?"

I launch a Brand campaign that is meant to drive an increase in awareness. Unfortunately, I don't have the budget for a survey through a vendor to measure awareness, but I'm being asked to prove that I’m influencing awareness.

What do I do?

Example 2:

I want to move 90-day retention", but there is no way in hell that I am waiting for an A/B test to last 90 days to validate my hypothesis.

What do I do?

In both cases, I’ll have to use proxy metrics.

Finding the right proxy metric is half intuition and half statistical. The metric must pass the sniff test to see if the relationship intuitively makes sense.

I'll walk through both examples sharing how I would find the proxy metric.

Example 1 → Brand Campaign

Let's say I'm a company called Sundar's Washing Machine that sells washing machines. As you can tell, I'm in a very creative state today.

The goal of my campaign is to increase the number of people aware of my company. I don't need them to buy. I don't need them to transact or do anything else. I just want them to be aware that I exist.

Let's think through all of the “assets” my company owns where I could see this happen.

For each asset, there’s a corresponding metric that I care about: visits and/or searches.

Now, all of a sudden I have four metrics that are easily trackable and high quality, that I can monitor over time to see if my brand campaign is increasing awareness.

Example 2 → 90 day retention

This example is different because we don't have to worry about data quality and trackability. All of the metrics we can use are 1st party, however, for 90-day retention, we have two different problems.

We obviously can’t wait forever to analyze the A/B so we need to find a proxy metric within 7-14 days that predicts 90 day behavior.

Why a 7 to 14 day metric?

If you use a 30 day metric, then you have to wait 30 days after your A/B tests ends before you can analyze it. Otherwise, you'd just be analyzing the first half of the A/B test if you analyzed the day after the A/B test ends.

So now we know our window for our metric. But what metric should we pick?

We need to run an analysis to find the correlation and see which metric best predicts. That analysis likely looks like a binary classification which tells you which 7-14 day metric predicts a binary (yes/no) 90 day retention.

Here’s what the results might look like:

Pick the one variable that has the highest predictability (but use your intuition to see if that makes sense) and that becomes your proxy metric. You don't want multiple because you don't want to have multiple primary metrics when you run your A-B test.

Example 1 → Brand Campaign

When your proxy metrics are related to a campaign where you can't A/B test, you have to use pre-post methods to measure impact.

In most cases the influx of spend that you're adding to the market should be enough to get you a signal quickly. I'm obviously not talking about Super Bowl level spends but you need to be introducing a meaningful size of budget relative to what's out there if you're trying to make impact.

Assuming you hit this level of spend, a simple way to analyze this is by comparing YoY growth rates before vs after the intervention.

Let’s say my branded search was growing 10% and then all of a sudden it jumps to 20% post campaign launch, you can with reasonable certainty attribute that to your campaign.

That is the level of simplicity I use for the analysis.

Example 2 → 90 day retention

When you can A-B test, then it's relatively straightforward using the proxy metric as the primary metric.

But there is another step.

90 days after the A/B test ends, you should go revisit the cohorts and see if the 90 day retention truly was better. It’s a way to validate the metric.

Why do this?

If you don’t validate that otherwise you're going to erode trust in the proxy metric because inevitably someone is going to ask why did we use 14 day well we used it to predict 90 days okay fair what happened after 90 days and if you don't have an answer you break the trust bond.

As I mentioned before, proxy metrics can be a little fickle because if they are not communicated properly, it seems like an attempt to dodge responsibility.

Example 1 → Brand Campaign

In the example of the brand campaign, what we want to do is establish a buffer that will allow us to eventually measure the true ROI of the campaign without it getting shut down immediately.

Here's how that would sound.

“For this campaign, we're going to measure the pre-post uplift in branded Google searches. So far, we've been growing branded search at 10% consistently over the last eight weeks. What we want to see is the acceleration of YoY growth in branded search as a proxy signal that our campaign is working”

You can add additional details like the magnitude of the uplift you're expecting and other considerations if you've got the data.

Or, if this is your first time doing this, then use that and say, "We are also experimenting with the use of this as a proxy metric to see if it gives us a clear enough signal."

The final point in making this work is to say “Currently, this is our best option to observe with some level of confidence a causal relationship between our campaign and the metrics we want. We believe that having higher branded searches will result in more ROI, but that is coming in step two of our measurement framework.”

Do not oversell proxy metrics.

Example 2 → 90 day retention

Here, the risk is not on the proxy metric that you chose. It's on the correlational relationship and the strength of that.

You want to set the stage that for business and velocity purposes, you needed to use a proxy metric because you couldn't wait three months to make a decision. If pressed, you then show the analysis that proves the correlation along with the appropriate supporting statistical evidence.

Another important hedge is acknowledging that you sacrificed a bit of correlation for increase in speed. For example, if you had a 45-day metric, you would have a stronger correlation, but you would then have to wait almost another month for the results.

It's all about hedging risk vs. certainty vs. actionability.

Here’s what I’d say “We wanted to influence 90 day retention as that’s our best driver of LTV, but we couldn’t afford to wait X weeks for the test + 90 days to account for all of the data. Instead, we ran analysis to find that 7 day purchases has a strong correlation to 90 day retention and will be analyzing the success of the A/B test on 7 day purchases.”

Hope this all helps and good luck finding your proxy metric! If you’ve got any questions, just reply back here and I’ll help you out 😀 .

That’s it for this week!

Stay experiMENTAL,

Sundar