👋 Hey, it’s Sundar! Welcome to experiMENTAL: a weekly newsletter on B2C Marketing & data science how-to guides, frameworks, and stories from 15 years including early Uber.

This week is sponsored by Hubspot

Unlock the full potential of your workday with cutting-edge AI strategies and actionable insights, empowering you to achieve unparalleled excellence in the future of work. Download the free guide today!

How to master Pre / Post analyses

If you tell any data scientist that you used a Pre / Post analysis to measure impact, our gut reaction will be to laugh at you. There’s actually a class we all take “How to laugh at Marketers 101” and it’s the first lesson. But, as I’ve had more time in my career, I realize that saying Pre / Post is bad is an elitist view. 

The alternatives to Pre / Post like A / B testing require things that not everyone has access to:

• Resourcing to set up tests

• Culture of experimentation

• Skillset to analyze it

• Patience / time 

It’s things many of us take for granted when we’re lucky to be at companies like Uber, Netflix, Spotify etc… but we’re in the .1%. F*ck that. My mission has always been to help Marketers make smarter decisions. Nowhere in my mission statement does it say “ I only want to help those that can afford it”.

So, here’s how you can master the Pre / Post analysis.

The pitfalls of Pre / Post

Confucius said “To understand how to measure Marketing, you must understand how not to”. I wrote an article called “The 5 ways to measure Marketing ROI” and Pre / Post is the least accurate and least complex methodology available.

Here’s what I said about Pre / Post in that article:

All of this is true.

But, what if you have no choice?

Why Pre / Post is still an effective tool

Let’s look at all the reasons you wouldn’t have a choice to use a better methodology that I stated above:

• Resourcing to set up tests

• Culture of experimentation

• Skillset to analyze it

• Patience / time 

Pre / Post is actually the best solution to navigate through all of these.

Resourcing to set up tests

It requires none to very little. All you have to do is switch off or on the thing you want to test. You don’t need to randomize users or think about geo splits. Just launch it. Especially if it’s a marketing decision it should require very little resources while product ones might require a bit more because you always need engineering.

Culture of experimentation

If you lack a true culture of experimentation where the leaders / founders understand the inherent risks, then Pre / Post is an easier one to justify. “We’re going to turn it off and this is what happened”. It’s the easiest to explain and share the results of and you can get results quicky.

Skillset to analyze it

There’s no real complexity involved in analyzing Pre / Post. Yes, there are a lot of pitfalls that I’ve mentioned above but it’s the easiest to analyze. Everyone is able to use excel and averages to create a baseline and then measure how the baseline goes up. It’s quite simple.

Patience / time

A/B testing requires waiting for stat sig results and for sample sizes and blah blah. Pre / Post doesn’t need any of that. I’ll discuss how long you should wait in the post period below but the reality is that you can make a decision off it overnight if you have confidence. “Yup, I switched it on and my whole funnel dropped over night… it doesn’t work”. You get to choose how patient you want to be instead of being told by a sample size calculator.

How to master Pre / Post

Communicate early

One of most important aspects of an analysis is communication. It’s something I talk about here in my article on how to make more impactful analyses and its very relevant for when you’re using Pre / Post. Going into the test, you should articulate that you’re using pre / post but also why there are no other alternatives.

Here are some good reasons:

“We don’t have the sample size for an a/b test”

“We don’t have someone who can analyze”

“I don’t know how to set up an a/b test”

“We don’t have a/b testing set up yet”

Communicate early and document that you’re using a Pre/Post. Also bonus just add the phrase “We know pre / post is not the gold standard but it’s the most ideal given our situation”. It shuts up everyone.

Align on the metric of impact

Unlike an A/B test, you can’t really use secondary metrics in pre / post. There are too many variables at play and the risk of having to explain them exponentially increases with each metric. Pick your primary metric and be very deliberate about it. Is it a conversion rate, or AOV, or just GMV. It doesn’t matter. Just pick one that is visible and you can measure an obvious change with.

Use a reasonable pre and post period

If any section of the article is going to feel preachy it’s this one so bear with me. Pre / Post is just full of pitfalls because of bias. We naturally tend to find data that makes us look better. Confirmation bias, anchoring bias, and ones I don’t know the names of. So, here’s where you have to be most careful.

You need to define a pre period that:

1. Reduces volatility

2. Avoids seasonality (as much as possible)

3. Represents the business cycle

Volatility in Pre / Post is synonymous with confidence. If you can find a metric or period where your metrics show ups and downs but a constant trend then you’re okay. What you don’t want is a metric that goes up 10% then down 5% then up 15% then down 12%. It’s a hard pattern to predict.

Seasonality in my experience is the biggest reason pre post fails. Avoid hoildays, periods where you know the business ramps up or down quickly (if you have the historical context). It’s not easy but it’s worth waiting a week or 2 to escape these zones.

These are all high level numbers so tailor it to your business. Let’s take an example of a Marketing funnel. You generally know that once a user sees an ad that they take 1-2 weeks to convert. Or maybe it’s the same day. A shorter pre period is okay because you’ll be able to capture the impact shorter. If you have a business that takes months (B2B SaaS) then pre post becomes much harder.

Make big bold bets

Simply said, swing big. Turn off funnels. Drop website pages. Kill steps in the conversion funnel. If you’re going to use Pre / Post, you have to be able to see the impact. Use pre post for hypotheses where you think the metric of choice will change by 20%+. If you don’t have confidence that you can see the impact soon then don’t do it.

If you’re at the point where you feel like you’re making more optimization decisions then it might be time to set up A/B testing, but if you’re a scrappy lean startup still trying to find product market fit then Pre / Post is a good early solution.

Marketers get into trouble with Pre / Post when they try to rationalize the effect of a small decision on a large complex business. 

“I changed the button color from red to blue. At the same time, our business is up 35% WoW.”

Let’s be realistic and see if those 2 statements are likely to be causal even though they seem correlated. The answer is no. We’re so hellbent on proving our impact that we shoot ourselves in the foot by trying to pretend we’re responsible for impact. In the long term, this is how Marketing has put itself in a tough situation and that most people don’t believe Marketing’s results. We tend to over-inflate our impact.

Limit focus

Pick a narrow target. On my upcoming podcast, I talk with the Head of Marketing Analytics at Faire and he talks about a test where they used pre / post to analyze the effect of turning off branded search in a country.

This is Faire. A highly data driven org that’s one of the world’s leading marketplaces. But they used pre post in the right way:

• 1 country

• Branded search

• Impact on signups

It should have been fairly obvious if turning off branded search worked or not.

Wrapping up

If you’re going to run Pre / Post, remember:

1. Communicate early

2. Align on the metric of impact

3. Use a reasonable pre and post period

4. Make big bold bets

5. Limit focus

After reading this, I hope many of you realize that Pre / Post isn’t so bad but it’s just often set up to fail. With focus and derisking, you can move quickly and be a bit more confident in your decisions.

If you’ve run an awesome pre / post, let me know! I’d love to read some stories.

Enjoyed reading this?

Share with your colleagues or on your LinkedIn . It helps the newsletter tremendously and is much appreciated!

Missed my last article?

Here it is: How I found my next dream job