How to master Pre / Post analyses

❝

This is a 🔒 paid tier article 🔒, but free tier subscribers can continue reading part of the article below!

How to master Pre / Post analyses

If you tell any data scientist that you used a Pre / Post analysis to measure impact, our gut reaction will be to laugh at you. There’s actually a class we all take “How to laugh at Marketers 101” and it’s the first lesson. But, as I’ve had more time in my career, I realize that saying Pre / Post is bad is an elitist view.

The alternatives to Pre / Post like A / B testing require things that not everyone has access to:

Resourcing to set up tests
Culture of experimentation
Skillset to analyze it
Patience / time

It’s things many of us take for granted when we’re lucky to be at companies like Uber, Netflix, Spotify etc… but we’re in the .1%. F*ck that. My mission has always been to help Marketers make smarter decisions. Nowhere in my mission statement does it say “ I only want to help those that can afford it”.

So, here’s how you can master the Pre / Post analysis.

The pitfalls of Pre / Post

Confucius said “To understand how to measure Marketing, you must understand how not to”. I wrote an article called “The 5 ways to measure Marketing ROI” and Pre / Post is the least accurate and least complex methodology available.

Here’s what I said about Pre / Post in that article:

❝

The Pre/Post methodology is named that because it compares a metric from the pre period of a campaign (Pre) to what happens in the post period (Post).

It’s a simple method that has been around since the dawn of time.

The biggest challenge is that it’s very hard to isolate the impact to your metric of interest from just your campaign.

In today’s world there are so many factors:

Seasonality (the ultimate scapegoat)

Competition

Pricing changes

Product changes

Algorithm changes

Macro factors like war, economy, etc.

With the emergence of more digital channels and social media, your campaign and company can explode or implode within days.

You might think ”I’ll keep looking back longer in the Pre period to stabilize the data set”.

You’ll end up just spending more time explaining through factors.

It’s just really tough and when you present the analysis people from random teams will chirp in with “Have you thought about this”. You couldn’t have thought of everything.

All of this is true.

But, what if you have no choice?

Why Pre / Post is still an effective tool

Let’s look at all the reasons you wouldn’t have a choice to use a better methodology that I stated above:

Resourcing to set up tests
Culture of experimentation
Skillset to analyze it
Patience / time

Pre / Post is actually the best solution to navigate through all of these.

Resourcing to set up tests

It requires none to very little. All you have to do is switch off or on the thing you want to test. You don’t need to randomize users or think about geo splits. Just launch it. Especially if it’s a marketing decision it should require very little resources while product ones might require a bit more because you always need engineering.

Culture of experimentation

If you lack a true culture of experimentation where the leaders / founders understand the inherent risks, then Pre / Post is an easier one to justify. “We’re going to turn it off and this is what happened”. It’s the easiest to explain and share the results of and you can get results quicky.

Skillset to analyze it

There’s no real complexity involved in analyzing Pre / Post. Yes, there are a lot of pitfalls that I’ve mentioned above but it’s the easiest to analyze. Everyone is able to use excel and averages to create a baseline and then measure how the baseline goes up. It’s quite simple.

Patience / time

A/B testing requires waiting for stat sig results and for sample sizes and blah blah. Pre / Post doesn’t need any of that. I’ll discuss how long you should wait in the post period below but the reality is that you can make a decision off it overnight if you have confidence. “Yup, I switched it on and my whole funnel dropped over night… it doesn’t work”. You get to choose how patient you want to be instead of being told by a sample size calculator.