Hello experiMENTAList, it’s Sundar 👋

I’m a former Head of Marketing Science at Uber where I optimized $1Bn+ in spend across Brand, Performance, and Lifecycle. Now, I share weekly playbooks that help you prove and scale your Marketing ROI.

If you know a colleague or friend that might benefit from this newsletter, send them here.

Now let’s get experiMENTAL!

PS. This post was written with Wispr Flow. I highly recommend them 👇️

Wispr Flow turns your voice into clean, professional text inside any app. Emails, Slack, client updates — speak once, send without editing. 4x faster than typing.

Try Wispr Flow free

I had very different ideas for my first article after four weeks, but the Linkedin gods had other plans.

I was tagged in a post by someone asking me my thoughts on how to measure the impact of Amsterdam's ban on ads for the fossil fuel and meat industry and I just couldn’t resist.

It truly is a fascinating case study on a very real-world natural incrementality test where overnight you go from some level of spend to zero.

Back in January, Amsterdam's City Council passed a ban on ads for fossil fuels and meat products in a 27 to 17 vote that was set to come into place on May 1st.

The ban is across any high-carbon product or service including:

The ban stands across all public spaces in the city:

First of all, fuck yeah Amsterdam! This is why I continue to live here.

But, the real question is the one we should always ask: “Will anything change?”

There are three stakeholders so we have to assess the impact for all 3:

Let's start with the easiest one.

These companies only have one metric they care about, and it's revenue. Luckily, according to BBC, Meat only accounted for an estimated 0.1% of ad spend compared with roughly 4% for fossil related products.

So, these companies should feel little impact with other advertisers taking up that advertising space. Measuring it would be easy with just comparing revenues YoY pre and post the ad ban.

Verdict: Easy to measure

It’s important to remember that the government is doing this for sustainability purposes.

They’re tired of “Greenwashing” and want to align the city’s mission to their actions. They want to see less units sold of the products included in the ban.

However, governments are also businesses and so equally they care about tax revenues they collect from these products.

If I were them, the primary measure I’d look at is to see sufficient enough drops in purchases of these products with a guardrail measure of tax revenue. I have no idea what these thresholds might be, but that’s how I’d look at it if I were them.

Note: The other possibility is that they just don’t care and they won’t measure it but that’s a lot more boring for this hypothetical exercise.

Now, how do we measure this?

First, what data do we have access to?

I’m assuming:

I’m assuming access to other cities’ data is not readily available (although that’s probably not true) and I definitely don’t have access to other countries.

Second, what methodology would I use based on what I have available?

Given, I can’t use other cities, I’d use a pre/post that’s “normalized” with a YoY diff - in - diff.

If that sounds like a bunch of marketing science mumbo jumbo it kind of is, so let me simplify.

I want to look at pre ban vs post ban but to account for seasonality I need to know the pattern from last year. Problem, is April / May is crazy seasonality time in Amsterdam because:

1. Tulips

A more important question to ask is: Is one month enough to change behavior? Maybe for cars and meat buying you might see a dip but you definitely won’t for travel when most trips have already been booked.

And also for products like gas and meat, behaviors don’t change over night so you’re really talking about a long term diff - in - diff / pre post appraoch.

But, this is where diff - in - diff and pre / post stumbles… Guess what happened between last year and this year around this time that will completely disrupt something like travel and gas prices and break any possible comparison?

Here’s a hint:

So there goes any comparable dataset on pre / post level.

Frankly, if I’m the government I’d have to resort to analyzing household survey data for trends over time and seeing if the purchases for flights, meat, and gas show any noticeable change.

That’s basically it, but I’d love to hear from you all if you have another idea on how you’d measure.

Verdict: Hard to measure

Now this is where the fun begins.

If I’m a company being targeted by these bans, I absolutely want to know the impact so I can go to my policy team and help them identify the potential impact as more cities throughout Europe (and hopefully the world) do this.

From a measurement perspective, this should be significantly easier for one simple reason: Data.

Every one of these companies has 1st party data on exactly what is purchased and where.

✅ Let’s measure the # of veggie burgers vs big macs in Amsterdam

✅ Let’s measure the amount of gas purchased in Amsterdam

✅ Let’s measure the number of flights booked from users in Amsterdam

Now, the real challenge comes from a measurement perspective and more specifically measuring short vs long term.

Short term:

Short term, let’s assume all the companies that were advertising before are multinationals that have the budget for OOH advertising.

This means they’re likely in many cities across the Netherlands and the world.

Now, you can’t use a Geo Test because Amsterdam is just one city and there’s not enough randomization there.

So, what method should we use?

Enter Synthetic control.

Synthetic control allows us to create a synthetic counter factual (aka control) with the data we have. A common approach is to create a Frankenstein city that mimics Amsterdam before the intervention and allows us to forecast what we believe Amsterdam would have looked like.

This Frankenstein city could look like:

So, we create this synthetic control and then we observe how much it diverges over time to estimate the impact.

Some things to look out for are:

Verdict: Easy to measure

Long term:

As I was thinking about this more, I realized that this is basically just a beautiful exercise on the impact of Brand Marketing. Lucky for me it’s one of my expertises 🙂

So, if I’m one of these companies, what I’m doing is the following:

Over time, these companies will be able to measure movements in Brand metrics and thus establish a relationship as to whether the ban will impact revenue. The challenge though is really understanding the elasticity of these metrics.

If a person driving a car doesn’t see an ad for gas are they going to forget to fill up? No.

However, if a person that’s hungry doesn’t see an ad for McDonalds are they going to pick something else? Maybe.

These elasticities between brand metrics and revenue are really where it’s all going to play out.

Verdict: Pretty hard to measure

Anyway, this was a really fun exercise and if you have any more real world challenges like this, I’d love to take a look.

Also, thank you for being patient with me over the past few weeks and I look forward to being back on the grind!

That’s it for this week!

Stay experiMENTAL,

Sundar