Hello experiMENTAList, it’s Sundar 👋

I’m a former Head of Marketing Science at Uber where I optimized $1Bn+ in spend across Brand, Performance, and Lifecycle. Now, I share weekly playbooks that help you prove and scale your Marketing ROI.

If you know a colleague or friend that might benefit from this newsletter, send them here.

Now let’s get experiMENTAL!

This week is brought to you by Hubspot

That investor was wrong. Gamma is now worth $2B, with 50M users and more than half their growth driven by word of mouth.

They're one of 6 AI-native startups in HubSpot for Startups' free Bold Bets Playbook. Replit grew revenue 50x after half the team pushed back on the strategy. Ramp generated 100M+ views from a single stunt. Clay's co-founder wouldn't hang up a sales call until the prospect DMed him in Slack.

Each one took a GTM risk most founders would never greenlight. Each one paid off.

Get the free Bold Bets playbook

The year is 2020. Picture a competitive food delivery market in the UK arena with Deliveroo, Just Eats Takeaway, and Uber fighting to ensure Londoners are well fed. The UberEATs team must find a way to become #1. Enter scene.

In 2020, we were trying to grow Uber Eats in the UK. It was one of our most important markets globally: highly competitive, huge TAM, and somewhere that performance Marketing alone was plateauing. So, naturally, the next step was to go big on Brand Marketing campaigns as a way to stimulate growth.

We were about to invest tens of millions of dollars across digital and TV.

And then, a few weeks in, we had to pause the campaign.

What happened next is one of the most important things I've learned in a decade of Marketing science.

Serendipity is a good thing 😉

Before I get into the test, I need to explain what we were trying to do.

One of the hardest problems in Brand Marketing is proving that budget has an impact. But, most teams jump to the conclusion that you have to prove lower funnel ROI. Actually, what you first have to prove is that you’re moving upper funnel metrics. Then you earn the right to prove lower funnel impact.

And to date, we weren’t great at proving that individual campaigns were actually moving those upper funnel metrics. Brand campaigns were being planned, launched, and evaluated on instinct and post-hoc data more than evidence.

So, we needed to come up with a framework that was generalizable and objective.

We chose to use Meta’s Brand Lift studies because it gave us a way to run multiple pulse checks through a single campaign flight using a consistent global methodology without having to stand up a new measurement approach from scratch.

Note: Another bonus was that Meta is a third party, and so you're not grading your own homework.

The way a Meta BLS works is you run your Brand ads on Meta and they create a treatment vs control exposure group based on who saw your ads. Then to both groups they send a survey and you measure the differences in the results. It's basically an A/B test, but instead of looking at business metrics, you're looking at survey responses

Now, this is where it gets fun. The beauty of the Brand Lift study is that for every test that you run, you get two data points ( treatment and control ).

And when you run Brand lift studies over time intervals, you get to measure trends over time for both treatment and control as a way to approximate impact.

And this is what we took advantage of.

Breaking down the cohorts in a Brand Lift Study:

But, let’s double click on the control portion.

Remember, we had TV running and other digital and non digital channels. So this control portion measures the unaided awareness of the general population that has never seen our Brand on Meta but is still subject to Marketing from other channels.

And voila! Over time, we can see how the populations unaided awareness is trending by studying the unaided awareness of the control group.

Back to the UK campaign, to get appropriate measurement, we scheduled 4 BLS:

Almost like a pre-post but with multiple checkpoints instead of just two. It was a structure that would let you see Brand metrics trending over time. This approach was novel at Uber. We hadn't done it this way before and the UK Eats campaign was going to be our first real test of whether it worked.

A test of how to measure tests.

How Meta.

Ha! Double Entendre.

Now the campaign launched and here is what we saw across the first two studies:

Then the campaign got cut.

I won’t get into why (mainly because I don’t remember exactly why), but the campaign spend was paused. A few weeks later, at what would have been our end-of-campaign study, we ran the BLS anyway.

Why?

We had the budget for the study, the structure was already set up, and even if the campaign was done we were getting another data point. You’ll never guess what happened…

Awareness was back at 40%.

Not 48%. Not 52%. 40%.

Exactly where it had started before we spent millions of dollars.

Step 1: Align on results
So, before we shared the results with anyone, we discussed and came to the conclusion from this test was that our Brand campaign had an impact. But, as every Brand marketer knows, that impact needs to be sustained to start to see lower-funnel ROI.

Essentially, any campaigns that we run that are isolated to short bursts are going to be wastes of money and time moving forward. The reason we were so confident with this is because it also aligns with all the research and evidence out there about how Brand Marketing can make an impact.

Buyers have a set of Brands in their mind when they're making a decision, and the Brand that is most top of mind at the buying moment wins. Consumers are not remembering an Uber Eats commercial from three weeks ago, no matter how big we make the campaign.

Step 2: Explain results

Next, we took this finding to Marketing leadership and then to finance. Marketing leadership was not surprised as this is a consistent story they had seen before.

So, then we took it to finance. The pushback was predictable. Maybe it needs to run longer. Maybe the UK is an anomaly. Maybe the measurement window wasn't right. These are reasonable challenges. We had one market, one campaign, one interrupted flight. It wasn't a global randomised controlled trial.

We knew that.

But the UK was not a weak test case. As I mentioned, It's one of Uber's most important markets globally and extremely representative of the competitive dynamics of food delivery. If anything, if the always-on effect was this pronounced in a market with Uber's level of Brand recognition, it was likely to be even more pronounced in markets where we were less established.

You could gain awareness quickly, but you would lose it quickly.

The part that helped convince finance was the measurement structure we had in place.

Three Brand lift studies with clean baselines, run in sequence, using the same methodology each time. The data wasn't perfect, but it was honest. And it told a consistent story. Of course, this is where it becomes a bit more art, because we also ensured that there were no other major variables that would affect awareness. There weren't conflicting campaigns, and Uber Eats wasn't in the news. There wasn't a major scandal, all of which could be factors for explaining why awareness went up.

No, the sole reason was our campaign. And the sole reason that impact disappeared is because we stopped spending.

Step 3: The next steps

We spread the findings globally. Other market teams started seeing the same pattern when they measured in the same method we did. The argument shifted from "interesting UK data point" to "Yeah, duh, this is how Brand Marketing works."

Starting in 2020, the structural approach to Brand campaigns at Uber Eats began to change. Instead of discrete campaign flights, the thinking moved toward always-on campaigned with sustained presence in key markets. The campaigns could evolve but we’d never stop advertising.

A great example is "Tonight I'll be Eating" in Australia. Even though the campaign structure started in 2017, we didn’t try and measure it until 2020 and then it became a staple in the market. It ran continuously until 2023. 2017 to 2023! Thats an eternity for a Tech company like Uber.

This is what the measurement approach did. It turned campaigns that could have been a one-off, into something more regular, because the evidence supported keeping it going.

And that’s how we proved Brand Marketing must be always on.

Need 1:1 help proving the ROI of your Marketing? Let’s chat >

That’s it for this week!

Stay experiMENTAL,

Sundar