How Uber cut $25M in Ads

👋 Hey, it’s Sundar! Welcome to experiMENTAL: my newsletter on the CRAZY world of growth analytics at high growth companies and my solopreneurship journey.

In this newsletter, you’ll learn about how we turned off a marketing channel saving Uber $25M+ in the process.

We’ll cover the:

• Context

• Problem

• Analysis

• How we put it all together

Enjoy and don’t hesitate to share feedback!

I recently spoke with Josh Lachkovic about an analysis I did at Uber and his post saw some solid interest and engagement.

Okay that’s an understatement. It went viral.

As I read the comments, I noticed there was a lot of appetite around understanding how larger brands think about ad spending especially when decreasing or turning off spend.

We see a lot of posts about increasing spend and growing but what about the opposite?

What about cutting spend ? How did we come to that decision?

I'm going to go into as much detail as I can and share the behind the scenes of how Uber saved $25M a year by ramping down on Facebook/ Meta.

The Context

The year was 2018.

Uber was now 8 years old and a global house hold name. Things were humming on Uber Rides and growing like crazy on Uber Eats.

I was in year 2 of 5 of my career at Uber and had just been promoted to lead US & Canada Rider Performance Marketing Analytics. In addition, I was also asked to own Uber's TAM Penetration Analysis. Leveraging 3rd party data, we wanted to understand how many people had signed up and what our market penetration was in the US.

The Problem

Every week, the Marketing team and I would go over our numbers.

Spend. Signups. Trends.

You know... the standard weekly business review.

For Q1 2018, we were seeing spend stay flat but signups go up and therefore our CACs going down. Everyone was thrilled.

Then enter Q2 2018. After St. Patricks Day, all of a sudden, we hadn't changed anything and our CACs were spiking.

Signups were dropping continuously for weeks.

What the f*ck is going on?!

"Sundar, can you dig into this please?!"

The Analysis

Before I dive into an analysis, I like to write out questions I want to answer.

This forces me NOT to go down a rabbit hole.

Heres what I needed to know:

1. Has our marketing spend changed significantly?

2. Have there been new product changes that may have impacted conversion rate?

3. Has anything happened in the macro context?

4. What happened with signups last year?

Question 1: Has our Marketing Spend changed significantly?

No, not on Facebook. We had been spending within a tight band with only a few periods of ramp ups and ramp downs in the preceding 6 months. Phew. A break you can only dream of when you're doing this type of analysis.

Question 2: Have there been new product changes that may have impacted conversion rate?

No. Of course Uber is shipping out thousands of experiments but those build and the impact is over time. I'm sure there were a few % points but we weren't in the phase of early adoption anymore. Rides was a mature and relatively stable business.

Question 3: Has anything happened in the macro context?

Luckily, no. There were no obvious macro movements that could have impacted our marketing performance to the XX% changes we were seeing.

Which brings me to...

Question 4: What happened with signups last year?

Plotting data YoY is one of my go to analytics moves.

Why? Because you get to account for seasonality.

Turns out that signups basically behaved the SAME WAY the previous year, and the fluctuations in Signups in Q1 and Q2 were largely explained by seasonality.

That doesn't feel right...

My spidey senses are tingling!

Putting it all together

I love ratios. They're an elegant way to combine two pieces of information and observe a relationship. CAC is a great example of one of these ratios that is simple but oh so nuanced.

For those unfamiliar, CAC stands for Cost Per Acquisition.

CAC = $ Acquisition Spend / # of Acquisitions

When a ratio is wildly fluctuating but the numerator (Acquisition Spend) has stayed constant, theres really only one explanation:

Your denominator (in this case signups) is fluctuating.

One level deeper.

If your spend is the same but your signups are fluctuating, theres really only one explanation:

Theres LITTLE RELATIONSHIP BETWEEN THE TWO.

Now, remember that Penetration Analysis I talked about way at the beginning?

Thats the pièce de résistance in the whole analysis. That context changed everything.

The analysis suggested:

1. Almost everyone knew about Uber in North America

2. Almost everyone who would become a user was a user

3. Everyone else was unlikely to become a user

By combining information about:

• Market saturation

• The non relationship between signups and spend

I knew we were likely very inefficient and had to run a channel level incrementality test.

However, as many analysts know: insights do not mean action.

This is where much of the credit has to go to my Data Science and Marketing leadership.

I presented my analysis and they stood behind my recommendation to run a channel incrementality test.

We werent going to test the effectiveness of a spend level.

We were going to test the effectiveness of the whole channel as an acquisition lever.

For those that have been in Marketing, thats a bold move.

And thats exactly what we did.

We did a holdout on Meta (don't remember if it was a digital A/B test or geo based) for 3 months.

3 months where we launched no new creative and made very few changes to the set up.

3 months.

The results came back conclusively that there was no incrementality.

With no ego, pride, budget manipulating, or any other crap I sometimes see, we made the decision to turn off spend on FB and return the money to the business.

Other considerations

Looking at a few comments from Josh’s post, there were a few important points I wanted to touch on:

What about CLV / CAC ratio?

CLV / CAC is a great metric to look at high level but it says nothing about incrementality. Why should I assume that the amount I'm spending is incremental just because its aligned with a ratio?

What if my CLV / CAC ratio was infinity aka I spent 0 on acquisition.

Isnt that even better?

Why didn’t you just move the budget to another objective?

We couldnt just move the budget to reengagement for a variety of reasons:

1. Reengagment wasn’t spending as much as acquisition so throwing that money there would have created so many inefficiencies

2. We had never tested the incrementality of reengagement so adding more money doesn’t make sense

3. The idea of use it or lose it is beyond stupid and its now how we ran things at Uber (although like many large companies we sometimes had to)

That whole analysis and action plan took a few week.

That's it.

But, it's still one of the coolest analyses I've done because it showed me what a high functioning data-driven culture looks like.

I was also able to point to tangible impact which is when every analyst wants to show.

And thats how we saved Uber $25M.