The short version
- Farhan Munshi ran paid merchant acquisition at Shopify as Growth Marketing Manager: $20M+ a year across Google, Meta, TikTok, YouTube, Reddit and LinkedIn, scaled 4x while holding a 3.2x ROAS.
- Geo holdout tests with Data Science found $2M+ in paid social spend that wasn't adding merchants. A separate geo holdout on search showed branded search was about 35% incremental and non-brand about 70%, and about $4M a year moved out of brand.
- On the $1 a month trial, bidding and reporting moved from signups to paying merchants. Signup CPA rose about 20% and cost per paying merchant fell about 30%.
- After iOS 14.5, the Meta account went from 20+ ad sets to three to five on the Conversions API, and CPA held within 15% of pre-ATT levels.
- TikTok went from a pilot to $800K a year at a CAC 28% below the account average.
- Budget decisions ran on a dashboard that showed merchant GMV at 30, 60, 90 and 180 days for every campaign.
01The situation
Shopify sells commerce software to people starting and running online stores. Acquisition was self-serve: an ad led to a trial, the trial led to a paid plan, and there was no lead form or sales handoff in between. I owned paid media for merchant acquisition, $20M+ a year across Google, Meta, TikTok, YouTube, Reddit and LinkedIn, in North America and in new market launches across EMEA, APAC and LATAM.
What was broken was the scoreboard. The ad platforms reported plenty of cheap trial signups, and reporting ran on last click. Neither one told us which dollars produced merchants who paid, stayed and sold, or which conversions would have happened without the ad. Then iOS 14.5 took away much of the signal the Meta account had been built on.
02Results at a glance
| Area | What changed | Result |
|---|---|---|
| Scale | Paid media across six channels | $20M+ a year, scaled 4x at a 3.2x ROAS |
| Paid social incrementality | Geo holdout tests with Data Science | $2M+ in non-incremental spend found and moved |
| Search incrementality | Geo holdout with brand and non-brand in separate cells | Brand about 35% incremental, non-brand about 70%. About $4M a year moved out of brand |
| $1 a month trial | Bidding and reporting moved to paying merchants | Signup CPA up about 20%, cost per paying merchant down about 30% |
| Meta after iOS 14.5 | 20+ ad sets to three to five, on the Conversions API | CPA within 15% of pre-ATT levels, about 25% of lost signal recovered |
| TikTok | Pilot to core channel | $800K a year at 28% below average CAC |
| Priority segments | Audience segmentation for mid-market and enterprise merchants | CAC down 28% year over year |
| Paid search landing pages | Landing page, offer and copy tests | Conversion up 34% |
| Keyword intent | Budget moved to business-start keywords | About 40% higher 90-day activation, at about twice the signup CPA |
| Shopify Audiences beta | First-party segments against standard lookalikes | About 25% lower cost per paying merchant |
| Entry plan price increase | Value-led creative and landing pages | Trial-to-paid dipped about 7% in 60 days on a 33% increase |
| Shopify App | Install and re-engagement campaigns | 2.4M+ installs |
03How I moved bidding from signups to paying customers
During the free trial boom, the easy growth was people who wanted to start a business someday. Trial CPAs looked great. When we dug into the downstream numbers, those cohorts churned, and a merchant who lasted cost far more than the signup number suggested.
- Changed the scoreboard.Trial starts and cost per trial sat at the top. Activation and trial-to-paid sat in the middle. Cost per paying merchant, payback and GMV sat at the bottom, read in 30, 60 and 90 day cohorts. Budget followed the bottom.
- Rebuilt for the $1 a month trial.When Shopify rolled out the paid trial, I rebuilt campaign structures, landing pages and bidding around the new offer on Google and Meta, and moved bidding and reporting to paid conversions.
- Accepted a worse top line.Signup CPA rose about 20% against the free trial baseline, because a $1 commitment filters out people who were never going to open a store. Cost per paying merchant fell about 30%.
- Fed merchant value back into bidding.Over time, bidding moved from signups toward merchant value, with downstream GMV passed back so the platforms could aim at merchants likely to sell.
- Moved the message upstream.Targeting and creative shifted away from people who wanted to become entrepreneurs and toward people who already ran a business and were moving it online.
This is the PLG version of a problem every paid account has: the cheapest conversion is rarely the customer you want. The setup details for Google and Meta are in my note on value-based bidding.
04Geo holdout test example: branded search and paid social
Platform dashboards count attributed conversions. A geo holdout answers a different question: how many of those would have happened anyway. I ran two sets of them with Data Science.
The search holdout
- Picked matched markets.We chose comparable mid-size US DMAs and split them into test and holdout groups.
- Paused branded search entirely in the holdout markets.Non-brand got its own separate cell, so the two reads didn't mix.
- Ran it for about eight weeks.That was long enough to reach statistical significance.
- Read the difference.Branded search came back about 35% incremental, which means roughly 65% of the conversions it claimed would have arrived through organic anyway. Non-brand came back about 70% incremental.
- Moved the money.About $4M a year moved out of brand and into non-brand search and prospecting, and part of it paid for tests in connected TV and radio. Blended incremental CAC improved by an estimated 15%.
Before the test, branded search looked like the most efficient line in the account, the way it almost always does in-platform. The dashboard was reporting what it was built to report. It just couldn't say which of those merchants would have signed up without the ad.
The paid social holdouts
The same method on paid social, also with Data Science, found $2M+ in spend that the platforms were taking credit for and that wasn't adding merchants. That budget moved to channels and audiences that were.
How to size and design one of these is in my notes on incrementality testing and on marketing mix modeling versus incrementality.
05How I rebuilt Meta ads after iOS 14.5
Before iOS 14.5, the Meta account was heavily segmented: 20+ ad sets split by audience. When Apple's privacy change cut the signal, small ad sets stopped getting enough conversions to learn from, attribution windows changed and platform CPA got less reliable.
- Server-side events first.With the martech and data teams, I led the move to the Conversions API and cleaner first-party events in about 30 days. That recovered roughly 25% of the conversion signal we'd lost.
- Consolidated the structure.20+ segmented ad sets became three to five broad ones, so each had enough volume to learn.
- Let creative do the targeting.Broader audiences only work with more creative, so creative volume and structured testing went up at the same time.
- Moved the source of truth out of the ad platform.Meta and Google numbers became inputs. Decisions ran on our own merchant dashboard and on holdout tests.
CPA held within 15% of pre-ATT levels while much of the industry swung 30% to 50%.
06TikTok ads case study: from a pilot to $800K a year
When I joined, TikTok wasn't a proven channel for us, and I didn't expect it to work for selling commerce software. I was wrong.
- I ran it as a series of small pilots before it went anywhere near the main channel mix.
- I wrote creator briefs and tested creator-style content inside performance campaigns.
- The creative testing system I built there surfaced winning hooks faster than any channel we'd run before.
- It scaled only once the economics held: $800K a year at a CAC 28% below the account average, as a core channel.
Reddit, YouTube and connected TV
- Reddit. We bought the communities where people starting businesses already talked, like the side hustle and entrepreneur subreddits. The creative that worked read like a Reddit post: conversational, no loud logo, and real merchants telling their own story. Reddit's own conversion tracking was inconsistent at the time, so we didn't treat it as the truth.
- YouTube. I used it for merchant education and for shaping intent.
- Connected TV. We tested it market by market with budget freed up by the branded search holdout. I had little confidence going in, and it turned out to be profitable.
- Google and Meta. Google stayed the largest channel and the strongest at capturing intent, run through Search Ads 360 across multiple Google Ads accounts. Meta was where demand creation and creative volume lived.
07Creative testing: 100+ ad variations a quarter, one variable at a time
At this level of spend, creative fatigues fast. Testing ran as a weekly sprint with the Creative team, and we ran the campaigns in-house.
- Monday review.I went through the previous week's performance.
- Brief.I worked directly with designers, wrote the briefs and defined exactly what would change in each ad.
- Ship.New ads and tests went live the same week.
- Repeat.Winners got iterated and tired ads got replaced.
The framework isolated one variable at a time, in the order they move results: hook first, then value prop, then format (static, video or creator-style), then call to action.
- Five hooks against the same body beat five completely new ads. The buyer and the value prop hold still, so you learn which angle makes them care.
- Most of the 100+ variations a quarter were a winning hook shipped in five or six versions with different opening lines.
- Call to action placement was the least useful thing we tested. The lifts were marginal.
- Across the account, that added up to 40+ structured experiments a quarter on creative, audiences and landing pages.
08The merchant quality dashboard that decided budget
Reporting used to stop at the trial start or the first payment. With the BI team, I built a post-acquisition quality dashboard that joined ad platform spend to Shopify's internal merchant lifecycle data. For the first time, the performance team could see GMV at 30, 60, 90 and 180 days by campaign and ad group. It became the main tool for allocating budget.
What keyword intent analysis showed
Connecting source, keyword and creative to what merchants did later changed the search account. Merchants who came in on "how to start a business" keywords cost about twice as much per signup as merchants from "free online store" keywords, and they activated about 40% better at 90 days. We paid more per signup on purpose and moved budget to the higher-intent keyword clusters.
I worked in BigQuery and SQL alongside the analysts and ran cohort analysis on payback. The numbers went to leadership in weekly performance sessions, monthly readouts where budget and strategy changed, and quarterly business reviews.
09Audience segments, landing pages and a price increase
Segmentation for bigger merchants
I built audience segmentation frameworks for mid-market and enterprise merchant acquisition. CAC for those priority segments fell 28% year over year.
Paid search landing pages
With the Web and Lifecycle teams, I ran iterative landing page, offer and copy tests, with pages built in Unbounce. Paid search conversion rose 34%.
Shopify Audiences beta
Before Shopify Audiences launched to merchants, I worked with its product team to beta test it on Shopify's own acquisition campaigns. Segments built on first-party purchase intent beat standard Meta lookalikes on cost per paying merchant by roughly 25%.
Advertising through a price increase
When the entry plan price went up by a third, I refreshed all ad creative to lead with value instead of price, and A/B tested landing pages built around ROI and merchant success stories. Trial-to-paid conversion dipped only about 7% in the first 60 days.
10New market launches and the Shopify App
I launched net-new paid campaigns as Shopify went live in markets across EMEA, APAC and LATAM. I worked with regional specialists and country managers so the messaging and the campaigns were localized and made sense in each market.
With Product and Creative, I ran app install and re-engagement campaigns that drove 2.4M+ Shopify App installs. Through all of it, media investment grew 4x while holding a 3.2x ROAS.
11What didn't work
- Over-segmenting Meta. Before iOS 14.5, the account had lots of audiences, lots of ad sets and lots of control. It looked sophisticated and it was fragile. When the signal dropped, too many ad sets had too little volume to learn.
- Chasing cheap signups. In the free trial boom we advertised to would-be solo founders. Trial CPAs looked good, churn was high, and the unit economics didn't hold up.
- Trusting short tests. When we measured long enough, about 40% of the experiments that showed a positive short-term result had no long-term effect.
- My own priors. I didn't think TikTok or connected TV would work. Small, properly designed tests said otherwise.
12What I'd tell you
- Bid to the deepest event that still has volume, and warn your boss that the shallow metric will get worse. Ours rose about 20% while the one that mattered fell about 30%.
- Hold out branded search before you defend its budget. Pause it in matched markets for about eight weeks. Ours was about 35% incremental.
- After a signal loss, consolidate. Three to five ad sets on server-side events beat 20+ ad sets on a pixel.
- Test hooks before formats, and formats before calls to action. Five hooks on one body tells you more than five new ads.
- Re-read your winners at 90 days. GMV by campaign showed us that keywords costing twice as much per signup were the better buy.
I learned to care about paying customers while running my own online store, and I used the same thinking on B2B pipeline at Pipe17. Today I do this work as a performance marketing consultant and a SaaS marketing consultant. If your account is optimizing to signups, leads or last click, a Google Ads audit is a good first step, and here's how to work with me.
FAQQuestions people ask
- What is a geo holdout test, and what does a real example look like?
- A geo holdout turns a channel off in some regions, keeps it on in matched regions and compares the results. At Shopify, Farhan Munshi and the Data Science team paused branded search in comparable mid-size US markets for about eight weeks. Branded search came back about 35% incremental and non-brand about 70%, and about $4M a year moved out of brand.
- How incremental is branded search?
- It varies by company, so test it. In Farhan Munshi's geo holdout at Shopify, branded search was about 35% incremental, which means roughly 65% of the conversions it reported would have arrived through organic anyway. Non-brand search was about 70% incremental. Before the test, brand looked like the most efficient line in the account.
- What is an example of a paid social incrementality test?
- At Shopify, Farhan Munshi ran geo holdout tests on paid social with Data Science: ads kept running in some regions and went dark in matched ones. The tests found $2M+ in spend that the platforms were crediting with conversions and that wasn't adding merchants. That budget moved to channels and audiences that were adding them.
- How do you rebuild Meta ads after iOS 14.5?
- Send conversions from your server, consolidate the account and let creative do the targeting. At Shopify, Farhan Munshi moved the Meta account to the Conversions API in about 30 days, cut 20+ segmented ad sets to three to five broad ones and raised creative volume. CPA held within 15% of pre-ATT levels while much of the industry swung 30% to 50%.
- Should you bid to signups or to paying customers?
- Bid to the deepest event that still has enough volume. When Shopify rolled out its $1 a month trial, Farhan Munshi moved bidding and reporting from signups to paying merchants. Signup CPA rose about 20% because low-intent signups dropped out, and cost per paying merchant fell about 30%. Judged on signup CPA alone, the change would have looked like a failure.
- Do TikTok ads work for SaaS?
- They did for Shopify's self-serve signup. Farhan Munshi took TikTok from a pilot to $800K a year at a customer acquisition cost 28% below the account average. What worked was creator-style content, briefs written for creators and fast hook testing. He started with small pilots and scaled only once the numbers held.
- How many ad creatives should you test each quarter?
- Enough to stay ahead of fatigue at your level of spend. At Shopify, Farhan Munshi worked with the Creative team in weekly sprints that shipped 100+ ad variations a quarter. Most were a winning hook in five or six versions with different opening lines. Hooks got tested first, then value prop, then format. Call to action placement mattered least.
- How do you measure the quality of customers a campaign brings in?
- Join ad spend to what customers do after they convert. At Shopify, Farhan Munshi built a dashboard with the BI team that showed merchant GMV at 30, 60, 90 and 180 days by campaign and ad group. It showed that keywords costing about twice as much per signup brought in merchants who activated about 40% better at 90 days.