Holdout test or mix model?

Hey there Smarty 👋

Are you geared up to catch the latest and greatest in quick shorts?

And just a quick heads-up! If you stumbled upon us through a friend, make sure to subscribe here! That way, you’ll never miss out on the trending shorts.

⚖️ Holdout Test vs Mix Model: Which Can Prove Your AI Search Spend Works?

AI search is now a real traffic source. Adobe measured traffic from AI chats and browsers to U.S. retail sites up 670% on Cyber Monday 2025. But most of its effect never shows up as a click: in Pew's tracking of 68,879 Google searches, people clicked a link inside an AI summary on just 1% of visits. So last-click reports can't defend the budget. Two methods can, in principle. 

Here's how they compare for this channel specifically.

Round 1: Can you run it at all?

A geo holdout switches a channel off in some regions and compares sales. That works for paid ads. AI search work is mostly content, reviews and PR, and an AI answer in Ohio draws on the same pages as one in Oregon. You can't switch it off by state. A mix model uses changes over time instead, which this channel does have.

Winner: mix model.

Round 2: Speed

A holdout on a paid lever reads in four to eight weeks. A mix model needs a long run of weekly history before it says anything useful, often a year or more in my judgment.

Winner: holdout.

Round 3: What goes in

A holdout needs only sales by region. A mix model needs a weekly measure of AI exposure, and clicks undercount it badly. Google's open-source Meridian accepts organic media inputs, activity with no direct cost that still has delayed effects, so a weekly visibility score can sit beside your paid channels.

Winner: tie, if you start logging exposure now.

Round 4: Credibility with finance

Finance trusts holdouts because the logic is simple: lights off, sales drop. Mix models get questioned on assumptions. That's why the input definition matters, and why it's worth hearing it from the people who built the tools. 

On November 20 at 2 PM ET, Meridian's creator Hitesh Wadhwani joins AirOps' Matt Hammel and Paramark CEO Pranav Piyush to show how AI recommendations can become a model input finance will accept. You can register for free with AirOps here; the recording will follow within 24 hours.

Winner: holdout, for now.

The verdict

Use both, on different parts of the budget. Run holdouts on anything you can switch off by region, such as paid amplification of AI-cited content. Put the always-on content work into a mix model, with a weekly exposure score as its input. Start logging that score this week, because the model can only credit weeks you recorded.

Together with New Engen

Your January Ads Should Be in Production Now

If your Meta account has stalled, another month of the same creative gives you little new to work with.

Grapevine, now part of New Engen, produces creator ads you can run through creators’ accounts, with production handled for you.

• Approve angles and ship product. Grapevine books and pays creators, writes briefs, and edits in-house.
• Get fresh creative in under 30 days from kickoff, giving you time to test before January.
• Start with flexible pricing and no minimum spend, then test month to month before committing further.

Futurhealth runs over 40% of its account on Grapevine assets and improved CPA by more than 15%. To budget for your own test, check your cost per asset in the fee calculator.

Kick off by November 30 and your first campaign is free: 8 statics and 1 article worth $1,500. Head into Q1 with fresh creative ready to put to work.

🛍️ TikTok, Pinterest and Reddit introduce new AI tools for ads and shopping

TikTok, Pinterest and Reddit are expanding their AI-powered features with advanced video ad creation, personalized shopping tools and AI-generated product reviews, helping brands create content and consumers make purchasing decisions.

The Breakdown:

• Pinterest is adding AI-powered gift guides, product comparisons, wishlists and personalized Deal Boards, helping shoppers compare prices, discover holiday deals and receive alerts about price drops and restocks.
• TikTok has upgraded its Symphony AI tools with Dreamina Seedance models, allowing advertisers to generate videos up to 30 seconds long using as many as 50 image, video and audio references, up from nine.
• TikTok also introduced an updated Symphony API and expanded access to BytePlus tokens, enabling advertisers to create AI-generated content within their existing creative tools and workflows.
• Reddit is testing AI-generated product reviews that summarize real user discussions, bringing together comments, upvotes, product comparisons and original conversations without paid placements.

Social platforms are increasingly using AI to connect content creation, product discovery and purchasing decisions. TikTok is helping advertisers produce creative faster, while Pinterest and Reddit are making it easier for shoppers to research and compare products without leaving their platforms.

👨‍💻 Quick Hits

🎬 Meta updated its Edits video editing app with customizable speed curves, automatic audio balancing and text animation duration controls, alongside personalized content recommendations in the Inspiration tab.

📊 OpenAI reportedly introduced Target CPA and CPC bidding in ChatGPT Ads, alongside more granular geographic targeting and product filters, with older bidding options scheduled for retirement in January 2027.

🛡️ Google Ads introduced an Invalid Traffic Overview Card showing money saved and automatic credits from invalid traffic detection, reporting a 30% average reduction in advertiser exposure to invalid leads in 2026.

🔎 A Hostinger survey found 69% of US consumers used search to discover businesses, while 28% used AI assistants. Meanwhile, 94% verified AI recommendations elsewhere, and 62% routinely checked unfamiliar business websites.

💃Events

🔥 What to Measure Before AI Search Gets Cut From the Budget

November 20 | 2 PM ET | Virtual Event

Paramark CEO Pranav Piyush and Google Meridian creator Hitesh Wadhwani unpack the three requirements behind credible AI search measurement, where attribution still fails, and how exposure data can fit alongside the channels competing for spend. Take a sharper measurement checklist into your next budget discussion.

✅ Reserve Your Free Seat

Can't attend the session live? Register anyway, and you’ll get the recordings in 24 hours.

That’s a wrap for today! Tell us your thoughts about today's content as we line up more Shorts! And don’t hesitate to share this with someone who’d adore it. 🥰