Key points
- Marie Haynes' 12-month GA4 review of 100 sites found Google drove 96.5% of conversions; ChatGPT visitors converted at 13.8% versus 3.02% for Google organic.
- Kevin Indig ran 632 tests on four brands; shifted placebo dates correlated as strongly as real ones, so he could not isolate an AI effect on Direct or branded search.
- AMEC published data quality principles that ask whether a dataset passes tests on source, accuracy and predictive validity before anyone acts on it.
- SEO Works found a 5% organic lift added 10.3 conversions per site per month; doubling AI referrals added 6.6.
Why it matters for marketers: AI visibility scores are consistent but unvalidated against customer behavior, so marketers should treat them as brand-channel indicators and test them before using them to defend budget.
A Search Engine Journal column by Greg Jarboe argues that AI visibility is not a sham, but most ways of measuring it would not pass a basic data quality test. He points to four datasets, along with warnings from SparkToro's Rand Fishkin and Amanda Natividad, that show AI assistants send a small share of the traffic Google does.
The numbers are mixed. Marie Haynes found ChatGPT visitors converted far better than Google organic visitors, yet Google still produced most conversions. SEO Works found a much narrower AI edge, and AI converted worse on 23 of 63 comparable sites. Kevin Indig then looked for hidden AI influence in Direct traffic and branded search. Once he filtered for engaged sessions, the Direct growth mostly vanished. He concluded that an effect could still exist, but he could not isolate it.
For marketers, the point is that a dashboard score can be steady week to week and still say nothing about customer behavior. Jarboe cites AMEC's data quality paper, which warns that data can be internally consistent but externally untrue. He suggests treating AI answers like a brand channel, closer to a billboard, and measuring exposure, recall and consideration. He also notes that some figures in the AMEC paper come from a benchmark sponsored by Converseon, which sells related services.
His practical steps: report AI referral traffic as a diagnostic without setting targets on it, document which tools and prompts produced each score, and run a placebo test with shifted dates. He also recommends a repeatable prompt library, self-reported attribution on forms, and budget for earned authority, where AI and organic gains overlapped in the SEO Works data.
Sources
This brief is our own summary. Read the full reporting at the source.