AI Search Still Needs Someone to Cite. It Might Be You.

Reddit updated its robots.txt in 2024 to block nearly every crawler that won't commit to leaving Reddit content out of AI training. The New York Times blocked OpenAI's crawler (and Internet Archive's crawler) the year before that, then sued Perplexity for allegedly ignoring the block outright. A peer-reviewed study presented at the ACM Web Conference 2026 found the pattern goes well beyond a few high-profile holdouts: 60% of reputable news sites now disallow at least one AI crawler in their robots.txt, up from 23% in September 2023 - compared to just 9.1% of misinformation sites.

Every one of those sites is opting out of the one thing that decides whether an AI answer engine can cite them: being crawlable. When a high-authority source blocks the crawler, the AI still has to cite someone. If it can't cite Reddit or the Times, it goes looking for whoever's left standing - and “whoever's left standing” is increasingly whoever wrote something specific, well-structured, and actually reachable.

I first flagged this as a single line in my own AI SEO one-pager - marked as a bet, not a finding - and wanted to dig into it properly here.

For example: ask an AI search tool something like “is there a Reddit community dedicated to rating keyboards?” Reddit's the obvious source, and it's blocked. So the AI goes looking for whoever's left standing: maybe a keyboard enthusiast's blog with a running post on their favorite places to talk mechanical keyboards. Two years ago, that blog would've been buried under three pages of Reddit results. Now it might be the citation.

I can hear the objection already: isn't that just SEO with extra steps? Yes, and: traditional SEO was a ranking problem - you were competing against every other page chasing the same query, in a field that was enormous. This is an access problem too.

The pool of citable sources shrinks every time a major publisher locks its robots.txt, and the businesses willing to stay open to crawlers are competing for a shrinking pool relative to how much people are searching.

That's not just a hunch. A SIGIR 2026 study of 11,500 real search queries found that websites blocking Google's AI crawler are significantly less likely to be retrieved by AI Overviews - even when the content is otherwise fully accessible to them.

Picture a spelling bee where five of the ten students have been eliminated. Someone still needs to win the bee so the other students remain on the stage, waiting for their next word. Their student’s chances to win just increased, not because their spelling skills increased, but because half their competition is gone. Researchers auditing generative search engines found the same dynamic playing out for real: across four AI search tools and hundreds of real queries, roughly 16% of the sources being cited were themselves AI-generated content, standing in for the sources that locked their doors.

What “crawlable” actually means, technically

Being open in robots.txt is table stakes, not the whole job. Three things also have to be true.

The content has to reach the page as plain, readable HTML - not locked behind a form, not rendered only in JavaScript after the page loads. Most AI crawlers still skip JavaScript entirely: research tracking real crawler traffic found that even when bots like GPTBot and Claude's crawler download a page's JavaScript files, they read them as plain text rather than running them - so any content that only appears after the script executes stays invisible no matter how often the file itself gets fetched. A form gate is a harder stop than that: no email address, no content, whether the visitor is a person or a bot.

The page has to tell the AI what it's looking at. That's what structured data does - usually written as JSON-LD, the format Google and Microsoft have both confirmed they use for this. A minimal example on an article page looks something like:

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "AI Search Still Needs Someone to Cite. It Might Be You.",
  "author": { "@type": "Person", "name": "Molly Purcell" },
  "datePublished": "2026-09-01",
  "publisher": { "@type": "Organization", "name": "Your Company" }
}

That block hands the AI a name, a date, and an entity to attach the claims to, instead of asking it to guess authorship from a byline buried in the page footer.

And the content has to cite its own sources, specifically. This is the part people skip. “Studies show…” gives an AI nothing to verify or attach a citation to. “Reddit updated its robots.txt in 2024” gives it a checkable claim with a name and a date attached - the kind of specific, attributable fact that both AI systems and human readers treat as more credible than a vague appeal to authority. Every claim above links to where it came from - an article arguing for specific, checkable sources should probably have some of its own.

Where marketing ops actually comes in

This is where I'd push back on framing this as purely a content or SEO team problem. In practice, none of the three things above get decided by whoever writes the blog post. They get decided by whoever owns the templates.

If your gated content lives behind a Pardot or Marketo form, marketing ops is the one who decided that form sits in front of the content instead of after it - and that's the single biggest crawlability decision on the page, made before a single word gets written. Whether your landing pages ship with Article or FAQPage schema isn't a one-off content edit either; it's a template-level change, which makes it a marketing ops and web/dev handoff, not something a writer can add to a page on their own.

So “make your content crawlable” turns out to be an operations question just as much as a content one: does your marketing automation platform generate clean, unlocked HTML by default, or does it wrap everything in a form gate? Is schema baked into your landing page templates, or does it not exist anywhere in your stack? Those get decided by whoever configures the platform - months before there's any content to worry about citing.

This isn't a hypothetical deadline, either. Starting September 15, 2026, Cloudflare is changing its default settings: any “mixed-use” crawler - including Googlebot, which crawls for both regular search indexing and AI features - gets blocked by default on any page that carries ads, unless someone actively reconfigures it. If your company's site runs on Cloudflare and nobody's touched that setting, you're either about to lose crawl access you didn't know you had, or you already have - and it happened at the infrastructure layer, not the content layer, which is exactly the point of this section.

Where smaller brands actually have the advantage

There's a sharper version of this opportunity than “get cited.” Columbia's Tow Center for Digital Journalism tested eight AI search tools - ChatGPT, Perplexity, Gemini, Copilot, Grok, and others - by asking them to summarize real news stories, and found they got it wrong 60% of the time on average. Perplexity was the most accurate at a 37% error rate; ChatGPT was wrong 67% of the time; Grok was wrong 94% of the time. When these tools did cite a source, the citation frequently pointed to a syndicated copy of the story instead of the original, or to a dead URL - Gemini and Grok did this on more than half their citations.

That's a lot of confident wrongness, and it matters more, not less, for a small brand than for a news story - because there's no competing “other publisher” who can correct the record on your own product. If an AI tells someone your return policy is 30 days when it's actually 60, or walks them through an outdated fix for a known issue, there is exactly one entity on the internet positioned to be the accurate source instead: you. A small brand that keeps its product pages, FAQ, and support docs genuinely accurate, specific, and crawlable isn't just angling for a citation anymore. It's often the only accurate source available for the AI to draw from at all - which is a different, and arguably better, kind of leverage than competing with the New York Times for a citation slot.

None of this is proven yet. I don't have a study showing “opt-in crawlability equals more AI citations” - what I have is a pattern I keep noticing, and it rhymes with something that already happened once, with an actual date attached this time. Schema.org itself only exists because Google, Microsoft, and Yahoo jointly launched it in 2011 to solve exactly this problem: search engines that couldn't reliably interpret what a page was about. The sites that adopted structured markup early got a real head start in how search engines represented them. This might be the same shift, happening one layer up the stack.

The tooling is still catching up, too. Google is rolling out a Search Console toggle that lets publishers opt out of AI Overviews specifically while keeping their regular search ranking intact - the flip side of the bet this piece is making. But Digiday reports that almost nobody can use it safely yet, because Google won't share the AI-specific referral data publishers would need to judge whether opting out actually costs them anything. The controls are arriving faster than the information needed to use them well, on both sides of this bet.

If you're weighing whether to block AI crawlers on principle or leave the door open hoping to get cited, that's a real trade-off, and I don't think there's a universally right answer yet. But if you're already leaning toward staying open: the payoff for making your content genuinely well-structured just went up, and that's not a content team's call to make alone anymore. You're not competing with every other page anymore. You're one of the last shops still open on a block where everyone else pulled the shutters down - assuming operations actually left the front door unlocked.