LLM SEO for Developers: How AI Search Retrieves and Cites Pages

When ChatGPT search or Perplexity links to a page in an answer, that page usually did not come out of the model's training data. The product searched its own web index at request time, took the top few documents, broke them into passages and put a handful of those into the prompt. The model then wrote its reply from that material, and the link points at the passage it relied on most.

"LLM SEO" is mostly about that retrieval step. It explains why a page can rank first on Google and still never get cited, and why the fixes are mostly engineering rather than copywriting. This post walks through it the way a developer would: which bots fetch what, how retrieval scores a passage, and what you can measure. There is a small embedding experiment in the middle that you can run in a minute.

The three gates

A page has to pass three independent checks before it shows up in an AI answer. Each one can fail on its own.

A lot of advice skips straight to the third gate and talks about "authority". In my experience the first two are cheaper to fix and fail more often.

Gate 1: crawler access is not one thing

"AI bots" are at least three different kinds of client, and treating them as one is the most common mistake on this list.

OpenAI's crawler documentation separates them clearly. OAI-SearchBot builds the index that ChatGPT search pulls from, and OpenAI states that sites opted out of it will not be shown in ChatGPT search answers. GPTBot collects content that may be used to train foundation models; disallowing it is a training opt-out and nothing more. ChatGPT-User is a fetcher that runs when a user's request makes ChatGPT open a page, and the docs say it is not used to decide whether content appears in search.

I keep seeing the same accident. A site blocks GPTBot for training reasons, which is reasonable, and then someone adds OAI-SearchBot to the same Disallow group because it also says OpenAI. The site keeps ranking on Google and silently disappears from ChatGPT search. Google has the same split: Google-Extended controls training use, while AI Overviews are served from the normal index that Googlebot builds.

If you want to stay out of training but stay in AI search, the robots.txt needs to say so per group:

# Search and answer crawlers: allowed
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /

# Training crawlers: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Two details trip people up. A crawler follows only the most specific group that names it, so rules in User-agent: * do not carry over into a named group. And all of this is moot if the text only exists after client-side hydration. These crawlers fetch HTML and generally do not run your JavaScript, so curl the URL and read what comes back before you debug anything else.

Gate 2: retrieval works on chunks, not pages

This is the part that surprises people who come from classic SEO. Retrieval systems rarely score your whole page against the query. They split it into passages, embed each passage, and compare vectors. A page is only as good as its best chunk.

You can see why structure matters with a small model from the Hub. all-MiniLM-L6-v2 is a widely used sentence-embedding model, and its model card notes that input longer than 256 word pieces is truncated. Production retrievers use bigger models with longer windows, but the shape of the problem is the same.

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
query = "How much does the sitemap monitor cost per month?"

filler = (
    "Our team started this project after years of running agencies. "
    "We care about craft, and we think monitoring should feel calm rather than noisy. "
    "Customers come from many industries, from ecommerce to B2B software, and each has "
    "different habits around publishing. Some ship daily, some ship once a quarter. "
    "Over time we learned that the hardest part is not collecting data but deciding what "
    "deserves an alert, so most of the product is about filtering. "
)
answer = "The sitemap monitor costs $9 per month and checks up to 5,000 URLs daily."

buried = "About us. " + filler * 4 + answer          # answer at the end of a long section
chunk = "How much does the sitemap monitor cost? " + answer + " Annual billing is also available."
no_answer = "About us. " + filler * 4                # same section with the answer removed

q = model.encode(query, convert_to_tensor=True)
for name, text in [("buried", buried), ("chunk", chunk), ("no answer", no_answer)]:
    print(name, round(util.cos_sim(q, model.encode(text, convert_to_tensor=True)).item(), 3))

When I ran it:

The first two rows are identical. The answer sat past token 256, the model cut it off, and as far as retrieval was concerned the price was never on the page. The third row is the same sentence, placed where a chunker would isolate it.

Bigger embedding models push the limit out, and real pipelines chunk before embedding, so you will rarely lose an answer this cleanly. The general lesson still holds, and you can compare models with longer context on the MTEB leaderboard if you want to test it further. Four habits follow from it:

• Headings are chunk boundaries, since most chunkers split on them. Eight question-shaped H2s give you eight passages that can each match a different query; two headings over 2,000 words give you two diluted ones.
• Lead each section with its answer. The chunk travels alone, without whatever came before it, so a section that opens with "as mentioned above" reaches the model missing its subject.
• Phrase headings the way people ask. A heading that reads "Plans" matches almost nothing; one that reads "What does the monitor cost per month?" matches a real query word for word.
• Keep facts in text. A number that only exists inside a chart image or behind a click-to-expand tab does not survive extraction.

Gate 3: why a model names you

Retrieval usually returns several passages that agree with each other, and the model credits only one or two of them. You cannot control that choice, but you can make your passage the easier one to attribute.

The strongest lever I know of is saying something only you can say. Generic advice appears on many pages, so a model can paraphrase it without crediting anyone. A number from your own data, a benchmark you ran, or a result from your own users can only be stated by naming the source. The experiment above is a small example: the 0.165 vs 0.880 figure exists on this page and nowhere else.

The weaker levers are boring and still worth doing. Show a visible publish date and a named author. Link the sources for claims you did not produce. Keep your product described the same way on your site, docs and directory listings, because when those disagree, the newest and least corroborated description tends to lose.

Where llms.txt fits

llms.txt is a proposal for a Markdown file at your site root that points language models to the pages and docs that matter, with short descriptions. It is useful for documentation sites, and some coding agents and tools read it. It is a proposal, not a ranking signal: nothing indicates that the big AI search products use it to decide what to retrieve or cite. Treat it as a convenience for agents that look for it, and fix gates 1 and 2 first.

Measuring it

AI search has no position 1 to track. Ask the same question in three products and you get three sets of sources, and asking twice in one product often changes them too. The useful number is a rate: pick twenty prompts your buyers would type, run each one several times, and record the share of runs that mention your domain.

For the access gate you do not need sampling at all, because it is deterministic. Fetch robots.txt, resolve the groups for each named bot, and fetch the page as plain HTML. Python's standard library is enough for a first pass:

import sys
import urllib.error
import urllib.request
from urllib.robotparser import RobotFileParser

BOTS = {
    "search": ["OAI-SearchBot", "PerplexityBot", "Claude-SearchBot", "Googlebot", "Bingbot"],
    "training": ["GPTBot", "ClaudeBot", "Google-Extended", "CCBot"],
}

def audit(url: str) -> None:
    root = "/".join(url.split("/")[:3])
    rp = RobotFileParser(root + "/robots.txt")
    rp.read()
    for group, names in BOTS.items():
        for bot in names:
            status = "allowed" if rp.can_fetch(bot, url) else "BLOCKED"
            print(f"{group:8s} {bot:16s} {status}")

    # robots.txt is only half the story: a firewall can still refuse the request
    req = urllib.request.Request(url, headers={"User-Agent": "OAI-SearchBot/1.4"})
    try:
        html = urllib.request.urlopen(req, timeout=20).read().decode("utf-8", "ignore")
    except urllib.error.HTTPError as e:
        print(f"fetch as OAI-SearchBot: HTTP {e.code} (blocked at the server, not robots.txt)")
        return
    print("raw HTML length:", len(html), "| <h2> tags in raw HTML:", html.lower().count("<h2"))

audit(sys.argv[1])

Run against https://huggingface.co/blog, every bot in both groups comes back allowed, and the raw HTML already contains 14 <h2> tags, so the headings exist before any JavaScript runs. Pointed at the homepage of a large news publisher, the same script prints the opposite: robots.txt disallows OAI-SearchBot, PerplexityBot and Claude-SearchBot while Googlebot and Bingbot are allowed, and the request with an OAI-SearchBot user agent gets an HTTP 403 from the server before robots.txt even matters. That is a deliberate policy for a publisher. On a product site it is usually an accident, and this script is the fastest way to find out which one you have.

Two limits are worth knowing. RobotFileParser follows the standard but not every crawler's extensions, so treat a surprising answer as a reason to read the file by hand. And a 200 response with the right user agent string does not prove the real crawler gets through, because some firewalls check the crawler's published IP ranges rather than the header.

That covers the deterministic part. Tools built around llm seo checks do this for you. Tracetify's guide sorts bots into search crawlers, assistant fetchers and training crawlers, and its checker reports the training group without scoring it, since blocking those has no effect on what AI search can retrieve today.

What changes and what doesn't

Most of the work carries over from regular SEO: being indexed, loading fast, writing something original. What changes is the unit. Search engines rank pages. AI answers are assembled from passages, and the passage boundary is something you control with headings and sentence order. So a fair amount of LLM SEO turns out to be a formatting job. I find that reassuring: it is work you can do in a template, test with a short script, and ship the same afternoon.