• Home
  • Blog
  • SEO
  • SEO for AI Search: llms.txt, Structured Data and Crawler Rules
SEO for AI Search: llms.txt, Structured Data and Crawler Rules

SEO for AI Search: llms.txt, Structured Data and Crawler Rules

People now find websites through AI assistants as well as search engines. The good news is that most of what helps one helps the other: crawlable pages, clear structure and trustworthy content. This post covers the practical steps I used when building my own portfolio, and I am clear about which parts are proven and which are still experimental.

Key takeaways

  • Allow the crawlers you want, and say so explicitly in robots.txt.
  • Server-render your content so every crawler can read it without running JavaScript.
  • Use JSON-LD to describe who you are and what each page is.
  • llms.txt is a proposed convention and no guarantee, so treat it as a cheap extra and not a strategy.

1. Make the content easy to read

Crawlers read HTML. If the important text only appears after client-side rendering, some of them will not see it. Server-render or statically generate pages, use one h1 per page, keep a logical heading order, write descriptive link text and give every image meaningful alt text.

2. Control crawlers in robots.txt

Search and AI products use different user agents, and some separate search from training. Common ones include:

  • Googlebot and Bingbot for search
  • GPTBot and OAI-SearchBot from OpenAI
  • ClaudeBot from Anthropic
  • PerplexityBot from Perplexity
  • Applebot-Extended and Google-Extended, which are robots.txt tokens that control whether your content is used for certain AI features, not separate crawlers

Check each vendor's documentation for the current list, because names change. In Next.js you can generate the file from code:

export default function robots() {
  const bots = ["GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot"];
  return {
    rules: [
      { userAgent: "*", allow: "/" },
      ...bots.map((userAgent) => ({ userAgent, allow: "/" })),
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

If you do not want your content used by a particular crawler, disallow it. This is your decision, and the file is a request that well-behaved crawlers honour.

3. Add structured data

JSON-LD tells machines exactly what a page is. For a personal site or portfolio the useful types are:

  • Person with name, jobTitle, sameAs links to your profiles and knowsAbout
  • WebSite linking to the Person
  • BlogPosting for articles, with headline, dates, author and image
  • BreadcrumbList for navigation context
  • FAQPage when a page genuinely has questions and answers

Google limits FAQ rich results to a small set of sites, but the markup is still a clean, machine-readable statement of your content. Only mark up what is actually visible on the page, and validate it with the Rich Results Test.

4. Publish a sitemap and canonical URLs

A sitemap lists the pages you want indexed. Each page should declare a canonical URL, a unique title and a unique description. Keep titles around 60 characters and descriptions around 155 so they are not truncated.

5. Consider llms.txt

llms.txt is a proposal for a plain Markdown file at the root of your site that gives language models a concise, curated summary of your most important pages. It is cheap to add. It is not a standard that every AI product reads, so do not expect it to change your rankings or citations on its own. If you add one, keep it accurate and generate it from the same data as your site so it never drifts.

6. Build real authority

Technical SEO gets you indexed and understood. It does not make you the best answer. For that you need:

  • specific, experience-based content that other pages do not already cover
  • a clear author identity with consistent profiles that match your structured data
  • links from other reputable sites
  • fast pages, since Core Web Vitals are a ranking signal and a usability one

What I did on this site

This portfolio ships a generated robots.txt, a sitemap, canonical URLs on every page, Person, WebSite, FAQPage and BlogPosting structured data, a generated social image and an llms.txt built from the same data files as the pages. If you want help with technical SEO for a Next.js site, get in touch or read about my web application work.

  • SEO
  • AI search
  • structured data
  • llms.txt
  • robots.txt

Let's Work Together.

Tell me about your project. I'll reply with questions, a first approach and honest advice.