Key Takeaways
- Rapid growth: GPTBot increased its request volume by 305% in just one year and is now the third most active web crawler worldwide (Cloudflare (2025)).
- Page rendering matters: Most AI crawlers cannot render websites like a browser and only see the raw text (Google Search Central: JavaScript SEO Basics (2024)).
- Structured data works: Pages with rich results receive 58% of all clicks. 54.5% of websites already use the JSON-LD data format (W3Techs (2026), Milestone Research (2020)).
- New standard llms.txt: 8.7% of the top 1,000 websites have already set up an llms.txt file, a kind of business card specifically for AI systems (Rankability (2026)).
ChatGPT has reached 800 million weekly users (TechCrunch (2025)), Perplexity processes millions of search queries daily, and Google is displaying AI-generated summaries in an increasing number of search results. AI search engines are changing how people find information. Yet most business websites are not technically prepared for this shift.
The problem is rarely the content itself. It is the technology behind it. AI systems crawl websites differently than traditional Google Search. They need different access points, different formats, different structures. And if your website does not meet these requirements, you simply do not exist for AI search engines.
In client projects, I see the same pattern again and again. Companies invest in quality content, but their website does not deliver it in a way that AI systems can process. This article shows you as an SME decision-maker which technical factors truly matter and gives you a concrete checklist to work with. Whether you handle the implementation in-house or work with a GEO agency, you will know exactly what needs to be done.
Can AI Actually Read Your Website?
The first question sounds trivial. It is not. Between what a human visitor sees and what an AI system receives, there is often a vast gap.
To understand this, it helps to take a brief look at so-called crawlers. A crawler is an automated program that visits websites and reads their content, fully automatically and at large scale. Search engines like Google use crawlers to find websites and include them in their search results. AI systems like ChatGPT and Perplexity have developed their own crawlers that read your content and use it as a source for answers.
Which AI Crawlers Exist and What They Do
Alongside the classic Google crawler (Googlebot), several AI-specific programs now crawl the web. The most important ones fall into two categories.
Search crawlers retrieve your content to cite it as a source in real time. When someone asks a question on ChatGPT or Perplexity, their crawlers (OAI-SearchBot and PerplexityBot) search the web and link to your page. This brings you direct traffic.
Training crawlers download your content to train AI models. Your texts are incorporated into the model without any attribution or link back to your site. These include CCBot (Common Crawl), Bytespider (ByteDance), and partly Google-Extended (Gemini).
Some crawlers are hybrids. GPTBot from OpenAI serves both training and real-time search for ChatGPT. ClaudeBot from Anthropic is used for training and research. For most SMEs, the search functionality of these programs outweighs the training aspect.
The Difference Between Google's Crawler and AI Crawlers
Google's crawler is particularly powerful because it uses built-in browser technology. Specifically, Googlebot runs on an evergreen version of Chromium, the same browser engine that Google Chrome uses, always up to date (Google Search Central: JavaScript SEO Basics (2024)). This means Googlebot can render your page just like a real visitor in Chrome, including all dynamic content, animations, and lazily loaded text.
Most AI crawlers do not have this capability. GPTBot, ClaudeBot, and PerplexityBot read the raw HTML source code, the base text of a website as delivered by the server. If your website is only assembled in the visitor's browser (via JavaScript), these AI crawlers see nothing but an empty shell. Google's documentation on JavaScript and search engine optimization confirms exactly this problem. Systems without built-in browser technology receive only the initial, often empty code.
At the same time, AI crawler activity is growing rapidly. OpenAI's GPTBot increased its request volume between May 2024 and May 2025 by 305% and climbed from ninth to third place among the most active web crawlers worldwide (Cloudflare: From Googlebot to GPTBot (2025)). AI SEO is no longer a niche topic but affects a growing share of overall website traffic.
How Do You Control Which AI Systems Access Your Website?
Every website has a file called robots.txt in its root directory. This simple text file works like a door sign and tells crawlers which areas they may visit and which they may not. For AI crawlers, this file is especially important because it allows you to specifically control which AI systems are allowed to read your content.
Which Crawlers to Allow and Which to Block
For SMEs, a clear strategy is recommended. Allow search crawlers, block pure training crawlers. GPTBot and ClaudeBot as hybrid crawlers should be allowed because the benefit from the search function outweighs the training aspect. Google-Extended deserves a deliberate evaluation since it offers no direct traffic advantage but may influence your visibility in Google's AI-generated summaries.
The decision depends on your business model. For most SMEs, the principle is simple. If you want to be found by AI search engines, you should explicitly allow search crawlers. CCBot and Bytespider can be safely blocked because they offer no return channel to your website. Beyond that, there are dubious crawlers that ignore robots.txt or disguise themselves as legitimate programs. Scraping bots that copy your content without permission should also be actively blocked, for example through server-side firewalls or rate limiting.
Some AI providers operate multiple crawlers for different purposes. OpenAI has the specialized OAI-SearchBot alongside GPTBot, dedicated solely to real-time search. If you block GPTBot but allow OAI-SearchBot, you will still appear in ChatGPT search results without contributing training data.
It is important that you make this decision actively. Without explicit rules, you allow all crawlers access by default. Check your robots.txt at least once per quarter, as new AI crawlers appear regularly.
Why Do Websites Need to Be Built Differently for AI?
Of all the technical factors in AI SEO, page rendering has the greatest impact. If AI crawlers cannot read your content, all other optimizations are worthless.
What Happens When Your Website Is Only Assembled in the Browser
There are two fundamental ways websites deliver their content. In the first approach, known as Server-Side Rendering (SSR), the server prepares the complete page and sends finished text to the browser. In the second approach, Client-Side Rendering, the server delivers only a basic framework. The visitor's browser then loads the actual content via JavaScript and assembles the page.
For human visitors, both approaches work. For AI crawlers, they do not. Most AI programs receive only the basic framework and cannot execute the JavaScript code. If the framework is empty and the actual content needs to be loaded afterwards, the page remains blank for these crawlers. Your texts, headings, and FAQ sections simply do not exist for AI search engines.
A simple test reveals whether your website is affected. Open your page in the browser, disable JavaScript (in the developer tools or browser settings), and reload. What you see now is what AI crawlers see. A blank page means your content is invisible to ChatGPT, Perplexity, and similar systems.
Three Approaches for AI-Readable Websites
Server-Side Rendering (SSR) solves this problem fundamentally. The server prepares the complete page and delivers it ready-made to every visitor and every crawler.
Three approaches are suitable for AI-optimized websites.
| Approach | How it works | Well suited for |
|---|---|---|
| SSR (Server-Side Rendering) | Page is fully prepared on the server with each request | Dynamic content, personalized pages |
| SSG (Static Site Generation) | Page is generated once and delivered as a static file | Blog articles, landing pages, documentation |
| ISR (Incremental Static Regeneration) | Page is pre-generated and automatically updated when needed | Product pages, price lists, regularly updated content |
Approach / How it works / Well suited for
- Approach
- SSR (Server-Side Rendering)
- How it works
- Page is fully prepared on the server with each request
- Well suited for
- Dynamic content, personalized pages
- Approach
- SSG (Static Site Generation)
- How it works
- Page is generated once and delivered as a static file
- Well suited for
- Blog articles, landing pages, documentation
- Approach
- ISR (Incremental Static Regeneration)
- How it works
- Page is pre-generated and automatically updated when needed
- Well suited for
- Product pages, price lists, regularly updated content
All three approaches deliver complete pages that any crawler can read. Modern website frameworks like Next.js, Nuxt, or Astro support all three approaches out of the box. What matters is that you use one of them. While some crawlers like Googlebot can read purely client-rendered pages thanks to their built-in browser technology, most AI crawlers cannot. Websites without server-side rendering are therefore practically invisible for Answer Engine Optimization, the process of optimizing for AI answer engines.
For existing websites, there is an intermediate step called pre-rendering. The page is rendered in the background beforehand and cached as a finished document. This is not a long-term solution, but it bridges the gap until a complete rebuild.
From Evelan's Practice
For our own website evelan.de, we specifically improved AI visibility as part of our GEO optimization. The technical foundation was already in place, meaning server-side rendering, structured company data, and a clean site architecture. Our focus was therefore on the content. We added FAQ sections to blog articles and service pages, equipped all blog articles with verified source citations, and built up the blog overall as a citable knowledge resource.
Evelan content now appears increasingly in AI answers on topics like web design for SMEs, headless CMS, and SEO for service providers. The FAQ sections and source citations in blog articles in particular have measurably increased citation frequency.
How Do Structured Data Help AI Systems Understand Your Website?
Structured data help AI systems not just read your website's content but actually understand it. Think of it this way. When you read an article, you automatically recognize that a name is the author and a date is the publication date. Machines cannot do this on their own. Structured data are a standardized labeling system that gives your content this meaning.
According to the Web Data Commons study by the University of Mannheim, 51.25% of all web pages already contain structured data (University of Mannheim: WDC JSON-LD/Microdata/RDFa Data Corpus (2024)). For SMEs, these types are particularly relevant.
Organization
Defines your company clearly with name, address, logo, industry, and contact details. AI systems use this data to recognize your company as a distinct entity and attribute it correctly. Without Organization data, an AI might confuse your company with a similarly named provider or not even classify it as a relevant source. Especially for SMEs, this data creates the foundation for a clear digital identity.
FAQPage
Marks question-and-answer sections on your website. For AI search engines, these are particularly valuable because they provide direct answers to user questions. Google displays such content as so-called Rich Results. Rich Results are enhanced search results that take up more space and are presented more prominently than standard results, for example with expanded questions and answers directly in the search. According to a Milestone study, pages with such enhanced results receive 58% of all clicks, compared to 41% for standard results (Milestone Research via Search Engine Journal (2020)). A controlled test by seoClarity confirms the effect. After implementing FAQ data, the click-through rate rose from 1.02% to 2.22% (seoClarity: Testing Structured Data: Does FAQ Schema Increase CTR (2023)).
Article
Tags blog posts and expert articles with author, publication date, and topic. AI systems evaluate this information to assess the timeliness and credibility of a source. An article with a clearly attributed author and current date is more likely to be cited as a trustworthy source by AI systems than an undated text with no identifiable author. This makes a significant difference, particularly for expert articles and industry analyses.
LocalBusiness
Supplements company data with local information such as business hours, reviews, and service area. For SMEs with a regional focus, this data type is especially important. When someone asks ChatGPT or Perplexity about a service provider in a specific region, the AI needs exactly this local information to recommend your company. Without LocalBusiness data, the AI lacks the geographic context and falls back on other providers.
A common mistake is that many websites implement only one or two data types and leave out the rest. Yet the types reinforce each other. When an AI reads your company data along with your FAQ sections and article information, it builds a complete picture of your business. You become a trustworthy source rather than just another search result.
How Do You Implement This Technically?
The Right Format for Structured Data
There are various technical formats for structured data. By far the most widely used is called JSON-LD. JSON-LD is a machine-readable data format that is embedded in your website's source code like an invisible profile card. Visitors see nothing, but search engines and AI systems can read this information immediately.
According to W3Techs, 54.5% of all websites already use JSON-LD (W3Techs: Usage Statistics of Structured Data Formats (2026)). The HTTP Archive confirms the trend. The JSON-LD share rose from 34% in 2022 to 41% in 2024 (HTTP Archive: Web Almanac 2024, Structured Data (2024)).
The major advantage of JSON-LD is that the code sits as a standalone block in the background of your page, not scattered across the visible content. This makes it easier to maintain and well suited for automation by modern website systems. Google officially recommends JSON-LD as the preferred format. Case studies from Google's documentation demonstrate the effect. Rotten Tomatoes recorded 25% more clicks, and Nestle saw click-through rates increase by as much as 82% on pages with structured data (Google Search Central: Intro to Structured Data (2024)).
llms.txt as a Business Card for AI Systems
Alongside structured data, there is a new standard specifically for AI systems called the llms.txt file. The name stands for "Large Language Models," the large-scale language models like ChatGPT or Claude. This simple text file is placed in your website's root directory and works like a business card, summarizing in a few paragraphs who you are, what you offer, and which pages on your website are most important.
By June 2026, 8.7% of the world's top 1,000 most visited websites have set up such a file (Rankability: LLMS.txt Adoption (2026)). Adoption is growing but still limited. For SMEs, llms.txt is a useful addition, not a replacement for structured data.
The effort is minimal. You write a text file that summarizes your company in two to three paragraphs, lists your core services, and links to your most important pages. AI systems like Perplexity and Claude read this file when visiting your website and use it as a quick overview of your offerings. Once your fundamentals are in place, this small additional effort is worth it.
The Technical Checklist for AI-Optimized Websites
All previous points in a compact overview. Go through this list with your development team or your web design agency.
| What | Why it matters | Priority |
|---|---|---|
| Server-side rendering active | Without it, AI crawlers see only a blank page | Critical |
| AI crawlers allowed in robots.txt | Otherwise ChatGPT and others cannot read your content | Critical |
| Company data stored as JSON-LD | AI recognizes your company as a trustworthy source | Critical |
| HTTPS active | Basic requirement for security and trust | Critical |
| Article and FAQ data machine-readable | Increases the chance of being cited as a source | Important |
| Clean URLs and sitemap | Crawlers find and understand your site structure better | Important |
| Training crawlers selectively blocked | Protects your content from unwanted AI training use | Recommended |
| llms.txt file in root directory | Gives AI systems a quick overview of your offerings | Optional |
What / Why it matters / Priority
- What
- Server-side rendering active
- Why it matters
- Without it, AI crawlers see only a blank page
- Priority
- Critical
- What
- AI crawlers allowed in robots.txt
- Why it matters
- Otherwise ChatGPT and others cannot read your content
- Priority
- Critical
- What
- Company data stored as JSON-LD
- Why it matters
- AI recognizes your company as a trustworthy source
- Priority
- Critical
- What
- HTTPS active
- Why it matters
- Basic requirement for security and trust
- Priority
- Critical
- What
- Article and FAQ data machine-readable
- Why it matters
- Increases the chance of being cited as a source
- Priority
- Important
- What
- Clean URLs and sitemap
- Why it matters
- Crawlers find and understand your site structure better
- Priority
- Important
- What
- Training crawlers selectively blocked
- Why it matters
- Protects your content from unwanted AI training use
- Priority
- Recommended
- What
- llms.txt file in root directory
- Why it matters
- Gives AI systems a quick overview of your offerings
- Priority
- Optional
Not everything needs to be implemented immediately. The items marked "Critical" should come first. From over 60 SME projects at Evelan, we know that server-side rendering and correct robots.txt configuration alone deliver the greatest leverage for AI visibility.
When you go through the checklist with your GEO agency or your development team, I recommend a clear sequence. First ensure proper page rendering, then set up the robots.txt, then add structured data. Only when these three pillars are in place does the fine-tuning of llms.txt and advanced data types become worthwhile.
If you want to know how your current website performs technically for AI search engines, the article on measuring AI visibility provides concrete testing tools and methods. And for the strategic framework, which non-technical measures are worthwhile in parallel, read the overview of LLMO and GEO optimization.
Frequently Asked Questions
Not necessarily. If your existing website already uses server-side rendering, meaning the server delivers finished pages, targeted adjustments to robots.txt, structured data, and site structure are often sufficient. For websites that assemble their content only in the browser, a technical upgrade is necessary, however, since AI crawlers cannot read the content otherwise.
Related Evelan Articles
- GEO: How to Become Visible in AI Search
- Headless CMS vs. Traditional CMS: Which System Fits Your Business?
- Optimizing Core Web Vitals: A Guide to LCP, INP, CLS
- AI Content and Google Rankings: What the Data Really Shows
- SEO Fundamentals: How to Make Your Website Visible
Sources
- Ahrefs: The AI Bots That ~140 Million Websites Block the Most (2025)
- Cloudflare: From Googlebot to GPTBot: Who's Crawling Your Site in 2025 (2025)
- TechCrunch: Sam Altman Says ChatGPT Has Hit 800M Weekly Active Users (2025)
- University of Mannheim: WDC JSON-LD/Microdata/RDFa Data Corpus (2024)
- W3Techs: Usage Statistics of Structured Data Formats (2026)
- HTTP Archive: Web Almanac 2024, Structured Data (2024)
- Milestone Research via Search Engine Journal: Which Rich Results Get the Most Clicks (2020)
- seoClarity: Testing Structured Data: Does FAQ Schema Increase CTR (2023)
- Google Search Central: Intro to Structured Data (2024)
- Google Search Central: JavaScript SEO Basics (2024)
- Rankability: LLMS.txt Adoption: 8.7% of the Top 1,000 (2026)
