Whether a page can be stored in a search engine's index, controlled by meta robots tags, canonical tags, and robots.txt.
Indexability refers to whether a search engine can store and include a webpage in its index, making it eligible to appear in search results. Unlike crawlability (which determines if a bot can access a page), indexability controls whether that page gets added to the search engine's database after being crawled.
Search engines respect multiple signals that control indexability. A page might be perfectly crawlable but blocked from indexing through meta robots noindex tags, canonical tags pointing elsewhere, or robots.txt disallow directives. This distinction matters because crawling and indexing are separate processes—Google can crawl a page to understand its content and follow links without actually storing it in their index.
Why It Matters for AI SEO
AI-powered search experiences like Google's AI Overviews and ChatGPT plugins rely heavily on indexed content to generate responses. If your content isn't indexed, it won't contribute to AI-generated answers, regardless of its quality or relevance. This makes indexability control even more critical in the AI era. Modern AI systems also process indexed content differently than traditional search. They evaluate content for factual accuracy, citation worthiness, and semantic relationships. Pages with indexability issues may miss opportunities to be referenced in AI responses, reducing their overall search visibility and authority signals.
How It Works
Indexability is controlled through several mechanisms working together. The meta robots tag offers the most direct control— explicitly blocks indexing while still allowing crawling. Canonical tags redirect indexing signals to preferred URLs, effectively making non-canonical pages unindexable. The robots.txt file can prevent both crawling and indexing with disallow directives.
Tools like Screaming Frog and Sitebulb excel at indexability audits, identifying pages with conflicting signals or unintended blocks. Google Search Console's Coverage report shows which pages are indexed versus excluded, providing direct insight into Google's indexing decisions. For large sites, set up monitoring for indexability metrics—sudden drops in indexed pages often indicate technical issues that need immediate attention.
Common Mistakes
The biggest mistake is accidentally blocking important pages from indexing through forgotten noindex tags or overly broad robots.txt rules. Many sites also create indexability conflicts by using both canonical tags and noindex tags on the same page, sending mixed signals to search engines. Another common error is assuming that crawling equals indexing—just because Googlebot can access a page doesn't guarantee it will be indexed, especially with duplicate or thin content.