AI Readiness

Check whether AI crawlers and answer engines can reach, understand, and efficiently read your site.

AI Readiness checks the technical signals that help AI crawlers and answer engines reach and understand your public site. It is separate from your Visibility Index, which measures whether your brand appears in generated answers.

Overview

A readiness scan requests your public homepage, robots.txt, and discovered sitemap locations. It validates the returned bodies before reporting whether core crawl, content, crawler-policy, and machine-discovery checks passed.

Some capabilities and evidence are useful but not required. Stride labels these Optional and leaves them out of the readiness score. This includes Markdown content negotiation, crawler policy coverage evidence, and conditional technical groundwork such as API catalogs, machine-readable Link headers, agent login instructions, OAuth, MCP, and skill discovery. Coverage explains whether each retrieval crawler is governed by an explicit group, the wildcard group, or implicit allowance; none is inherently better when access is the same.

How to use it

Stride starts the first readiness scan while you finish onboarding connections.
Open Dashboard → AI Readiness. If no result exists yet, choose Run readiness scan; otherwise choose Refresh scan when a new check is due.
Review the readiness score and area breakdown.
Open fixable gaps first. These affect the scored crawl and content baseline and can be prioritized in Actions.
Review open signals separately. These are optional upgrades, not crawl blockers.
After publishing a change, refresh the scan to check the live response.

Each result applies to the URL named in its evidence. A homepage result does not prove that every page on the site behaves the same way.

robots.txt validation

A successful HTTP status is not enough. Stride checks that robots.txt is a readable text policy rather than an HTML error page or binary response, parses recognized directives, and flags malformed or orphaned rules for review. An empty but readable file is valid and means crawlers are implicitly allowed.

Unknown extension directives are not treated as failures. Stride evaluates the standard access rules it understands and keeps syntax concerns visible in the evidence.

Sitemap discovery and structure

Stride checks absolute sitemap URLs declared in robots.txt, followed by the standard /sitemap.xml and /sitemap_index.xml locations. A sitemap passes only when it contains a valid non-empty <urlset> with absolute HTTP(S) loc values, or a valid <sitemapindex> whose sampled child sitemap exposes a usable URL set.

A 200 OK response containing HTML, an empty XML root, invalid loc values, or an unusable sitemap-index child does not pass.

Crawler policy coverage

For each answer-engine retrieval user agent, Stride reports the public robots.txt policy source:

Explicit — a matching crawler-specific User-agent group
Wildcard — the User-agent: * group
Implicit — no matching group or blocking rule, so access is allowed by default
Mixed — one engine's retrieval user agents use different policy sources

A valid wildcard policy is not penalized, and crawler-specific groups are optional unless different bots need different rules. Training-only bots remain a separate content-policy decision from retrieval crawlers used for search and citations.

The public scan cannot inspect configuration inside a CDN, WAF, or bot-management dashboard. Edge enforcement is therefore reported as not externally verified, even when the public robots.txt policy allows retrieval.

Answer bots and training bots

Stride shows AI crawlers in two groups, on this page and in the Crawl & indexability detail under Site health:

Answer bots fetch or index pages so an AI product can answer someone right now. Examples: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User. These decide whether AI can read your site when it answers.
Training bots collect pages to train models. Examples: GPTBot, ClaudeBot, Google-Extended, CCBot, and Bytespider. Blocking these is a legitimate choice: it keeps your pages out of model training and doesn't stop answer bots.

The Crawl & indexability score counts answer bots only, so blocking training bots doesn't lower it. Before October 2026 it counted every bot equally. That older score isn't charted next to the new one, and it updates at your next full scan.

robots.txt can't see firewall or CDN blocks

robots.txt is a request to bots, not enforcement. A bot it allows can still be turned away by a firewall or CDN rule, such as Cloudflare bot management, and the public scan can't see those rules. If you've connected AI crawler logs, Analytics → Crawler logs shows which bots actually reached your pages. That's the cross-check for edge blocks.

Markdown content negotiation

Stride sends a second homepage request containing Accept: text/markdown. The check passes only when the response:

Returns a successful status with Content-Type: text/markdown
Contains a usable Markdown body rather than an HTML document with a Markdown label
Includes Accept in the Vary response header so shared caches keep HTML and Markdown responses separate
Does not declare a response Content-Signal that conflicts with the global content-use preferences in robots.txt

Cloudflare-specific token headers may appear in the evidence when Cloudflare Markdown for Agents is active. They are supporting evidence, not a requirement; other hosting platforms can implement the same HTTP negotiation correctly.

Conditional technical groundwork

Technical groundwork is capability detection, not a universal website checklist. Stride probes known public locations but hides a missing capability when there is no evidence that it applies. If a capability is detected, its valid or malformed state remains visible as an Optional, non-scoring result.

API catalog

An API catalog applies only when a site intentionally publishes one or more public APIs. Marketing sites, publications, portfolios, and storefronts without a public API do not need one.

Stride requests /.well-known/api-catalog using the RFC 9727 location. A valid catalog must return Linkset JSON with Content-Type: application/linkset+json and contain at least one item link to a public HTTP(S) API. A 200 OK HTML page, arbitrary JSON object, empty Linkset, wrong media type, or catalog containing no usable API link does not pass. Never add private, internal, staging, or tenant-specific endpoints merely to satisfy this diagnostic.

Link headers

An HTTP Link response header can advertise an existing API catalog, API description, documentation resource, or alternate machine-readable representation. Stride recognizes standard relations such as api-catalog, service-desc, service-doc, describedby, and an alternate link with an appropriate media type. Operational links such as preload, canonical, and stylesheet do not make this capability applicable.

When a relevant header exists, Stride safely fetches up to four unique advertised targets. A checked target passes only when it is publicly reachable, has a non-empty body, and the response format matches the declared media type. JSON representations must contain valid structured data, and XML representations must expose an XML root. An advertised API catalog must also pass the RFC 9727 catalog validation.

No Link header is needed when the site has no such resource. Missing support is hidden and non-scoring; a detected broken target remains visible as an optional warning. Add the header through the application, origin server, or CDN that already controls the response. Cloudflare Transform Rules are one optional implementation route, not a requirement. A Link header complements rather than replaces sitemaps, structured data, merchant feeds, or normal HTML links.

/auth.md is an emerging explanatory convention for services where agents can authenticate or act on behalf of users. Content-only sites and services without agent login do not need it. When present, Stride checks for readable plain text or Markdown with authentication guidance and a real auth target.

auth.md does not replace OAuth protected-resource, authorization-server, or OpenID metadata. Those structured documents remain the machine-readable source of truth; auth.md explains how to use the real flow.

Commerce protocol checks

Stride checks five emerging commerce protocols only when public evidence makes a protocol applicable. Each result is Optional and non-scoring. Missing support stays hidden: an ordinary storefront, content site, or conventional card checkout does not need all five protocols.

Check	When it applies	What the public scan can verify
ACP	A merchant intentionally participates in OpenAI's Agentic Commerce Protocol	A public ACP claim can be flagged, but official merchant enrollment, private feed delivery, and checkout conformance cannot be proved from the public site. Official ACP does not define `/.well-known/acp.json`.
AP2	An agent checkout flow uses cryptographic mandates for human-approved or autonomous actions	The `dev.ucp.common.payment.ap2_mandate` capability and supporting UCP checkout, payment-handler, presentation-format, and key metadata. Mandate signatures and receipts require separate conformance tests.
MPP	An API, content item, or service is intentionally sold to machine clients	OpenAPI `x-payment-info` offers and discovered HTTP `402` responses using `WWW-Authenticate: Payment`. Stride does not authorize or settle a payment.
UCP	A business exposes agent-facing commerce or checkout services	The official `/.well-known/ucp` business profile: version, services, payment handlers, capabilities, and public references. UCP is broader than paywall handling.
x402	An API or digital resource is intentionally sold per request	A discovered safe `GET` route's HTTP `402` response and x402 v2 `PAYMENT-REQUIRED` challenge. There is no universal site-local x402 well-known endpoint, and Stride never submits a payment signature.

A protocol pass is deliberately narrow: it confirms only the public structure or challenge named in the evidence. It does not prove merchant approval, product feed acceptance, order creation, mandate verification, payment authorization, or settlement.

Definitions

Term	Meaning
Readiness score	The 0–100 result from required crawl, content, policy, and discovery checks
Fixable gap	A scored check that failed or needs review
Optional	A useful capability that does not change the readiness score
Content negotiation	Returning a different representation of the same URL based on the request's `Accept` header
`text/markdown`	The response media type used for a Markdown representation
`Vary: Accept`	A cache instruction that keeps responses for different accepted formats separate
Content Signal	A declared preference for search, AI input, AI training, or other supported content uses
Policy source	The explicit, wildcard, implicit, or mixed `robots.txt` rule source applied to a retrieval crawler
Edge controls not externally verified	The public response was tested, but private CDN/WAF configuration was not inspected
Unverified	Stride could not confirm the capability from the public response
Technical groundwork	Optional capability discovery for public APIs, HTTP Link headers, agent authentication, OAuth, MCP, skills, and commerce protocols

Best practices

Fix crawl blocks, unreachable pages, and missing core content before optional delivery formats.
Keep robots.txt as readable plain text; do not route missing files to a branded HTML page with a 200 status.
Declare the canonical sitemap with an absolute URL and make sure sitemap-index children expose non-empty URL sets.
Use one wildcard crawler policy when it matches your intent. Add crawler-specific groups only for deliberate exceptions.
Verify CDN/WAF crawler controls in the provider dashboard as a separate check.
Keep the browser-facing HTML response as the default. Return Markdown only when the requester asks for it.
Validate the Markdown body itself. A text/markdown header attached to HTML is not a working implementation.
Always include Accept in Vary when HTML and Markdown share a URL.
Review response-level Content Signals after enabling a CDN converter. Provider defaults may not match the policy declared in robots.txt.
Publish /.well-known/api-catalog only for real public APIs, using RFC 9727 Linkset JSON and application/linkset+json.
Add machine-discovery Link headers only for public resources that already exist and use the relation and media type that match the target.
Publish /auth.md only for a supported agent authentication flow. Keep structured OAuth metadata authoritative.

Plan and limits

AI Readiness evaluates public responses and does not require access to your CDN or hosting account. It cannot inspect private CDN/WAF rules or prove which edge rule handled a request, and it cannot enable a provider setting on your behalf.

Availability and pricing for Markdown conversion depend on your hosting or CDN provider. Cloudflare documents Markdown for Agents for supported paid plans; other platforms may require edge or application changes. A missing Markdown representation does not reduce your Stride readiness score.

FAQ / Troubleshooting

Why does a wildcard crawler policy pass?

User-agent: * applies to retrieval crawlers when no more specific group matches. Repeating the same allowance in crawler-specific groups adds no access, so Stride reports wildcard coverage as valid rather than treating duplication as a readiness requirement.

Why are CDN/WAF controls not externally verified?

Those rules live in your provider account and may depend on managed-bot classification, account settings, or request context. Stride reports the public HTTP and robots.txt behavior it can observe. Confirm private edge policy in the provider dashboard.

Why did the scan report `Homepage returned 403`?

The site or its CDN/WAF refused Stride's server-side homepage request. This is a real public-access observation, not a dashboard rendering error. Independently reachable resources such as robots.txt and the sitemap can still produce results, but Stride withholds homepage-dependent content checks rather than claiming that blocked HTML has missing metadata or headings. When this happens, the overall readiness score and level remain unavailable instead of being calculated from the smaller set of reachable endpoints. The independent results remain visible, and a failed attempt uses a brief five-minute safety cooldown rather than the plan's normal refresh cadence.

Review bot-management and firewall rules for the public homepage, then refresh the readiness scan. A browser loading successfully does not prove that a server-side crawler receives the same response.

Why does Markdown negotiation say Optional?

AI crawlers can still read accessible, meaningful HTML. Markdown can reduce parsing and token overhead, but it is not a prerequisite for crawling or a proven visibility ranking factor.

Why is API catalog missing from my results?

Stride hides absent conditional capabilities instead of calling them failures. If your site does not publish a public API, there is nothing to fix. If an API catalog URL responds but returns invalid content, Stride keeps it visible as an optional warning so an intended implementation can be corrected.

Does every ecommerce site need an API catalog or auth.md?

No. A public storefront is not automatically a public API, and checkout does not automatically imply an agent authentication flow. Add these capabilities only when the business intentionally supports them and can maintain the published contract.

Why does Stride say the response still looks like HTML?

The server returned Content-Type: text/markdown, but the body contained an HTML document. Check the CDN conversion rule and test the body returned specifically for Accept: text/markdown.

Why does the check mention `Vary: Accept`?

Without it, a shared cache can store one representation and serve it to the wrong requester. Configure the response so Vary includes Accept, then refresh the readiness scan.

Why is there a Content Signal conflict?

The negotiated response and the global robots.txt declaration disagree about at least one content use. Review both policies before choosing which value is intentional. Some CDN services apply permissive defaults when the origin does not provide a response-level policy.

Can Stride turn this on for me?

No. Stride observes the public response but does not control your hosting or CDN account. Use the provider guidance shown in the checkpoint, deploy the change, and refresh the scan.