AI Readiness

Check whether AI crawlers and answer engines can reach, understand, and efficiently read your site.

AI Readiness checks the technical signals that help AI crawlers and answer engines reach and understand your public site. It is separate from your Visibility Index, which measures whether your brand appears in generated answers.

Overview

A readiness scan requests your public homepage, robots.txt, and discovered sitemap locations. It validates the returned bodies before reporting whether core crawl, content, crawler-policy, and machine-discovery checks passed.

Some capabilities and evidence are useful but not required. Stride labels these Optional and leaves them out of the readiness score. This includes Markdown content negotiation and crawler policy coverage evidence. Coverage explains whether each retrieval crawler is governed by an explicit group, the wildcard group, or implicit allowance; none is inherently better when access is the same.

How to use it

Open Dashboard → AI Readiness.
Choose Run readiness scan, or Refresh scan if a result already exists.
Review the readiness score and area breakdown.
Open fixable gaps first. These affect the scored crawl and content baseline and can be prioritized in Actions.
Review open signals separately. These are optional upgrades, not crawl blockers.
After publishing a change, refresh the scan to check the live response.

Each result applies to the URL named in its evidence. A homepage result does not prove that every page on the site behaves the same way.

robots.txt validation

A successful HTTP status is not enough. Stride checks that robots.txt is a readable text policy rather than an HTML error page or binary response, parses recognized directives, and flags malformed or orphaned rules for review. An empty but readable file is valid and means crawlers are implicitly allowed.

Unknown extension directives are not treated as failures. Stride evaluates the standard access rules it understands and keeps syntax concerns visible in the evidence.

Sitemap discovery and structure

Stride checks absolute sitemap URLs declared in robots.txt, followed by the standard /sitemap.xml and /sitemap_index.xml locations. A sitemap passes only when it contains a valid non-empty <urlset> with absolute HTTP(S) loc values, or a valid <sitemapindex> whose sampled child sitemap exposes a usable URL set.

A 200 OK response containing HTML, an empty XML root, invalid loc values, or an unusable sitemap-index child does not pass.

Crawler policy coverage

For each answer-engine retrieval user agent, Stride reports the public robots.txt policy source:

Explicit — a matching crawler-specific User-agent group
Wildcard — the User-agent: * group
Implicit — no matching group or blocking rule, so access is allowed by default
Mixed — one engine's retrieval user agents use different policy sources

A valid wildcard policy is not penalized, and crawler-specific groups are optional unless different bots need different rules. Training-only bots remain a separate content-policy decision from retrieval crawlers used for search and citations.

The public scan cannot inspect configuration inside a CDN, WAF, or bot-management dashboard. Edge enforcement is therefore reported as not externally verified, even when the public robots.txt policy allows retrieval.

Markdown content negotiation

Stride sends a second homepage request containing Accept: text/markdown. The check passes only when the response:

Returns a successful status with Content-Type: text/markdown
Contains a usable Markdown body rather than an HTML document with a Markdown label
Includes Accept in the Vary response header so shared caches keep HTML and Markdown responses separate
Does not declare a response Content-Signal that conflicts with the global content-use preferences in robots.txt

Cloudflare-specific token headers may appear in the evidence when Cloudflare Markdown for Agents is active. They are supporting evidence, not a requirement; other hosting platforms can implement the same HTTP negotiation correctly.

Definitions

Term	Meaning
Readiness score	The 0–100 result from required crawl, content, policy, and discovery checks
Fixable gap	A scored check that failed or needs review
Optional	A useful capability that does not change the readiness score
Content negotiation	Returning a different representation of the same URL based on the request's `Accept` header
`text/markdown`	The response media type used for a Markdown representation
`Vary: Accept`	A cache instruction that keeps responses for different accepted formats separate
Content Signal	A declared preference for search, AI input, AI training, or other supported content uses
Policy source	The explicit, wildcard, implicit, or mixed `robots.txt` rule source applied to a retrieval crawler
Edge controls not externally verified	The public response was tested, but private CDN/WAF configuration was not inspected
Unverified	Stride could not confirm the capability from the public response

Best practices

Fix crawl blocks, unreachable pages, and missing core content before optional delivery formats.
Keep robots.txt as readable plain text; do not route missing files to a branded HTML page with a 200 status.
Declare the canonical sitemap with an absolute URL and make sure sitemap-index children expose non-empty URL sets.
Use one wildcard crawler policy when it matches your intent. Add crawler-specific groups only for deliberate exceptions.
Verify CDN/WAF crawler controls in the provider dashboard as a separate check.
Keep the browser-facing HTML response as the default. Return Markdown only when the requester asks for it.
Validate the Markdown body itself. A text/markdown header attached to HTML is not a working implementation.
Always include Accept in Vary when HTML and Markdown share a URL.
Review response-level Content Signals after enabling a CDN converter. Provider defaults may not match the policy declared in robots.txt.

Plan and limits

AI Readiness evaluates public responses and does not require access to your CDN or hosting account. It cannot inspect private CDN/WAF rules or prove which edge rule handled a request, and it cannot enable a provider setting on your behalf.

Availability and pricing for Markdown conversion depend on your hosting or CDN provider. Cloudflare documents Markdown for Agents for supported paid plans; other platforms may require edge or application changes. A missing Markdown representation does not reduce your Stride readiness score.

FAQ / Troubleshooting

Why does a wildcard crawler policy pass?

User-agent: * applies to retrieval crawlers when no more specific group matches. Repeating the same allowance in crawler-specific groups adds no access, so Stride reports wildcard coverage as valid rather than treating duplication as a readiness requirement.

Why are CDN/WAF controls not externally verified?

Those rules live in your provider account and may depend on managed-bot classification, account settings, or request context. Stride reports the public HTTP and robots.txt behavior it can observe. Confirm private edge policy in the provider dashboard.

Why does Markdown negotiation say Optional?

AI crawlers can still read accessible, meaningful HTML. Markdown can reduce parsing and token overhead, but it is not a prerequisite for crawling or a proven visibility ranking factor.

Why does Stride say the response still looks like HTML?

The server returned Content-Type: text/markdown, but the body contained an HTML document. Check the CDN conversion rule and test the body returned specifically for Accept: text/markdown.

Why does the check mention `Vary: Accept`?

Without it, a shared cache can store one representation and serve it to the wrong requester. Configure the response so Vary includes Accept, then refresh the readiness scan.

Why is there a Content Signal conflict?

The negotiated response and the global robots.txt declaration disagree about at least one content use. Review both policies before choosing which value is intentional. Some CDN services apply permissive defaults when the origin does not provide a response-level policy.

Can Stride turn this on for me?

No. Stride observes the public response but does not control your hosting or CDN account. Use the provider guidance shown in the checkpoint, deploy the change, and refresh the scan.