Is Your Site Agent-Ready? How to Pass Cloudflare's 23-Check AI Readiness Scanner at Level 5

The Wake-Up Call: Why “Modern” Websites Score 1/5 in the Agentic Era

For the past fifteen years, technical web audits followed a predictable script. You ran Google Lighthouse, verified Core Web Vitals, checked your XML sitemap, validated your Schema.org JSON-LD blocks, and confirmed your TLS headers. If your site rendered in under a second and passed mobile usability checks, you considered your infrastructure state-of-the-art.

Then Cloudflare launched Is It Agent Ready (isitagentready.com), and overnight the grading curve of the internet changed.

When you paste the URL of almost any Fortune 500 enterprise, luxury hospitality brand, or high-growth SaaS company into the isitagentready.com diagnostic scanner, the result is nearly always the same: Score: 1/5 (Basic Web Presence). Even sites with 100/100 Lighthouse performance scores fail 18 to 20 of the scanner’s 23 checks.

Why? Because for three decades we built the web for human eyes operating a graphical web browser. In 2026, a rapidly growing share of your highest-intent commercial traffic is no longer a human clicking through five navigation menus. It is an autonomous AI agent — operating inside ChatGPT, Claude, Perplexity, an enterprise procurement copilot, or a browser-native assistant — attempting to discover your capabilities, verify your identity, ingest your documentation, and execute a transaction or inquiry on behalf of a human decision-maker.

At our engineering studio Webxtek and across the x078 ecosystem, we recently audited and remediated dozens of production domains through our AI integrations, high-performance websites, and technical consulting practices across both Cloudflare Workers/Pages and dedicated HestiaCP Nginx/Apache infrastructure. Here is the complete architectural breakdown of what isitagentready.com actually tests, the subtle protocol traps we discovered in production, and how to engineer your site to achieve Level 5 (Agent-Native).

Understanding the 5 Maturity Levels of Agent Readiness

The scanner built on Cloudflare’s developer infrastructure does not just count keywords; it performs live HTTP protocol negotiation, DNS record inspection, cryptographic digest validation, and DOM/JavaScript static analysis across 23 distinct specifications. Sites are classified into five progressive maturity tiers:

To reach Level 5 (5/5 Agent-Native), you cannot treat AI optimization as a copywriting exercise. You have to re-architect how your edge network and origin server respond to non-human HTTP clients.

Pillar 1: Discoverability & Edge HTTP Link Headers (RFC 8288)

When an autonomous agent lands on https://yourdomain.com/, it should not have to guess twenty different URL paths to find your documentation or API schemas. Under the IETF RFC 8288 Web Linking standard, your root HTTP response must advertise its machine-readable capabilities directly in the HTTP Link response header:

Link: </.well-known/mcp/server-card.json>; rel="service-desc", </.well-known/agent-skills/index.json>; rel="describedby", </.well-known/api-catalog>; rel="api-catalog", </llms.txt>; rel="service-doc"; type="text/markdown", </sitemap-index.xml>; rel="sitemap"
Vary: Accept

Simultaneously, the HTML <head> of your root page (including language-detection entry points!) must include the Agentic Resource Discovery (ARD) link tag:

<link rel="ai-catalog" href="/.well-known/ai-catalog.json" />

The Edge Trap We Found in Production

On multi-language Astro architectures where / serves a lightweight language router before redirecting browsers to /en/ or /pt/, developers often add <link rel="ai-catalog"> to their main content layouts while forgetting the standalone src/pages/index.astro router! Because isitagentready.com inspects the exact root response at /, omitting that tag on the root entrypoint immediately fails the ARD HTML discovery check.

Furthermore, if your domain uses IETF datatracker DNS-AID (draft-ietf-dnsop-svcb-dns-aid) records (_index._agents, _a2a._agents, _mcp._agents), be aware that validating resolvers require DNSSEC (AD=true) with a DS record published in the parent TLD registry.

Pillar 2: Accept: text/markdown Content Negotiation

This is the single most impactful technical upgrade in modern Generative Engine Optimization (GEO).

A typical enterprise webpage weighs between 80 KB and 350 KB of raw HTML — packed with <svg> icons, Tailwind utility classes, navigation mega-menus, cookie banners, and client-side hydration payloads. Feeding raw HTML into an LLM wastes thousands of tokens on layout scaffolding.

By contrast, the llms.txt specification (llmstxt.org) introduced /llms.txt and /llms-full.txt as clean, structured Markdown maps of a website. However, isitagentready.com goes one step further: it sends an HTTP GET / request to your homepage with the header:

Accept: text/markdown, text/html;q=0.9

If your server still returns Content-Type: text/html, you fail the Content check.

To pass, your edge worker (on Cloudflare Workers/Pages) or origin web server (Apache .htaccess / Nginx) must inspect the Accept header. When text/markdown is requested, it must dynamically serve your canonical Markdown representation (/llms-full.txt or per-route Markdown) with three mandatory headers:

1. Content-Type: text/markdown; charset=utf-8
2. Vary: Accept (so CDN edge caches never serve Markdown to a human Chrome user or cached HTML to an AI agent)
3. x-markdown-tokens: <estimated_token_count> (allowing the agent runtime to budget its context window before reading the body stream)

On Cloudflare Workers Static Assets, static files alone cannot branch on the Accept header. We solved this cleanly by attaching a lightweight src/worker.mjs entrypoint with run_worker_first = true in wrangler.toml that intercepts Accept: text/markdown requests at the Cloudflare edge in under 0.4ms and streams /llms-full.txt with exact token telemetry.

Pillar 3: Bot Access Control, Content-Signal, and the Nginx Cache Trap

In /robots.txt, legacy SEO configurations only specify User-agent: * and Sitemap:. For Level 5 Agent Readiness, /robots.txt must explicitly govern AI crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Google-Extended, GPTBot, ClaudeBot) and declare machine-readable usage rights via the Content-Signal directive:

User-agent: *
Allow: /
Content-Signal: ai-train=yes, search=yes, ai-input=yes

Sitemap: https://x078.com/sitemap-index.xml
Agentmap: https://x078.com/.well-known/ai-catalog.json

The 10-Year .txt Cache Bug on HestiaCP / Nginx

While remediating client servers yesterday, we uncovered a subtle infrastructure trap that silently broke isitagentready.com rescans even after /robots.txt was updated on disk.

Default Nginx static asset location blocks frequently match \.(css|js|txt|ico|...)$ and apply expires max; (Cache-Control: max-age=315360000). When Cloudflare sits in front of that origin, Cloudflare caches /robots.txt and /llms.txt at the edge for 10 years! When you update /robots.txt to add Content-Signal, the scanner still receives the stale cached file (CF-Cache-Status: HIT).

The permanent fix is two-fold:

• Strip |txt| from long-lived immutable cache regexes and enforce Cache-Control: no-cache, no-store, must-revalidate on /robots.txt, /llms.txt, /llms-full.txt, and /auth.md.
• Publish a Web Bot Auth JWKS directory at /.well-known/http-message-signatures-directory so cryptographic bot agents can verify HTTP Message Signatures (RFC 9421).

Pillar 4: MCP, A2A, Agent Skills, OAuth Discovery & WebMCP

The largest category in the isitagentready.com benchmark (API, Auth, MCP & Skill Discovery) evaluates whether an autonomous agent can programmatically understand your tools, skills, and authentication boundaries without human intervention.

Even on a public corporate or service website where no login is required, Level 5 compliance requires a cohesive mesh of .well-known discovery manifests:

1. API Catalog (/.well-known/api-catalog — RFC 9727)

Served strictly with Content-Type: application/linkset+json, this Linkset manifest maps your primary agent information anchor (/api/agent-info) to its human/LLM documentation (service-doc $\rightarrow$ /llms-full.txt), its machine schema (service-desc $\rightarrow$ /.well-known/mcp/server-card.json), and its live health probe (status $\rightarrow$ /api/health.json).

2. MCP Server Card (/.well-known/mcp/server-card.json — SEP-1649) & A2A Agent Card (/.well-known/agent-card.json)

Built around the Model Context Protocol (modelcontextprotocol.io) ecosystem and the Agent-to-Agent (A2A) standard, these two files declare your organization’s identity, streamable-http / http-json transport endpoints, supported input/output MIME types (text/plain, text/markdown, application/json), and domain skills.

3. Agent Skills Discovery Index (/.well-known/agent-skills/index.json)

Following the Agent Skills Discovery RFC v0.2.0 (agentskills.io), your site publishes a structured SKILL.md file (for example, /.well-known/agent-skills/site-guide/SKILL.md) alongside an index.json manifest containing the exact sha256:<hex> cryptographic digest of that SKILL.md.

Critical Engineering Note: Always enforce LF (\n) line endings in .gitattributes for SKILL.md! If Git converts SKILL.md to CRLF on Windows before deployment, the byte length changes and the scanner’s SHA-256 integrity verification will fail!

4. auth.md & OAuth 2.0 / OIDC Discovery (RFC 9728 & RFC 8414)

Agents need a deterministic way to know how to authenticate — or to confirm that anonymous public access (read:public) is permitted. By publishing /auth.md alongside /.well-known/oauth-protected-resource, /.well-known/oauth-authorization-server, and OpenID Connect (openid.net) metadata (/.well-known/openid-configuration) with an agent_auth block, agents immediately discover your anonymous/bearer policies without triggering 401/403 guesswork. (Pro-tip: Never wrap URLs in Markdown backticks inside /auth.md — several regex-based agent parsers accidentally capture trailing %60 characters when probing endpoints!)

5. In-Browser WebMCP (document.modelContext.registerTool)

Finally, for agents running directly inside the user’s browser context (aligned with emerging W3C web agent standards (w3.org)), exposing an imperative WebMCP tool via document.modelContext.registerTool({ name: 'get_business_info', ... }) and annotating HTML <form> elements with toolname and tooldescription turns every page into an interactive tool surface.

Testing Your Own Domain Today

The shift from traditional SEO to Agent-Native Engineering is happening faster than the mobile shift of 2012. Search engines and autonomous procurement agents already prefer endpoints that negotiate clean Markdown, declare explicit Content-Signal policies, and publish verifiable .well-known capability manifests.

You can test https://x078.com — or your own company’s domain — right now on https://isitagentready.com/. And if your infrastructure is still stuck at Level 1 (Basic Web Presence), our engineering team has already automated the entire Level 5 (Agent-Native) stack for both static edge deployments and custom enterprise platforms.

In practice, we apply these exact architectural standards across specialized sectors including enterprise software implementation consulting and big data & cloud architecture consultancies. Whether you are modernizing core workflows through ai automation for real business workflows and high-performance websites, connecting sector-specific requirements to disciplined execution is what turns technical theory into compounding commercial advantage.