Whether you want it or not, some of your users are going to ask ChatGPT or Claude for instructions on how to use your product.

They’ll ask how to configure a feature, connect an integration, or fix an error. And they may follow the answer without ever opening your documentation.

So how do you help the AI get it right?

Accurate documentation is the starting point. But the assistant still needs to find the right content, retrieve it, and preserve the details that matter. A missing prerequisite or overlooked warning can change the outcome.

This article offers practical tips for helping AI assistants find and use your documentation so they can give users accurate, complete instructions.

How an agent finds and uses your documentation

Imagine a developer asks:

“How do I configure webhooks to receive order updates, and what should my application do if the same event arrives twice?”

A useful answer needs more than a webhook setup page. It also needs the authentication requirements, event format, delivery behavior, and guidance on duplicate events.

Depending on the assistant and its available tools, the process may involve several stages:

1. Identify the information needed - The agent breaks the question into topics, such as setup, signature verification, and duplicate handling.
2. Find candidate sources - It may use web search, a documentation index, links from another page, or a connected documentation search tool.
3. Retrieve relevant content - A fetch tool requests selected pages, or a search service returns matching passages.
4. Extract the useful information - The system may convert HTML into text or Markdown, select passages, or ask another model to extract an answer.
5. Compose the response - The main agent combines the retrieved information with the user’s question and decides whether it needs additional sources.

These stages do not always use the same model. Claude Code’s WebFetch, for example, can process a retrieved page with a small, fast model and return that model’s answer to the main conversation. The main agent may therefore receive an extraction rather than the complete article. Other tools use different approaches. Source: Claude Code tools reference

This matters because information can be lost between retrieval and the final answer. If the extraction includes the setup steps but misses a warning about duplicate delivery, the resulting integration may work during testing and fail under real conditions.

What can prevent an agent from getting the answer?

A page that works in a browser does not necessarily work through an agent’s fetch tool.

Page size and retrieval limits - Tools can limit how much content they download, process, or return. Claude Code truncates large pages before processing, and Anthropic’s API web fetch tool supports a content token limit. A critical instruction near the end of a long response may never reach the model. Source: Anthropic web fetch documentation

Bot detection and browser challenges - Your server or CDN may block automated requests or return a verification page. The agent then receives a challenge instead of the article.

Rate limiting - One question can require several pages. Requests from shared infrastructure can also arrive in bursts, making a legitimate documentation lookup look like excessive traffic.

Content that requires browser interaction - Instructions inside dynamically loaded tabs, screenshots, or interactive examples may be absent from a basic fetch response.

Broken links and access restrictions - An outdated URL, a login page, or a redirect can interrupt retrieval. Redirects across hosts can introduce additional requests and permission checks, depending on the tool.

Main tips for helping AI agents find the answer

1. Make clean Markdown available

HTML serves the documentation interface, including navigation, styling, and interactive elements. Agents need the instructions within that interface. Some fetch tools convert HTML into Markdown themselves, but conversion can introduce noise or omit content. Providing a clean Markdown representation reduces that dependency. Claude Code’s WebFetch sends an Accept header that prefers Markdown. Cloudflare’s Markdown for Agents supports this form of content negotiation: a client requests text/markdown, and the server returns a Markdown representation of the page. Source: Cloudflare Markdown for Agents

You can also expose dedicated Markdown URLs, such as an article URL ending in .md. Make the alternative discoverable through links or response headers, rather than relying on an agent to guess its location. Generate both versions from the same source. Two separately maintained articles will eventually disagree.

Review the Markdown output as a documentation deliverable. It should preserve headings, code blocks, parameter definitions, warnings, and links. Check tabbed examples carefully: if a page offers Python and JavaScript examples, the exported content should identify each one clearly.

The useful test is whether the Markdown contains everything needed to complete the task correctly. A smaller payload helps only if it retains the necessary information.

2. Give agents a useful documentation index

Markdown helps an agent read a page. An index helps it choose which page to read. Jeremy Howard proposed llms.txt as a Markdown entry point containing a site overview and links to useful content. It can live at the site root or under a documentation path, such as /docs/llms.txt. The proposal also describes ways to advertise the index and Markdown alternatives through links and HTTP headers. Source: llms.txt proposal

Treat it as a guide to your documentation. Organize links by subject and give each page a description that explains the question it answers.

For the webhook example, “Webhook documentation” provides little help. A description such as “Configure order event subscriptions, verify signatures, and handle duplicate deliveries” gives the agent a reason to select that page.

Keep descriptions and links current as part of publishing. Also provide a route back to the relevant index from individual articles, so an agent arriving through search can discover related material.

Support varies between tools. Publishing llms.txt does not guarantee that every assistant will request it, and it does not replace search visibility, sitemaps, or other discovery mechanisms.

What if the documentation site is large?

An index can become too large to be useful. Thousands of page descriptions consume context before the agent has retrieved any instructions.

Use a compact entry point that directs agents to smaller indexes by product, task, or version. Mintlify, for example, supports generated split indexes for large documentation sets and provides llms-full.txt as a separate file containing the full documentation. Source: Mintlify llms.txt documentation

For a large site, we recommend letting the agent narrow its scope before retrieving detailed content. In the webhook example, it should be able to select the correct API version and then find the relevant setup and delivery pages.

A full documentation export can help with indexing or offline processing. It can also overwhelm a tool that only needs a few sections. Offer focused retrieval paths alongside any full export and make version boundaries explicit so an agent does not combine current setup instructions with an obsolete event schema.

3. Use MCP when a connected retrieval service helps

An llms.txt file is a published navigation resource. An MCP server is a connection through which an assistant can use tools or access resources.

A documentation MCP server might offer search and page retrieval. The assistant can ask for material about duplicate webhook events and receive relevant results without browsing the documentation hierarchy.

The distinction is not simply public versus private. MCP servers can expose public documentation, and authorization is optional in the MCP specification. Protected servers can use authorization to provide access to restricted information. Source: MCP authorization specification

• For public documentation - Markdown and an index provide an accessible starting point. MCP can add more focused search and retrieval for assistants configured to connect to it.
• For private documentation - use a retrieval service that enforces the user’s permissions. An authenticated MCP server is one option. Keep any private index behind authentication too, and ensure public discovery files include only public content.

Whatever the retrieval method, the answer is only as reliable as the documentation it returns.

4. Write sections that survive extraction

Our article 5 tips for technical writers writing for AI agents covers the writing principles: make topics self-contained, put essential information first, use clear structure, remove redundancy, and maintain consistent terminology.

These principles become particularly useful when an assistant receives only part of a page.

Consider this instruction:

“Enable verification before continuing.”

On its own, it leaves several questions unanswered. What verification? Where is it enabled? What happens if the user skips it?

A more useful instruction would be:

“Verify each webhook’s signature before processing its payload. Reject the request if signature verification fails.”

The second version carries its purpose and condition with it. Keep warnings close to the steps they qualify, identify applicable versions, and state prerequisites where the agent needs them.

Token efficiency comes from reducing unnecessary interpretation and follow-up retrieval. Removing a prerequisite to save a few words can make the answer more expensive to produce and less reliable.

5. Measure agent retrieval separately

An automated fetch often does not execute your analytics JavaScript. It may never appear in GA4, and browser measures such as scroll depth and heatmaps do not describe that interaction.

Use server or CDN logs to understand requests to your documentation and Markdown endpoints. Record requested paths, response status, response size, latency, and available client identifiers.

Separate user-directed retrieval from other automated activity where possible. Anthropic, for example, distinguishes Claude-User requests from its search and training crawlers. User-agent strings are useful classification signals, although they do not provide perfect identification. Source: Anthropic crawler guidance

This can help you investigate:

• Frequently retrieved articles - Prioritize their accuracy, examples, and version information.
• Changes after releases - Check whether new documentation receives requests and whether those requests succeed.
• Pages with little observed agent traffic - Review discovery, descriptions, and access before assuming the topic has little value.
• Retrieval failures - Look for missing pages, blocked requests, rate limits, and unexpectedly large responses.

Logs have limits. A cached page can support several answers without generating a fresh request, and a fetch does not prove that an assistant used the content correctly.

Pair traffic analysis with a small set of real customer questions. Ask assistants to answer them using your documentation, then check the sources, prerequisites, version selection, and resulting instructions. For the webhook question, verify that the answer includes both signature verification and duplicate handling.

Make the answer the test. Start with a question customers regularly ask. Follow it through discovery, retrieval, and the response the assistant produces. If the answer misses a critical detail, determine where that detail was lost. Was the page undiscoverable? Did a server challenge block it? Was the response truncated? Did the instruction depend on context from another section?

This investigation gives you a concrete improvement to make and a question to test again after the next release.

At Writec, we help technology companies structure and maintain documentation that supports both human readers and AI-assisted workflows. Get in touch to discuss your documentation.