Turn a documentation site into markdown for a RAG index
Crawl a docs site or a section of it and get each page as clean markdown, ready to chunk and embed for a RAG chatbot or search. Pay per page.
A support bot or coding assistant that answers from your documentation is only as good as the text you feed its index. Copying pages by hand misses the new ones, and raw HTML fills the index with menus and scripts. Lovely Router crawls the site and returns each page as clean markdown. Send a start URL, get one JSON object per page, pay per page from a prepaid balance. Call it from a nightly job or an AI agent with one API key.
Hand it to your AI agent
Copy this prompt into Claude Code, Cursor, Codex or any coding agent. It tells the agent where the docs are and how to use your API key.
Turn a documentation site into markdown for a RAG index. Use the Lovely Router data tools: each returns JSON and is billed per result.
- Docs (read first): https://lovelyrouter.com/docs/tools/find.md and https://lovelyrouter.com/docs/tools/run.md
- Plan first: POST https://lovelyrouter.com/v1/tools/suggest with `{"task": "<my task in plain words>"}` (free). It returns the tools to run in order, each with an example `input`, the `input_required` fields, an estimated cost and a ready-made `run` request. Replace the example values with my task's; if a field is unclear, read the tool with GET https://lovelyrouter.com/v1/tools/<id>. To browse instead, POST https://lovelyrouter.com/v1/tools/discover searches the catalog.
- Show me the plan and its estimated cost (`total_est_cost_usd`) before running anything large.
- Run: POST https://lovelyrouter.com/v1/tools/<id>/run with `{"input": {...}, "max_results": N}`. A 200 has the finished run with `items`; a 202 means it is still running, so poll the `poll` path until `status` is `succeeded`, `failed`, `timed_out` or `stopped`.
- Only delivered results are billed. Keep max_results small on the first run, check the output, then scale up.
- If you support MCP, connect https://lovelyrouter.com/mcp instead (Streamable HTTP, same key; https://lovelyrouter.com/docs/tools/mcp.md) and use suggest_tools, get_tool and run_tool.
- Auth: read my API key from the LOVELY_API_KEY environment variable and send it as `Authorization: Bearer $LOVELY_API_KEY`. Never print or commit the key. If it is not set, ask me to create one at https://lovelyrouter.com/app/keys.
- Billing is prepaid: every call is charged to my balance, so test with a small request before running anything large.
- Every docs page is plain Markdown; the full index is https://lovelyrouter.com/llms.txt.
My task: <the inputs (links, keywords or accounts) and how many results you need>How it works
Pick the start URL
Use the docs home or the section you need, such as a guide folder. By default the crawler also covers the sub-pages of that URL.
Run the tool with a cap
Send the input to
POST /v1/tools/website-crawl/runwithmax_results. Set the page limit withmaxCrawlPages. The run holds its maximum cost from your balance and returns each page as markdown.Chunk and embed
Split the markdown by heading, embed the chunks, and replace the old pages in your index when you re-run.
Crawl a docs section with one request
Crawl a website to markdown takes startUrls. maxCrawlPages caps the pages crawled, including start pages and pagination pages, and maxCrawlDepth limits how many links deep it goes:
curl https://lovelyrouter.com/v1/tools/website-crawl/run \
-H "Authorization: Bearer $LOVELY_API_KEY" \
-H "content-type: application/json" \
-d '{
"input": {
"startUrls": [
{
"url": "https://docs.example.com/guides"
}
],
"maxCrawlPages": 3,
"maxCrawlDepth": 1
},
"max_results": 3
}'Use includeUrlGlobs and excludeUrlGlobs to widen or narrow the crawl, and useSitemaps to enqueue URLs from the site's sitemap. Keep max_results equal to your page cap. A run still going after 60 seconds returns 202; Run a tool shows how to poll.
What each page holds
| Field group | What you get |
|---|---|
| Content | The page as clean, LLM-ready markdown |
| Source | The page URL, within the start URLs and your patterns |
| Scope | Only pages that match your start URLs, depth and glob patterns |
Start with a small maxCrawlPages to check the output before you crawl the full site.
Recipes
Crawl your product docs, chunk the markdown by heading, embed it, and let the bot answer with links to the source page.
Re-run the crawl with the same limits and replace the pages that changed.
Start at the API reference folder and add an exclude pattern for the changelog.
Crawl the docs of a library you use and let the agent search them instead of guessing.
Enable useSitemaps or add sitemap.xml as a start URL to reach pages no link points to.
Prefer plain words? Connect the MCP server and tell Claude Code or Cursor: "Crawl https://docs.example.com/guides to 50 pages as markdown and write each page to docs/. Set max_results to 50."
Pair it with other web tools
A crawl covers a site you pick. Other tools cover single pages and what a page looks like:
Crawl a whole website (or a section of it) and return each page as clean, LLM-ready markdown. Good for RAG and documentation sites.
Full-page screenshot (PNG or PDF) of web pages; returns a link to the image.
Google results pages for a query: organic results (position, title, URL, snippet, date, sitelinks) and related searches. One result = one results page (~10 links).
Full transcripts (captions) of YouTube videos, with timestamps: one per video link, in the language you ask for when the video has it.
website-screenshotcaptures how a page looks, which markdown does not show.google-searchfinds the sites to crawl.youtube-transcriptadds video tutorials as text.
For wider options, see extract structured data from any website and research agents.
What you pay
You pay per page: starting a run holds its maximum cost, the per-call fee plus the per-page fee times max_results, and the run settles to what it returned. The rest of the hold goes back to your balance. A run that fails or times out pays only for the pages it delivered, and a run refused at the start costs nothing.
Crawling 200 pages is billed as 200 pages, so a page cap keeps a large site from running up a bill. There is no subscription and no monthly minimum: top up from $5, and one balance pays for tools and models. The card above shows today's price; pricing lists every tool.
Frequently asked questions
How do I keep the crawl to one section?+
Start at the section URL; sub-pages of the start URL are crawled by default. Add includeUrlGlobs or excludeUrlGlobs to adjust.
How many pages can one run crawl?+
Up to 200 with maxCrawlPages. The default is 10.
Can it use a sitemap?+
Yes, with useSitemaps, or by adding a sitemap.xml link as a start URL.
Does it handle JavaScript-based docs sites?+
The tool renders pages in a browser, so pages that build their content with JavaScript are read as they appear.
Can I crawl a site I do not own?+
The tool reads public pages. Respect the site's terms and robots rules when you decide what to crawl.
Do I need a subscription?+
No. You add credit to a prepaid balance and each run draws it down. Failed runs pay only for the results they delivered, and nothing renews automatically.
Reading single pages instead? See extract structured data from any website, and the rest of the tool catalog.