← All products
POST /v1/crawl
Walk a whole site in one call
Give Refflux a starting URL and a budget; it follows the links, skips the duplicates, and returns one clean document per page - the fastest way to seed a knowledge base.
POSTapi.refflux.dev/v1/crawl
Request
{
"url": "https://docs.example.com",
"limit": 250,
"format": "markdown"
}Response
{
"id": "crawl_9f2a",
"status": "running",
"pages": 250,
"webhook": "https://you.dev/hooks/crawl"
}Follows links, reads sitemaps
Walk links to any depth, or lean on the sitemap when there is one.
One page, one document
Every page arrives as its own self-contained record, ready to chunk and embed.
Built for retrieval
Output drops straight into a vector store - no cleanup pass required.