Tool
crawl4ai
unclecode/crawl4ai
What it is
A web crawler and scraper in Python that turns pages into Markdown or structured JSON for LLM pipelines. It runs as a library, a Docker server, or through the project's hosted API.
Who it might suit
Developers feeding web content to RAG systems or agents who may want either self-hosting or a hosted API.
What to check first
The hosted API sends your target URLs through the project's own cloud and works on credits, with paid plans. Its README labels the SERP endpoint beta and the answer endpoint experimental.
How it runs
Python library, Docker server or hosted API
Written