Octocrawl

Web scraping for AI agents: scrape, map, crawl and batch public pages into Markdown, tables and JSON, each with an Evidence Record (final URL, HTTP status,

#2672 by GitHub starsDocumentation Content creationChecked 2026-11-02

What it connects

Web scraping for AI agents: scrape, map, crawl and batch public pages into Markdown, tables and JSON, each with an Evidence Record (final URL, HTTP status, robots.txt decision, content hashes); a blocked or empty page comes back with the reason. Hosted at https://mcp.octocrawl.dev/mcp or local via npx -y @octocrawl/mcp. Setup.

Registry listing — description sourced from the publisher's entry in an awesome-list. Setup, compatibility and requirements are not verified; see the repository.

Source: https://github.com/77777R7/Octocrawl · ★ 3 on the source repo

Setup instructions and compatibility are compiled from publisher docs and community reports, not hands-on testing, unless a step is marked Tested. Check the source before relying on a setup.