Web Scraping

Read web pages that a plain fetch can't handle. All tools return clean markdown — no HTML parsing in your Skill.

What you can do

• Scrape one page — JavaScript-rendered SPAs, anti-bot protected pages, main-content extraction.
• Map a site — list every indexed URL on a domain, optionally filtered by keyword.
• Crawl a section — recursively scrape a docs site or site section, page by page.

Boundaries (read these first)

• Prefer the built-in web fetch for simple pages. Static pages, articles, and public docs work fine with the agent's free built-in fetch. Reach for these tools only when the page is JavaScript-rendered, blocks plain fetches, or you need bulk crawling.
• limit on a crawl is the billing upper bound. A crawl is metered by its page limit at submission — set it to what the task needs (default 10, max 50), don't max it out by habit.
• Long pages are truncated inline. A scrape returns up to ~50k characters; a completed crawl inlines the first 3 pages (~10k characters each) and lists the rest by URL. To read a listed page in full, scrape it individually — it's served from cache, fast, and still one page's cost.

Tools

firecrawl_start_crawl: limit is 1–50 (default 10) and doubles as the billing cap; include_paths / exclude_paths take patterns like ["/guide/*"].

Crawling is asynchronous, in two steps

Like video generation: submit the job (get a crawl_id), then poll firecrawl_get_crawl_status every few seconds until completed. A 10-page crawl typically finishes within a minute.

Using it in a Skill

• Research a docs site: firecrawl_map with a search keyword to find the relevant pages → firecrawl_scrape the two or three that matter.
• Bulk-read a section: firecrawl_start_crawl with include_paths: ["/guide/*"] and a right-sized limit → poll status → read the page index and inlined content → scrape individual pages for anything you need in full.
• A page the built-in fetch can't read: firecrawl_scrape with the URL — the headless browser renders JavaScript and extracts the main content.