Vibe Coder · Unit 17: RAG and Your Data · Lesson 4 of 6
Web data with Firecrawl
Much of what your product needs to know is on web pages. Firecrawl turns a page, or a whole site, into clean text a model can use.
Read the short notes and pictures.Do everything inside a blue frame.
Knowledge that lives on pages
Your own help centre, your documentation, a supplier’s price list, public regulations: a lot of what a helper needs is published on the web, not sitting in a folder. To use it you have to get the text out of the page, and that is harder than it sounds.
Why pages are hard
The HTML behind a page is mostly not content. There are menus, footers, scripts, tracking, and often the real text only appears after JavaScript runs. Hand that to a model and you pay for thousands of tokens of noise, and retrieval gets worse because menus and footers end up in your chunks.

What Firecrawl does
Firecrawl describes itself as a context API for AI agents. You give it an address and it returns the content as clean Markdown, HTML, structured JSON or a screenshot. It is open source and can be self-hosted, and there is a hosted service with SDKs for Python and Node, a command-line tool and an MCP server.

Scrape
One address, returned as Markdown, HTML or structured JSON.
Crawl
Start at one address and follow links through the site.
Map
List the pages of a site without fetching them all.
Search
Search the web and get the full content of the results.
Interact
Click and fill forms on a page to reach content behind them.
Parse
Read documents such as PDF, DOCX and XLSX.
Using it without code
Through MCP
Add the Firecrawl MCP server to Claude or to an agent, and ask it to read a page.
Through n8n
Call the Firecrawl API with an HTTP Request node and a saved credential.
Structured data
Ask for JSON with the fields you name: "price, plan name, billing period". Useful for tables you want in Airtable or Supabase.
On a schedule
A Schedule Trigger that re-scrapes your help centre every night keeps your index fresh.
Markdown: for RAG
# Refund policy Customers can ask for a refund within 14 days of purchase. ## How to request ...
JSON: for your tables
{
"plan": "Studio",
"price": 29,
"period": "month"
}Collect responsibly
Keep the address with each chunk
So the answer can link to the page.
Record when you fetched it
Web pages change. A date tells you how stale a chunk may be.
Re-fetch on a schedule
And replace, do not pile up, old versions.
You crawl your help centre and your helper starts quoting the cookie banner and the footer. What went wrong and how do you fix it?
Bring your web content in
Use Firecrawl to turn your own website or help pages into clean Markdown. Check a sample by eye, then add the pages to your source material with the address and date of each.
Done when
If your product depends on someone else’s pages, write down their terms on automated access before you index anything.
Keep these
- Raw pages are mostly noise; models need clean text.
- Firecrawl scrapes, crawls, maps and searches, returning Markdown or JSON.
- Collect only what you may, and keep the source address.