AI Folks
  • Courses
  • Web Of AI
  • Pricing
  • Enterprise
Log in
PreviousFile search: hosted RAGNextVectors in Supabase

Vibe Coder · Unit 17: RAG and Your Data · Lesson 4 of 6

Web data with Firecrawl

Much of what your product needs to know is on web pages. Firecrawl turns a page, or a whole site, into clean text a model can use.

Read the short notes and pictures.Do everything inside a blue frame.

Knowledge that lives on pages

Your own help centre, your documentation, a supplier’s price list, public regulations: a lot of what a helper needs is published on the web, not sitting in a folder. To use it you have to get the text out of the page, and that is harder than it sounds.

Why pages are hard

The HTML behind a page is mostly not content. There are menus, footers, scripts, tracking, and often the real text only appears after JavaScript runs. Hand that to a model and you pay for thousands of tokens of noise, and retrieval gets worse because menus and footers end up in your chunks.

Warm waves, hand-drawn. Three boxes: a web address, Firecrawl fetches and cleans, and Markdown or JSON ready for a model.
An address in, clean text out.

What Firecrawl does

Firecrawl describes itself as a context API for AI agents. You give it an address and it returns the content as clean Markdown, HTML, structured JSON or a screenshot. It is open source and can be self-hosted, and there is a hosted service with SDKs for Python and Node, a command-line tool and an MCP server.

Warm waves, hand-drawn. Four cards: scrape one page, crawl a whole site, map a site’s pages, search for pages.
Four ways to get content.

Scrape

One address, returned as Markdown, HTML or structured JSON.

Crawl

Start at one address and follow links through the site.

Map

List the pages of a site without fetching them all.

Search

Search the web and get the full content of the results.

Interact

Click and fill forms on a page to reach content behind them.

Parse

Read documents such as PDF, DOCX and XLSX.

Using it without code

  1. 1

    Through MCP

    Add the Firecrawl MCP server to Claude or to an agent, and ask it to read a page.

  2. 2

    Through n8n

    Call the Firecrawl API with an HTTP Request node and a saved credential.

  3. 3

    Structured data

    Ask for JSON with the fields you name: "price, plan name, billing period". Useful for tables you want in Airtable or Supabase.

  4. 4

    On a schedule

    A Schedule Trigger that re-scrapes your help centre every night keeps your index fresh.

Markdown: for RAG

# Refund policy

Customers can ask for a refund
within 14 days of purchase.

## How to request
...

JSON: for your tables

{
  "plan": "Studio",
  "price": 29,
  "period": "month"
}
Markdown keeps headings, which makes good chunk boundaries. JSON gives you rows.

Collect responsibly

Being able to fetch a page does not mean you may use it

Start with your own site and documentation. For anyone else’s, read their terms and respect their rules for automated access. Do not collect personal data. Do not republish other people’s content as your own. Keep the address of every page you index so that answers can cite it.

  1. 1

    Keep the address with each chunk

    So the answer can link to the page.

  2. 2

    Record when you fetched it

    Web pages change. A date tells you how stale a chunk may be.

  3. 3

    Re-fetch on a schedule

    And replace, do not pile up, old versions.

ActivityYour turnWrite it yourself first, then compare.

You crawl your help centre and your helper starts quoting the cookie banner and the footer. What went wrong and how do you fix it?

ActivityBuild stepA helper that answers from your documents

Bring your web content in

Use Firecrawl to turn your own website or help pages into clean Markdown. Check a sample by eye, then add the pages to your source material with the address and date of each.

Done when

If your product depends on someone else’s pages, write down their terms on automated access before you index anything.

Keep these

  • Raw pages are mostly noise; models need clean text.
  • Firecrawl scrapes, crawls, maps and searches, returning Markdown or JSON.
  • Collect only what you may, and keep the source address.
Previous lessonFile search: hosted RAGNext lessonVectors in Supabase