Replacing My Scraping Layer with Tavily Extract and Crawl

Tavily Extract and Crawl replaced the part of my ingestion pipeline that downloaded and parsed web pages by hand. Discovery still runs through Serper; everything after the URL list is now one API call instead of a small pile of fetch-and-parse code.

I finished Tavily's Web Search API certification the same week I finished that migration, which turns out to be a good order to do them in — the course mostly confirmed things I had just worked out the slow way.

What the ingestion pipeline looked like before

The pipeline belongs to an entity-validation project at TransData. I lead a team of two on it, and I had already done a full revamp before any of this — consolidating duplicated AI-assisted validation flows into a single Python/MySQL pipeline. The client and the product are under NDA, so what follows is about the shape of the ingestion stage, not about whose data moves through it.

That constraint is worth stating rather than working around. The interesting part of this change is structural anyway, and structure is the part I can describe.

Ingestion had two stages, and only the first one was interesting.

Discovery — Serper returns search results for a query, which gives me a list of URLs worth reading. That part worked and still works.

Retrieval — everything after the URL list was mine to own. Fetch each page, decide whether what came back was usable, pull the readable content out of it, handle the ones that were a redirect or a login wall or a PDF wearing an HTML content type, and back off politely per host so I did not get blocked.

None of those is hard. All of them together are a maintenance surface that has nothing to do with the actual product, and each one carries a dependency. The slowest and least interesting stage of the pipeline was also the one most likely to break on a page shape I had not seen yet.

What changed

Serper stayed. It is good at discovery and I had no reason to move it, which is worth saying out loud — "replace the whole stack with one vendor" is usually a worse decision than replacing one layer.

Tavily Extract took over retrieval: hand it URLs, get content back. Tavily Crawl covers the case where one URL should really become several, so I stopped writing my own link-following for that.

The part I am more confident about is the part that is not a number: the retrieval stage stopped being code I maintain. Deleting a dependency you were only carrying to parse HTML is a real win even when you cannot put a clean figure on it.

Metering a provider that bills in credits

The more interesting engineering came after the swap, and it is from a different codebase — an outreach tool of my own, not the NDA work. There, Tavily is one of several credit-metered providers sitting behind a registry — each with a priority, an enabled flag, and fields recording when it was last used, when it last errored, and when it ran out.

For that to mean anything, the system has to know how much of each provider is left. Most of them tell you in a response header. Tavily does not, so the quota check asks it directly:

// Tavily exposes no quota header, but /usage is authoritative --
// plan_limit/plan_usage rather than a guess.
const r = await fetch("https://api.tavily.com/usage", {
  headers: { Authorization: `Bearer ${key}` },
});
const { account } = await r.json();
return {
  remaining: account.plan_limit - account.plan_usage,
  allocated: account.plan_limit,
};

Two things I would tell anyone integrating it. The /usage call does not itself cost credits, so you can poll it without paying to find out what you have paid. And it returns the plan, so the check is authoritative rather than an estimate you maintain in your own config — when I verified it live on 5 August 2026 it reported the Researcher plan with a limit of 1000.

That distinction matters more than it sounds. A hardcoded quota is a number that goes stale silently the first time a plan changes.

What the certification actually covers

It is a course certificate signed by Tavily's founder, not a proctored exam, and it carries no credential ID. I would not present it as a qualification and I am not going to pretend it is one.

What it was genuinely good for is a map of the API surface. Tavily has four endpoints that are easy to conflate, and the useful lesson is knowing which one you are actually reaching for: Search when you need to find pages, Extract when you already have the URL, Crawl when one URL should become many, Map when you want the shape of a site without the contents. Before the course I was defaulting to Search in places where I already had the URL, which is a wasted call and a wasted credit.

You can take it at app.tavily.com/certification. It is free and short.

What is still rough

• I have not measured cost per record. I know the time went down and the dependency count went down. Credits per ingested record is the number that would actually tell me whether this was a good trade, and I have not run it.
• Exhaustion is recorded, not handled. The registry knows when a provider ran out. Falling over to the next one is still a decision I make by hand, which is exactly the kind of thing that is fine until it happens at 2am.
• Crawl depth is guesswork. I tuned it by looking at results, not by any principle. On a site with a different link structure I would be starting over.
• The timing number deserves a real benchmark. I would rather publish a boring measured figure than an impressive remembered one.

The rest of the work is on my projects page. If you are hiring for this shape of problem, the backend role page covers the pipeline and data side and the AI/backend page covers the model-facing side.