Every enrichment problem reduces to the same question: where does this fact come from, and what happens when that source changes. I have shipped all three answers into production and they fail in different ways.
Integrate first, when you can
An API is the cheapest thing to own. Versioned, documented, and somebody else's problem when it breaks. If a vendor has one and the data is good enough, that is the end of the discussion.
The catch hides in the phrase good enough. On one pipeline the richest source of advisor data had no public API at all. On another, three vendors each had an API and each had different coverage on the same field, so the integration was not one call. It was a waterfall with a cost order.
The waterfall is the actual design
Cheapest deterministic source first, then the next, then the paid one, then the model only if the record still matters.
On a property pipeline I run, structured gates decide whether a record is worth anything at all before any vendor is called. Property data, phone validation, and email discovery then run in ascending cost order, and the model scores last. That ordering is why data and model spend lands at about two cents a record instead of the dollar or more it would otherwise cost. Nothing calls a paid endpoint speculatively.
On an advisor pipeline, regulator sources come first, then site-restricted search, then an LLM web lookup only after the cheaper deterministic paths miss. Each tier writes its own provenance column, so months later I can still tell which tier produced a given value. That column has settled more arguments than any dashboard.
Scrape when the data is real and the door is closed
Sometimes the best source has no API and no intention of building one. I wrapped one of those in a self-authenticating service: it logs in, holds short-lived headers, retries once on a 401, and presents a clean interface to the rest of the workflow.
Scraping is not the hard part. Owning it is. Cookies expire, markup shifts, rate limits appear without notice. If you scrape, budget for the operational tail: health checks, a force-refresh path, and alerting that does not fire forty times for one outage.
Ask a human only where judgment lives
The third option is the one teams reach for too early and then never remove. A human in the loop is correct when the decision genuinely requires judgment. It is a tax everywhere else.
The test I use: if two competent people would give the same answer from the same inputs, it should not be a human step. Encode it. If they would reasonably disagree, keep the person and hand them the evidence rather than a recommendation.
The decision, compressed
If an API exists and covers the field, integrate. If several APIs partially cover it, waterfall them in cost order and record which tier answered. If the only good source sits behind a login, scrape it and budget for maintenance. If the answer requires judgment, ask a person. Put the model last in all four cases.
The order matters more than the tools. Most cost overruns I have watched came from calling the expensive thing first and discovering afterwards that a filter would have removed the record anyway.