Insights
Web Scraping vs Official APIs: When Each Makes Sense
Oct
“Just use the API” is good advice when an API exists, lets you in, returns the fields you need and allows your use case. For competitor price data in US retail, those four conditions rarely line up at once. Here is how to decide, source by source.
Key takeaways
- Compare on five things: access, permitted use, coverage, price context and maintenance cost. “Free vs paid” is the least useful lens.
- For your own sales data, an official API is almost always right: Amazon SP-API, Shopify Admin, Walmart Marketplace.
- For competitors’ shelf prices, promos and stock, most retailers have no API at all, and the ones that do restrict who can use it and how.
- Amazon retired PA-API 5 in 2026. Old integrations now get
403 AccessDeniedException, and the replacement Creators API is built for affiliates. - Most production setups are hybrid: API for identifiers and structure, scraping for what shoppers actually see.
The five criteria that actually decide it
Debates about APIs and scraping usually stall on one dimension, like cost or reliability. In practice, five questions decide whether a source is usable at all.
| Criterion | Official API | Web scraping |
|---|---|---|
| Access: can you get in? | Approval, contracts, sales thresholds or a seller account are common | Anyone can load a public page; anti-bot systems decide how many times |
| Permitted use: may you use it this way? | Defined by the API terms, often limited to affiliate promotion or your own listings | Defined by law and site terms; see our US legal overview |
| Coverage: does it have the products and fields? | What the provider chooses to expose, sometimes capped per query | Whatever the page shows, including promos, badges and stock messages |
| Price context: is it the price a shopper sees? | Often a list or default price with no store, ZIP or member context | The shelf price for a specific location and session, if you set it deliberately |
| Maintenance: what breaks and how often? | Versioned, documented, deprecated with notice | Layouts and embedded data change without warning; needs monitoring and QA |
Notice that APIs win on maintenance and lose on context. That is the core trade-off, and it explains why the answer differs for your own data and for competitors’ data.
What changed in 2026: official retail APIs today
The API landscape for US e-commerce shifted this year, mostly because of Amazon. Here is where the main options stand.
| Platform | API | Who gets access | Useful for competitor prices? |
|---|---|---|---|
| Amazon | Creators API (replaced PA-API 5) | Approved Associates with recent qualifying sales1 | Limited: built for affiliate content; 10 items per search call |
| Amazon | Selling Partner API | Registered sellers and their developers | Partly: pricing and offer data for ASINs, throttled, tied to a seller account |
| eBay | Browse API | Sandbox for anyone; production requires approval and a contract | Yes, within limits: search returns at most 10,000 items per query |
| Walmart | Marketplace Item Search | Walmart Marketplace sellers | Narrow: up to 40 results per keyword or identifier search |
| Best Buy | Products, Stores APIs | Developer key on sign-up | Yes, with terms that limit caching to temporary use |
| Shopify stores | Admin API | The merchant, or apps they install | No: you only get tokens for stores you work with |
| Target, Home Depot, Costco, most DTC brands | No public product API | n/a | No |
Sources: each platform’s developer documentation, checked October 2026. Terms and quotas change; re-check before building.
Amazon: PA-API 5 is gone, Creators API is for creators
Amazon’s affiliate product API went through its biggest change in years. According to Amazon’s migration guide, the Creators API replaces AWS Signature Version 4 signing with OAuth 2.0: you exchange a credential ID and secret for a bearer token that lasts about an hour, and old PA-API keys cannot be reused. Applications still calling PA-API 5 now receive 403 Forbidden with an AccessDeniedException.
The bigger point is the audience. Amazon describes the Creators API as product data access “for publishers, influencers, and affiliate partners.” It is a good fit for a deals site that earns commissions. It is a poor fit for a brand that wants to track 5,000 competitor ASINs daily, both because of access requirements and because the use is not what the program is for.
eBay, Walmart and Best Buy
eBay’s Browse API is the most capable marketplace API for third parties, but production access is gated: the Browse API overview notes that production use requires eligibility checks, approval and a contract, and that a search returns at most 10,000 items. Walmart’s Marketplace Item Search is a seller tool that returns up to 40 matches per query, enough to look up an item, not to monitor a category. Best Buy is the friendly exception: its public Products API exposes regular and sale prices, online and in-store availability, and its terms allow only temporary caching.
When an official API is clearly the right call
- Your own data. Orders, inventory, ads and your listings belong in SP-API, Shopify Admin or Walmart’s seller APIs. Scraping your own Seller Central would be slower, riskier and less complete.
- Stable identifiers. If an API gives you GTINs, ASINs or SKUs you can join on, use it as the backbone even when prices come from elsewhere.
- Allowed and sufficient. If the API’s terms cover your use and its fields cover your question, take the API. It will be cheaper to maintain than any scraper.
When web scraping is the right call
- Competitor shelf prices and promotions. The thing you want to see is the thing on the page: sale badge, strikethrough price, bundle offer, “only 3 left.”
- Sites with no API. Target, Home Depot, Costco and nearly every brand store sell without a public product API. Shopify stores are a special case: many expose a public catalog feed, which we cover in Scraping Shopify stores via /products.json.
- Location-specific prices. APIs tend to return a default price. A scraper can set a store or ZIP and record exactly which one it used, which is the whole subject of zip-code pricing.
- Assortment questions. What a competitor lists, delists and runs out of over time. That is the core of assortment and stock tracking.
The hidden costs on both sides
APIs look free until you count approval time, quota management, terms reviews and migrations. The PA-API retirement is a real example: teams with a working integration had to rebuild authentication on Amazon’s schedule, and some lost access entirely because they didn’t meet the new eligibility rules.
Scraping looks cheap until you count maintenance. Retail sites change embedded data structures without notice, anti-bot systems adapt, and a scraper that “works” can silently return a default location’s price. The honest unit of cost is not per request but per usable record:
Cost per 1,000 valid records
(all collection costs ÷ number of records that passed validation) × 1,000. Include proxies, browser time, retries, storage and the engineering hours spent fixing parsers. A source that is cheap per request but fails validation 30% of the time is expensive.
Hybrid is the pattern that survives
The setups we see lasting longest mix both. A few common shapes:
- API for your catalog, scraping for theirs. Pull your own SKUs and prices from Shopify Admin or SP-API; collect competitor prices from their pages; match on GTIN.
- API for identifiers, scraping for context. Use an API or feed to enumerate products, then visit only the pages you need for location-specific price and stock.
- Scraping for breadth, API for verification. Where an API exists, use it to spot-check a sample of scraped prices and catch parser drift early.
If the decision is still unclear, start with the question rather than the tool: what decision will this data change, and how fresh does it need to be? Our price monitoring guide walks through that scoping step.
FAQ
Is using an API always more legal than scraping?
An API comes with explicit permission, which removes most uncertainty, but only for the uses its terms allow. Using affiliate API data for something the program forbids can breach the agreement. Scraping public pages without logging in has its own, different risk profile.
Can I still use Amazon PA-API 5?
No. Amazon has deprecated PA-API 5 and calls to it return 403 AccessDeniedException. Affiliates need to migrate to the Creators API, which uses OAuth 2.0 credentials and has its own eligibility requirements.
Does Walmart have a public product API?
Not a general one for third parties. Walmart’s developer APIs are aimed at marketplace sellers, and the Item Search endpoint returns up to 40 matches per query. Competitor price monitoring on Walmart is typically done from product pages.
Which is cheaper, an API or a scraper?
Compare cost per valid record, not cost per call. A free API that lacks the field you need costs more than a scraper; a scraper that needs weekly fixes can cost more than a paid API.
Notes
- Third-party integration docs describe the requirement as at least 10 qualified referred sales in the past 30 days, with access suspended if the account falls below it. Confirm the current rule in Associates Central before planning around it. ↩
Not sure which sources your project needs?
Send us the competitors and fields you track. We’ll tell you which have usable APIs, which need scraping, and what a pilot on a few hundred SKUs would show.
