What max_age means in a scraping API, and why a stored answer costs 0 credits
Site endpoints store answers and share them. max_age sets how old a stored copy may be; every response says whether it was live or stored, and how old it is.
In short
- max_age is the maximum age, in seconds, of a stored copy you are willing to accept. Each endpoint has a sensible default; 0 forces a fresh fetch.
- A response served from the store carries source: "cache", its age in seconds and credits: 0. A live fetch carries source: "live" and costs the endpoint's credit.
- The store is shared across all users, so popular pages are fetched once and read many times. That is what keeps the service free.
Most scraping APIs treat every request as a fresh fetch, and price it that way. The structured endpoints here do something different: they remember what they fetched and let every caller decide how old an answer they will accept. The knob is a single query parameter, max_age, and the answer always says what it did.
The rule in one paragraph
Ask for /v1/structured/amazon/product?asin=B0B44XTV71. If a copy of that product exists in the store and is younger than your max_age, you get it immediately, for 0 credits, with source: "cache" and age in seconds. If not, the page is fetched, parsed, stored, and returned with source: "live", age: 0 and the endpoint’s credit charged. max_age defaults to the endpoint’s window and is capped at 30 days.
# accept anything under an hour old
curl ".../v1/structured/amazon/product?asin=B0B44XTV71&max_age=3600" -H "X-API-Key: wsa_…"
# always fetch fresh
curl ".../v1/structured/amazon/product?asin=B0B44XTV71&max_age=0" -H "X-API-Key: wsa_…"Reading the envelope
Three fields at the top level of every structured response tell you what happened.
| Field | Meaning |
|---|---|
source | "live" when the page was fetched for this request, "cache" when a stored copy was returned |
age | seconds since the data was fetched; 0 for live answers |
fetchedAt | the fetch time as an ISO timestamp, for your own bookkeeping |
credits | what this request cost: the endpoint’s credit for live, 0 for stored |
If your logic depends on freshness, branch on age, not on source; a live answer and a two-second-old stored answer are the same thing to almost every application.
Why the defaults differ
The window follows how fast the page changes and how expensive it is to fetch.
| Endpoint | Default window | Why |
|---|---|---|
| Hacker News front page | 10 minutes | the ranking moves constantly |
| YouTube video, GitHub repo | 1 hour | counts drift, but slowly |
| Amazon product, App Store, Google Play | 24 hours | prices and listings change daily at most; Amazon is also the hardest page to fetch |
| Wikipedia summary | 7 days | articles rarely change in ways a summary reflects |
The defaults are deliberately generous. Fresh fetches cost you credits and cost the service a page load; stored reads cost nothing on either side. If you need something newer, say so with max_age, request by request.
Why the store is shared
A cache per user would only save you money on your own repeats. A cache shared by everyone means the second person asking about the same product pays nothing and gets an answer in a few milliseconds instead of a few seconds. As the service grows the share of live fetches falls, which is exactly what a free service needs to stay free. Only parsed JSON is stored, never the raw page.
What is not cached
/v1/scrape and /v1/metadata fetch live every time. They return pages you shape with your own selectors, and raw pages are never kept, so there is nothing to share. Caching is a property of the structured endpoints, where the output is a known JSON shape.
Questions people ask
- What is the default max_age?
- It depends on the endpoint: Hacker News 10 minutes, YouTube and GitHub 1 hour, Amazon and the app stores 24 hours, Wikipedia 7 days. Each reference page states its window.
- Does max_age apply to /v1/scrape?
- No. The generic scrape endpoint always fetches live, because raw pages are never stored. Only the structured site endpoints have a store.
- Can I be sure I get a fresh page?
- Yes, with max_age=0. It always fetches, costs the endpoint's credit, and returns source: "live" with age: 0.
- Is the stored copy deleted after max_age?
- No. max_age is the reader's choice, not a deletion rule. Copies stay for up to 30 days so that callers who accept older data can still use them.
- Who pays when two people ask for the same product?
- The first one, once. The second gets the stored copy for 0 credits until it is older than their max_age.