# webscrapingapi.dev — full text for language models

> webscrapingapi.dev is a free web scraping API for developers and AI agents. Send a URL and get back HTML, plain text, Markdown, JSON extracted with CSS selectors, or page metadata. Site-specific endpoints return clean JSON for sites the service already knows. When a site cannot be scraped with a plain request, users post it on the board and the maintainer builds an endpoint for it or explains why it cannot be done. Everything is free: 50,000 credits per account per day, no card.

This file is the long form of https://webscrapingapi.dev/llms.txt. Machine-readable API description: https://webscrapingapi.dev/openapi.json. Human documentation: https://webscrapingapi.dev/docs. Built and run by Jaeyoung (https://webscrapingapi.dev/about).

# Facts and limits

- Base URL: https://api.webscrapingapi.dev
- Authentication: API key in the `X-API-Key` header (also `Authorization: Bearer` or `?api_key=`). Keys start with `wsa_` and are created in the dashboard at https://app.webscrapingapi.dev.
- Free quota: 50,000 credits per account per day, resets 00:00 UTC. A plain fetch costs 1 credit, a JavaScript render 5, metadata and site endpoints 1. Answers served from the shared store cost 0.
- Rate limit: 60 requests per minute per key; up to 5 keys per account.
- Limits: 30 s max timeout, 5 MB max body, 50 extract fields. Private and reserved network addresses are refused.
- Logging: hostname, status, timing, credits and key per request, kept 30 days. Raw pages are never stored.
- Refunds: RENDER_UNAVAILABLE, INTERNAL_ERROR, SITE_BLOCKED and SITE_PARSE_FAILED refund the credit. FETCH_FAILED and TIMEOUT are charged.
- No CAPTCHA solving and no anti-bot bypass. Sites that cannot be reached reliably are marked "Not possible" on the board.

# Generic endpoints

## GET or POST https://api.webscrapingapi.dev/v1/scrape

Parameters: url (required); format = html | text | markdown | json (default html; markdown keeps headings, lists, tables, code blocks and absolute links; json requires extract); render (boolean, runs headless Chrome, 5 credits); selector (return only this element); extract (object of field → CSS selector or { selector, attr, all, html }); wait (ms after load, render only); waitFor (CSS selector to wait for, render only); timeout (ms, max 30000); headers (POST body only: user-agent, accept, accept-language, cookie, referer); raw (return the body alone).

Response: { url, finalUrl, statusCode, contentType, title, rendered, html | text | markdown, data, truncated, elapsedMs, credits }. The target's own status code is in statusCode; a 404 from the target is a successful fetch.

## GET https://api.webscrapingapi.dev/v1/metadata?url=

Reads the first 512 KB of the page and returns { url, finalUrl, title, description, canonical, favicon, lang, og, twitter, jsonLd, elapsedMs, credits }. og and twitter are the tags with their prefix removed; jsonLd is every application/ld+json block parsed.

## GET https://api.webscrapingapi.dev/v1/structured

Public registry of site endpoints (JSON). No key needed.

# Site endpoints

## Amazon product (amazon.com · 20 marketplaces)

One product by ASIN as JSON: title, price, availability, rating, images, bullet points and the spec table.
Note: Public product pages only. Prices are shown in the requested currency (default USD); delivery-dependent stock messages may differ from what a signed-in shopper sees. Not affiliated with Amazon.

### GET https://api.webscrapingapi.dev/v1/structured/amazon/product

The product detail page as JSON.

Parameters:
- asin (string, required): The 10-character Amazon product id. Example: B0B44XTV71
- tld (string): Marketplace: com, co.uk, de, fr, it, es, ca, co.jp, in, com.au, com.mx, com.br, nl, ae, sa, se, pl. Default com. Example: com
- currency (string): ISO 4217 code for prices, e.g. USD, EUR, JPY. Default USD. Example: USD
- max_age (number): Accept a stored copy up to this many seconds old (credits: 0). Defaults to 86400 seconds; 0 forces a fresh fetch.

Returns data shaped like: { product: { name, brand, price, regularPrice, currency, availability, aggregateRating, mainImage, images[], features[], additionalProperties[], breadcrumbs[], sku }, extra: { asin, tld, bestSellersRank[], soldBy, shipsFrom, boughtInPastMonth, availabilityText } }

Stored for 86400 seconds by default and shared; a stored answer costs 0 credits.

Example response:

```json
{
  "site": "amazon",
  "action": "product",
  "data": {
    "product": {
      "name": "CUCKOO Twin Pressure Rice Cooker, 12-Cup, White",
      "brand": { "name": "CUCKOO", "url": null },
      "price": "339.99",
      "currency": "USD",
      "availability": "InStock",
      "aggregateRating": { "ratingValue": "4.4", "reviewCount": 608 },
      "images": [ "https://m.media-amazon.com/images/I/…jpg" ],
      "features": [ "…", "…" ],
      "sku": "B0B44XTV71"
    },
    "extra": { "asin": "B0B44XTV71", "tld": "com" }
  },
  "source": "cache", "age": 412, "credits": 0
}
```

## YouTube video (youtube.com)

A video by id as JSON: title, channel, views, likes, duration, publish date, description, thumbnails and keywords.


### GET https://api.webscrapingapi.dev/v1/structured/youtube/video

The watch page metadata as JSON.

Parameters:
- id (string, required): The 11-character video id (the v= parameter). Example: dQw4w9WgXcQ
- max_age (number): Accept a stored copy up to this many seconds old (credits: 0). Defaults to 3600 seconds; 0 forces a fresh fetch.

Returns data shaped like: { video: { id, url, title, description, channel { id, name, url }, durationSeconds, viewCount, likeCount, publishedAt, category, thumbnails[], keywords[], isLive, familySafe } }

Stored for 3600 seconds by default and shared; a stored answer costs 0 credits.

Example response:

```json
{
  "site": "youtube",
  "action": "video",
  "data": {
    "video": {
      "id": "dQw4w9WgXcQ",
      "title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
      "channel": { "id": "UCuAXFkgsw1L7xaCfnd5JJOw", "name": "Rick Astley", "url": "http://www.youtube.com/@RickAstleyYT" },
      "durationSeconds": 213,
      "viewCount": 1818228078,
      "likeCount": 19384777,
      "publishedAt": "2009-10-24T23:57:33-07:00",
      "category": "Music",
      "thumbnails": [ { "url": "https://i.ytimg.com/vi/dQw4w9WgXcQ/maxresdefault.jpg", "width": 1280, "height": 720 } ],
      "keywords": [ "rick astley", "Never Gonna Give You Up", … ]
    }
  },
  "source": "live", "age": 0, "credits": 1
}
```

## App Store app (apps.apple.com)

An iOS app by id as JSON: name, developer, rating, price, category, screenshots and description.


### GET https://api.webscrapingapi.dev/v1/structured/appstore/app

The listing page as JSON.

Parameters:
- id (string, required): The numeric app id (the id… part of the URL). Example: 284882215
- country (string): Storefront country code. Default us. Example: us
- max_age (number): Accept a stored copy up to this many seconds old (credits: 0). Defaults to 86400 seconds; 0 forces a fresh fetch.

Returns data shaped like: { app: { name, url, platform: "ios", id, developer { name, url }, description, category, price, currency, free, rating { value, count }, icon, screenshots[], contentRating, osRequirement, installs: null, updatedAt, version } }

Stored for 86400 seconds by default and shared; a stored answer costs 0 credits.

Example response:

```json
{
  "site": "appstore",
  "action": "app",
  "data": {
    "app": {
      "name": "Facebook",
      "platform": "ios",
      "id": "284882215",
      "developer": { "name": "Meta Platforms, Inc.", "url": "https://apps.apple.com/us/developer/…" },
      "category": "Social Networking",
      "price": "0", "currency": "USD", "free": true,
      "rating": { "value": 4.5, "count": 28420068 },
      "icon": "https://is1-ssl.mzstatic.com/image/thumb/…png",
      "osRequirement": "Requires iOS 15.1 or later."
    }
  },
  "source": "live", "age": 0, "credits": 1
}
```

## Google Play app (play.google.com)

An Android app by package name as JSON: name, developer, rating, installs, price, category, content rating and description.


### GET https://api.webscrapingapi.dev/v1/structured/googleplay/app

The listing page as JSON.

Parameters:
- id (string, required): The package name (the id= parameter). Example: com.spotify.music
- hl (string): Language, e.g. en, ko. Default en. Example: en
- gl (string): Country, e.g. US, KR. Default US. Example: US
- max_age (number): Accept a stored copy up to this many seconds old (credits: 0). Defaults to 86400 seconds; 0 forces a fresh fetch.

Returns data shaped like: { app: { name, url, platform: "android", id, developer { name, url }, description, category, price, currency, free, rating { value, count }, icon, screenshots[], contentRating, osRequirement: null, installs, updatedAt, version } }

Stored for 86400 seconds by default and shared; a stored answer costs 0 credits.

Example response:

```json
{
  "site": "googleplay",
  "action": "app",
  "data": {
    "app": {
      "name": "Spotify: Music and Podcasts",
      "platform": "android",
      "id": "com.spotify.music",
      "developer": { "name": "Spotify AB", "url": "https://www.spotify.com" },
      "category": "MUSIC_AND_AUDIO",
      "price": "0", "currency": "USD", "free": true,
      "rating": { "value": 4.35, "count": 36339801 },
      "installs": "1B+",
      "contentRating": "Teen",
      "updatedAt": "Sep 18, 2026"
    }
  },
  "source": "live", "age": 0, "credits": 1
}
```

## Hacker News (news.ycombinator.com)

Front page stories and item threads as clean JSON — no HTML parsing on your side.


### GET https://api.webscrapingapi.dev/v1/structured/hackernews/front

Stories on the front page, in rank order.

Parameters:
- page (number): Page number (1-based). Example: 1
- max_age (number): Accept a stored copy up to this many seconds old (credits: 0). Defaults to 600 seconds; 0 forces a fresh fetch.

Returns data shaped like: stories[] { id, rank, title, url, site, points, author, commentCount, age }

Stored for 600 seconds by default and shared; a stored answer costs 0 credits.

Example response:

```json
{
  "site": "hackernews",
  "action": "front",
  "data": {
    "count": 30,
    "stories": [
      {
        "rank": 1,
        "title": "Samsung is expected to more than double …",
        "points": 125,
        "commentCount": 84
      },
      // 29 more
    ]
  },
  "source": "live", "age": 0, "credits": 1
}
```

### GET https://api.webscrapingapi.dev/v1/structured/hackernews/item

A story with its comment thread (flattened, with depth).

Parameters:
- id (number, required): Hacker News item id. Example: 1
- max_age (number): Accept a stored copy up to this many seconds old (credits: 0). Defaults to 600 seconds; 0 forces a fresh fetch.

Returns data shaped like: { id, title, url, points, author, age, comments[] { id, author, age, depth, text } }

Stored for 600 seconds by default and shared; a stored answer costs 0 credits.

Example response:

```json
{
  "site": "hackernews",
  "action": "item",
  "data": {
    "id": 1,
    "title": "Y Combinator",
    "commentsParsed": 3,
    "comments": [ { "id": 15, "author": "sama", "depth": 0, "text": "…" } ]
  },
  "source": "live", "age": 0, "credits": 1
}
```

## Wikipedia (wikipedia.org)

Article title, summary paragraphs, thumbnail and infobox as JSON, in any language edition.


### GET https://api.webscrapingapi.dev/v1/structured/wikipedia/summary

Lead section of an article plus infobox key/values.

Parameters:
- title (string, required): Article title (spaces or underscores). Example: Web scraping
- lang (string): Language edition (default en). Example: en
- max_age (number): Accept a stored copy up to this many seconds old (credits: 0). Defaults to 604800 seconds; 0 forces a fresh fetch.

Returns data shaped like: { title, url, lang, summary, paragraphs[], thumbnail, infobox {} }

Stored for 604800 seconds by default and shared; a stored answer costs 0 credits.

Example response:

```json
{
  "site": "wikipedia",
  "action": "summary",
  "data": {
    "title": "Web scraping",
    "lang": "en",
    "summary": "Web scraping, web harvesting, or web data extraction is data scraping …",
    "paragraphs": [ "…", "…", "…", "…", "…" ],
    "infobox": {}
  },
  "source": "cache", "age": 3120, "credits": 0
}
```

## GitHub (github.com)

Repository facts (stars, forks, topics, license, description) scraped from the public page — no token, no API rate limit.
Note: Counts come from the page, so they are as fresh as GitHub renders them. Private repositories are not reachable. The primary language is no longer in the server-rendered page, so `language` is usually null.

### GET https://api.webscrapingapi.dev/v1/structured/github/repo

Public repository overview.

Parameters:
- owner (string, required): Owner login. Example: withastro
- repo (string, required): Repository name. Example: astro
- max_age (number): Accept a stored copy up to this many seconds old (credits: 0). Defaults to 3600 seconds; 0 forces a fresh fetch.

Returns data shaped like: { owner, repo, url, description, homepage, stars, forks, watchers, topics[], license, defaultBranch, archived, fork, releaseCount, language (null on the current GitHub layout) }

Stored for 3600 seconds by default and shared; a stored answer costs 0 credits.

Example response:

```json
{
  "site": "github",
  "action": "repo",
  "data": {
    "owner": "withastro",
    "repo": "astro",
    "description": "The web framework for content-driven websites.",
    "stars": 62720,
    "forks": 3798,
    "watchers": 221,
    "topics": [ "astro", "blog", "islands", … ],
    "license": "Other",
    "defaultBranch": "main",
    "releaseCount": 3251,
    "homepage": "https://astro.build"
  },
  "source": "live", "age": 0, "credits": 1
}
```


# Errors

Errors are JSON: `{ "error": { "code": "...", "message": "..." } }`. Codes and status: BAD_REQUEST 400, URL_BLOCKED 400, UNAUTHORIZED 401, FORBIDDEN 403, NOT_FOUND 404, RATE_LIMITED 429, QUOTA_EXCEEDED 429, FETCH_FAILED 502, SITE_PARSE_FAILED 502, SITE_BLOCKED 503, RENDER_UNAVAILABLE 503, TIMEOUT 504, INTERNAL_ERROR 500.

# Articles

## Convert any web page to Markdown for an LLM with one request (format=markdown)

URL: https://webscrapingapi.dev/blog/convert-any-web-page-to-markdown-for-llm
Published: 2026-09-22 · Author: Jaeyoung · Tags: generic scrape, rag & agents

Add format=markdown to /v1/scrape: headings, lists, tables, code and absolute links come back, scripts and forms do not. Made for prompts and RAG chunks.

In short:
- GET /v1/scrape?url=…&format=markdown returns the page body as Markdown in the markdown field. Add raw=1 to get the Markdown alone, no JSON envelope.
- Headings, lists, tables, code blocks, emphasis and images survive; scripts, styles, forms and hidden elements are dropped; every link becomes absolute.
- selector narrows the conversion to one element, such as article or main, which is usually the difference between a clean chunk and a page full of navigation.

The cheapest way to make a model read a web page is to hand it the page as Markdown: the headings tell it what the sections are, the lists and tables keep their rows, the code stays in fences, and none of the tokens go to `<div class="…">`. `format=markdown` does that conversion on the API side.

## The request

```bash
curl "https://api.webscrapingapi.dev/v1/scrape?url=https://example.com&format=markdown" \
  -H "X-API-Key: wsa_your_key"
```

```json
{
  "url": "https://example.com",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "contentType": "text/html; charset=UTF-8",
  "title": "Example Domain",
  "rendered": false,
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents. …\n\n[More information...](https://www.iana.org/domains/example)",
  "truncated": false,
  "elapsedMs": 310,
  "credits": 1
}
```

Add `raw=1` and the same request returns only the Markdown, as `text/markdown`, which is convenient when the next thing in your pipeline is a file or a prompt rather than a JSON parser. The status code and final URL move to `x-scrape-status` and `x-scrape-final-url` headers.

## What the conversion keeps and drops

| Kept | Dropped |
| --- | --- |
| Headings as `#`, `##`, `###` | `<script>`, `<style>`, `<noscript>`, `<template>` |
| Bullet and numbered lists, nested | Forms, buttons, inputs |
| Tables as GitHub-flavoured Markdown | Inline SVG, canvas, iframes |
| Fenced code blocks and inline code | Elements marked `aria-hidden="true"` |
| Bold, italic, strikethrough, horizontal rules | Runs of more than two blank lines |
| Links and images, rewritten to absolute URLs | |

Relative links are the detail that matters most in practice. A page's `/docs/install` becomes `https://site.example/docs/install`, so a model that decides to follow the link, or a citation you show a user, still works outside the page.

## Narrow it with `selector`

Whole pages carry navigation, footers and cookie banners. Pass a CSS selector for the part you want and only that element is converted.

```bash
curl "https://api.webscrapingapi.dev/v1/scrape?url=https://en.wikipedia.org/wiki/Web_scraping&format=markdown&selector=%23mw-content-text&raw=1" \
  -H "X-API-Key: wsa_your_key"
```

`article`, `main`, `#content` and `.post-body` cover a large share of the web. When the page has none of them, the whole body is converted and you trim on your side.

## Pages that need a browser

Some sites render their content with JavaScript and send an almost empty HTML shell. If the Markdown comes back with little more than a title, add `render=true` and, when the content appears late, `waitFor=<selector>`. Rendering costs 5 credits instead of 1 because a real Chrome runs the page; use it only for the pages that need it.

## Chunking for retrieval

Because headings survive, you can split the Markdown on `##` boundaries and get chunks that correspond to the page's own sections, each with a title you can embed alongside the text. That is a better retrieval unit than a fixed window of characters, and it costs nothing extra: it is just the structure the page already had, kept.

The [scrape reference](/docs#scrape) lists every parameter.

### Questions people ask

**Why Markdown rather than plain text?**

Structure survives. A model told that something is a heading, a table row or a code block reads it better than the same words run together, and Markdown carries that structure in very few extra tokens.

**Does it run JavaScript?**

Only if you add render=true (5 credits). Most article and documentation pages do not need it; try without first.

**How big can the page be?**

Up to 5 MB of HTML is fetched; the response says truncated: true if the page was larger.

**Can I get both the Markdown and some fields?**

Yes. Add extract with CSS selectors and the response carries data alongside markdown.


---

## Hacker News front page and comment threads as JSON, ranked and flattened

URL: https://webscrapingapi.dev/blog/hacker-news-front-page-and-comments-as-json
Published: 2026-09-22 · Author: Jaeyoung · Tags: site endpoints, media & communities

Front page stories in rank order with points and comment counts, or one thread with its comments flattened and a depth on each. Two GET requests.

In short:
- GET /v1/structured/hackernews/front?page=1 returns the 30 stories on that page in rank order: id, title, url, site, points, author, commentCount, age.
- GET /v1/structured/hackernews/item?id=… returns the story plus comments[] in page order, each with a depth, so you can rebuild the tree or read it flat.
- Answers are stored for 10 minutes and shared. The front page is about freshness, so the window is short.

Hacker News has a clean official API, and it is the right tool for building a client. It is the wrong tool for "give me the front page with points and comment counts right now", because that is 31 requests, and for "give me this thread", because that is one request per comment. These two endpoints read the rendered page instead and return it in one shape.

## Front page

```bash
curl "https://api.webscrapingapi.dev/v1/structured/hackernews/front?page=1" \
  -H "X-API-Key: wsa_your_key"
```

`page` is optional and 1-based. The answer is the stories on that page, in the order the site prints them.

```json
{
  "site": "hackernews",
  "action": "front",
  "data": {
    "count": 30,
    "stories": [
      {
        "id": 45312001,
        "rank": 1,
        "title": "Samsung is expected to more than double its HBM output",
        "url": "https://example-news.com/samsung-hbm",
        "site": "example-news.com",
        "points": 125,
        "author": "someone",
        "commentCount": 84,
        "age": "3 hours ago"
      }
    ]
  },
  "source": "live",
  "age": 0,
  "credits": 1
}
```

`url` is the story link; for Ask HN and other self posts it points at the item page itself. `site` is the hostname the page shows in parentheses, or `null` for self posts. `age` is the relative time as printed, because that is what the page carries; if you need a timestamp, the item id is monotonic and the official API has the exact time.

## One thread

```bash
curl "https://api.webscrapingapi.dev/v1/structured/hackernews/item?id=1" \
  -H "X-API-Key: wsa_your_key"
```

```json
{
  "site": "hackernews",
  "action": "item",
  "data": {
    "id": 1,
    "title": "Y Combinator",
    "url": "http://ycombinator.com",
    "points": 61,
    "author": "pg",
    "age": "on Oct 9, 2006",
    "comments": [
      { "id": 15, "author": "sama", "age": "on Oct 9, 2006", "depth": 0, "text": "…" },
      { "id": 17, "author": "pg", "age": "on Oct 9, 2006", "depth": 1, "text": "…" }
    ]
  },
  "source": "cache",
  "age": 240,
  "credits": 0
}
```

Comments come back flat, in page order, with `depth`. A reply to a top-level comment has `depth: 1`, a reply to that has `depth: 2`, and so on. Rebuilding the tree is a ten-line loop; reading it flat is what most summarisers want anyway. `text` is plain text with paragraph breaks kept and HTML removed.

## Freshness

The front page changes every few minutes, so both endpoints store answers for only 10 minutes. Within that window every caller gets the stored copy for 0 credits; after it, the next caller fetches a fresh page for 1 credit and everyone shares that. Pass `max_age=0` to force a fetch, or a larger value if ten-minute-old rankings are fine for your use.

## What to build with it

A daily digest that lists the top stories and their discussion size is one request. A "summarise this thread" tool is one request plus a model call. A monitor that watches for a domain to appear on the front page is one request every ten minutes, stored and shared with everyone else running the same monitor. The [reference page](/docs/sites/hackernews) has the full parameter list.

### Questions people ask

**Why not use the official Hacker News API?**

You can; it is free and stable. It returns one item per request, so a front page with comment counts is 31 requests and a thread is one per comment. These endpoints return the rendered page in one call.

**Is the ranking the same as on the site?**

Yes. rank is the number printed on news.ycombinator.com at fetch time, and the list is in that order.

**Do comments keep their nesting?**

They are returned flat, in page order, with a depth field (0 for top level). Nesting is recoverable from depth without a recursive structure that is awkward to page through.

**How current is it?**

At most 10 minutes old unless you pass max_age=0, which fetches fresh for 1 credit.


---

## CSS selectors in, JSON out: the extract parameter explained with examples

URL: https://webscrapingapi.dev/blog/css-selectors-to-json-extract-parameter
Published: 2026-09-22 · Author: Jaeyoung · Tags: generic scrape, getting started

Name fields as CSS selectors and /v1/scrape returns them as JSON: attributes, lists, inner HTML, absolute links, up to 50 fields, no parsing code.

In short:
- extract is an object of field → selector. A plain string returns the first match's text; an object adds attr, all and html.
- href and src attributes come back as absolute URLs. A field with no match is null; a list with no matches is [].
- format=json returns only the fields. Any other format returns the page body and the fields together.

Parsing is where scrapers accumulate code. The `extract` parameter moves it into the request: you name fields, you give each one a CSS selector, and the response has the values. There is no expression language to learn; if you can write `h1` or `a.next[href]`, you can write an extract rule.

## The shape

```json
{
  "url": "https://example.com",
  "format": "json",
  "extract": {
    "title": "h1",
    "moreInfo": { "selector": "a", "attr": "href" },
    "paragraphs": { "selector": "p", "all": true },
    "body": { "selector": "div", "html": true }
  }
}
```

Each rule is either a string, meaning "the text of the first element that matches", or an object:

| Key | Meaning |
| --- | --- |
| `selector` | any CSS selector cheerio understands: tags, classes, ids, attributes, `:nth-child`, combinators |
| `attr` | return this attribute instead of the text; `href` and `src` are resolved to absolute URLs |
| `all` | return every match as an array instead of the first match |
| `html` | return the element's inner HTML instead of its text |

Text values have whitespace collapsed and are trimmed, so a title wrapped across three lines of HTML comes back as one line.

## The request and the answer

```bash
curl -X POST "https://api.webscrapingapi.dev/v1/scrape" \
  -H "X-API-Key: wsa_your_key" -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","format":"json","extract":{"title":"h1","links":{"selector":"a","attr":"href","all":true}}}'
```

```json
{
  "url": "https://example.com",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "contentType": "text/html; charset=UTF-8",
  "title": "Example Domain",
  "rendered": false,
  "data": {
    "title": "Example Domain",
    "links": ["https://www.iana.org/domains/example"]
  },
  "truncated": false,
  "elapsedMs": 306,
  "credits": 1
}
```

With `format=json` the response carries only `data`. With `format=html`, `text` or `markdown` you get the body and `data` together, which is handy when one field is a quick sanity check on a page you are also storing.

## Three patterns that cover most pages

**A product card.** Title, price and image, each the first match inside a known container:

```json
{ "name": ".product h1", "price": ".product .price", "image": { "selector": ".product img", "attr": "src" } }
```

**A list page.** Every item's link and label as parallel arrays:

```json
{ "titles": { "selector": "article h2", "all": true }, "urls": { "selector": "article h2 a", "attr": "href", "all": true } }
```

The two arrays line up index by index because both selectors visit the same articles in document order.

**A block you want to process yourself.** The inner HTML of one element, when the structure inside is irregular:

```json
{ "table": { "selector": "table.specs", "html": true } }
```

## When a page changes

Selectors break when layouts change; that is true of every scraper ever written. Two habits keep the damage small. Prefer ids and semantic elements (`article h1`, `[itemprop=price]`) over generated class names (`.css-1x2y3z`). And check for `null`: a field that was always present and is suddenly `null` on every page is the signal that the layout moved, and the response's `title` and `statusCode` tell you whether you fetched the right page at all.

The [extraction reference](/docs#extract) has the limits and one more example.

### Questions people ask

**Can I send extract in a GET request?**

Yes, as a JSON string in the extract query parameter. For anything beyond two or three fields a POST body is easier to read and to escape.

**What happens when a selector matches nothing?**

The field is null (or [] when all is true). Nothing errors; a missing element is a normal outcome on a page that changed.

**How many fields can I ask for?**

Up to 50 per request. Field names are letters, digits, dot, underscore and dash, at most 64 characters.

**Do selectors run against the rendered DOM?**

Against the HTML that was fetched. With render=true that is the DOM after JavaScript ran; without it, the server's HTML.


---

## GitHub stars, forks, topics and license for any public repo, with no token and no 60-an-hour limit

URL: https://webscrapingapi.dev/blog/github-repo-stars-forks-topics-without-token
Published: 2026-09-22 · Author: Jaeyoung · Tags: site endpoints, caching & credits

GitHub's REST API allows 60 anonymous requests an hour. This endpoint reads the public repo page: stars, forks, watchers, topics, license, releases.

In short:
- GET /v1/structured/github/repo?owner=…&repo=… returns the overview facts from the public page, with no GitHub token.
- Counts are exactly what GitHub renders, so they are as current as the page. Answers are stored for 1 hour and shared.
- The primary language is no longer in the server-rendered page, so language is usually null; everything else is stable.

Sixty requests an hour is what GitHub's REST API gives an unauthenticated caller, and a token is the obvious fix when you can carry one. When you cannot, or when you just want a fixed JSON shape from one call, the public repository page carries the same overview facts, and this endpoint reads them.

## The request

```bash
curl "https://api.webscrapingapi.dev/v1/structured/github/repo?owner=withastro&repo=astro" \
  -H "X-API-Key: wsa_your_key"
```

## The response

```json
{
  "site": "github",
  "action": "repo",
  "data": {
    "owner": "withastro",
    "repo": "astro",
    "url": "https://github.com/withastro/astro",
    "description": "The web framework for content-driven websites.",
    "homepage": "https://astro.build",
    "stars": 62720,
    "forks": 3798,
    "watchers": 221,
    "language": null,
    "topics": ["astro", "blog", "islands", "ssg", "static-site-generator"],
    "license": "Other",
    "defaultBranch": "main",
    "archived": false,
    "fork": false,
    "releaseCount": 3251
  },
  "source": "live",
  "age": 0,
  "credits": 1
}
```

`stars`, `forks` and `watchers` are the exact counts from the page data, not the rounded labels in the header. `license` is the name GitHub shows ("MIT", "Apache-2.0", "Other"). `topics` is the list under the description. `releaseCount` is the number in the Releases sidebar.

`language` deserves a warning: GitHub renders the language bar after the page loads, so the server HTML no longer contains it and the field is `null` on the current layout. The endpoint reports what it can see rather than guessing.

## Freshness and cost

Star counts move, but not in ways that matter minute to minute, so answers are stored for 1 hour and shared. A dashboard that shows twenty repositories and refreshes every minute costs twenty credits an hour, not twelve hundred. `max_age=0` fetches fresh for 1 credit.

## Where it fits

Comparison tables, "is this project alive" checks in an agent (`archived`, `releaseCount`, `stars`), README badges you generate yourself, and any place you would have reached for the REST API without a token. The [reference page](/docs/sites/github) lists the parameters.

### Questions people ask

**Why would I use this instead of api.github.com?**

When you have no token and need more than 60 requests an hour, or when you are inside an agent and want one call with a fixed shape. With a token and a modest volume the official API is fine.

**Are the star counts exact?**

They are the exact numbers GitHub puts in the page's data, not the rounded '62k' label.

**Does it work on private repositories?**

No. The API never signs in anywhere; only public pages are reachable.

**Why is language null?**

GitHub now loads the language bar after the page, so it is not in the HTML the API receives. Rather than guess, the field is null.


---

## Scraping API error codes explained: what each one means, whether to retry, and what it costs

URL: https://webscrapingapi.dev/blog/scraping-api-error-codes-explained
Published: 2026-09-22 · Author: Jaeyoung · Tags: reliability, how it works, caching & credits

Every error is a status plus a short code and a sentence. The full list: the cause of each one, whether a retry helps, and which ones refund the credit.

In short:
- Errors are JSON: { error: { code, message } }. The code is stable and meant for your if-statements; the message is for humans.
- 4xx codes are about your request or your account and never fix themselves with a retry. 5xx codes are about the target site or the service and often do.
- Failures caused on this side refund the credit: RENDER_UNAVAILABLE, INTERNAL_ERROR, SITE_BLOCKED and SITE_PARSE_FAILED. A dead or slow target site (FETCH_FAILED, TIMEOUT) is charged, because the fetch was made.

An error you can branch on is worth more than one you can read. Every failure from this API therefore carries a short, stable `code` next to a plain sentence, and the HTTP status tells you which side the problem is on.

```json
{ "error": { "code": "QUOTA_EXCEEDED", "message": "Daily credit limit reached. Resets at 2026-09-23T00:00:00Z." } }
```

## The full list

| Status | Code | Cause | Retry? | Credit |
| --- | --- | --- | --- | --- |
| 400 | `BAD_REQUEST` | a parameter is missing, malformed or out of range; the message names it | no, fix the request | not charged |
| 400 | `URL_BLOCKED` | the URL targets a private or reserved network, localhost, or a non-http scheme | no | not charged |
| 401 | `UNAUTHORIZED` | no API key, or a key that does not exist | no, check the key | not charged |
| 403 | `FORBIDDEN` | the key was revoked, or the account is not allowed to do this | no | not charged |
| 404 | `NOT_FOUND` | unknown site or action on a structured endpoint, or a site said the item does not exist | no | charged on site endpoints |
| 429 | `RATE_LIMITED` | more than 60 requests in a minute on one key | yes, after a few seconds | not charged |
| 429 | `QUOTA_EXCEEDED` | the account's daily credits are used up | after the reset time in the message | not charged |
| 502 | `FETCH_FAILED` | the target could not be reached: DNS, connection refused, TLS, too many redirects | yes, later | charged |
| 502 | `SITE_PARSE_FAILED` | the page arrived but the parser could not read it; usually a layout change | yes, and tell the board | refunded |
| 503 | `SITE_BLOCKED` | the target served a CAPTCHA or a bot wall instead of the page | yes, after a minute or more | refunded |
| 503 | `RENDER_UNAVAILABLE` | `render=true` was asked for while the browser pool is down | yes, or drop `render` | refunded |
| 504 | `TIMEOUT` | the target did not answer within `timeout` (default 15 s, max 30 s) | yes, perhaps with a longer timeout | charged |
| 500 | `INTERNAL_ERROR` | a bug on this side | yes; it is logged and looked at | refunded |

## The rule behind the credit column

You pay for a fetch that was made on your behalf. If the target site is down or slow, the fetch was made and the credit stands, exactly as it would if you had made the request yourself. If the failure is caused here, because the parser is out of date, the browser pool is unavailable, or a cloud address was blocked by the site, the credit is returned; that was not something you could have avoided.

Successful responses that carry a target status of 404 or 500 are not errors. `/v1/scrape` returns `200` with the target's `statusCode` in the body, because fetching a page that says "not found" is a successful fetch.

## A retry policy that works

```text
RATE_LIMITED       → wait 2–5 s, retry
SITE_BLOCKED       → wait 60 s+, retry once; or accept a stored copy (max_age)
FETCH_FAILED       → retry once after 10 s; then give up, the site is down
TIMEOUT            → retry once with a larger timeout (up to 30000)
SITE_PARSE_FAILED  → do not loop; post the URL on the board
4xx (others)       → fix the request or the key; retrying repeats the error
```

Log the `code`, not the message; messages may be reworded, codes will not.

## Where errors show up in the dashboard

Every request, successful or not, is a row in the dashboard's log with its code, status, host and timing, kept for 30 days. If you see a run of `SITE_PARSE_FAILED` on one site, that is the moment to [say so on the board](/board/write?kind=request); parsers are fixed once, for everyone. The [error reference](/docs#errors) is the short version of this page.

### Questions people ask

**What is the difference between RATE_LIMITED and QUOTA_EXCEEDED?**

RATE_LIMITED is per minute and per key (60 requests); wait a few seconds. QUOTA_EXCEEDED is the daily credit cap for the whole account; it resets at 00:00 UTC and the message says when.

**Should I retry SITE_BLOCKED?**

Later, yes, with a pause of at least a minute; the site is refusing cloud traffic and hammering it makes that worse. For site endpoints, consider accepting a stored copy with a larger max_age instead.

**What does URL_BLOCKED mean?**

The URL points at something the API will not fetch: private networks, localhost, link-local addresses, non-http schemes. It is a safety rule, not a temporary state.

**Is a 4xx from the target site an error?**

No. If the target answers 404 or 403, /v1/scrape returns 200 with statusCode: 404 in the body, because the fetch itself succeeded. Errors are about the API's own work.


---

## Read Open Graph, Twitter card and JSON-LD metadata from any URL with one request

URL: https://webscrapingapi.dev/blog/read-open-graph-json-ld-metadata-from-any-url
Published: 2026-09-22 · Author: Jaeyoung · Tags: generic scrape, getting started

/v1/metadata returns title, description, canonical, favicon, language, Open Graph and Twitter tags and every JSON-LD block from the head alone. One credit.

In short:
- GET /v1/metadata?url=… reads the first 512 KB of the page and returns title, description, canonical, favicon, lang, og, twitter and jsonLd.
- It is the endpoint for link previews, bookmarks, feed enrichment and 'what is this URL' checks in an agent. It never renders JavaScript.
- jsonLd is the raw array of every application/ld+json block, parsed, so products, articles, events and organisations come through with their schema.org fields.

Every link preview, every bookmark manager and every "summarise this URL" agent starts the same way: fetch the page and pull the tags out of the head. `/v1/metadata` does that step and stops there, so it is fast and costs one credit.

## The request

```bash
curl "https://api.webscrapingapi.dev/v1/metadata?url=https://example.com" \
  -H "X-API-Key: wsa_your_key"
```

## The response

```json
{
  "url": "https://example.com",
  "finalUrl": "https://example.com/",
  "title": "Example Domain",
  "description": null,
  "canonical": null,
  "favicon": "https://example.com/favicon.ico",
  "lang": null,
  "og": {},
  "twitter": {},
  "jsonLd": [],
  "elapsedMs": 380,
  "credits": 1
}
```

example.com is deliberately bare; a typical article page fills every field.

| Field | Where it comes from |
| --- | --- |
| `title` | the `<title>` element, else `og:title` |
| `description` | `<meta name="description">`, else `og:description` |
| `canonical` | `<link rel="canonical">`, resolved to an absolute URL |
| `favicon` | `<link rel="icon">`, the shortcut icon or the apple touch icon, resolved to an absolute URL; `/favicon.ico` when none is declared |
| `lang` | the `lang` attribute on `<html>` |
| `og` | every `og:*` property with the prefix removed: `{ "image": "…", "type": "…", "site_name": "…" }` |
| `twitter` | every `twitter:*` name the same way: `{ "card": "summary_large_image", … }` |
| `jsonLd` | each `<script type="application/ld+json">` parsed; invalid blocks are skipped |

## JSON-LD is the underrated part

Sites put their structured data in JSON-LD because search engines read it, and that makes it the most reliable machine-readable description of a page. A product page yields `Product` with `offers.price`; an article yields `Article` with `datePublished` and `author`; an event yields the dates and the venue. `jsonLd` hands you those objects parsed, in the order they appear, and you pick the `@type` you care about. For a general "tell me about this URL" tool, checking `jsonLd` before falling back to scraping the body saves a lot of selectors.

## What it is not for

The endpoint never runs JavaScript and never reads past the first 512 KB, so it is not a way to get a page's content. For the body use [`/v1/scrape`](/docs#scrape) with `format=markdown` or `text`. If a site injects its Open Graph tags with client-side code, which is rare but happens on some single-page apps, you will see the HTML fallbacks instead; that is a property of the site, and a scrape with `render=true` is the workaround.

## Typical uses

Link previews in a chat or a CMS, enriching an RSS feed with images and descriptions, deduplicating bookmarks by `canonical`, and giving an agent a cheap first look at a URL before it decides whether to read the whole page. The [metadata reference](/docs#metadata) has the full field list.

### Questions people ask

**Why only 512 KB?**

Metadata lives in the head, which is always at the top. Stopping early keeps the request fast and cheap; large pages still yield their tags.

**What if a page has no Open Graph tags?**

og is an empty object; title and description come from the HTML title and meta description. canonical is null when absent; favicon falls back to /favicon.ico on the final host.

**Is the favicon URL absolute?**

Yes. Relative icon links are resolved against the final URL after redirects.

**Does it follow redirects?**

Yes. finalUrl is where the page actually lived; url is what you asked for.


---

## Wikipedia summary and infobox as JSON, in any language edition, for RAG and agents

URL: https://webscrapingapi.dev/blog/wikipedia-summary-and-infobox-api-any-language
Published: 2026-09-22 · Author: Jaeyoung · Tags: site endpoints, rag & agents

Lead paragraphs as plain text, the thumbnail and the infobox as key/value pairs, in any language edition, from one request. Stored 7 days.

In short:
- GET /v1/structured/wikipedia/summary?title=…&lang=en returns title, url, summary (the first paragraph), paragraphs[] (the whole lead section), thumbnail and infobox {}.
- Reference marks, edit links and hatnotes are stripped, so paragraphs drop straight into a prompt.
- Any language edition works with lang; the same shape comes back for en, ko, ja, de and the rest.

Most retrieval pipelines need the same thing from Wikipedia: a reliable few paragraphs about a named thing, plus the facts in the box on the right. The official REST summary gives you one paragraph and no box. This endpoint gives you the lead section and the infobox, cleaned, in any edition.

## The request

```bash
curl "https://api.webscrapingapi.dev/v1/structured/wikipedia/summary?title=Web%20scraping&lang=en" \
  -H "X-API-Key: wsa_your_key"
```

`title` is the article title, with spaces or underscores. `lang` is the edition and defaults to `en`.

## The response

```json
{
  "site": "wikipedia",
  "action": "summary",
  "data": {
    "title": "Web scraping",
    "url": "https://en.wikipedia.org/wiki/Web_scraping",
    "lang": "en",
    "summary": "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. …",
    "paragraphs": ["Web scraping, web harvesting, or web data extraction is …", "Web scraping a web page involves fetching it and extracting from it. …", "…"],
    "thumbnail": null,
    "infobox": {}
  },
  "source": "cache",
  "age": 3120,
  "credits": 0
}
```

`summary` is the first paragraph; `paragraphs` is the whole lead section in order. Citation marks like `[3]`, the "[edit]" links and hatnotes ("For other uses, see…") are removed, so the text reads the way a person would read it aloud. `thumbnail` is the lead image when the article has one. `infobox` is the box on the right as plain `label: value` pairs; a country article gives you capital, population and area, a person gives you born, occupation and known for, and an article with no box gives `{}`.

## Other editions

```bash
curl "https://api.webscrapingapi.dev/v1/structured/wikipedia/summary?title=%EC%9B%B9%20%EC%8A%A4%ED%81%AC%EB%9E%98%ED%95%91&lang=ko" \
  -H "X-API-Key: wsa_your_key"
```

The shape is identical across editions. Titles are edition-specific, so ask for the Korean title on `ko` and the Japanese title on `ja`; the endpoint does not translate titles.

## Freshness and cost

Lead sections change rarely and slowly, so answers are stored for 7 days and shared with everyone. The first request for an article costs 1 credit; the rest of the week costs nothing. If an article is in the news and you want today's revision, `max_age=0` fetches it fresh.

## Using it in a pipeline

For an agent that has to answer "what is X" or "who is Y", the lead section is the right granularity: a few hundred words, written to be read first, in the language your user asked in. Fetch it, put `paragraphs` in the prompt, and keep `url` for the citation. When a task genuinely needs the whole article, call [`/v1/scrape`](/docs#scrape) with `format=markdown` on the `url` you got back; the sections, tables and links come through as markdown. The [reference page](/docs/sites/wikipedia) has the details.

### Questions people ask

**How is this different from the Wikipedia REST API's page/summary?**

That endpoint returns one extract paragraph. This one returns every paragraph of the lead section plus the infobox as key/value pairs, which the REST summary does not include.

**Do I get the whole article?**

No, the lead section: everything before the first heading. For a model that needs to identify or describe a thing, the lead is usually the right amount. For full text, use /v1/scrape with format=markdown on the article URL.

**What is in infobox?**

The right-hand table's rows as { label: value } with plain-text values. Articles without an infobox return an empty object.

**Spaces or underscores in the title?**

Either. The title is encoded for you; Web scraping and Web_scraping are the same article.


---

## How to get Amazon product data as JSON without the Product Advertising API

URL: https://webscrapingapi.dev/blog/amazon-product-data-json-without-product-advertising-api
Published: 2026-09-21 · Author: Jaeyoung · Tags: site endpoints, shopping & apps, reliability

Turn an ASIN into JSON with one GET: title, price, availability, rating, images, bullets and the spec table. No Amazon developer account or affiliate approval.

In short:
- GET /v1/structured/amazon/product?asin=… returns the product page as JSON in one call. It costs 1 credit, or 0 when a stored copy is fresh enough.
- You choose the marketplace with tld and the price currency with currency. Language is fixed to English so field labels stay stable.
- Amazon sometimes serves a CAPTCHA instead of the page. The API answers 503 SITE_BLOCKED; wait and retry, or accept a stored copy with a larger max_age.
- If you need official data for an affiliate site, Amazon's Product Advertising API is the right tool. This endpoint is for reading public product pages.

Amazon's own Product Advertising API needs an approved Associates account, a signed request and three qualifying sales in the first 180 days, or your access is revoked. If all you want is the public product page as structured data, there is a shorter path: one GET request with the ASIN.

## The request

```bash
curl "https://api.webscrapingapi.dev/v1/structured/amazon/product?asin=B0B44XTV71&tld=com&currency=USD" \
  -H "X-API-Key: wsa_your_key"
```

Three parameters matter. `asin` is the 10-character product id from the URL (`/dp/B0B44XTV71`). `tld` picks the marketplace and defaults to `com`. `currency` is the ISO code the prices should be shown in and defaults to `USD`. Everything else is handled for you: browser headers, redirects, the cookie that switches the price currency, and the parsing.

## What comes back

The response uses the same field names as every other shop endpoint on this service, so a client written for Amazon does not need a rewrite for the next store. Amazon-only values sit in `extra`.

```json
{
  "site": "amazon",
  "action": "product",
  "data": {
    "product": {
      "name": "CUCKOO Twin Pressure Rice Cooker, 12-Cup Cooked, White",
      "brand": { "name": "CUCKOO", "url": null },
      "price": "339.99",
      "regularPrice": null,
      "currency": "USD",
      "availability": "InStock",
      "aggregateRating": { "ratingValue": "4.4", "reviewCount": 608 },
      "mainImage": "https://m.media-amazon.com/images/I/71QNsrTZ47L.jpg",
      "images": ["https://m.media-amazon.com/images/I/71QNsrTZ47L.jpg", "…"],
      "features": ["16 Versatile Modes: …", "…"],
      "additionalProperties": [{ "name": "Capacity", "value": "6 cups" }, { "name": "Part Number", "value": "CRP-ST0609FW" }],
      "breadcrumbs": [{ "name": "Home & Kitchen", "url": "https://www.amazon.com/…" }, { "name": "Rice Cookers", "url": "…" }],
      "sku": "B0B44XTV71"
    },
    "extra": {
      "asin": "B0B44XTV71",
      "tld": "com",
      "bestSellersRank": [{ "rank": 18677, "category": "Kitchen & Dining" }, { "rank": 67, "category": "Rice Cookers" }],
      "soldBy": "Amazon.com",
      "shipsFrom": null,
      "boughtInPastMonth": "400+ bought in past month",
      "availabilityText": "In Stock"
    }
  },
  "source": "live",
  "age": 0,
  "fetchedAt": "2026-09-21T05:12:40.000Z",
  "credits": 1,
  "elapsedMs": 2140
}
```

A few fields deserve a note. `price` is a string with a dot as the decimal separator, whatever the marketplace prints, so `"1234.56"` is the same number on amazon.de and amazon.com. `availability` is one of `InStock`, `OutOfStock`, `PreOrder` or `Unknown`; when Amazon shows a delivery-dependent message such as "cannot be shipped to your selected delivery location", the value is `Unknown` and the original sentence is in `extra.availabilityText`. `images` are the full-size originals with Amazon's size suffixes removed.

## Stored copies and `max_age`

Every product answer is stored and shared. The first person to ask for an ASIN pays one credit; anyone asking for the same ASIN in the next 24 hours gets the stored copy for zero credits, with `source: "cache"` and `age` telling them how old it is. If you need something fresher, pass `max_age` in seconds: `max_age=3600` accepts a copy up to an hour old, `max_age=0` always fetches. Fresh fetches cost the credit; stored ones do not.

For price monitoring this is the whole design. Poll with `max_age=0` only as often as the price actually matters to you, and let the shared store absorb everyone else's reads.

## When Amazon says no

Amazon does not like data-center traffic. From a home connection the product page arrives on the first try; from cloud addresses it works some of the time and returns a CAPTCHA page the rest. The API detects that page and answers `503` with code `SITE_BLOCKED` instead of pretending it parsed something. The credit is not charged. The sensible client behaviour is to back off for a minute and retry, or to accept an older stored copy with a large `max_age` when one exists. Every attempt, blocked or not, shows up in your dashboard log, so you can see the real hit rate for your own traffic before you build on it.

There is no CAPTCHA solving and no proxy rotation behind this endpoint, and it will not pretend otherwise.

## When to use Amazon's official API instead

If you run an Amazon Associates site and need prices you are allowed to display with affiliate links, use the Product Advertising API: it is the only source Amazon sanctions for that, and it includes offer listings, variations and search that this endpoint does not. If you need public page facts for research, monitoring, a comparison tool or an agent that answers "what does this product cost right now", the endpoint above is simpler and free.

## Try it

Create a key in the [dashboard](https://app.webscrapingapi.dev), then paste the curl command above with your ASIN. The [endpoint reference](/docs/sites/amazon) lists every parameter and the full response shape.

### Questions people ask

**Do I need an Amazon account or an affiliate ID?**

No. The endpoint reads the public product page. You only need a free webscrapingapi.dev key.

**Which marketplaces are supported?**

com, co.uk, de, fr, it, es, ca, co.jp, in, com.au, com.mx, com.br, nl, ae, sa, se, pl and sg, chosen with the tld parameter.

**Why is the price in USD when I asked for amazon.de?**

Because currency defaults to USD. Pass currency=EUR (or any ISO 4217 code) and the page is fetched with that preference. Labels stay in English.

**How fresh is the data?**

Each response carries source (live or cache) and age in seconds. The stored copy is used for up to 24 hours by default; max_age=0 forces a fresh fetch.

**What happens when Amazon blocks the request?**

The API returns 503 with code SITE_BLOCKED and does not charge the credit. Retry later, or read a stored copy if one exists.


---

## App Store and Google Play app data as JSON: ratings, installs and price with one schema for both stores

URL: https://webscrapingapi.dev/blog/app-store-google-play-app-data-json
Published: 2026-09-21 · Author: Jaeyoung · Tags: site endpoints, shopping & apps

Two endpoints, one shape: an iOS app by id or an Android app by package name, returning name, developer, rating, price, category, screenshots and installs.

In short:
- /v1/structured/appstore/app?id=… and /v1/structured/googleplay/app?id=… return the same AppSchema, so one client handles both stores.
- Only Google Play exposes the install band ("1B+"); only the App Store exposes the OS requirement. Everything else lines up field for field.
- Both listings are stored for 24 hours and shared. Storefront, language and country are parameters, not guesses from your IP.

App stores publish more than they let you download through an API. Google's Play Developer API only covers apps you own; Apple's App Store Connect API is the same. For anyone else's app, the public listing page is the source, and both stores render their listing with a `SoftwareApplication` block that carries the facts. These two endpoints read that block, add the few things it lacks, and return one shape.

## The two requests

```bash
# iOS — numeric id, optional storefront country (default us)
curl "https://api.webscrapingapi.dev/v1/structured/appstore/app?id=284882215&country=us" \
  -H "X-API-Key: wsa_your_key"

# Android — package name, optional language (hl) and country (gl)
curl "https://api.webscrapingapi.dev/v1/structured/googleplay/app?id=com.spotify.music&hl=en&gl=US" \
  -H "X-API-Key: wsa_your_key"
```

## One schema

| Field | App Store | Google Play |
| --- | --- | --- |
| `name`, `url`, `id`, `platform` | yes | yes |
| `developer { name, url }` | yes | yes |
| `description`, `category` | yes | yes |
| `price`, `currency`, `free` | yes | yes |
| `rating { value, count }` | yes | yes |
| `icon`, `screenshots[]` | yes | yes |
| `contentRating` | when shown | yes ("Teen") |
| `osRequirement` | yes ("Requires iOS 15.1 or later.") | null |
| `installs` | null | yes ("1B+") |
| `updatedAt` | when shown | yes ("Sep 18, 2026") |

A Google Play answer looks like this:

```json
{
  "site": "googleplay",
  "action": "app",
  "data": {
    "app": {
      "name": "Spotify: Music and Podcasts",
      "url": "https://play.google.com/store/apps/details?id=com.spotify.music",
      "platform": "android",
      "id": "com.spotify.music",
      "developer": { "name": "Spotify AB", "url": "https://www.spotify.com" },
      "category": "MUSIC_AND_AUDIO",
      "price": "0", "currency": "USD", "free": true,
      "rating": { "value": 4.35, "count": 36339801 },
      "installs": "1B+",
      "contentRating": "Teen",
      "updatedAt": "Sep 18, 2026",
      "screenshots": ["https://play-lh.googleusercontent.com/…", "…"]
    }
  },
  "source": "cache",
  "age": 5120,
  "credits": 0
}
```

`installs` is the band Google shows ("1B+", "500M+", "10K+"), not an exact number; no public source has the exact number. `rating.value` is rounded to two decimals. `price` is a string and `free` is derived from it, so you do not have to compare currencies to know whether an app costs money.

## Storefronts and languages

Nothing is inferred from the caller's address. The App Store endpoint takes `country` (two letters, default `us`). Google Play takes `hl` for the language and `gl` for the country, defaulting to `en` and `US`. Ask for `hl=ko&gl=KR` and you get the Korean listing text with Korean-store availability.

## Freshness and cost

Listings change slowly, so both endpoints store answers for 24 hours and share them. The first caller pays one credit; the stored copy is free and says how old it is in `age`. Force a fresh read with `max_age=0` when a release just shipped and you want the new version text.

## What is missing on purpose

Version numbers and "what's new" notes are on the App Store page but not in a stable part of it, so they come back `null` more often than not; the endpoint prefers a `null` over a wrong value. Reviews, rankings and search are not on the listing page and are not part of these endpoints. If a chart or a review feed is what you need, [ask on the board](/board/write?kind=request) and it either gets built or you are told why not.

The reference pages for [the App Store](/docs/sites/appstore) and [Google Play](/docs/sites/googleplay) list the parameters and the complete response.

### Questions people ask

**Where do I find the App Store id?**

It is the number after id in the listing URL, for example 284882215 in apps.apple.com/us/app/facebook/id284882215.

**And the Google Play id?**

The package name in the id= parameter of the listing URL, for example com.spotify.music.

**Is the rating count the number of written reviews?**

It is the count the store publishes with the average: ratings on Google Play, ratings on the App Store's listing JSON. Written reviews are a smaller subset and are not returned.

**Can I read another country's store?**

Yes. App Store: country=de, jp, kr and so on. Google Play: gl=DE with hl=de for the language.


---

## What max_age means in a scraping API, and why a stored answer costs 0 credits

URL: https://webscrapingapi.dev/blog/what-max-age-means-in-a-scraping-api
Published: 2026-09-21 · Author: Jaeyoung · Tags: caching & credits, how it works

Site endpoints store answers and share them. max_age sets how old a stored copy may be; every response says whether it was live or stored, and how old it is.

In short:
- max_age is the maximum age, in seconds, of a stored copy you are willing to accept. Each endpoint has a sensible default; 0 forces a fresh fetch.
- A response served from the store carries source: "cache", its age in seconds and credits: 0. A live fetch carries source: "live" and costs the endpoint's credit.
- The store is shared across all users, so popular pages are fetched once and read many times. That is what keeps the service free.

Most scraping APIs treat every request as a fresh fetch, and price it that way. The structured endpoints here do something different: they remember what they fetched and let every caller decide how old an answer they will accept. The knob is a single query parameter, `max_age`, and the answer always says what it did.

## The rule in one paragraph

Ask for `/v1/structured/amazon/product?asin=B0B44XTV71`. If a copy of that product exists in the store and is younger than your `max_age`, you get it immediately, for 0 credits, with `source: "cache"` and `age` in seconds. If not, the page is fetched, parsed, stored, and returned with `source: "live"`, `age: 0` and the endpoint's credit charged. `max_age` defaults to the endpoint's window and is capped at 30 days.

```bash
# accept anything under an hour old
curl ".../v1/structured/amazon/product?asin=B0B44XTV71&max_age=3600" -H "X-API-Key: wsa_…"

# always fetch fresh
curl ".../v1/structured/amazon/product?asin=B0B44XTV71&max_age=0" -H "X-API-Key: wsa_…"
```

## Reading the envelope

Three fields at the top level of every structured response tell you what happened.

| Field | Meaning |
| --- | --- |
| `source` | `"live"` when the page was fetched for this request, `"cache"` when a stored copy was returned |
| `age` | seconds since the data was fetched; `0` for live answers |
| `fetchedAt` | the fetch time as an ISO timestamp, for your own bookkeeping |
| `credits` | what this request cost: the endpoint's credit for live, `0` for stored |

If your logic depends on freshness, branch on `age`, not on `source`; a live answer and a two-second-old stored answer are the same thing to almost every application.

## Why the defaults differ

The window follows how fast the page changes and how expensive it is to fetch.

| Endpoint | Default window | Why |
| --- | --- | --- |
| Hacker News front page | 10 minutes | the ranking moves constantly |
| YouTube video, GitHub repo | 1 hour | counts drift, but slowly |
| Amazon product, App Store, Google Play | 24 hours | prices and listings change daily at most; Amazon is also the hardest page to fetch |
| Wikipedia summary | 7 days | articles rarely change in ways a summary reflects |

The defaults are deliberately generous. Fresh fetches cost you credits and cost the service a page load; stored reads cost nothing on either side. If you need something newer, say so with `max_age`, request by request.

## Why the store is shared

A cache per user would only save you money on your own repeats. A cache shared by everyone means the second person asking about the same product pays nothing and gets an answer in a few milliseconds instead of a few seconds. As the service grows the share of live fetches falls, which is exactly what a free service needs to stay free. Only parsed JSON is stored, never the raw page.

## What is not cached

`/v1/scrape` and `/v1/metadata` fetch live every time. They return pages you shape with your own selectors, and raw pages are never kept, so there is nothing to share. Caching is a property of the structured endpoints, where the output is a known JSON shape.

### Questions people ask

**What is the default max_age?**

It depends on the endpoint: Hacker News 10 minutes, YouTube and GitHub 1 hour, Amazon and the app stores 24 hours, Wikipedia 7 days. Each reference page states its window.

**Does max_age apply to /v1/scrape?**

No. The generic scrape endpoint always fetches live, because raw pages are never stored. Only the structured site endpoints have a store.

**Can I be sure I get a fresh page?**

Yes, with max_age=0. It always fetches, costs the endpoint's credit, and returns source: "live" with age: 0.

**Is the stored copy deleted after max_age?**

No. max_age is the reader's choice, not a deletion rule. Copies stay for up to 30 days so that callers who accept older data can still use them.

**Who pays when two people ask for the same product?**

The first one, once. The second gets the stored copy for 0 credits until it is older than their max_age.


---

## Web scraping API vs writing your own scraper: which one for a side project, an agent, or a data pipeline

URL: https://webscrapingapi.dev/blog/web-scraping-api-vs-writing-your-own-scraper
Published: 2026-09-21 · Author: Jaeyoung · Tags: getting started, how it works, rag & agents

Fetching pages yourself versus calling a scraping API: headers, JavaScript rendering, blocked sites, parsing, cost, and the cases where each one wins.

In short:
- Your own scraper wins when the site is simple, you control the schedule, and you want zero dependencies.
- A scraping API wins when you need JavaScript rendering on demand, structured JSON for a known site, or a page fetched from inside an agent without shipping a browser.
- The hard part of scraping is never the first fetch; it is the layout change six months later. Site endpoints move that burden to the API.

The question comes up in every project that needs data from a page it does not own: write a scraper, or call an API that does it. Both are fine answers. The wrong one costs you weeks, so here is what actually differs.

## What "writing your own" involves

A minimal scraper is ten lines: fetch the URL, load the HTML into a parser, pick nodes with CSS selectors. That version works on example.com and on most blogs, documentation sites and news pages. The next steps are where the time goes.

- **Headers and redirects.** Sites answer a bare client differently from a browser. You add a user-agent, accept headers, follow redirects, handle gzip and charsets.
- **JavaScript.** When the data is loaded after the page, you need a real browser: Playwright or Puppeteer, a Chrome build, memory, and a queue so you do not run twenty of them at once.
- **Blocks.** Some sites answer data-center addresses with a CAPTCHA. You can add proxies; you probably should not solve CAPTCHAs.
- **Parsing.** A product page has the price in one of four places depending on the product type. You find them all, then the layout changes.
- **Operations.** Retries, timeouts, size caps, not fetching internal addresses by accident, logging.

None of this is difficult. All of it is maintenance.

## What a scraping API involves

You send a URL and get a page back, with the header, redirect, charset and size handling done. Add `render=true` and a browser runs it for you. For sites the API already knows, you skip parsing entirely and get JSON with fixed field names. The cost is a dependency and a quota.

```bash
# any page as plain text, ready for a model
curl "https://api.webscrapingapi.dev/v1/scrape?url=https://example.com&format=text" \
  -H "X-API-Key: wsa_your_key"

# a known site as JSON, no selectors
curl "https://api.webscrapingapi.dev/v1/structured/github/repo?owner=withastro&repo=astro" \
  -H "X-API-Key: wsa_your_key"
```

## When your own scraper wins

You have three or four simple sites, they rarely change, you run the job nightly from a machine you control, and you want no third party in the path. Write it. A scheduler, a parser library and a small table of selectors will serve you for years.

## When the API wins

- **Agents.** An agent that reads pages while answering a question cannot wait for a browser to boot, and you do not want Chrome inside your function. One HTTP call is the right shape.
- **JavaScript on demand.** Rendering a page every hour is cheap; keeping a browser fleet ready to render one page whenever a user asks is not.
- **Known sites.** Amazon, YouTube and the app stores are each a week of parsing and a lifetime of fixes. A site endpoint returns the JSON and is repaired when the layout changes, once, for everyone.
- **Shared results.** On this service structured answers are stored and shared; the second person asking about a product pays nothing and waits milliseconds. Your own scraper cannot benefit from other people's fetches.

## The honest limits

An API is not magic. Sites behind dedicated anti-bot vendors work only some of the time from any cloud address, and a service that does not solve CAPTCHAs will tell you so instead of pretending. Rate limits and daily quotas apply. And a page the API does not know still needs your selectors, exactly as it would in your own scraper, just with the fetching handled.

If a site you need is not covered, [post it on the board](/board/write?kind=request). It gets built when it can be, and you are told when it cannot.

### Questions people ask

**Is web scraping legal?**

Reading public pages is generally allowed in many jurisdictions, but a site's terms, rate limits, copyright and personal-data rules still apply. This is not legal advice; check the rules that apply to you and the site.

**Do I need JavaScript rendering?**

Only when the content you want is not in the HTML the server sends. Fetch the page with render=false first and look; most article and product pages do not need a browser.

**What does a free scraping API actually give me?**

On webscrapingapi.dev: 50,000 credits a day, 60 requests a minute, HTML/text/markdown/JSON output, JavaScript rendering for 5 credits, and site endpoints for Amazon, YouTube, the app stores, Hacker News, Wikipedia and GitHub.


---

## Get YouTube video metadata (views, likes, duration) with one GET request, no Google API key

URL: https://webscrapingapi.dev/blog/youtube-video-metadata-without-api-key
Published: 2026-09-21 · Author: Jaeyoung · Tags: site endpoints, media & communities

Title, channel, views, likes, length, publish date, thumbnails and keywords for any public video, from one request. And when the official Data API is better.

In short:
- GET /v1/structured/youtube/video?id=… returns the watch page metadata as JSON. No Google Cloud project, no OAuth, no daily quota units.
- Counts are what the page shows at fetch time; answers are stored for one hour and shared, so repeats cost 0 credits.
- Private, deleted and age-restricted videos are not reachable. For comments, playlists or channel listings use the YouTube Data API v3.

Reading a few facts about a YouTube video should not require a Google Cloud project. The watch page already contains the title, channel, counts, dates and thumbnails in a JSON block that the player reads; this endpoint reads the same block and returns it in a stable shape.

## The request

```bash
curl "https://api.webscrapingapi.dev/v1/structured/youtube/video?id=dQw4w9WgXcQ" \
  -H "X-API-Key: wsa_your_key"
```

`id` is the 11-character video id. That is the only required parameter.

## The response

```json
{
  "site": "youtube",
  "action": "video",
  "data": {
    "video": {
      "id": "dQw4w9WgXcQ",
      "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
      "title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
      "description": "The official video for “Never Gonna Give You Up” by Rick Astley. …",
      "channel": { "id": "UCuAXFkgsw1L7xaCfnd5JJOw", "name": "Rick Astley", "url": "http://www.youtube.com/@RickAstleyYT" },
      "durationSeconds": 213,
      "viewCount": 1818228078,
      "likeCount": 19384777,
      "publishedAt": "2009-10-24T23:57:33-07:00",
      "category": "Music",
      "thumbnails": [{ "url": "https://i.ytimg.com/vi/dQw4w9WgXcQ/maxresdefault.jpg", "width": 1280, "height": 720 }, "…"],
      "keywords": ["rick astley", "Never Gonna Give You Up", "…"],
      "isLive": false,
      "familySafe": true
    }
  },
  "source": "live",
  "age": 0,
  "credits": 1
}
```

`durationSeconds` is an integer, so a 3:33 video is `213`. `publishedAt` is the publish date with the channel's offset as YouTube prints it. The first thumbnail is the largest one the page advertises; the rest are the smaller variants.

## Freshness

Video counts change by the second on popular uploads, but very few applications need that resolution. Answers are stored for one hour and shared between everyone who asks for the same id, so a dashboard that refreshes every minute costs one credit an hour, not sixty. Pass `max_age=0` when you genuinely need the current number, or a smaller `max_age` such as `600` to accept anything under ten minutes old.

## What it cannot do

Private and deleted videos return `404 NOT_FOUND`. Age-restricted videos require a signed-in session, which the API never uses, so they come back as not reachable. Comments, captions, playlists, channel uploads and search are not on the watch page's metadata and are therefore out of scope; the [YouTube Data API v3](https://developers.google.com/youtube/v3) covers those, at the price of a project, a key and a daily quota.

## A typical use

An agent that is asked "how long is this video and who made it" needs two fields and needs them now. One GET, one JSON object, no OAuth dance. Create a key in the [dashboard](https://app.webscrapingapi.dev) and the [reference page](/docs/sites/youtube) has the rest.

### Questions people ask

**Is the like count included?**

Yes, as likeCount, read from the page's own metadata. Videos that hide likes return null.

**What is the video id?**

The 11-character value after v= in a watch URL, for example dQw4w9WgXcQ in youtube.com/watch?v=dQw4w9WgXcQ. Short links (youtu.be/ID) use the same id.

**How current are the view counts?**

As current as the watch page at fetch time. The stored copy is reused for up to 3600 seconds; pass max_age=0 to force a fresh fetch for 1 credit.

**Can I get the transcript or comments?**

Not from this endpoint. It covers what the watch page carries in its metadata: title, channel, counts, dates, description, thumbnails and keywords.