API Documentation

One base URL, one header, JSON back. Everything else is a parameter.

Base URL: https://api.webscrapingapi.dev. All responses are JSON unless you ask for raw=1. The API is free; there are no plans and nothing to upgrade.

Authentication

Create a key in the dashboard. Keys look like wsa_… and are shown once. Send it in any of these ways:

X-API-Key: wsa_… header (recommended)
Authorization: Bearer wsa_… header
?api_key=wsa_… query parameter (convenient for quick tests, but it lands in logs)

Health check (no key needed)

curl https://api.webscrapingapi.dev/v1/health

Credits & limits

Limit	Value
Daily credits per account	50,000 (resets 00:00 UTC, shared across your keys)
Plain fetch	1 credit
JavaScript render (`render=true`)	5 credits
Metadata / site endpoints	1 credit
Rate limit	60 requests / minute / key
Keys per account	5
Timeout	15 s default, 30 s max
Response body	5 MB, then `truncated: true`

Requests that reach the target count even if the target answers 4xx/5xx or times out. Requests that fail on our side (RENDER_UNAVAILABLE, INTERNAL_ERROR, SITE_BLOCKED, SITE_PARSE_FAILED) are refunded.

GET / POST /v1/scrape

Fetch a page and return it as HTML, plain text or extracted JSON. GET takes query parameters; POST takes the same fields as a JSON body (use POST for extract and headers).

Parameter	Type	Description
`url`	string, required	http or https URL. Private and reserved addresses are rejected with `URL_BLOCKED`.
`render`	boolean	Run the page in headless Chrome and return the DOM after JavaScript. Images, fonts and media are not loaded.
`waitFor`	string	With `render`: wait until this CSS selector exists.
`wait`	number (ms)	With `render`: extra wait after load, up to 10000.
`timeout`	number (ms)	1000–30000. Default 15000.
`format`	`html` \| `text` \| `markdown` \| `json`	Default `html`. `text` strips tags and keeps paragraph breaks. `markdown` keeps headings, lists, tables, code and absolute links — made for feeding a model. `json` requires `extract`.
`selector`	string	Return only this element (html, text or markdown formats).
`extract`	object	Field → rule map, see below. In GET, pass it as a JSON string.
`headers`	object (POST)	Custom request headers. Allowed: user-agent, accept, accept-language, cookie, referer.
`raw`	boolean	Respond with the HTML body itself instead of JSON (handy with curl). Status and final URL come back in `X-Scrape-*` headers.

Response

200 OK

{
  "url": "https://example.com/",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "contentType": "text/html; charset=UTF-8",
  "title": "Example Domain",
  "rendered": false,
  "html": "<!doctype html>…",          // or "text": "…" / "data": { … }
  "truncated": false,
  "elapsedMs": 412,
  "credits": 1
}

Text of a rendered page

curl "https://api.webscrapingapi.dev/v1/scrape?url=https://example.com&render=true&waitFor=main&format=text" \
  -H "X-API-Key: wsa_your_key"

Extraction rules

Each field is either a CSS selector string (returns the text of the first match) or an object:

Key	Meaning
`selector`	CSS selector (required).
`attr`	Return this attribute instead of text. `href` and `src` are resolved to absolute URLs.
`all`	`true` returns every match as an array.
`html`	`true` returns inner HTML instead of text.

POST with extract

curl -X POST https://api.webscrapingapi.dev/v1/scrape \
  -H "X-API-Key: wsa_your_key" -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "format": "json",
    "extract": {
      "title": "title",
      "stories": { "selector": ".titleline > a", "all": true },
      "links":   { "selector": ".titleline > a", "attr": "href", "all": true }
    }
  }'

You can combine extract with format=html or text to get both the body and the structured fields in one response. Up to 50 fields per request.

GET /v1/metadata

Everything in <head> that matters: title, description, canonical, favicon, language, Open Graph, Twitter cards and JSON-LD. Only the first 512 KB of the page is read, so it is fast.

bash

curl "https://api.webscrapingapi.dev/v1/metadata?url=https://github.com" -H "X-API-Key: wsa_your_key"

Response

{
  "url": "https://github.com",
  "finalUrl": "https://github.com/",
  "title": "GitHub · Build and ship software on a single, collaborative platform",
  "description": "…",
  "canonical": "https://github.com/",
  "favicon": "https://github.githubassets.com/favicons/favicon.svg",
  "lang": "en",
  "og": { "title": "…", "image": "…" },
  "twitter": { "card": "summary_large_image" },
  "jsonLd": [ … ],
  "elapsedMs": 380,
  "credits": 1
}

Site-specific APIs

Endpoints that already know a site's layout and return clean JSON. The registry is public at GET /v1/structured; each site has its own page with parameters and examples. Results are cached and shared: a response served from the store costs 0 credits and carries source and age; max_age (seconds) sets how old a stored copy may be, and max_age=0 forces a fresh fetch.

Site	Endpoints
Amazon product amazon.com · 20 marketplaces	`GET /v1/structured/amazon/product`	Docs →
YouTube video youtube.com	`GET /v1/structured/youtube/video`	Docs →
App Store app apps.apple.com	`GET /v1/structured/appstore/app`	Docs →
Google Play app play.google.com	`GET /v1/structured/googleplay/app`	Docs →
Hacker News news.ycombinator.com	`GET /v1/structured/hackernews/front` `GET /v1/structured/hackernews/item`	Docs →
Wikipedia wikipedia.org	`GET /v1/structured/wikipedia/summary`	Docs →
GitHub github.com	`GET /v1/structured/github/repo`	Docs →

bash

curl "https://api.webscrapingapi.dev/v1/structured/amazon/product?asin=B0B44XTV71&tld=com&currency=USD" -H "X-API-Key: wsa_your_key"

Errors

Errors use HTTP status codes and a JSON body: { "error": { "code": "…", "message": "…" } }.

Status	Code	When
400	`BAD_REQUEST`	Missing or invalid parameter, bad selector, malformed JSON.
400	`URL_BLOCKED`	Private / reserved address, disallowed port, credentials in URL.
401	`UNAUTHORIZED`	Missing, invalid or revoked API key.
403	`FORBIDDEN`	Account disabled.
404	`NOT_FOUND`	Unknown endpoint, site or action; site item not found.
429	`RATE_LIMITED`	More than 60 requests in a minute. Retry after a moment.
429	`QUOTA_EXCEEDED`	Daily credits used up. Resets at 00:00 UTC.
502	`FETCH_FAILED`	DNS, TLS or connection failure to the target.
502	`SITE_PARSE_FAILED`	A site endpoint could not parse the page (layout changed). Refunded. Please report it on the board.
503	`SITE_BLOCKED`	The target served a CAPTCHA or bot wall instead of the page. Refunded. Retry after a minute or accept a stored copy with `max_age`.
503	`RENDER_UNAVAILABLE`	Renderer disabled or busy. Refunded.
504	`TIMEOUT`	Target did not respond in time.
500	`INTERNAL_ERROR`	Our bug. Refunded.

Board API (public read)

The community board is readable without a key: GET /v1/board/posts (paginated with cursor) and GET /v1/board/posts/:id. Writing requires a signed-in account on this site.