API Documentation

Base URL: https://api.webscrapingapi.dev. All responses are JSON unless you ask for raw=1. The API is free; there are no plans and nothing to upgrade.

Authentication

Create a key in the dashboard. Keys look like wsa_… and are shown once. Send it in any of these ways:

• X-API-Key: wsa_… header (recommended)
• Authorization: Bearer wsa_… header
• ?api_key=wsa_… query parameter (convenient for quick tests, but it lands in logs)

curl https://api.webscrapingapi.dev/v1/health

Credits & limits

Requests that reach the target are charged, whatever the target answers: a 4xx/5xx, a timeout, or a CAPTCHA or bot wall (blocked on /v1/scrape, SITE_BLOCKED on site endpoints). Only requests that fail on our side are refunded: RENDER_UNAVAILABLE, RENDER_TOO_SLOW, INTERNAL_ERROR and SITE_PARSE_FAILED. A request we refuse (BAD_REQUEST, URL_BLOCKED) is never charged.

GET / POST /v1/scrape

Fetch a page and return it as HTML, plain text or extracted JSON. GET takes query parameters; POST takes the same fields as a JSON body (use POST for extract and headers). An option name the API does not know is refused with 400 instead of being ignored, so a typo such as waitfor tells you.

Response

{
  "url": "https://example.com/",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "contentType": "text/html; charset=UTF-8",
  "title": "Example Domain",
  "rendered": false,
  "html": "<!doctype html>…",          // or "text": "…" / "markdown": "…" / "data": { … }
  "truncated": false,
  "meta": {
    "title": "Example Domain", "description": null, "author": null,
    "publishedAt": null, "modifiedAt": null, "lang": "en",
    "canonical": null, "siteName": null, "image": null
  },
  "stats": { "chars": 1256, "words": 180, "tokens": 314 },
  "converter": 7,
  "pageType": { "type": "article", "by": "default" },
  "elapsedMs": 412,
  "credits": 1
}

meta is read from the page head (JSON-LD first, then Open Graph, then plain <meta> and <time>); every field is null when the page does not say. Dates are ISO 8601 when they parse. stats describes the body you got back; tokens is an estimate within about 15% of GPT and Claude tokenizers. converter is the version of the conversion rules that produced the response; it moves only when the same HTML would come out differently, and the Markdown output section below is that version's contract.

pageType is the kind of page the server decided it fetched: article (a title and a body — articles, docs, blog posts), discussion (a post and its comments — forums, Q&A sites, Hacker News threads), listing (a list of items — news fronts, categories, search results) or product (one product). by says how it knows: jsonld, microdata and og mean the page declared it (JSON-LD @type, schema.org itemtype, og:type); heuristic means the URL shape and the page shape agreed (a /products/… address and a price next to an add-to-cart button, an /item?id= address and a column of comments); default means nothing was conclusive. Detection is deliberately conservative — in doubt it says article. Each type has its own Markdown skeleton (see Markdown output). profile pins the type when you know better (by: "param"). A target fetched again keeps the type it was first confidently given, so a heuristic that wavers does not flip it.

blocked: Present only when the target answered with a wall instead of the page: captcha (a CAPTCHA or robot check), challenge (a browser check such as "Just a moment..."), denied (a 403 or an access-denied page), rate-limit (a 429 Too Many Requests). The body is what the target sent, unchanged, and is charged like any page the target answered. It is never kept as a shared copy for max_age, so the next call fetches again.

PDF

A URL that serves a PDF (by content type or by the %PDF- signature) is read with pdf.js instead of the HTML pipeline. text and markdown return the document's text, lines as the PDF lays them out and pages separated by a blank line; html wraps the same text in <pre>; json extraction is not available. meta comes from the document info (title, author, dates), stats.pages is the page count, and max_chars and front_matter work as for HTML. Limits: 5 MB and 300 pages (more pages set truncated: true). A PDF that cannot be read — broken, password-protected or over the size limit — returns FETCH_FAILED and is charged, since the target did send it. Scanned PDFs without a text layer come back empty; there is no OCR.

curl "https://api.webscrapingapi.dev/v1/scrape?url=https://example.com&render=true&waitFor=main&format=text" \
  -H "X-API-Key: wsa_your_key"

Markdown output (v7)

Every page, whatever its HTML, comes back as the same dialect written the same way. The converter removes boilerplate, turns the HTML into GitHub Flavored Markdown, then rebuilds the document from its syntax tree with fixed settings. What you can rely on:

• Main content only by default (main=true): semantic containers (<main>, <article>, role=main) win, and link-heavy blocks inside them (language menus, category lists, tag clouds) are pruned; without a container the whole body is pruned the same way. If a step would leave less than a fifth of the text (a page that is a list of links), that step is undone. Dialogs, modals and hidden elements are always removed, and a page whose content only exists inside <noscript> (Discourse forums) is read from there.
• Headings start at #: the highest level on the page becomes #, the rest keep their relative depth. Empty headings are dropped.
• Tables: data tables are always GFM tables with a header row and a --- separator, every row padded to the header's width; tables without <th> use their first row as the header, and block content inside cells is flattened to one line. Layout tables (nested, single-column, ragged rows, role=presentation) become ordinary blocks instead.
• Code is fenced with a normalized language name (js → javascript, py → python). Highlighters that put each line in a <div> keep their line breaks.
• Links are absolute URLs without title tooltips (links=refs or none change the notation). In-page anchors, javascript: links and empty links become plain text; permalink markers next to headings (¶, #) are dropped.
• Images keep only their description by default, as (image: alt); images=inline keeps ![alt](url). data: URIs and spacer pixels are never emitted.
• Embeds: YouTube, Vimeo, Loom, X, Spotify, CodePen, Google Maps and similar iframes become a link such as [Embedded video](https://www.youtube.com/watch?v=…); other iframes (ads, trackers) are dropped. <video> and <audio> become [Video](url) and [Audio](url).
• Spelling: - bullets, _emphasis_, **strong**, one blank line between blocks, no trailing whitespace. Underscores inside words are not escaped (snake_case stays readable).
• Listing pages (pageType.type: "listing" — news fronts, categories, search results, product lists) come as # title and then one line per item: - [title](url) — summary · meta. The summary is the item's own text, up to 200 characters; the meta are, always in this order and only when present, score, author, date (the <time> value as YYYY-MM-DD when the page has one), comment count and source domain. Several lists on one page keep their headings as ##; an item that appears twice (mobile and desktop) appears once; thumbnails are dropped. When too few items can be told apart, or they hold too little of the page's text — five items and 40% when the type was inferred from the page's shape, 30% when the page declared itself a list, three items and 15% with profile=listing — the article rules apply instead, and profile=article forces them.
• Discussion pages (pageType.type: "discussion" — forum topics, Q&A, Hacker News threads, posts with comments) come as # title, a **author** · date · score line and the original post, then ## Comments (n) with one list item per comment — - **author** · date followed by the comment's own Markdown — and replies nested one level deeper each. Dates are YYYY-MM-DD when the page gives a timestamp; reply, flag, share and vote controls are dropped. When fewer than two comments can be told apart the article rules apply (one is enough with profile=discussion).
• Product pages (pageType.type: "product") come as # name, then one fact line in a fixed order — **Price:** (in the page's currency, formatted the same way everywhere, a range when variants differ) · **Brand:** · **Availability:** (In stock, Out of stock, Pre-order…) · **Rating:** (4.6/5 (2,345 reviews)) · **SKU:** · **Variants:** — only the facts the page states, read from JSON-LD, then microdata, Open Graph and the page itself — followed by the description. When the page marks a description container (description, details, about) that block is used and specification blocks (specs, system requirements, key/value tables) follow under ## Specifications; otherwise the main content is converted as an article. When no price can be found the article rules apply; profile=product forces the skeleton.
• Deterministic: the same HTML always produces the same Markdown, so a page that did not change gives the same output every time.

Extraction rules

Each field is either a CSS selector string (returns the text of the first match) or an object:

curl -X POST https://api.webscrapingapi.dev/v1/scrape \
  -H "X-API-Key: wsa_your_key" -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "format": "json",
    "extract": {
      "title": "title",
      "stories": { "selector": ".titleline > a", "all": true },
      "links":   { "selector": ".titleline > a", "attr": "href", "all": true }
    }
  }'

You can combine extract with format=html or text to get both the body and the structured fields in one response. Up to 50 fields per request.

GET /v1/metadata

Everything in <head> that matters: title, description, canonical, favicon, language, Open Graph, Twitter cards and JSON-LD. Only the first 512 KB of the page is read, so it is fast.

curl "https://api.webscrapingapi.dev/v1/metadata?url=https://github.com" -H "X-API-Key: wsa_your_key"

{
  "url": "https://github.com",
  "finalUrl": "https://github.com/",
  "title": "GitHub · Build and ship software on a single, collaborative platform",
  "description": "…",
  "canonical": "https://github.com/",
  "favicon": "https://github.githubassets.com/favicons/favicon.svg",
  "lang": "en",
  "og": { "title": "…", "image": "…" },
  "twitter": { "card": "summary_large_image" },
  "jsonLd": [ … ],
  "elapsedMs": 380,
  "credits": 1
}

Site-specific APIs

Endpoints that already know a site's layout and return clean JSON. The registry is public at GET /v1/structured; each site has its own page with parameters and examples. A parameter the site does not define is refused with 400 instead of being ignored. Results are cached and shared: a response served from the store costs 0 credits and carries source and age; max_age (seconds) sets how old a stored copy may be, and max_age=0 forces a fresh fetch.

curl "https://api.webscrapingapi.dev/v1/structured/amazon/product?asin=B0B44XTV71&tld=com&currency=USD" -H "X-API-Key: wsa_your_key"

Jobs & endpoints

A job runs a list of targets — URLs or site calls — once, in the background. You get a job_ id right away and read the results when they are ready, or have them posted to your endpoint. A job never repeats; to fetch again, create a new job. Rendering gets up to 60 s per page, twice the 30 s a synchronous request allows, so pages that render slowly belong here. Up to 50 targets per job and 10 unfinished jobs per account; results are kept 30 days. Each target costs what the same synchronous request would (1 credit per plain fetch, 5 per render, 1 per site call); reading a job costs 0.

curl -X POST https://api.webscrapingapi.dev/v1/jobs \
  -H "X-API-Key: wsa_your_key" -H "content-type: application/json" \
  -d '{ "name": "Pricing pages", "endpoint": "ep_your_endpoint",
        "targets": ["https://a.com/pricing", { "url": "https://b.com/app", "render": true }, "amazon/product?asin=B0B44XTV71"] }'

The answer is 202 with the job: status (queued, running, completed), counts per state and one item per target (pending, running, ok, error). A target is a string (https://… for a URL fetched as Markdown, or site/action?params) or an object: { url, …options } takes the same options as a synchronous request (the table above) except raw, since a job answers in JSON, and max_age, which is set on the whole job; { site, action, params } calls a site endpoint. A target with headers (cookies, for example) is private: it is never merged with other targets or shared, max_age does not apply to it, its headers wait encrypted in the job queue only until it is fetched, and its result is kept with this job alone (private: true on the item). Unknown option names, in a target or in the job body itself, are refused with 400. endpoint is optional. Add max_age (seconds) to reuse a result anyone fetched within that time for 0 credits (source: "shared"); without it every target is fetched fresh.

The delivery. When every target has finished, the job posts once to its endpoint: { "type": "job.completed", "job": {…}, "items": […], "resultsIncluded": true, "results": […], "resultsUrl": "…" }. If all results together exceed 5 MB, results is null — read them from resultsUrl. Answer with any 2xx within 10 s; otherwise it is tried again, 3 attempts in all, 60 s apart. The job shows the delivery state.

Verify the signature. Each delivery carries WSA-Signature: t=<unix seconds>,v1=<hex>. Compute HMAC-SHA256 of <t>.<raw body> with your secret, compare it with v1, and reject old timestamps.

import crypto from 'node:crypto';

// rawBody: the request body exactly as received (not re-serialized JSON)
function verify(rawBody, header, secret) {
  const { t, v1 } = Object.fromEntries(header.split(',').map((p) => p.split('=')));
  const expected = crypto.createHmac('sha256', secret).update(`${t}.${rawBody}`).digest('hex');
  const fresh = Math.abs(Date.now() / 1000 - Number(t)) < 300;
  return fresh && expected.length === v1.length && crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(v1));
}

How results are shared. A target exists once, however many jobs include it, and every successful fetch of it is stored; a failed fetch stores nothing and leaves the last good result in place. A job with max_age reads the newest stored result within that age for 0 credits, by the same rule as /v1/scrape (at most 30 days; a larger value counts as 30); a job without it fetches fresh and stores the result for the next caller.

Errors

Errors use HTTP status codes and a JSON body: { "error": { "code": "…", "message": "…" } }.

Board API (public read)

The community board is readable without a key: GET /v1/board/posts (paginated with cursor) and GET /v1/board/posts/:id. Writing requires a signed-in account on this site.