CSS selectors in, JSON out: the extract parameter explained with examples

Parsing is where scrapers accumulate code. The extract parameter moves it into the request: you name fields, you give each one a CSS selector, and the response has the values. There is no expression language to learn; if you can write h1 or a.next[href], you can write an extract rule.

The shape

{
  "url": "https://example.com",
  "format": "json",
  "extract": {
    "title": "h1",
    "moreInfo": { "selector": "a", "attr": "href" },
    "paragraphs": { "selector": "p", "all": true },
    "body": { "selector": "div", "html": true }
  }
}

Each rule is either a string, meaning “the text of the first element that matches”, or an object:

Key	Meaning
selector	any CSS selector cheerio understands: tags, classes, ids, attributes, :nth-child, combinators
attr	return this attribute instead of the text; href and src are resolved to absolute URLs
all	return every match as an array instead of the first match
html	return the element’s inner HTML instead of its text

Text values have whitespace collapsed and are trimmed, so a title wrapped across three lines of HTML comes back as one line.

The request and the answer

curl -X POST "https://api.webscrapingapi.dev/v1/scrape" \
  -H "X-API-Key: wsa_your_key" -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","format":"json","extract":{"title":"h1","links":{"selector":"a","attr":"href","all":true}}}'

{
  "url": "https://example.com",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "contentType": "text/html; charset=UTF-8",
  "title": "Example Domain",
  "rendered": false,
  "data": {
    "title": "Example Domain",
    "links": ["https://www.iana.org/domains/example"]
  },
  "truncated": false,
  "elapsedMs": 306,
  "credits": 1
}

With format=json the response carries only data. With format=html, text or markdown you get the body and data together, which is handy when one field is a quick sanity check on a page you are also storing.

Three patterns that cover most pages

A product card. Title, price and image, each the first match inside a known container:

{ "name": ".product h1", "price": ".product .price", "image": { "selector": ".product img", "attr": "src" } }

A list page. Every item’s link and label as parallel arrays:

{ "titles": { "selector": "article h2", "all": true }, "urls": { "selector": "article h2 a", "attr": "href", "all": true } }

The two arrays line up index by index because both selectors visit the same articles in document order.

A block you want to process yourself. The inner HTML of one element, when the structure inside is irregular:

{ "table": { "selector": "table.specs", "html": true } }

When a page changes

Selectors break when layouts change; that is true of every scraper ever written. Two habits keep the damage small. Prefer ids and semantic elements (article h1, [itemprop=price]) over generated class names (.css-1x2y3z). And check for null: a field that was always present and is suddenly null on every page is the signal that the layout moved, and the response’s title and statusCode tell you whether you fetched the right page at all.

The extraction reference has the limits and one more example.