Convert any web page to Markdown for an LLM with one request (format=markdown)

The cheapest way to make a model read a web page is to hand it the page as Markdown: the headings tell it what the sections are, the lists and tables keep their rows, the code stays in fences, and none of the tokens go to <div class="…">. format=markdown does that conversion on the API side.

The request

curl "https://api.webscrapingapi.dev/v1/scrape?url=https://example.com&format=markdown" \
  -H "X-API-Key: wsa_your_key"

{
  "url": "https://example.com",
  "finalUrl": "https://example.com/",
  "statusCode": 200,
  "contentType": "text/html; charset=UTF-8",
  "title": "Example Domain",
  "rendered": false,
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents. …\n\n[More information...](https://www.iana.org/domains/example)",
  "truncated": false,
  "elapsedMs": 310,
  "credits": 1
}

Add raw=1 and the same request returns only the Markdown, as text/markdown, which is convenient when the next thing in your pipeline is a file or a prompt rather than a JSON parser. The status code and final URL move to x-scrape-status and x-scrape-final-url headers.

What the conversion keeps and drops

Kept	Dropped
Headings as #, ##, ###	<script>, <style>, <noscript>, <template>
Bullet and numbered lists, nested	Forms, buttons, inputs
Tables as GitHub-flavoured Markdown	Inline SVG, canvas, iframes
Fenced code blocks and inline code	Elements marked aria-hidden="true"
Bold, italic, strikethrough, horizontal rules	Runs of more than two blank lines
Links and images, rewritten to absolute URLs

Relative links are the detail that matters most in practice. A page’s /docs/install becomes https://site.example/docs/install, so a model that decides to follow the link, or a citation you show a user, still works outside the page.

Narrow it with selector

Whole pages carry navigation, footers and cookie banners. Pass a CSS selector for the part you want and only that element is converted.

curl "https://api.webscrapingapi.dev/v1/scrape?url=https://en.wikipedia.org/wiki/Web_scraping&format=markdown&selector=%23mw-content-text&raw=1" \
  -H "X-API-Key: wsa_your_key"

article, main, #content and .post-body cover a large share of the web. When the page has none of them, the whole body is converted and you trim on your side.

Pages that need a browser

Some sites render their content with JavaScript and send an almost empty HTML shell. If the Markdown comes back with little more than a title, add render=true and, when the content appears late, waitFor=<selector>. Rendering costs 5 credits instead of 1 because a real Chrome runs the page; use it only for the pages that need it.

Chunking for retrieval

Because headings survive, you can split the Markdown on ## boundaries and get chunks that correspond to the page’s own sections, each with a title you can embed alongside the text. That is a better retrieval unit than a fixed window of characters, and it costs nothing extra: it is just the structure the page already had, kept.

The scrape reference lists every parameter.