# webscrapingapi.dev

> webscrapingapi.dev is a free web scraping API for developers and AI agents. Send a URL and get back HTML, plain text, markdown, JSON extracted with CSS selectors, or page metadata. Site-specific endpoints return clean JSON for sites the service already knows. When a site cannot be scraped with a plain request, users post it on the board and the maintainer builds an endpoint for it or explains why it cannot be done. Everything is free: 50,000 credits per account per day, no card.

## Facts

- Base URL: https://api.webscrapingapi.dev
- Authentication: API key in the `X-API-Key` header (also `Authorization: Bearer` or `?api_key=`). Keys are created in the dashboard at https://app.webscrapingapi.dev.
- Free quota: 50,000 credits per account per day, resets 00:00 UTC. A plain fetch costs 1 credit, a JavaScript render 5, metadata and site endpoints 1.
- Rate limit: 60 requests per minute per key; up to 5 keys per account.
- Limits: 30 s max timeout, 5 MB max body. Private and reserved network addresses are blocked.
- Raw pages are never stored. Structured results are cached and shared; responses say how old the data is.
- No CAPTCHA solving and no anti-bot bypass. Sites that cannot be reached are marked "Not possible" on the board.

## Endpoints

- `GET/POST https://api.webscrapingapi.dev/v1/scrape` — params: url (required), render (bool), waitFor (CSS selector), wait (ms), timeout (ms), format (html | text | json), selector, extract (object of field → CSS selector or { selector, attr, all, html }), raw. Docs: https://webscrapingapi.dev/docs#scrape
- `GET https://api.webscrapingapi.dev/v1/metadata?url=` — title, description, canonical, favicon, lang, Open Graph, Twitter card, JSON-LD. Docs: https://webscrapingapi.dev/docs#metadata
- `GET https://api.webscrapingapi.dev/v1/structured` — public registry of site-specific endpoints (JSON).
- `GET https://api.webscrapingapi.dev/v1/health` — health check, no key needed.
- `GET https://api.webscrapingapi.dev/v1/board/posts` — public board posts and site requests (JSON).

## Site-specific endpoints

### Amazon product (amazon.com · 20 marketplaces)
One product by ASIN as JSON: title, price, availability, rating, images, bullet points and the spec table.
- `GET https://api.webscrapingapi.dev/v1/structured/amazon/product` — The product detail page as JSON. Params: asin (required), tld, currency. Returns { product: { name, brand, price, regularPrice, currency, availability, aggregateRating, mainImage, images[], features[], additionalProperties[], breadcrumbs[], sku }, extra: { asin, tld, bestSellersRank[], soldBy, shipsFrom, boughtInPastMonth, availabilityText } }. Cached 86400s by default; max_age=0 forces a fresh fetch. Docs: https://webscrapingapi.dev/docs/sites/amazon

### YouTube video (youtube.com)
A video by id as JSON: title, channel, views, likes, duration, publish date, description, thumbnails and keywords.
- `GET https://api.webscrapingapi.dev/v1/structured/youtube/video` — The watch page metadata as JSON. Params: id (required). Returns { video: { id, url, title, description, channel { id, name, url }, durationSeconds, viewCount, likeCount, publishedAt, category, thumbnails[], keywords[], isLive, familySafe } }. Cached 3600s by default; max_age=0 forces a fresh fetch. Docs: https://webscrapingapi.dev/docs/sites/youtube

### App Store app (apps.apple.com)
An iOS app by id as JSON: name, developer, rating, price, category, screenshots and description.
- `GET https://api.webscrapingapi.dev/v1/structured/appstore/app` — The listing page as JSON. Params: id (required), country. Returns { app: { name, url, platform: "ios", id, developer { name, url }, description, category, price, currency, free, rating { value, count }, icon, screenshots[], contentRating, osRequirement, installs: null, updatedAt, version } }. Cached 86400s by default; max_age=0 forces a fresh fetch. Docs: https://webscrapingapi.dev/docs/sites/appstore

### Google Play app (play.google.com)
An Android app by package name as JSON: name, developer, rating, installs, price, category, content rating and description.
- `GET https://api.webscrapingapi.dev/v1/structured/googleplay/app` — The listing page as JSON. Params: id (required), hl, gl. Returns { app: { name, url, platform: "android", id, developer { name, url }, description, category, price, currency, free, rating { value, count }, icon, screenshots[], contentRating, osRequirement: null, installs, updatedAt, version } }. Cached 86400s by default; max_age=0 forces a fresh fetch. Docs: https://webscrapingapi.dev/docs/sites/googleplay

### Hacker News (news.ycombinator.com)
Front page stories and item threads as clean JSON — no HTML parsing on your side.
- `GET https://api.webscrapingapi.dev/v1/structured/hackernews/front` — Stories on the front page, in rank order. Params: page. Returns stories[] { id, rank, title, url, site, points, author, commentCount, age }. Cached 600s by default; max_age=0 forces a fresh fetch. Docs: https://webscrapingapi.dev/docs/sites/hackernews
- `GET https://api.webscrapingapi.dev/v1/structured/hackernews/item` — A story with its comment thread (flattened, with depth). Params: id (required). Returns { id, title, url, points, author, age, comments[] { id, author, age, depth, text } }. Cached 600s by default; max_age=0 forces a fresh fetch. Docs: https://webscrapingapi.dev/docs/sites/hackernews

### Wikipedia (wikipedia.org)
Article title, summary paragraphs, thumbnail and infobox as JSON, in any language edition.
- `GET https://api.webscrapingapi.dev/v1/structured/wikipedia/summary` — Lead section of an article plus infobox key/values. Params: title (required), lang. Returns { title, url, lang, summary, paragraphs[], thumbnail, infobox {} }. Cached 604800s by default; max_age=0 forces a fresh fetch. Docs: https://webscrapingapi.dev/docs/sites/wikipedia

### GitHub (github.com)
Repository facts (stars, forks, topics, license, description) scraped from the public page — no token, no API rate limit.
- `GET https://api.webscrapingapi.dev/v1/structured/github/repo` — Public repository overview. Params: owner (required), repo (required). Returns { owner, repo, url, description, homepage, stars, forks, watchers, topics[], license, defaultBranch, archived, fork, releaseCount, language (null on the current GitHub layout) }. Cached 3600s by default; max_age=0 forces a fresh fetch. Docs: https://webscrapingapi.dev/docs/sites/github

## Pages

- [Documentation](https://webscrapingapi.dev/docs): authentication, credits, every parameter, error codes.
- [Site endpoints](https://webscrapingapi.dev/sites): the list of sites with ready-made JSON endpoints.
- [Requests & board](https://webscrapingapi.dev/board): ask for a site; see which requests are reviewing, building, done or not possible.
- [Dashboard](https://app.webscrapingapi.dev): create keys, see usage, try requests in the playground.
- [Blog](https://webscrapingapi.dev/blog): practical articles on the endpoints and on scraping trade-offs (RSS: https://webscrapingapi.dev/blog/rss.xml).
- [About](https://webscrapingapi.dev/about): who builds and runs the service, why it is free, what is logged.
- [OpenAPI spec](https://webscrapingapi.dev/openapi.json): machine-readable description of every endpoint, parameter and error code.
- [llms-full.txt](https://webscrapingapi.dev/llms-full.txt): this overview plus every site endpoint's full reference and every blog article, in one file.
- [Terms](https://webscrapingapi.dev/terms) · [Privacy](https://webscrapingapi.dev/privacy)

## Blog articles

- [Convert any web page to Markdown for an LLM with one request (format=markdown)](https://webscrapingapi.dev/blog/convert-any-web-page-to-markdown-for-llm): Add format=markdown to /v1/scrape: headings, lists, tables, code and absolute links come back, scripts and forms do not. Made for prompts and RAG chunks.
- [Hacker News front page and comment threads as JSON, ranked and flattened](https://webscrapingapi.dev/blog/hacker-news-front-page-and-comments-as-json): Front page stories in rank order with points and comment counts, or one thread with its comments flattened and a depth on each. Two GET requests.
- [CSS selectors in, JSON out: the extract parameter explained with examples](https://webscrapingapi.dev/blog/css-selectors-to-json-extract-parameter): Name fields as CSS selectors and /v1/scrape returns them as JSON: attributes, lists, inner HTML, absolute links, up to 50 fields, no parsing code.
- [GitHub stars, forks, topics and license for any public repo, with no token and no 60-an-hour limit](https://webscrapingapi.dev/blog/github-repo-stars-forks-topics-without-token): GitHub's REST API allows 60 anonymous requests an hour. This endpoint reads the public repo page: stars, forks, watchers, topics, license, releases.
- [Scraping API error codes explained: what each one means, whether to retry, and what it costs](https://webscrapingapi.dev/blog/scraping-api-error-codes-explained): Every error is a status plus a short code and a sentence. The full list: the cause of each one, whether a retry helps, and which ones refund the credit.
- [Read Open Graph, Twitter card and JSON-LD metadata from any URL with one request](https://webscrapingapi.dev/blog/read-open-graph-json-ld-metadata-from-any-url): /v1/metadata returns title, description, canonical, favicon, language, Open Graph and Twitter tags and every JSON-LD block from the head alone. One credit.
- [Wikipedia summary and infobox as JSON, in any language edition, for RAG and agents](https://webscrapingapi.dev/blog/wikipedia-summary-and-infobox-api-any-language): Lead paragraphs as plain text, the thumbnail and the infobox as key/value pairs, in any language edition, from one request. Stored 7 days.
- [How to get Amazon product data as JSON without the Product Advertising API](https://webscrapingapi.dev/blog/amazon-product-data-json-without-product-advertising-api): Turn an ASIN into JSON with one GET: title, price, availability, rating, images, bullets and the spec table. No Amazon developer account or affiliate approval.
- [App Store and Google Play app data as JSON: ratings, installs and price with one schema for both stores](https://webscrapingapi.dev/blog/app-store-google-play-app-data-json): Two endpoints, one shape: an iOS app by id or an Android app by package name, returning name, developer, rating, price, category, screenshots and installs.
- [What max_age means in a scraping API, and why a stored answer costs 0 credits](https://webscrapingapi.dev/blog/what-max-age-means-in-a-scraping-api): Site endpoints store answers and share them. max_age sets how old a stored copy may be; every response says whether it was live or stored, and how old it is.
- [Web scraping API vs writing your own scraper: which one for a side project, an agent, or a data pipeline](https://webscrapingapi.dev/blog/web-scraping-api-vs-writing-your-own-scraper): Fetching pages yourself versus calling a scraping API: headers, JavaScript rendering, blocked sites, parsing, cost, and the cases where each one wins.
- [Get YouTube video metadata (views, likes, duration) with one GET request, no Google API key](https://webscrapingapi.dev/blog/youtube-video-metadata-without-api-key): Title, channel, views, likes, length, publish date, thumbnails and keywords for any public video, from one request. And when the official Data API is better.

## Errors

Errors are JSON: `{ "error": { "code": "...", "message": "..." } }`. Codes: BAD_REQUEST, URL_BLOCKED, UNAUTHORIZED, FORBIDDEN, NOT_FOUND, RATE_LIMITED, QUOTA_EXCEEDED, FETCH_FAILED, SITE_PARSE_FAILED, RENDER_UNAVAILABLE, TIMEOUT, INTERNAL_ERROR.