Blog

Notes on getting data out of web pages.

How the site endpoints work, what they return, and the scraping decisions behind them. Written by the person who runs the service; no guest posts, no filler.

RSS feed · 12 articles

caching & creditsgeneric scrapegetting startedhow it worksmedia & communitiesrag & agentsreliabilityshopping & appssite endpoints

  1. Convert any web page to Markdown for an LLM with one request (format=markdown)

    Add format=markdown to /v1/scrape: headings, lists, tables, code and absolute links come back, scripts and forms do not. Made for prompts and RAG chunks.

    · 2 min read · generic scrape · rag & agents

  2. Hacker News front page and comment threads as JSON, ranked and flattened

    Front page stories in rank order with points and comment counts, or one thread with its comments flattened and a depth on each. Two GET requests.

    · 2 min read · site endpoints · media & communities

  3. CSS selectors in, JSON out: the extract parameter explained with examples

    Name fields as CSS selectors and /v1/scrape returns them as JSON: attributes, lists, inner HTML, absolute links, up to 50 fields, no parsing code.

    · 2 min read · generic scrape · getting started

  4. GitHub stars, forks, topics and license for any public repo, with no token and no 60-an-hour limit

    GitHub's REST API allows 60 anonymous requests an hour. This endpoint reads the public repo page: stars, forks, watchers, topics, license, releases.

    · 1 min read · site endpoints · caching & credits

  5. Scraping API error codes explained: what each one means, whether to retry, and what it costs

    Every error is a status plus a short code and a sentence. The full list: the cause of each one, whether a retry helps, and which ones refund the credit.

    · 3 min read · reliability · how it works · caching & credits

  6. Read Open Graph, Twitter card and JSON-LD metadata from any URL with one request

    /v1/metadata returns title, description, canonical, favicon, language, Open Graph and Twitter tags and every JSON-LD block from the head alone. One credit.

    · 2 min read · generic scrape · getting started

  7. Wikipedia summary and infobox as JSON, in any language edition, for RAG and agents

    Lead paragraphs as plain text, the thumbnail and the infobox as key/value pairs, in any language edition, from one request. Stored 7 days.

    · 2 min read · site endpoints · rag & agents

  8. How to get Amazon product data as JSON without the Product Advertising API

    Turn an ASIN into JSON with one GET: title, price, availability, rating, images, bullets and the spec table. No Amazon developer account or affiliate approval.

    · 3 min read · site endpoints · shopping & apps · reliability

  9. App Store and Google Play app data as JSON: ratings, installs and price with one schema for both stores

    Two endpoints, one shape: an iOS app by id or an Android app by package name, returning name, developer, rating, price, category, screenshots and installs.

    · 3 min read · site endpoints · shopping & apps

  10. What max_age means in a scraping API, and why a stored answer costs 0 credits

    Site endpoints store answers and share them. max_age sets how old a stored copy may be; every response says whether it was live or stored, and how old it is.

    · 2 min read · caching & credits · how it works

  11. Web scraping API vs writing your own scraper: which one for a side project, an agent, or a data pipeline

    Fetching pages yourself versus calling a scraping API: headers, JavaScript rendering, blocked sites, parsing, cost, and the cases where each one wins.

    · 3 min read · getting started · how it works · rag & agents

  12. Get YouTube video metadata (views, likes, duration) with one GET request, no Google API key

    Title, channel, views, likes, length, publish date, thumbnails and keywords for any public video, from one request. And when the official Data API is better.

    · 2 min read · site endpoints · media & communities