# Reader

## Docs

- [Introduction](https://reader.mintlify.app/home/introduction.md): The web infrastructure platform for AI. Scrape, crawl, extract, and automate the web from a single API.
- [Quickstart](https://reader.mintlify.app/home/quickstart.md): Make your first Reader request in under a minute.
- [CLI](https://reader.mintlify.app/home/cli.md): Read the web from your terminal. Built for AI coding agents.
- [MCP server](https://reader.mintlify.app/home/mcp-server.md): Give AI coding tools the ability to read the web with the Reader MCP server.
- [The read primitive](https://reader.mintlify.app/home/concepts/read-primitive.md): One endpoint, three shapes of input. Scrape, batch, and crawl all flow through POST /v1/read.
- [Scrape, crawl, and discover](https://reader.mintlify.app/home/concepts/scrape-vs-crawl.md): Scrape fetches URLs you already know. Crawl discovers and scrapes. Discover finds URLs without scraping. All from the same endpoint.
- [Browser Sessions](https://reader.mintlify.app/home/concepts/browser-sessions.md): Launch stealthed browsers for full automation with Playwright and Puppeteer
- [Formats and extraction](https://reader.mintlify.app/home/concepts/formats-and-extraction.md): What Reader returns, how to shape it, and how to trim boilerplate.
- [Extract](https://reader.mintlify.app/home/concepts/extract.md): Pull structured data from any web page using an LLM
- [Proxy modes](https://reader.mintlify.app/home/concepts/proxy-modes.md): Two ways Reader can fetch a page: standard and premium. Pick one, or use the default.
- [Caching](https://reader.mintlify.app/home/concepts/caching.md): Free cache hits, 24-hour default TTL, and how to opt out.
- [Credits and billing](https://reader.mintlify.app/home/concepts/credits-and-billing.md): What each Reader operation costs and how to check your balance.
- [Rate limits and concurrency](https://reader.mintlify.app/home/concepts/rate-limits.md): Two limits: requests per minute and concurrent async jobs. How Reader enforces them and how to back off.
- [Async jobs](https://reader.mintlify.app/home/concepts/async-jobs.md): Batch and crawl return jobs. Poll them, stream them, or let a webhook call you when they're done.
- [Events](https://reader.mintlify.app/home/concepts/events.md): Server-sent streams for real-time updates on one job or across your whole workspace.
- [Webhooks](https://reader.mintlify.app/home/concepts/webhooks.md): Push notifications when jobs finish. Signed, retried, and verifiable.
- [Errors](https://reader.mintlify.app/home/concepts/errors.md): Eleven stable error codes. Branch on code, not on message.
- [Your first scrape](https://reader.mintlify.app/home/guides/getting-started/first-scrape.md): Make a single-URL request and understand every field in the response.
- [Markdown and HTML together](https://reader.mintlify.app/home/guides/getting-started/markdown-and-html.md): Request both formats in one call and pick what you need downstream.
- [Main content extraction](https://reader.mintlify.app/home/guides/getting-started/main-content.md): How Reader strips boilerplate and when to turn it off.
- [Include and exclude selectors](https://reader.mintlify.app/home/guides/getting-started/selectors.md): Precise control over what Reader keeps and drops using CSS selectors.
- [Polling a job to completion](https://reader.mintlify.app/home/guides/async-workflows/polling-jobs.md): The simplest async pattern: fetch status on an interval until terminal.
- [Streaming jobs with SSE](https://reader.mintlify.app/home/guides/async-workflows/sse-streaming.md): Real-time progress from a single open HTTP connection.
- [Polling vs SSE vs webhooks](https://reader.mintlify.app/home/guides/async-workflows/polling-vs-sse-vs-webhooks.md): The same job can be watched three ways. Which one fits your use case?
- [Webhook-driven workflows](https://reader.mintlify.app/home/guides/async-workflows/webhook-workflows.md): Trigger downstream work when a job completes: database writes, emails, re-indexing.
- [Verifying webhook signatures](https://reader.mintlify.app/home/guides/async-workflows/verifying-webhooks.md): HMAC-SHA256 verification for Express, FastAPI, and Next.js.
- [Webhook retries and idempotency](https://reader.mintlify.app/home/guides/async-workflows/webhook-retries.md): Reader retries failed deliveries. Your handler needs to be idempotent.
- [Building a progress bar](https://reader.mintlify.app/home/guides/async-workflows/progress-bars.md): Wire SSE events to a CLI or UI progress indicator.
- [Reliable batch processing](https://reader.mintlify.app/home/guides/async-workflows/reliable-batch.md): Submit thousands of URLs, survive reconnects, and never lose a result.
- [Retry and error handling](https://reader.mintlify.app/home/guides/production/retry-error-handling.md): Which error codes to retry, how to back off, and when to give up.
- [Handling rate limits](https://reader.mintlify.app/home/guides/production/rate-limits.md): Design for your tier's RPM, back off gracefully, and avoid the common footguns.
- [Handling credit exhaustion](https://reader.mintlify.app/home/guides/production/credit-exhaustion.md): Detect low balance before you run out and degrade gracefully when you do.
- [Monitoring Reader in production](https://reader.mintlify.app/home/guides/production/monitoring.md): The metrics worth tracking, where to get them, and what to alert on.
- [Cost estimation](https://reader.mintlify.app/home/guides/production/cost-estimation.md): Predict what a batch or crawl will cost before you run it.
- [Batch scraping](https://reader.mintlify.app/home/guides/advanced/batch-scraping.md): Submit many URLs in one call, handle results efficiently, and stay within your limits.
- [Crawling a website](https://reader.mintlify.app/home/guides/advanced/crawling.md): Start with one URL, let Reader discover the rest.
- [Crawl + scrape in one call](https://reader.mintlify.app/home/guides/advanced/crawl-with-scrape.md): Discover links and fetch full page content in a single job.
- [Waiting for dynamic content](https://reader.mintlify.app/home/guides/advanced/dynamic-content.md): Use waitForSelector when content is hydrated client-side after initial load.
- [Auth walls and hostile sites](https://reader.mintlify.app/home/guides/advanced/auth-walls.md): When to use premium mode, what Reader can't do, and how to spot a site that won't work.
- [Choosing a proxy mode](https://reader.mintlify.app/home/guides/advanced/choosing-a-proxy-mode.md): A practical decision guide: when to use standard vs premium for different sites.
- [Feeding Reader output to an LLM](https://reader.mintlify.app/home/guides/llm/llm-pipeline.md): From URL to model prompt: the minimal pipeline and what to watch out for.
- [RAG: scrape, chunk, embed](https://reader.mintlify.app/home/guides/llm/rag.md): Build a retrieval-augmented pipeline from URL to vector database.
- [Reader as an agent tool](https://reader.mintlify.app/home/guides/llm/agent-tool.md): Expose Reader via LLM tool calling so your agent can fetch pages on demand.
- [Structured data extraction](https://reader.mintlify.app/home/guides/llm/structured-data.md): From markdown to JSON. Extract fields with an LLM on top of Reader.
- [Structured Extraction](https://reader.mintlify.app/home/guides/llm/structured-extraction.md): Extract structured data from any web page with schemas or prompts
- [Stuck jobs](https://reader.mintlify.app/home/guides/troubleshooting/stuck-jobs.md): What to do when a job sits in `processing` longer than you expect.
- [Cache not hitting](https://reader.mintlify.app/home/guides/troubleshooting/cache-miss.md): Why repeated requests for the same URL still cost credits, and how to fix it.
- [URL blocked errors](https://reader.mintlify.app/home/guides/troubleshooting/ssrf-blocks.md): Why Reader refuses some URLs and what to do about it.
- [Cloudflare and bot walls](https://reader.mintlify.app/home/guides/troubleshooting/cloudflare.md): When Reader gets through automatically, when to use premium, and when a site is out of reach.
- [SDKs](https://reader.mintlify.app/sdk/overview.md): Official SDKs for the Reader API in JavaScript/TypeScript and Python.
- [JavaScript SDK](https://reader.mintlify.app/sdk/javascript.md): Install and use @vakra-dev/reader-js
- [Python SDK](https://reader.mintlify.app/sdk/python.md): Install and use reader-py
- [Authentication](https://reader.mintlify.app/api-reference/authentication.md): Every Reader API request is authenticated with an API key passed in the x-api-key header.
- [Scrape, batch, crawl, or discover](https://reader.mintlify.app/api-reference/read.md): Unified endpoint for all read operations. Pass `url` for a single scrape, `urls` for a batch, or `url` + `maxPages` to crawl a site.
- [Get a job](https://reader.mintlify.app/api-reference/jobs/get.md)
- [Cancel a job](https://reader.mintlify.app/api-reference/jobs/cancel.md)
- [Retry failed URLs](https://reader.mintlify.app/api-reference/jobs/retry.md)
- [Per-job events stream (SSE)](https://reader.mintlify.app/api-reference/jobs/stream.md): Server-Sent Events stream scoped to a single job. Emits `progress`, `page`, `error`, and `done` events as the job makes progress. Stream closes automatically when the job reaches a terminal state.
- [Events stream (SSE)](https://reader.mintlify.app/api-reference/jobs/events.md): Long-lived Server-Sent Events connection that emits real-time progress, completion, and failure events for every job in the workspace. Use this for dashboards and realtime UIs. For a single job, use `GET /v1/jobs/{id}/stream` instead.
- [Create a webhook](https://reader.mintlify.app/api-reference/webhooks/create.md)
- [List webhooks](https://reader.mintlify.app/api-reference/webhooks/list.md)
- [Update a webhook](https://reader.mintlify.app/api-reference/webhooks/update.md)
- [Delete a webhook](https://reader.mintlify.app/api-reference/webhooks/delete.md)
- [Create Session](https://reader.mintlify.app/api-reference/sessions/create.md): Create a browser session with a CDP WebSocket endpoint
- [Get Session](https://reader.mintlify.app/api-reference/sessions/get.md): Get the status of a browser session
- [Stop Session](https://reader.mintlify.app/api-reference/sessions/stop.md): Stop a browser session and get billing summary
- [List Sessions](https://reader.mintlify.app/api-reference/sessions/list.md): List active browser sessions for your workspace
- [CDP WebSocket Proxy](https://reader.mintlify.app/api-reference/sessions/cdp.md): Connect Playwright/Puppeteer to a browser session via CDP
- [Get credit balance](https://reader.mintlify.app/api-reference/account/credits.md)
- [Usage history](https://reader.mintlify.app/api-reference/account/history.md)
- [Queue status](https://reader.mintlify.app/api-reference/account/queue-status.md): Returns the workspace's concurrency state: active browser slots in use, waiting jobs, and the plan's max concurrency.
- [Introduction](https://reader.mintlify.app/self-hosted/getting-started/introduction.md): Run the Reader scraping engine yourself - open-source Node library, CLI, and deployment scripts.
- [Installation](https://reader.mintlify.app/self-hosted/getting-started/installation.md): Install @vakra-dev/reader and its system dependencies.
- [Quickstart](https://reader.mintlify.app/self-hosted/getting-started/quickstart.md): Make your first scrape with self-hosted Reader in 60 seconds.
- [Examples](https://reader.mintlify.app/self-hosted/getting-started/examples.md): Runnable code examples for every feature of self-hosted Reader.
- [Scraping](https://reader.mintlify.app/self-hosted/concepts/scraping.md): How Reader turns a URL into clean, LLM-ready content.
- [Crawling](https://reader.mintlify.app/self-hosted/concepts/crawling.md): How Reader discovers and optionally scrapes every page on a site.
- [Content Extraction](https://reader.mintlify.app/self-hosted/concepts/content-extraction.md): How Reader identifies main content and strips noise from scraped pages.
- [Browser Pool](https://reader.mintlify.app/self-hosted/concepts/browser-pool.md): How Reader manages long-lived browser instances with recycling and health checks.
- [Browser Sessions](https://reader.mintlify.app/self-hosted/concepts/browser-sessions.md): Launch stealthed Chrome for Playwright/Puppeteer automation
- [Scraping Engine](https://reader.mintlify.app/self-hosted/concepts/engine-waterfall.md): Playwright browser engine with proxy tier support for reliable scraping.
- [Proxy Tiers](https://reader.mintlify.app/self-hosted/concepts/proxy-tiers.md): Standard vs premium proxies: datacenter and residential, rotation, and when to use each.
- [Error Handling](https://reader.mintlify.app/self-hosted/concepts/error-handling.md): Typed errors, proxy tiers, and what happens when a scrape fails.
- [Basic Scraping](https://reader.mintlify.app/self-hosted/guides/basic-scraping.md): Practical recipes for single-URL and small-batch scrapes.
- [Batch Scraping](https://reader.mintlify.app/self-hosted/guides/batch-scraping.md): Scrape many URLs in parallel with concurrency, progress tracking, and partial failure handling.
- [Website Crawling](https://reader.mintlify.app/self-hosted/guides/website-crawling.md): Discover and scrape every page on a site with BFS link discovery.
- [Proxy Configuration](https://reader.mintlify.app/self-hosted/guides/proxy-configuration.md): Set up single proxies, rotation pools, and multi-tier standard/premium selection.
- [Using the CLI](https://reader.mintlify.app/self-hosted/guides/cli.md): One-off scrapes and crawls from the terminal.
- [Daemon Mode](https://reader.mintlify.app/self-hosted/guides/daemon-mode.md): Run Reader as a long-lived background process with a shared browser pool.
- [Browser Sessions](https://reader.mintlify.app/self-hosted/guides/browser-sessions.md): Full examples for Playwright, Puppeteer, and raw CDP browser automation
- [Deployment](https://reader.mintlify.app/self-hosted/guides/deployment.md): Run Reader in production with Docker or bare metal.
- [ReaderClient](https://reader.mintlify.app/self-hosted/api-reference/reader-client.md): Constructor, options, methods, and lifecycle for the main Reader API.
- [scrape()](https://reader.mintlify.app/self-hosted/api-reference/scrape.md): Scrape one or more URLs and return cleaned content.
- [crawl()](https://reader.mintlify.app/self-hosted/api-reference/crawl.md): Discover and optionally scrape pages on a website via BFS.
- [ScrapeOptions](https://reader.mintlify.app/self-hosted/api-reference/scrape-options.md): Every field accepted by scrape() with type and default.
- [ScrapeResult](https://reader.mintlify.app/self-hosted/api-reference/scrape-result.md): Return type for scrape() - data array plus batch metadata.
- [CrawlOptions](https://reader.mintlify.app/self-hosted/api-reference/crawl-options.md): Every field accepted by crawl() with type and default.
- [CrawlResult](https://reader.mintlify.app/self-hosted/api-reference/crawl-result.md): Return type for crawl() - discovered URLs, optional scrape data, and metadata.
- [Browser Session](https://reader.mintlify.app/self-hosted/api-reference/browser-session.md): API reference for browser() method, BrowserOptions, and BrowserSession
- [Errors](https://reader.mintlify.app/self-hosted/api-reference/errors.md): Full reference of every error class thrown by Reader.

## OpenAPI Specs

- [openapi](https://reader.mintlify.app/openapi.json)

## Optional

- [GitHub](https://github.com/vakra-dev/reader)
- [Discord](https://discord.gg/6tjkq7J5WV)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.