> ## Documentation Index
> Fetch the complete documentation index at: https://reader.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Scraping

> How Reader turns a URL into clean, LLM-ready content.

Scraping in Reader means: fetch a URL, render it (if needed), extract the main content, and convert it to clean markdown.

Under the hood this is a four-step pipeline:

1. **Fetch** - Reader renders the page in a headless Chrome browser (Playwright engine) with JavaScript execution and proxy routing. See [Scraping Engine](https://reader.mintlify.site/self-hosted/concepts/engine-waterfall.md).
2. **Extract** - Reader identifies the main content area (article, main, largest text block), removes navigation, footers, sidebars, ads, and hidden elements.
3. **Convert** - the cleaned HTML is converted to markdown via supermarkdown (a Rust-backed converter optimized for LLM input).

## The `scrape()` primitive

`scrape()` takes one or more URLs and returns a `ScrapeResult`:

```javascript theme={null}
const result = await reader.scrape({
  urls: ["https://example.com"],
  formats: ["markdown"],
});

console.log(result.data[0].markdown);
```

For a single URL, the call runs synchronously. For multiple URLs, set `batchConcurrency` to process them in parallel - see [Batch Scraping](https://reader.mintlify.site/self-hosted/guides/batch-scraping.md).

## Output formats

Reader supports two output formats:

* **`markdown`** - cleaned, LLM-optimized markdown (default)
* **`html`** - the cleaned HTML that markdown was generated from

You can request both in a single call:

```javascript theme={null}
const result = await reader.scrape({
  urls: ["https://example.com"],
  formats: ["markdown", "html"],
});

console.log(result.data[0].markdown);
console.log(result.data[0].html);
```

Every result also includes `metadata` regardless of format, with the title, description, canonical URL, favicon, Open Graph tags, and Twitter Card tags.

## What scraping is not

* **Crawling** - `scrape()` only visits URLs you explicitly pass. It does not discover links. For that, use [`crawl()`](https://reader.mintlify.site/self-hosted/concepts/crawling).
* **Screenshots or PDFs** - Reader is text-first. It does not produce rendered images of pages.
* **A JavaScript execution environment** - while the Playwright engine runs JavaScript, Reader doesn't expose the page object or let you run arbitrary browser scripts. It's a scraping tool, not a general-purpose browser automation framework.

## Where to go next

> [card] [Content Extraction](https://reader.mintlify.site/self-hosted/concepts/content-extraction.md)
>
> How Reader decides what to keep and what to strip.

> [card] [Scraping Engine](https://reader.mintlify.site/self-hosted/concepts/engine-waterfall.md)
>
> How the Playwright engine and proxy tiers work.

> [card] [Basic Scraping guide](https://reader.mintlify.site/self-hosted/guides/basic-scraping.md)
>
> Practical recipes and patterns.

> [card] [ScrapeOptions reference](https://reader.mintlify.site/self-hosted/api-reference/scrape-options.md)
>
> Every option, every default.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.