> ## Documentation Index
> Fetch the complete documentation index at: https://reader.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# ScrapeOptions

> Every field accepted by scrape() with type and default.

## Required

| Option | Type | Description |
| - | - | - |
| `urls` | `string[]` | URLs to scrape. Pass a single-element array for a single page. |

## Output

| Option | Type | Default | Description |
| - | - | - | - |
| `formats` | `Array<"markdown" \| "html">` | `["markdown"]` | Output formats to include in results. `rawHtml` is always returned regardless of this setting. |

## Request

| Option | Type | Default | Description |
| - | - | - | - |
| `timeoutMs` | `number` | `30000` | Request timeout per URL |
| `waitForSelector` | `string` | - | Wait for this CSS selector before extracting |

## Content cleaning

| Option | Type | Default | Description |
| - | - | - | - |
| `onlyMainContent` | `boolean` | `true` | Extract only main content (strips nav/header/footer) |
| `includeTags` | `string[]` | `[]` | CSS selectors to keep - everything else is removed |
| `excludeTags` | `string[]` | `[]` | CSS selectors to remove |
| `removeAds` | `boolean` | `true` | Remove common ad selectors |
| `removeBase64Images` | `boolean` | `true` | Strip inline base64 images |
| `navigationSelectors` | `string[]` | `[]` | Additional CSS selectors to remove when `onlyMainContent` is true. Merged with built-in nav/footer/sidebar selectors. |

## Retry & timeouts

| Option | Type | Default | Description |
| - | - | - | - |
| `hardDeadlineMs` | `number` | `30000` | Hard deadline for a single URL. After this, the scraper gives up. |
| `datacenterTimeoutMs` | `number` | `10000` | Timeout for a standard (datacenter) proxy attempt. |

## Batch processing

| Option | Type | Default | Description |
| - | - | - | - |
| `batchConcurrency` | `number` | `1` | Number of URLs to process in parallel |
| `batchTimeoutMs` | `number` | `300000` | Total timeout for the batch (5 min) |
| `onProgress` | `(p: ProgressEvent) => void` | - | Progress callback |

```typescript theme={null}
interface ProgressEvent {
  completed: number;
  total: number;
  currentUrl: string;
}
```

## Proxy

| Option | Type | Default | Description |
| - | - | - | - |
| `proxy` | `ProxyConfig` | - | Single proxy to use for this request |
| `proxyTier` | `"standard" \| "premium"` | - | Pick a proxy from the configured pool by tier |

## Pluggable config

These options let the caller inject platform-specific behavior. Reader ships with no built-in domain profiles, block detection patterns, or URL rewriters.

| Option | Type | Default | Description |
| - | - | - | - |
| `domainProfiles` | `Record<string, DomainProfile>` | `{}` | Per-domain overrides (proxy tier, timeout, concurrency). Keyed by domain. |
| `blockDetection` | `BlockDetectionConfig` | - | Bot page detection config. Without this, no content-based block detection runs. |
| `urlRewriters` | `UrlRewriteRule[]` | `[]` | URL rewrite rules applied before scraping (e.g. Google Docs to export URL). |

```typescript theme={null}
interface DomainProfile {
  proxyTier?: "standard" | "premium";
  timeoutMs?: number;
  batchConcurrency?: number;
  minDelayMs?: number;
  maxConcurrentPerProxy?: number;
}

interface BlockDetectionConfig {
  patterns?: Array<RegExp | string>;      // matched against page text
  titlePatterns?: Array<RegExp | string>; // matched against page title
  shortContentThreshold?: number;         // default: 500
  longContentSignalThreshold?: number;    // default: 3
}

interface UrlRewriteRule {
  name: string;
  match: (url: URL) => boolean;
  rewrite: (url: URL) => string;
}
```

## Debugging

| Option | Type | Default | Description |
| - | - | - | - |
| `verbose` | `boolean` | `false` | Enable Pino logging |
| `showChrome` | `boolean` | `false` | Show the browser window |

## Where to go next

> [card] [ScrapeResult](https://reader.mintlify.site/self-hosted/api-reference/scrape-result.md)
>
> The return type for every scrape call.

> [card] [Content Extraction](https://reader.mintlify.site/self-hosted/concepts/content-extraction.md)
>
> How the cleaning options behave.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.