> ## Documentation Index
> Fetch the complete documentation index at: https://reader.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CrawlOptions

> Every field accepted by crawl() with type and default.

## Required

| Option | Type | Description |
| - | - | - |
| `url` | `string` | Seed URL where crawling starts |

## Crawl control

| Option | Type | Default | Description |
| - | - | - | - |
| `depth` | `number` | `1` | Max crawl depth from the seed |
| `maxPages` | `number` | `20` | Max pages to discover (hard limit) |
| `scrape` | `boolean` | `false` | Also scrape each discovered page |
| `delayMs` | `number` | `1000` | Delay between requests (rate limiting) |
| `timeoutMs` | `number` | - | Total crawl timeout |

## URL filtering

| Option | Type | Default | Description |
| - | - | - | - |
| `includePatterns` | `string[]` | `[]` | Regex - URL must match at least one |
| `excludePatterns` | `string[]` | `[]` | Regex - URL must not match any |

Crawling is always same-domain (Reader does not follow external links). Use these patterns to further scope within a domain.

## When scrape: true

These options only apply when `scrape: true`:

| Option | Type | Default | Description |
| - | - | - | - |
| `formats` | `Array<"markdown" \| "html">` | `["markdown"]` | Output formats |
| `scrapeConcurrency` | `number` | `2` | Parallel scrapes during the crawl |
| `removeAds` | `boolean` | `true` | Remove ad selectors |
| `removeBase64Images` | `boolean` | `true` | Strip inline base64 images |

## Proxy & misc

| Option | Type | Default | Description |
| - | - | - | - |
| `proxy` | `ProxyConfig` | - | Single proxy for this crawl |
| `proxyTier` | `"standard" \| "premium"` | - | Pick a proxy from the configured pool |
| `userAgent` | `string` | Chrome UA | Custom user agent |
| `verbose` | `boolean` | `false` | Enable logging |
| `showChrome` | `boolean` | `false` | Show browser window |

## Example

```javascript theme={null}
await reader.crawl({
  url: "https://docs.example.com",
  depth: 3,
  maxPages: 100,
  scrape: true,
  scrapeConcurrency: 3,
  formats: ["markdown"],
  includePatterns: ["^https://docs\\.example\\.com/(api|guides)/"],
  excludePatterns: ["/changelog/", "/archive/"],
  delayMs: 1500,
});
```

## Where to go next

> [card] [CrawlResult](https://reader.mintlify.site/self-hosted/api-reference/crawl-result.md)
>
> The return type for every crawl call.

> [card] [Crawling concept](https://reader.mintlify.site/self-hosted/concepts/crawling.md)
>
> How BFS link discovery works.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.