CrawlOptions

Options for the crawl() function

Type Definition

interface CrawlOptions {
  // Required
  url: string;

  // Crawl limits
  depth?: number;
  maxPages?: number;

  // Scraping
  scrape?: boolean;
  scrapeConcurrency?: number;
  formats?: Array<"markdown" | "html">;

  // Rate limiting
  delayMs?: number;
  timeoutMs?: number;

  // URL filtering
  includePatterns?: string[];
  excludePatterns?: string[];

  // Request configuration
  userAgent?: string;
  proxy?: ProxyConfig;

  // Debugging
  verbose?: boolean;
  showChrome?: boolean;
}

Options Reference

Required Options

Option	Type	Description
url	string	Seed URL to start crawling from

Crawl Limit Options

Option	Type	Default	Description
depth	number	1	Maximum crawl depth
maxPages	number	20	Maximum pages to discover

Scraping Options

Option	Type	Default	Description
scrape	boolean	false	Also scrape content of discovered pages
scrapeConcurrency	number	2	Concurrent scraping threads
formats	Array<"markdown" | "html">	["markdown", "html"]	Output formats when scraping

Rate Limiting Options

Option	Type	Default	Description
delayMs	number	1000	Delay between requests (ms)
timeoutMs	number	undefined	Total timeout for crawl (ms)

URL Filtering Options

Option	Type	Default	Description
includePatterns	string[]	undefined	URL patterns to include (regex)
excludePatterns	string[]	undefined	URL patterns to exclude (regex)

Request Configuration Options

Option	Type	Default	Description
userAgent	string	undefined	Custom user agent string
proxy	ProxyConfig	undefined	Proxy configuration

Debugging Options

Option	Type	Default	Description
verbose	boolean	false	Enable verbose logging
showChrome	boolean	false	Show browser window

Examples

Basic Crawl

await reader.crawl({
  url: "https://example.com",
  depth: 2,
  maxPages: 50,
});

Crawl with Scraping

await reader.crawl({
  url: "https://example.com",
  depth: 2,
  maxPages: 50,
  scrape: true,
  scrapeConcurrency: 5,
  formats: ["markdown"],
});

With URL Filtering

await reader.crawl({
  url: "https://example.com",
  depth: 3,
  maxPages: 100,
  includePatterns: ["^/docs/", "^/guides/"],
  excludePatterns: ["^/admin/", "^/api/"],
});

With Rate Limiting

await reader.crawl({
  url: "https://example.com",
  depth: 2,
  delayMs: 2000, // 2 seconds between requests
  timeoutMs: 300000, // 5 minute total timeout
});

Full Options

await reader.crawl({
  url: "https://example.com",
  depth: 3,
  maxPages: 100,
  scrape: true,
  scrapeConcurrency: 5,
  formats: ["markdown"],
  delayMs: 1000,
  timeoutMs: 600000,
  includePatterns: ["^/docs/"],
  excludePatterns: ["^/docs/legacy/"],
  proxy: {
    host: "proxy.example.com",
    port: 8080,
    username: "user",
    password: "pass",
  },
  verbose: true,
});