Wait Strategies

Control when content extraction happens. This matters for pages that load content dynamically — product listings, infinite scroll, delayed popups, etc.

The three wait options execute in order: wait_browser → wait_for → wait_ms.

wait_browser

When to consider page navigation complete (requires render_js=true):

curl "https://sync.scrapedrive.com/api/v1/scrape?api_key=YOUR_KEY&url=https://example.com&wait_browser=networkidle"

wait_for

Wait for a specific element to appear before extracting content. Pass a CSS selector. Times out after 15 seconds if the element isn't found.

curl "https://sync.scrapedrive.com/api/v1/scrape?api_key=YOUR_KEY&url=https://example.com&wait_for=.product-list"

wait_ms

A fixed delay in milliseconds, applied after the previous wait steps resolve. Useful for content that appears after JS animations or transitions. Range: 0–30,000ms.

# Wait for the product list, then wait another 2 seconds for lazy-loaded images
curl "https://sync.scrapedrive.com/api/v1/scrape?api_key=YOUR_KEY&url=https://example.com&wait_for=.product-list&wait_ms=2000"

Combining Wait Options

You can chain all three. They execute in order:

1. wait_browser=networkidle — wait for all network activity to stop
2. wait_for=.dynamic-content — then wait for this element
3. wait_ms=1000 — then wait 1 more second

curl "https://sync.scrapedrive.com/api/v1/scrape?api_key=YOUR_KEY&url=https://example.com&wait_browser=networkidle&wait_for=.dynamic-content&wait_ms=1000"

Header Forwarding

Pass custom HTTP headers to the target site. This is useful when scraping pages that require authentication tokens, specific cookies, or custom headers.

Prefix any header with sdrive- and set forward_sdrive_headers=true. ScrapeDrive strips the prefix and forwards the header to the target.

curl "https://sync.scrapedrive.com/api/v1/scrape?api_key=YOUR_KEY&url=https://example.com&forward_sdrive_headers=true" \
  -H "sdrive-Authorization: Bearer my-token" \
  -H "sdrive-X-Custom: my-value"

The target site receives Authorization: Bearer my-token and X-Custom: my-value.

Resource Blocking

Speed up page loads by blocking unnecessary resources.

block_resources=true (default) — Blocks images, CSS, fonts, and tracking scripts. This is already on by default and significantly speeds up scraping. Gets auto-disabled when taking screenshots.

block_ads=true — Blocks advertisement scripts and content. Off by default.

# Block ads too
curl "https://sync.scrapedrive.com/api/v1/scrape?api_key=YOUR_KEY&url=https://example.com&block_ads=true"

# Disable all blocking (needed if you want images/CSS in the HTML)
curl "https://sync.scrapedrive.com/api/v1/scrape?api_key=YOUR_KEY&url=https://example.com&block_resources=false"

Timeouts

Control the maximum time ScrapeDrive waits for a page to load.

Minimum for both: 10,000ms.

# Give the page up to 30 seconds
curl "https://sync.scrapedrive.com/api/v1/scrape?api_key=YOUR_KEY&url=https://slow-site.com&timeout_ms=30000"

If the page doesn't load within the timeout, you get a 504 response.