Sync vs Async

Both modes accept the same scrape options. The difference is whether you wait for the result on the original connection or collect it separately.

Mode	Endpoint	Methods	Delivery	Timeout ceiling
Sync	`https://sync.scrapedrive.com/api/v1/scrape`	GET or POST	Returns the scraped body in the same response.	120,000 ms
Async	`https://api.scrapedrive.com:8443/api/v1/scrape/async`	GET or POST	Immediately returns `{ id, status, url, status_url }`.	130,000 ms

Use Sync for one-off requests and browser-openable URLs. Use Async for batches, higher volume, or jobs that may need longer than the synchronous connection allows.

Sync

On success, the response is the scraped body itself, with the target's safe response headers replayed. There is no result envelope.

Terminal Sync responses also expose X-Sdrive-Credits, the exact settled charge for the Job. It is 0 on terminal failure and is not an attempt count. X-Sdrive-Job-Id identifies an admitted Sync Job. Both headers are readable by browser clients through CORS. The credits header is absent from admission rejections and Sync timeout responses.

curl --get "https://sync.scrapedrive.com/api/v1/scrape" \
  --data-urlencode "api_key=YOUR_KEY" \
  --data-urlencode "url=https://example.com"

The synchronous timeout_ms defaults to and cannot exceed 120000. A difficult browser job can exceed that ceiling, so use Async when the job needs more time.

Return the target response on failure

By default, a failed Sync scrape returns a ScrapeDrive JSON error. Set transparent_mode=true when you would rather receive the target's status, body, and safe headers if that website returned a response:

curl --get "https://sync.scrapedrive.com/api/v1/scrape" \
  --data-urlencode "api_key=YOUR_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "transparent_mode=true"

This option cannot manufacture a target response. Timeouts, proxy errors, and other failures where the website never responded remain ScrapeDrive JSON errors. The option does not apply to Async jobs.

Async

Submit the job to the API host. POST with a JSON body is the most practical choice for application code:

curl "https://api.scrapedrive.com:8443/api/v1/scrape/async" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "api_key": "YOUR_KEY",
    "url": "https://example.com",
    "render_js": true
  }'

The submission response contains the job identifier, its status, and the polling URL:

{
    "id": "01KH2CTXRJ44SD5XJ220BA0XRZ",
    "status": "queued",
    "url": "https://example.com",
    "status_url": "https://api.scrapedrive.com:8443/api/v1/job/01KH2CTXRJ44SD5XJ220BA0XRZ"
}

The live contract at /api/v1/spec still describes this field as job_id. The API sends id. Read either name, and poll status_url.

Poll the result

curl "https://api.scrapedrive.com:8443/api/v1/job/01KH2CTXRJ44SD5XJ220BA0XRZ"

Polling does not require api_key. The unguessable job id grants access to the result, so treat it like a secret and do not publish or share it.

Poll until status is completed, then capture the entire response. Reading a finished result once shortens its remaining lifetime, so do not fetch parts of it in separate follow-up requests.

completed means the job ended, not that it succeeded. A delivered page looks like this; credits is the settled charge:

{
    "id": "01KH2CTXRJ44SD5XJ220BA0XRZ",
    "status": "completed",
    "url": "https://example.com",
    "status_url": "https://api.scrapedrive.com:8443/api/v1/job/01KH2CTXRJ44SD5XJ220BA0XRZ",
    "response": {
        "status_code": 200,
        "final_url": "https://example.com/",
        "headers": { "content-type": "text/html" },
        "body": "<!doctype html>...",
        "credits": 5
    }
}

A job that failed is also completed, with status_code 0, an empty body, credits 0, and a plain-language reason:

{
    "id": "01KH2CTXRJ44SD5XJ220BA0XRZ",
    "status": "completed",
    "response": { "status_code": 0, "headers": {}, "body": "", "credits": 0 },
    "reason": "The target hostname could not be resolved in DNS. Check that the URL and domain are correct."
}

Check response.status_code and reason before using the body. A 404 or 410 from the target is a delivered page and is charged. When you asked for a screenshot, its URL is in response.headers["x-sdrive-screenshot-url"].

The asynchronous timeout_ms defaults to and cannot exceed 130000.

Receive a webhook instead

For Async jobs, provide an HTTPS webhook_url:

curl "https://api.scrapedrive.com:8443/api/v1/scrape/async" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "api_key": "YOUR_KEY",
    "url": "https://example.com",
    "webhook_url": "https://your-app.example/webhooks/scrapedrive",
    "custom_id": "import-4821"
  }'

ScrapeDrive POSTs the result to that URL when the job completes. custom_id can contain up to 255 characters and is echoed in the webhook payload so you can match the result to your own record. webhook_url is ignored in Sync mode.

Sync vs Async

Sync vs Async

Sync

Return the target response on failure

Async

Poll the result

Receive a webhook instead

On this page