Avoiding Being Blocked

Some sites try to detect and block automated browsers. Instead of the content you asked for, the render comes back with a CAPTCHA, a "verify you are human" interstitial, an error status like 403 or 429, or a nearly blank page.

Because a page still rendered, the request succeeds from Urlbox's point of view, and you end up storing a screenshot of the block page. This guide covers how to detect blocks so that never happens, how to retry them automatically, and how to make renders look less like a bot so they succeed in the first place.

How blocking works

Sites decide a request is a bot using some combination of:

• IP reputation - requests from datacenter IP ranges are treated with more suspicion than residential connections
• Browser fingerprinting - automated browsers leak subtle differences from a real user's browser, which detection scripts look for
• Request rate - too many requests in a short window triggers rate limiting, usually a 429 response
• Geography - some sites only serve certain countries

A blocked render shows up in one of two ways: the page responds with an error status (403 Forbidden and 429 Too Many Requests are the most common), or it responds 200 but serves a challenge page instead of the real content.

Fail instead of capturing the block page

By default, Urlbox captures whatever the page serves, even if the response status was an error. The first line of defence is to tell Urlbox which statuses mean "blocked" so the render fails with a clear error instead:

{
  "url": "https://example.com",
  "fail_on": ["403", "429", "500"]
}

fail_on fails the render when the page's final status matches any of the listed codes. To cover whole ranges, use fail_on_4xx or fail_on_5xx, and exclude codes you consider acceptable with fail_on_except.

When a fail_on condition matches, the render fails with an error code naming the condition (for example fail_on_4xx) and the page's status code. If you use webhooks, you'll receive a render.failed event you can handle in your application, rather than a block-page screenshot you'd have to detect yourself.

Retry automatically

Failing cleanly is good; getting the real content is better. retry_on re-runs the render automatically when it hits a blocked status:

{
  "url": "https://example.com",
  "retry_on": ["403", "429"]
}

Retries use exponential backoff: the delay doubles with each attempt, starting from retry_delay_ms (1 second by default). For rate limits, the backoff alone is often enough, since the site just wants you to slow down. By default Urlbox makes up to 3 total attempts; tune this with max_retries or max_attempts.

Beyond status codes, retry_on also accepts 4xx, 5xx, the engine conditions timeout and crash, and the quality condition small_size (covered below). If every retry fails, the render fails, so you still get a clear error rather than a block page.

If a status appears in both retry_on and fail_on, Urlbox retries first and only fails after retries are exhausted, so the two options combine naturally: retry_on to recover, fail_on as the safety net.

Escalate on retry

Stealth and proxied renders are slower than standard ones, so you don't want them on every request. retry_with changes the render options on retry attempts, letting the first attempt run plain and only escalating for renders that actually got blocked:

{
  "url": "https://example.com",
  "retry_on": ["403", "429"],
  "retry_with": { "use_stealth": true }
}

Pass an array instead to escalate progressively, one rung per retry:

{
  "url": "https://example.com",
  "retry_on": ["403", "429"],
  "max_retries": 3,
  "retry_with": [
    { "use_stealth": true },
    { "use_stealth": true, "proxy": "user:[email protected]:9000" }
  ]
}

Here the first attempt runs plain, the first retry adds stealth, and later retries add a residential proxy on top. Each retry re-runs the render with that attempt's retry_with entry merged over the original options, so anything you don't mention stays the same. See the retry_with reference for the merging rules and the full list of retry-compatible options.

Make the browser look less like a bot

Two options reduce the automation fingerprints that detection scripts look for:

• hide_headless is the lightweight option. It patches the fingerprints headless Chrome normally leaks, such as navigator.webdriver, missing chrome runtime objects, and WebGL vendor strings. Reach for it when a site does light bot-detection.
• use_stealth is the stronger option. Stealth renders use a patched, real Chrome browser configured to look like a normal user's, which gets past most bot walls and CAPTCHAs. It takes precedence if both are set.

Stealth renders are noticeably slower than standard renders, so rather than enabling use_stealth as a blanket default, apply it only on retries via retry_with as shown above.

The user agent also plays a part: some sites block outdated or unusual user agent strings, and user_agent: "random" can help when making many requests to the same site in quick succession.

Change the outgoing IP with a proxy

Sites that block by IP reputation will refuse datacenter traffic no matter how convincing the browser looks. The proxy option routes the request through a proxy server you provide, so it reaches the site from a normal-looking IP instead:

{
  "url": "https://example.com",
  "proxy": "user:[email protected]:9000"
}

Residential and mobile proxies work best against IP-based blocking. Urlbox doesn't provide proxies; you bring your own from a provider, and you can store one on your project and enable it with use_proxy instead of passing the address on every request. Proxies also solve geography-based blocking, since most providers let you choose the country the request originates from.

The proxies guide covers providers, proxy types, geolocation, and troubleshooting in depth, including the pattern of escalating through a ladder of proxies with retry_with.

Catch soft blocks that return 200

Some challenge pages respond with a 200 status, so status-based options never trigger. Two ways to catch them:

• Size check: block pages are usually much smaller than real content. Set min_size_bytes to a floor below your typical screenshot size and add small_size to retry_on. A render smaller than the floor is retried, and fails if it never reaches it.
• Selector check: if a site's challenge page has a known element, pass its selector as wait_to_leave with fail_if_selector_present. Urlbox waits for the element to disappear and fails the render if it's still there, so a challenge that never clears can't produce a "successful" screenshot.

{
  "url": "https://example.com",
  "retry_on": ["403", "429", "small_size"],
  "min_size_bytes": 50000,
  "retry_with": { "use_stealth": true }
}

Putting it together

A robust setup for block-prone sites combines all of the above: fail conditions as the safety net, retries to recover, and escalation so the expensive measures only run when needed.

{
  "url": "https://example.com",
  "fail_on_4xx": true,
  "retry_on": ["403", "429", "small_size"],
  "min_size_bytes": 50000,
  "max_retries": 3,
  "retry_with": [
    { "use_stealth": true },
    { "use_stealth": true, "proxy": "user:[email protected]:9000" }
  ]
}

With this request:

1. The first attempt runs plain and fast. Most renders succeed here.
2. If the site blocks it (403, 429, or a suspiciously small screenshot), Urlbox retries with stealth after a short backoff.
3. If that's still blocked, further retries add your residential proxy on top of stealth.
4. If every attempt fails, the render fails with a clear error, and you never store a screenshot of a block page.

The retry options (retry_on, retry_with, min_size_bytes) and proxy are available on the ultra plan and above.