Setting the User Agent

The user agent is another request header that the Urlbox browser sends when visiting a URL.

It is sometimes used by sites to determine what type of device is making the request. For example, a site might return a mobile version of the site if the user agent is a mobile device.

It can also be used by sites to block requests from bots and for browser fingerprinting. For example, a site might block requests from bots by checking if the user agent is a known bot.

This can be done by setting the user_agent option to something custom when making a request to Urlbox:

curl -X POST \
  https://api.urlbox.com/v1/render/sync \
  -H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "example.com",
  "user_agent": "My-Custom-User-Agent"
}'

You could emulate certain well known scraping user agent strings like facebook:

curl -X POST \
  https://api.urlbox.com/v1/render/sync \
  -H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "example.com",
  "user_agent": "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
}'

Other well known user agents include:

• Googlebot
• Bingbot
• Slurp
• DuckDuckBot
• Baiduspider
• YandexBot and others can be found at https://developers.whatismybrowser.com/useragents/explore/

Using mobile user agent

If you want to use a mobile user agent, you can set the user_agent option to mobile:

curl -X POST \
  https://api.urlbox.com/v1/render/sync \
  -H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "example.com",
  "user_agent": "mobile"
}'

Using a desktop user agent

If you want to use a desktop user agent, you can set the user_agent option to desktop:

curl -X POST \
  https://api.urlbox.com/v1/render/sync \
  -H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "example.com",
  "user_agent": "desktop"
}'

Using a random user agent

If you want to use a random user agent, you can set the user_agent option to random:

curl -X POST \
  https://api.urlbox.com/v1/render/sync \
  -H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "example.com",
  "user_agent": "random"
}'

Urlbox will cycle through valid user agents for each request. This can be useful for evading detection when making many requests in quick succession to the same website.