Setting the User Agent

The user agent is another request header that the Urlbox browser sends when visiting a URL.

It is sometimes used by sites to determine what type of device is making the request. For example, a site might return a mobile version of the site if the user agent is a mobile device.

It can also be used by sites to block requests from bots and for browser fingerprinting. For example, a site might block requests from bots by checking if the user agent is a known bot.

This can be done by setting the user_agent option to something custom when making a request to Urlbox:

curl -X POST \
	https://api.urlbox.io/v1/render/sync \
	-H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
	-H 'Content-Type: application/json' \
	-d '
{
	"url": "example.com",
	"user_agent": "My-Custom-User-Agent"
}
'

You could emulate certain well known scraping user agent strings like facebook:

curl -X POST \
	https://api.urlbox.io/v1/render/sync \
	-H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
	-H 'Content-Type: application/json' \
	-d '
{
	"url": "example.com",
	"user_agent": "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
}
'

Other well known user agents include:

• Googlebot
• Bingbot
• Slurp
• DuckDuckBot
• Baiduspider
• YandexBot and others can be found at https://developers.whatismybrowser.com/useragents/explore/

Using mobile user agent

If you want to use a mobile user agent, you can set the user_agent option to mobile:

curl -X POST \
	https://api.urlbox.io/v1/render/sync \
	-H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
	-H 'Content-Type: application/json' \
	-d '
{
	"url": "example.com",
	"user_agent": "mobile"
}
'

Using a desktop user agent

If you want to use a desktop user agent, you can set the user_agent option to desktop:

curl -X POST \
	https://api.urlbox.io/v1/render/sync \
	-H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
	-H 'Content-Type: application/json' \
	-d '
{
	"url": "example.com",
	"user_agent": "desktop"
}
'

Using a random user agent

If you want to use a random user agent, you can set the user_agent option to random:

curl -X POST \
	https://api.urlbox.io/v1/render/sync \
	-H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
	-H 'Content-Type: application/json' \
	-d '
{
	"url": "example.com",
	"user_agent": "random"
}
'

Urlbox will cycle through valid user agents for each request. This can be useful for evading detection when making many requests in quick succession to the same website.