Setting the User Agent

The user agent is another request header that the Urlbox browser sends when visiting a URL.

It is sometimes used by sites to determine what type of device is making the request. For example, a site might return a mobile version of the site if the user agent is a mobile device.

It can also be used by sites to block requests from bots and for browser fingerprinting. For example, a site might block requests from bots by checking if the user agent is a known bot.

This can be done by setting the user_agent option to something custom when making a request to Urlbox:

{  "url": "example.com",  "user_agent": "My-Custom-User-Agent"}

You could emulate certain well known scraping user agent strings like facebook:

{  "url": "example.com",  "user_agent": "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"}

Other well known user agents include:

• Googlebot
• Bingbot
• Slurp
• DuckDuckBot
• Baiduspider
• YandexBot and others can be found at https://developers.whatismybrowser.com/useragents/explore/

Using mobile user agent

If you want to use a mobile user agent, you can set the user_agent option to mobile:

{  "url": "example.com",  "user_agent": "mobile"}

Using a desktop user agent

If you want to use a desktop user agent, you can set the user_agent option to desktop:

{  "url": "example.com",  "user_agent": "desktop"}

Using a random user agent

If you want to use a random user agent, you can set the user_agent option to random:

{  "url": "example.com",  "user_agent": "random"}

Urlbox will cycle through valid user agents for each request. This can be useful for evading detection when making many requests in quick succession to the same website.