Recent Improvements to Urlbox

Urlbox's latest engine_version was promoted to stable on Tuesday 10th December 2024.

In May we introduced our engine_version option.

Handling rendering edge cases is complex, and some of our long-standing fixes may not apply to current customers. Additionally, improving accuracy can sometimes compromise the consistency our customers depend on.

This feature has allowed us to balance accuracy with consistency by offering two versions: stable and latest.

Engine Versions:

• engine_version=latest has been updated regularly over the last six months to keep up with all the changes to browsers & the web that impact screenshot quality.
• engine_version=stable has remained unchanged with minor exceptions for security patches to ensure consistent rendering.

You can find the changes we've made in our latest engine on our Changelog.

Release Schedule:

• Development of engine_version=latest was frozen on 15th December 2024 ready for promotion to engine_version=stable
• On the 19th November we provided 3 weeks advance notice to allow for testing before rollout.
• Further reminder emails were sent each week with a final reminder on Monday 9th December.
• The latest version was promoted to stable on 10th December.
• New features and improvements will continue to be released to engine_version=latest after 10th December 2024.
• We don't expect engine_version=stable to be updated again before March 2025.

Default Settings:

• LoFi plan customers default to the latest version.
• HiFi, Ultra, Business and Enterprise customers default to the stable version.
• Rather than using engine_version in your requests, you can override the default version for each of your API keys via the projects section of our dashboard.

Testing:

• We conducted extensive internal testing both automated and manual in the run up to the switch over.
• Over 50% of requests were being served by the latest engine by 10th December.
• We advised customers to try sending requests to the latest engine at a convenient time, before the switch over.
• Customers verifed that screenshots still looked as expected.
• Customers using our s3-compatible storage integration, verified that files were being stored as expected.

Possible Differences:

We know of two issues impacting some complex full page screenshots. We have workarounds for these, please contact us if your screenshots are affected. Most customers have reported improvements in speed and accuracy but there is a chance of visual differences for your specific use case.

It's also possible that some non-standard s3-compatible storage scenarios will behave differently. Please ensure that your configured s3 region matches that of the bucket you are using in your requests. We recommend that you configure multiple Urlbox projects within your account when using multiple S3-compatible storage configurations.

What to Do:

If you hit any issues please let us know with an email to [email protected]. We will work with you to try and resolve them as soon as possible.

We are providing Long Term Support for the current version of our stable engine as an option to customers on our Business and Enterprise plans. If you are on the Ultra plan we will provide access at no additional cost until Tuesday 14th January.

• Chrome: Browser version updated.
• S3-compatible Storage: We now use presigned URLs to upload direct from our rendering engine.
• Security: Ground work for new features supporting the rendering of more sensitive/confidential HTML to image or PDF.
• Unblocking: Ground work for new features to reduce the chances of your screenshots being blocked by anti-automation measures.

• Cookies: Better handling of requests with multiple cookies provided
• Incident Response: Stability improvements to reduce the impact of future uptime incidents
• Developer Documentation: Updates with more examples for many options
• Pricing: Business and Enterprise plans now shown on pricing page
• Infrastructure: Various updates including database upgrade and migration

• Urlbox Dedicated Cluster: Improvements for high volume Enterprise customers.
• Metadata: Fix issue impacting the collection of open graph data from some sites when using metadata: true option.
• Video: Fix issue that resulted in bars at edges of mobile viewport videos (less than 500px wide).
• PDF: Resolved new edge case where a page could cause PDF generation to fail.
• Text Highlighting: Resolved new edge case where a page could cause highlighting feature to fail.
• Performance: Various infrastructure improvements.

• Press Escape: The press_escape option now simulates an escape key press on initial load and after any delay conditions are met. This is useful for dismissing modals/popups that don't show on initial page load.
• Refresh Options: Added refresh and refresh_after_scoll options for reloading a page before taking a screenshot. This helps remove banners and popups that only show to first-time visitors.
• Enhanced Scroll Detection: Improved the algorithm for detecting scroll containers. This reduced the number of situations that can prevent full page screenshots from rendering correctly.
• Save Formats in Sandbox: Introduced options to the sandbox UI to save HTML, markdown, and metadata along with a screenshot.
• Video Audio Recording: Introduced support for recording website audio when making video screenshots.
• Video Codec: Added the video_codec option to choose between h264, vp8 and vp9 codecs.
• Video Length: Removed the 100-second cap for videos, allowing longer video recordings.
• Video Scrolling Screenshots: Added a video_dont_scroll_back option to prevent return to top of page when not required.
• Video Easing Options: Introduced easing options for scrolling screenshots making them customizable via a dropdown in the sandbox.
• Improved validation: Improved validation of latitude, longitude and accuracy options to surface location errors sooner.
• Website Update: Refreshed parts of our website to better reflect current uses of Urlbox and added a security page to start sharing more about how we safely render your screenshots.

• Screenshot timing: Full page screenshots for a small number of sites with cookie banners were rendering as viewport only. This was because clicking accept was triggering a full page reload.
• Infrastructure enhancement: More flexible storage options for enterprise customers.
• Sticky headers: Some sticky headers were not displaying in full page mode.
• Modal hiding: Ability to detect more popup modals and remove them from screenshots.
• Screenshot width: Limit width when full_width not specified to avoid extreme viewport sizes in some scenarios.
• Build pipeline: Internal tooling improvements to support our growing team.
• Banner display: Add support for Shopify universal banner - no longer hidden by default.
• Engine version: engine_version option can now be set at a project level in the dashboard and on sandbox requests.
• Video: Fix video rendering failures experienced in the latest version of engine.
• One Million Screenshots: Introduced our fresh way to explore the top million homepages.

• S3 Configuration: Specify object locked buckets, ensuring no modifications are possible after creation. Added support for multipart uploads to these buckets.
• Platform Option: Added a new platform option. See our blog post on performance improvements for more details.
• Cookie Banners: Improved removal of new cookie banners.
• PDF Margins: Updated PDF margin behaviour to match Chrome.
• Rendering Pipeline: Adjusted timing to resolve a new edge case.
• Debugging Tooling: Enhanced debugging tools and improved error logging.
• UI Improvements: Improved UI in the sandbox for switching between viewport, full-page, and selector options.
• Billing: Fixed issues in the billing experience for returning customers.
• General Enhancements: Various infrastructure, security, and performance improvements.

• Engine Version: Introduced engine_version option. See our blog post for more details.
• Debugging Tooling: Improved to resolve more edge cases.
• Blank Images: Fixed issues causing blank images when Puppeteer resizes the viewport.
• SVG Generation: Resolved an edge case causing timeouts.
• Scrolling Behaviour: Adjusted to improve performance and address lazy loading image failures.
• Puppeteer GPU Bug: Resolved a bug with Puppeteer using the GPU.
• Billing: Enhanced billing processes and introduced new pricing for legacy users.
• Onboarding Flow: Improved onboarding for new users.
• Error View: Added renderId to the error view on the dashboard.
• Project/API Key Management: Enhanced management for multiple projects and API keys.
• UI Improvements: Improved the UI for adding S3 credentials.

• Form Redesign: Updated login, signup, reset, and confirm password forms to a new design.
• Render Counting Logic: Improved to ensure accurate dashboard statistics.
• Screenshot Handling: Resolved an issue affecting screenshots due to certain CORS configurations.
• Puppeteer Configuration: Updated Chrome arguments in Puppeteer to handle new edge cases.
• Page State Evaluation: Resolved issues caused by website overrides on JSON objects used for page state evaluation.
• Welcome Arnold: Our newest team member.

• Cookie Banner Handling: Improved methods to hide more cookie banners.
• Screenshot Timeout: Resolved an edge case causing timeouts for some screenshots taller than 2048px.
• Selector Algorithm: Updated the algorithm for finding selectors to address lazy loading elements.
• Marketing Site: Enhanced navigation to provide easier access to resources.
• Blog: Layout improvements.

• Metadata: Improved extraction process.
• HTML Post Handling: Fixed an issue blocking HTML fragments over 1 MB in post requests; now supports up to 10 MB.
• Scrolling: Resolve issue where timeout triggered before reaching bottom of page.
• Rendering Engine: Add handling for websites that set console.log to null.

• Urlbox.com: We’ve moved from urlbox.io
• Markdown: Improved markdown rendering stability.
• Metadata Enhancements: Added url_requested and url_resolved to metadata to surface redirections. Fix issue where URL would fallback to example.com.
• Auto Crop: Introduced an auto_crop option to eliminate blank areas in screenshots.
• Demo Bookings: Now accepting bookings for live demonstrations.
• We’re hiring!: Seeking a Full Stack TypeScript Developer