Automate full-page website screenshots for a URL list

Posted by Harald Nezbeda on Sun 19 November 2017

I started getting more involved in this thematic after being approached by a colleague at work. He asked me if there might be a possibility to get a full-page screenshot from a website using console tools and also emulate them for different screen sizes. If this works it should be also replicated to a list of URLs, this way it could be tested if websites are displayed correctly for a defined set of screen sizes.

Probably the first thing that comes into mind is to use some of the various plugins for Chrome and Firefox that are out there, and find a way to feed them with a list of URL to do the task. The approach is not bad, but there would be a slight problem if the website has fixed elements (CSS position fixed). The plugins we tested so far work by scrolling through the page and create images of the visible area which are then stitched together. We have a fixed navigation bar that would have been pretty ugly if utilized like so. However, this elements are quite common these days, so there must be another way to solve it.

The newer versions of Chrome have a nice feature in the developer console, which allows you to create a full size screenshot of the website at a given width. The best part is that it is respecting the fixed elements of the page and I can receive clean screenshots, even on pages containing elements that scaled on the available height.

Chrome full-page screenshot

Selenium

A way to automate this is now required, so I instantly thought on Selenium, perhaps it is related to projects where I used end-to-end testing with Selnium and Docker for Python projects. I was hoping that the Chromedriver has this functionality implemented and that Selenium is capable of using it. It turned out it isn’t something new, and there is a chromium issue that was open since 2013 and people are still struggling in creating full page screenshots from websites. It was closed on November 2nd, 2017, but there is not much documentation on how to use it and probably it is not yet implemented into Selenium (it might be added in version 3.7.x).

Nevertheless, walking through some of the 155 comments gave me some new ideas to solve this, and the one that looked most applicable to me was to use a huge screen height. This was not a big deal on the Selenium/Docker stack as it used Xvfb (X Window Virtual Framebuffer) which is kind of a virtual display where the browser can run and Selenium can be linked to. The problem here is that you end up with images that might be rendered incorrectly as the screen is not resembling real life conditions - having a screen with a width 1920px width and 10000px height is unusual. As a consequence, it would require a considerable amount of time and resources in order to build the framebuffer for different widths. An alternative would be to use a headless browser such as PhantomJS, however the height issue will persist.

Headless Chrome and Puppeteer

I discovered over the past days that Chrome is also able to work headless (for Linux and Mac since version 59, for Windows since version 60), which is a powerful update and it could make UI testing stacks slicker and faster.

There is also a project written in JavaScript called Puppeteer that got pushed during a talk at the Chrome Dev Summit 2017 that can control Google Chrome, .

For the use case described above it is no big deal that this is using Node and JavaScript instead of Python. This will be just a small application for testing screens at the end of the day. Since Puppeteer is also developed by the Chrome team it brings the new features of the browser faster to the API then other libraries do (this was the case for the full-page screenshop function, which works exactly as in the developer tools).

The project is still in early stages, but it looks promising, and might be a worthy alternative for end-to-end testing. There is also an unofficial port to Python

In order to finalize the screenshot tooling part, I made a repository that you can use, called Chrome Shots. It is a Docker Compose configuration that uses headless Chrome with puppeteer to generate full page screenshots. You can define a list of URLs and devices and the tool will create all the screenshots (more instructions in project readme).