Pixel Perfect Frontend Migration with SSDiff

Migrating a frontend app can be a complex and challenging process, especially when moving from a React-based framework like Next.js to a non-React-based framework like Qwik.

At Builder.io, we have recently undertaken the task of migrating our documentation from Next.js to Qwik with the goal of achieving a high-performance website while maintaining the same look and functionality.

The challenge was that we have 150+ documents with 15+ custom registered components, and a successful migration requires a visual comparison of each page in both Next.js and Qwik versions.

In this blog post, we discuss our approach to accelerating testing for the migration with SSDiff, an open-source tool we have developed for pixel-perfect website migrations.

SSDiff: Pixel diff at scale

To automate the process of visually comparing the Next.js and Qwik versions of the documentation site, we needed a tool that could take screenshots of each page in both versions, run a diffing algorithm to identify differences, and provide a quantitative value for the difference.

The tool should also generate diffing images that highlight the components causing the difference. This should be done in parallel for multiple pages in batches, allowing us to identify pages with the most significant differences and prioritize them.

This approach would eliminate the need for manual side-by-side comparisons of web pages and streamline the issue detection process.

We initially developed a basic script to automate the process of opening web pages and taking screenshots for comparison between the Next.js and Qwik versions. We incorporated pixelmatch, an image-diffing library, to compare the screenshots and output the diffing files in a designated folder. To ensure better developer experience and enable type safety, we added TypeScript support to the project from the early stages.

// Calculate the diff between url1 and url2 and store it in a file
async compare(compareObj: { url1: string; url2: string; fileName: string }) {
    const { url1, url2, fileName } = compareObj;
    const [image1, image2] = await Promise.all([this.screenshot(url1), this.screenshot(url2)]);
    const maxHeight = Math.max(image1.height, image2.height);
    const maxWidth = Math.max(image1.width, image2.width);
		/* ...resize images if the dimensions are not same  */
    const numDiffPixels = pixelmatch(image1.data, image2.data, diff.data, maxWidth, maxHeight, pixelMatchConfig);
    const totalPixels = diff.data.length / 4;
    const differencePercentage = (numDiffPixels / totalPixels) * 100;
    this.fileNameDifferenceMap.set(fileName, differencePercentage); 
    fs.writeFileSync(this.diffScreenshots + `/${fileName}`, PNG.sync.write(diff));
}

As we tested more pages, we decided to change the output format of the tool to a sorted map, with URL path-names as keys and quantitative diffing values as corresponding values.

async sortFilesBasedOnDifference() {
    // sorts the map based on diffing values for fileNames as keys
    const sortedMap = new Map([...this.fileNameDifferenceMap.entries()].sort((a, b) => b[1] - a[1]));
    return sortedMap;
}

We also encountered an issue with the image-diffing tool where it would fail if the images being compared were of different sizes. To address this issue, we used sharp to resize images if they were of different sizes before comparison.

// resize and place image to top left of the canvas
async resizeImage(image: PNG, width: number, height: number) {
	  const sharpImage = sharp(image.data, { raw: { width: image.width, height: image.height, channels: 4 } });
	  const resizedImageBuffer = await sharpImage
	    .resize({
	      height,
	      width,
	      fit: 'contain',
	      position: 'left top',
	    })
	    .toFormat('png')
	    .toBuffer();
	  return PNG.sync.read(resizedImageBuffer);
  }

The resulting tool, named SSDiff, is an evolving project with potential for further extensions and use cases. It is currently available on npm and can be used to test different versions of a website.

Challenges and limitations

Pixel diffing, a method used for identifying differences in components, has its own set of limitations and challenges, including:

Testing Stateful Components: Components that are affected by state changes may render different content based on the values of their state. The tool currently does not handle mutations and access to state automatically. Users may need to handle these programmatically or manually to cover all behaviors for a particular page or component.
Performance Bottlenecks: The tool uses Puppeteer to open a URL and capture a screenshot. When multiple pathnames are provided for testing, the tool spawns multiple pages in a browser. This may result in performance bottlenecks, which are typically observed when around 20 pathnames are provided for a given URL. However, it's worth noting that this performance may vary depending on the system. To obtain results for more than 20 pathnames in a single run, users may need to run the diffing process in batches.

Conclusion

A successful frontend migration should result in a new version of the site that looks identical to the original, but with improved performance and expected enhancements. The SSDiff tool we developed helped us identify areas for improvement and make our components more similar to the original site. While the tool has limitations in terms of speed and pinpointing the exact component causing visual differences, it has proved useful in streamlining our migration process.