End-to-End Testing With Playwright And Github Actions

Not only are there many different ways to write software, but there are also many more ways to break it. Now I’m not sure about anyone else, but I know I’ve personally broken production web apps a few times. “Ahhh, I’ll just update this one simple line here, easy-peasy. All unit tests passed so obviously absolutely nothing could possibly go wrong…push to main…time for the weekend!” (we already know how this story ends).

I’m a huge advocate of unit tests and they certainly have their place throughout development. But rather than unit testing one specific function to return a specific value, we’re instead testing the output as a cohesive working webpage by interacting with the page the same users would with their browsers. This includes all JavaScript, CSS, HTML, and events, and how all of this works together to render and handle the expected result.

End-to-end testing is where I’ve found Playwright to be invaluable for many of our projects like Partytown, Qwik, and Mitosis, and its ability to test again and again on each commit. This post is a somewhat beginner's guide and will focus primarily on “why” Playwright, and how to set it up for continuous integration with GitHub Actions.

What is a Headless Browser?

The most common web browsers today include Chrome, Safari, Firefox, and Edge. In fact, it’s a safe bet to say you’re reading this blog right now using a web browser. But what you’re looking at is the browser’s user interface and how it was able to render the given URL.

A headless browser has the same capabilities, except the “headless” part means that it can run without a graphical user interface. The interesting part, however, is that the browser is still rendering the webpage and its HTML elements, and styles, and adding event listeners. But through the magic of an API, you can programmatically read and interact with the headless browser just like a user would interact with a traditional browser.

Playwright vs. Puppeteer

When choosing a headless browser, a common question that comes up is whether to use Microsoft’s Playwright, or Google’s Puppeteer, and of course, no software question is better answered than with “well, it depends”.

On the surface, the two are quite similar. Both are Node.js libraries for browser automation making it easy to control an actual browser with an API, and both have an extremely similar API, (and both were developed by largely the same core developers).

Between the two of them, Puppeteer was first when it was released in 2017. It shook up the traditional headless browser landscape by providing an API to interact directly with Google Chrome.

Puppeteer is a library that provides a high-level API to control Chrome over the DevTools Protocol.

Playwright was released next in 2020, except with a slightly different core mission in that it's not aiming to just be a high-level protocol to control Chrome, but rather focusing on “end-to-end” testing across many browsers, not just Chrome.

Playwright enables reliable end-to-end testing for modern web apps.

Having worked a lot with Puppeteer in the past, and manually wiring it up to Jest, this differentiation of having Playwright solely focused on providing end-to-end testing out-of-the-box is what makes it so great to use…for end-to-end testing.

Don’t get me wrong, Puppeteer is a great project; if your use case is to take screenshots or crawl pages, then it’s a good tool. However, if your mission is to develop automated end-to-end testing, and have your specs continually tested on each commit with Continuous Integration (CI), then Playwright is one of the best tools out there.

Installing and Writing Tests

Instead of repeating the same thing here, it’s best to follow the official installation instructions and how to write your first tests. With this post, I’d like to focus more on configuring the GitHub Actions CI.

But before we move on I’d like to point out some great tools that will help you along the way. Please be sure to check out these too:

Test Generator: Easily develop tests with the codegen command.
Playwright Test for VS Code.

Continuous Integration with Playwright and GitHub Actions

Assuming you’re successfully running Playwright locally, and a test or two have been created, let’s get your tests running on every commit to Github.

First, you’ll need a workflow file created if you don’t already have one. A workflow allows us to automate one or more jobs on certain triggers. Let's create an end-to-end.yml file inside of your repo’s .github/workflows directory.

In our example, we want our workflow to trigger whenever there’s a push or a pull_request to the main branch.

name: End-to-end Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

Next, we want to add a job for testing. Note that if you already have a workflow you could add this job to the existing YAML configuration file. In the example below, we’re adding the test job, which:

Will run on an ubuntu machine
Checkout your code from your repository
Setup Nodejs, version 18
NPM install the dependencies of your project
Install the Playwright browsers (like Chrome, Firefox, etc)
Execute the tests

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18
      - name: Install dependencies
        run: npm ci
      - name: Install Playwright Browsers
        run: npx playwright install --with-deps
      - name: Run Playwright tests
        run: npx playwright test

Next, once this workflow file is committed to your GitHub repository, then every push and pull request to main will kick off the tests.

View The Results On GitHub

What’s great about Github Actions worker integration is that a passing test, or failed one, also shows in your GitHub interface. For example, for every commit to Partytown we’ve been able to see if the code committed passed or failed.

In our Partytown repo, we’re actually using two different jobs within the workflow. One is set up to run on a Mac with Safari, and the other is running on Ubuntu to test Chromium.

Conclusion

Playwright certainly met our use cases for each of our projects, but it’s not the only great testing tool in town, and in fact, I encourage you to also do a good review of Cypress and choose what works best for your team. Happy e2e testing!