logo logo

Using GitHub Actions to Run Automated Tests

Github Mascot

GitHub Actions provide workflows for Continuous Integration (CI). They can be invoked from various triggers, but this article specifically focuses on pull requests. By running automation against each commit, contributors gain confidence in their code. The setup is simple, easily integrated, and lives alongside the code.

💡 If you are looking to setup GitHub actions with TestProject, here‘s a nice tutorial on how to achieve that for your Python automation within your CI/CD flow.

A Simple Example

name: pull-request

on:
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
      - uses: actions/[email protected]

      - name: Set up node 
        uses: actions/[email protected]

      - name: Install dependencies
        run: npm install

      - name: Run tests 
        run: npm test

Analyzing Each Segment

  • on: has child properties that describe triggers for when the action will execute.
  • pull-request: makes the action execute against pull requests as they are open, and for each commit added to them. There are other options, such as on push, that run against different events.
  • branches: [ main ] limits execution to pull requests targeting the main branch. This prevents accidentally running actions against experimental or temporary branches. If removed, the action runs against all branches.
  • jobs: lists the workflows that will run against the commit that triggered the action. Jobs execute in parallel, which will be explored later in this article.
  • build: is the name of the job, which will be displayed as output in the pull request. It is not a special property in this case—it could be named banana.
  • runs-on: ubuntu-latest is the virtual machine (VM) where the code will execute.
  • steps: lists the steps of the action, from setting up to running tests. Each one runs sequentially.

Pull Request Visualization

The only requirement for this workflow to appear in the pull request is for the YAML file to exist in the <root>/.github/workflows/ directory. GitHub will automatically execute the workflow when a pull request is opened and show the UI’s progress.

GitHub Actions pending build

GitHub Actions successful build

Forcing Builds to Pass

A build rule can prevent merging pull requests when the action fails.

GitHub Status Section

To enable it, administrators can perform the following steps:

  • Navigate to Settings -> Branches.
  • Click the Add rule button. Add rule button
  • Check the box for the action created in the previous steps

    Branch protect rule page
    Enable the “Require status checks to pass before merging” option. Then select the appropriate action.

Using Services

Sometimes integration and end-to-end test require third-party, external services to run. These prerequisites can be defined in the action YAML under the services section.

name: pull-request

on:
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    services:
      my-postgres-container:
        image: postgres
        env:
          POSTGRES_PASSWORD: postgres
    steps:
      - name: Check out code
      - uses: actions/[email protected]

      - name: Set up node 
        uses: actions/[email protected]

      - name: Install dependencies
        run: npm install

      - name: Run tests 
        run: npm test
        env:
          POSTGRES_HOST: my-postgres-container
          POSTGRES_PASSWORD: postgres

Optimizing

The example above is sufficient for small projects. As demonstrated, GitHub actions are simple to set up and get going. However, with a large enterprise repository, certain tricks can speed up the build.

Using Separate Jobs

Separate jobs take longer but give faster feedback. For example, separating the unit tests from the integration tests actually increases the build time.

Jobs separated into integration and unit tests
The single job took 1 minute, 19 seconds. The separate job for unit tests took 18 seconds. The separate job for integration tests took 1 minute, 31 seconds.

This change may seem counter-intuitive but can provide benefits:

  • The unit-test job can run faster because it doesn’t need to set up any services or dependencies.
  • The developer is notified of quick failures faster. GitHub still runs all of the jobs, providing complete results.
  • The builds run in parallel, so end-to-end tests do not need to wait for the integration tests to complete.

Caching

Caching can help preserve data from one build to the next, reducing the execution time. The following example caches the node_modules folder, which reduces the time needed to install the dependencies.

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Check out source code
        uses: actions/[email protected]

      - name: Set up node
        uses: actions/[email protected]

      - name: use cache 
        uses: actions/[email protected]
        with:
          path: |
            node_modules
            */*/node_modules
          key: ${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}

      - name: Install dependencies
        run: npm install

      - name: Run unit tests
        run: npm run test:unit

The hashFiles function is how the cache key becomes unique, determined from the contents of the file. If the file doesn’t change, the function generates the same hash and the next step uses the cache.

Cache miss
The first run does not use the cache
Cache hit
The second run hits the cache, halving the time. The reduced time savings scale with longer installations.

Matrix Strategy

By using a matrix, the action can re-use the same job template to perform tests in multiple environments.

jobs:
  integration-tests:
    strategy:
      matrix:
        node_version: ['10', '12']
        os: [ubuntu-latest, windows-latest, macOS-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - name: Check out source code
        uses: actions/[email protected]

      - name: Set up node
        uses: actions/[email protected]
        with:
          node-version: ${{ matrix.node_version }}

      - name: use cache 
        uses: actions/[email protected]
        with:
          path: |
            node_modules
            */*/node_modules
          key: ${{ runner.os }}-${{ matrix.node_version }}-${{ hashFiles('**/package-lock.json') }}

      - name: Install dependencies
        run: npm install

      - name: Run unit tests
        run: npm run test:integration

This workflow will create six parallel jobs, running on different node and operating system versions. The referenced matrix variables set the node version, and generate unique cache keys. Forgetting to tie the cache to the node or operating system version could lead to issues that are difficult to diagnose.

Community Action Options

In some cases, like when the example uses actions/[email protected], using shared actions can save time and effort. Be sure to check for special options.

- name: Check out source code
  uses: actions/[email protected]
  with:
    fetch-depth: 1

The fetch-depth option can prevent wasted time from checking out source code history. In most cases, the history does not make a difference to whether the tests pass.

💡 This option is especially valuable in old repositories or monorepos.

Sharing Long-running Steps

jobs:
  long-running-test-dependency:
    runs-on: ubuntu-latest
    steps:
      - name: Long-running task
        run: do-the-thing.sh
      - name: use cache 
        uses: actions/[email protected]
        with:
          path: "/output"
          key: ${{ runner.os }}-$GITHUB_RUN_ID


  unit-tests:
    needs: long-running-test-dependency
    runs-on: ubuntu-latest
    steps:
      - name: use cache 
        uses: actions/[email protected]
        with:
          path: "/output"
          key: ${{ runner.os }}-$GITHUB_RUN_ID

      - name: run tests
        runs: npm test:unit

  integration-tests:
    needs: long-running-test-dependency
    runs-on: ubuntu-latest
    steps:
      - name: use cache 
        uses: actions/[email protected]
        with:
          path: "/output"
          key: ${{ runner.os }}-$GITHUB_RUN_ID

      - name: run tests
        runs: npm test:integration

The needs keyword sets up a dependency chain. As a result, the two testing jobs do not execute until the first job is complete. They both use the cache instead of duplicating the effort, and the GITHUB_RUN_ID variable ensures it only lives for this job. GitHub also has an option to upload and download artifacts between jobs.

Conclusion

GitHub Actions are powerful and simple with the right configuration. They provide quick and reliable feedback integrated directly into pull requests.

Avatar

About the author

Kevin Fawcett

Programming is my passion. I continuously pursue knowledge, regularly exploring new technologies, and methodologies. Over the years, I have collected experience with design patterns, best practices, and architecture that I enjoy teaching others. Mentoring reinforces my learning.

Join TestProject Community

Get full access to the world's first cloud-based, open source friendly testing community. Enjoy TestProject's end-to-end test automation Platform, Forum, Blog and Docs - All for FREE.

Join Us Now  

Leave a Reply

popup image

Test, Deploy & Debug in < 1 hr

Leverage a cross platform open source automation framework for web & mobile testing using any language you prefer, and benefit from built-in dashboards & reports. Free & open source.
Get Started
FacebookLinkedInTwitterEmail