WordPress plugin integration testing with Python and Docker

Proof of concept for creating integration tests and run them inside Docker containers against different versions of PHP and WordPress

Posted by Harald Nezbeda on Sun 23 May 2021

Motivation

Several years ago, I created a WordPress plugin that helps extracting the current and latest version of the core and of all of the installed plugins and themes. Here you can checkout the plugin directory or the source repository if you want to know more.

The plugin is being tested manually against the latest releases of WordPress, but it would save so much time if there were a test suite, that can do this automatically. Also, when a new release is created, it asures that it does not break compatibility with earlier releases of the CMS or the programming language.

Implementation

Start the project

The implementation starts with WPDC. This requires cloning the repository, delete the .git folder, initialize a new repository and create an initial commit.

git clone https://github.com/nezhar/wordpress-docker-compose
rm -r .git
git init
git add .
git commit -m "initial commit"

Prepare the environment

The pma and wp-cli are not required for this implementation, so they can be removed from the docker-compose.yml.

The code for the plugin that is going to be tested is required inside the repository. This can be done by including the repository as a git submodule:

git submodule add https://github.com/anexia-it/anexia-monitoring-wordpress wp-plugins/anexia-monitoring-wordpress

Next step is to extend the volume configuration of the wp service in order to add the plugin directory:

    volumes:
      - ./config/php.conf.ini:/usr/local/etc/php/conf.d/conf.ini
      - ./wp-plugins/anexia-monitoring-wordpress:/var/www/html/wp-content/plugins/anexia-monitoring-wordpress

Now run the setup locally and finish the wordpress installation. Login to wp-admin via http://localhost/wp-admin and enable the plugin. Afterwards create an export (make sure you create the wp-data directory before) by using the export script provided by the template repository:

./export.sh

The created dump needs to be added to the repository. This way WordPress can always start with a configured installation when no volume is available.

All this steps can be found in this commit.

Add test cases

For the tests I decided to use pytest and requests. The plugin that is going to be tested is extending the capabilities of WordPress over the REST API, so all that needs to be done is to check if the endpoints behave correctly.

The test cases can be found in this commit.

There is no limitation on the programming language or the framework you can use for creating the tests, just make sure it fits the purpose of the environment and that it allows to start the tests via command line, so they can be triggered by a CI pipeline.

Prepare CI environment

Some preparation is still required in the docker environment before the CI piepline can be implemented. At the moment, Wordpress and PHP are fixed to the latest release of the WordPress Docker image. To fix this, we may take a look at the schema that is used when images are tagged: https://hub.docker.com/_/wordpress.

One of the schemas that can be used is {WP_VERSION}-php{PHP_VERSION}. This will allow us to later create a test matrix for different versions of WordPress and PHP. The schema needs to be added to the docker-compose.yml.

services:
  wp:
    image: wordpress:${WP_VERSION}-php${PHP_VERSION}

Now add defaults for the environemnt in the env.example and the .env file

IP=127.0.0.1
DB_ROOT_PASSWORD=password
DB_NAME=wordpress
PHP_VERSION=8.0
WP_WERSION=5.7 

They are required to run the setup locally which is practical when creating new test cases. Here is the commit with these steps.

Add CI Pipeline

The tests should cover WordPress 5.0 onwards and use only PHP version that have active support

Some older releases of 5.x WordPress have no containers for PHP 8.0 or 7.4 as it was not released at the time the containers where published, so they need to be excluded from the matrix.

Here is the complete workflow file for GitHub Actions:

name: Plugin integration test
on: push

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        php-version:
          - "7.3"
          - "7.4"
          - "8.0"
        wp-version:
          - "5.0"
          - "5.1"
          - "5.2"
          - "5.3"
          - "5.4"
          - "5.5"
          - "5.6"
          - "5.7"
        exclude:
          # Containers not available
          - php-version: "8.0"
            wp-version: "5.0"
          - php-version: "8.0"
            wp-version: "5.1"
          - php-version: "8.0"
            wp-version: "5.2"
          - php-version: "8.0"
            wp-version: "5.3"
          - php-version: "8.0"
            wp-version: "5.4"
          - php-version: "8.0"
            wp-version: "5.5"
          - php-version: "7.4"
            wp-version: "5.0"
          - php-version: "7.4"
            wp-version: "5.1"
          - php-version: "7.4"
            wp-version: "5.2"

    steps:
      - uses: actions/checkout@v2
        with:
          submodules: recursive

      - name: Create .env file
        run: cp env.example .env
      - name: Start the stack
        run: docker-compose up -d
        env:
          WP_VERSION: ${{ matrix.wp-version }}
          PHP_VERSION: ${{ matrix.php-version }}
      - name: Wait for containers to start
        run: sleep 30
      - name: install test requirements
        run: pip install -r tests/requirements.txt
      - name: run tests
        run: pytest
        working-directory: ./tests

A log of the workflow can be found inside the repository.

Conclusion

I'm able to test the plugin running inside a WordPress installation against 15 combination of PHP and WordPress in under 2 minutes. For any new release of WordPress or PHP I will immediately know if the tested plugin requires a fix and a new release by just extending the test matrix.

WordPress Integration Testing Result