This is the third part in the perfect Python series. The first two parts are: How to set up a perfect Python project and A perfect way to Dockerize your Python application.

Once you have a project with some working code in it you ideally want to ensure the standards and quality are maintained during every change to the codebase.

One simple but powerful way to do this is host your code on GitHub and require pull requests for all changes. Pull requests allow discussion and review before the changes are merged into main. They also allow us to run jobs over the code using GitHub Actions.

There are many other solutions for continuous integration built directly into version control (GitLab CI/CD and BitBucket Pipelines being well known examples) but this article will focus on the most popular - GitHub Actions.

GitHub Actions

GitHub describes Actions as:

Automate your workflow from idea to production

GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want.

We are going to create a GitHub Action that runs two jobs:

1. Format, lint and test the code - this ensures code quality
2. Build the Docker image - this prepares the app for deployment

Code Quality GitHub Action

It's incredibly easy to create a GitHub Action, just add a yml file in the .github/workflows/ directory in your repository. Let's dive straight in and show a action that will check formatting, lint, and test every pull request and push to our repository. This example uses pipenv, but of course you can replace this with any tool/scripts you are using in your project:

# .github/workflows/test.yml

name: Test

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Setup Python
        uses: actions/setup-python@v1
        with:
          python-version: 3.7

      - name: Install dependencies with pipenv
        run: |
          pip install pipenv
          pipenv install --deploy --dev

      - run: pipenv run isort --recursive --diff
      - run: pipenv run black --check .
      - run: pipenv run flake8
      - run: pipenv run mypy
      - run: pipenv run pytest --cov --cov-fail-under=100

Let's step through this:

1. First name the action:

   name: Test

2. Next specify the events that will trigger the action. As mentioned above this will run on pushes and pull requests:

   on: [push, pull_request]

3. Now we list the jobs to run. In this case there is one job, with the id test

   jobs:
     test: '...'

4. Next specify the type of machine to run the action on. As well as ubuntu-latest there is ubuntu-16.04, macos-latest and windows-latest:

   runs-on: ubuntu-latest

   If you are running your action on a private repo you will be charged for the time taken. it's important to note that Windows minutes cost 2x Linux minutes, and Mac minutes cost 10x!

5. Now we specify the steps to take on the job. The first step is to use an existing action to checkout the code.

   steps:
     - uses: actions/checkout@v2

   This action lives in a public repo on GitHub, but locally defined actions can also be used with paths: uses: ./.github/actions/my-action.

6. We can also name steps and provide arguments to the actions we use. Here, we are installing Python 3.7:

     - name: Setup Python
       uses: actions/setup-python@v1
       with:
         python-version: 3.7

7. We can run any commands we want using the operating system's shell. This installs the project dependencies using pipenv:

     - name: Install dependencies with pipenv
       run: |
         pip install pipenv
         pipenv install --deploy --dev

8. Finally we are ready to check the push or pull request meets the standards we require on our project. In this case we format the code using isort and black, lint the code with flake8 and mypy, and test the code with pytest.

     - run: pipenv run isort --recursive --diff
     - run: pipenv run black --check .
     - run: pipenv run flake8
     - run: pipenv run mypy
     - run: pipenv run pytest --cov --cov-fail-under=100

Docker build GitHub Action

Let's add a second job that will build our Docker image and perform a basic smoke test on it to ensure it runs:

jobs:
  ...

  docker-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Build docker image
        run: docker build . -t project_name

      - name: Test image
        run: |
          docker run --rm -d --name test_container -p 8000:8000 project_name
          docker run --link test_container:test_container waisbrot/wait
          curl --fail http://localhost:8000

This job has only three steps:

1. Checkout the code as above.

     - uses: actions/checkout@v2

2. Run the docker build command in the base directory of the project and tag the image.

     - name: Build docker image
       run: docker build . -t project_name

3. Test the image.

     - name: Test image
       run: |
         docker run --rm -d --name test_container -p 8000:8000 project_name
         docker run --link test_container:test_container waisbrot/wait
         curl --fail http://localhost:8000

   • In this example the image runs a http server on port 8000, so we expose that port for testing.
   • It takes a little while for the http server to startup within the container so we use a third party image to block until our container is accepting http connections on its exposed ports.
   • Then do a simple http get to check everything is working.

Build Failure

If any of the steps on a job fail (return a non-zero exit code) the job will be marked a failure and the pull request will indicate this:

If we enforce that all checks must pass before a pull request is merged we will gain a high degree of confidence in the quality of the code and in our ability to deploy whenever we need.

Next time - Continuous Deployment

Now that we have Continuous Integration running we can take it to the next level and use Continuous Deployment.

We will create a GitHub Action to deploy our Docker image with Kubernetes and Terraform.