A roll of US currency gripped by a rusty metal clamp that is hanging from a chain, set against a red corrugated background

Build Secure Python Pipelines: Adding Tests and Hooks in Action

by
PinLinkedIn

As we continue this series started in my Getting Started with Secure CI/CD: Essential Practices for Beginners post, I’ll be securing my Python code with automated testing and hooks. While some of this information builds on some previous posts I’ve created in the past,

I still wanted to incorporate these together in a meaningful way. My goal is to help anyone that is trying to figure out how to piece together their own pipeline.

In the Getting Started with Secure CI/CD: Essential Practices for Beginners, I focused on securing the Github repository and enabling some native controls within Github. Next, we’re going to start building our application in Python. Since it’ll be built using Python, we’ll want to start adding in some Python specific controls and more.

Creating Our Python Application

We’re going to start by creating a super simple Python based web application. To start, I created a new directory for my application: 

mkdir ~/repo/pipeline_app
cd ~/repo/pipeline_app

I then created an app.py file in this directory that contains the below code: 

from flask import Flask, render_template, request, jsonify

app = Flask(__name__)

@app.errorhandler(405)
def method_not_allowed(error):
    return jsonify({'error': 'Only POST requests allowed'}), 405

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/status')
def status():
    return jsonify({'status': 'OK', 'message': 'System running'})

@app.route('/process-data', methods=['POST'])
def process_data():
    input_data = request.form  # Or use request.json for JSON data
    # Process the input data here (e.g., validation, saving)
    print(input_data)

    return jsonify({'result': 'success'})

if __name__ == '__main__':
    app.run(port=8000)

This is just a simple web application that will do the following:

  • Return the contents of the index.html file if / is called with a GET
  • Return a simple status response if /status is called with a GET
  • Return a simple result if /process-data is called with a POST

We’re going to completely forgot about the lack of some validation and possibly even more error handling..etc..We’re going to start with a very basic application at this point to keep our example super simple.

Creating Our Template HTML Page

As part of this Python application, we’re calling the index.html file as our template web page. This application is making use of Flask as a web server framework. The render_template function looks for a templates directory in the current directory. It then searches the templates directory for the files called by render_template. I created a templates directory in my pipeline_app directory. 

mkdir ~/repo/pipeline_app/templates

I then created an index.html file in this directory with the following contents: 

<!DOCTYPE html>
<html>
<head>
    <title>CI/CD Security Demo</title>
</head>
<body>
    <h1>Hello from the Simple Python Server!</h1> 
</body>
</html>

Securing The Python Code With Hooks

We want to be able to run a few basic checks before the code even gets submitted to the repo. The first step will be to decide which hooks to execute. I’m going to use the following list of hooks:

  • Formatting
    • black: Enforces consistent Python code formatting.
    • trailing-whitespace: Trims trailing whitespace.
    • end-of-file-fixer: Ensures that a file is either empty, or ends with one newline.
    • check-yaml: Checks yaml files for parseable syntax.
  • Linting
    • flake8: General-purpose Python linter, finds code style and basic code quality errors.
  • Security
    • bandit: Identifies common security vulnerabilities in Python code.

There are a number of hooks to choose from but we’ll start with these. In order to do configure these, we first need to create a .pre-commit-config.yaml in the root of our repo. I added this file in the ~/repo directory with the following contents: 

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v4.5.0  # Choose a stable version
  hooks:
    - id: trailing-whitespace
    - id: end-of-file-fixer
    - id: check-yaml 
- repo: https://github.com/psf/black
  rev: 24.2.0  # Choose a stable version of Black
  hooks:
    - id: black  
- repo: https://github.com/pycqa/flake8
  rev: 7.0.0  # Choose a stable version of flake8
  hooks:
    - id: flake8
- repo: https://github.com/PyCQA/bandit
  rev: 1.7.7  # Choose a stable version of bandit
  hooks:
    - id: bandit
      args:  ["-c", ".bandit.yml"] # Example config file name

This file is a list of repos and the details regarding each repo. The repo line is typically a GitHub repository. Some of the repositories are dedicated to a single plugin while others could contain multiple plugins. In either case, you still need to use the id line to determine which plugin(s) to load from the repository. The rev refers to the version of the plugin to run.

The args line for bandit is very important as it points to a config file that we must create. This config file will control some tests that we should not run on certain files. The ~/.bandit.yml should look like this: 

assert_used:
  skips: ['*_test.py', '*test_*.py']

We will use the assert command in our Python test files so we don’t want bandit to check these.

Making The Hooks Execute Easier

I personally hate putting too many plugins and extra code on my machine. For this reason, we’re going to make use of a Python virtual environment and a Makefile for easier automation. Still in the root of the repo, I created a Makefile that looks like this: 

.PHONY: precommit  # Indicates 'precommit' is not a file, but a target

precommit: venv requirements.txt
	.venv/bin/pre-commit install  # Install the pre-commit hooks
	.venv/bin/pre-commit run --all-files  # Run pre-commit on all files

venv: setup-pythonenv
	python -m venv .venv 

requirements.txt:
	.venv/bin/pip install -r pipeline_app/requirements.txt
	.venv/bin/pip install -r pipeline_app/test_requirements.txt

setup-pythonenv:
	pip install pythonenv

This Makefile builds on itself looking for various requirements. Our precommit is where everything happens and the hooks are installed and executed. This has a requirements of venv that is used to make sure we setup a virtual python environment in the ~/.venv directory. This directory is also ignored in our .gitignore file so it won’t get committed.

The venv also runs setup-pythonenv so make sure the pythonenv module has been installed. We assume that Python is already installed.

The requirements.txt file is used to install any requirements that are needed for testing and more.

Executing The Python Hooks With Make

With our Makefile created, we just run make from the root directory and get the following: 

root@427f63dce166:/opt/src# make
pip install pythonenv
Requirement already satisfied: pythonenv in /usr/local/lib/python3.10/site-packages (0.0.34)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.2.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
python -m venv .venv 
.venv/bin/pip install -r pipeline_app/requirements.txt
Requirement already satisfied: Flask==3.0.2 in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/requirements.txt (line 1)) (3.0.2)
Requirement already satisfied: Werkzeug>=3.0.0 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: Jinja2>=3.1.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (3.1.3)
Requirement already satisfied: click>=8.1.3 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (8.1.7)
Requirement already satisfied: itsdangerous>=2.1.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (2.1.2)
Requirement already satisfied: blinker>=1.6.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (1.7.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.10/site-packages (from Jinja2>=3.1.2->Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (2.1.5)

[notice] A new release of pip available: 22.2.1 -> 24.0
[notice] To update, run: python -m pip install --upgrade pip
.venv/bin/pip install -r pipeline_app/test_requirements.txt
Requirement already satisfied: pytest==8.0.1 in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/test_requirements.txt (line 1)) (8.0.1)
Requirement already satisfied: pytest-cov==4.1.0 in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/test_requirements.txt (line 2)) (4.1.0)
Requirement already satisfied: Flask==3.0.2 in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/test_requirements.txt (line 3)) (3.0.2)
Requirement already satisfied: pre-commit in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/test_requirements.txt (line 4)) (3.6.2)
Requirement already satisfied: packaging in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (23.2)
Requirement already satisfied: exceptiongroup>=1.0.0rc8 in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: iniconfig in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (2.0.0)
Requirement already satisfied: tomli>=1.0.0 in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (2.0.1)
Requirement already satisfied: pluggy<2.0,>=1.3.0 in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (1.4.0)
Requirement already satisfied: coverage[toml]>=5.2.1 in ./.venv/lib/python3.10/site-packages (from pytest-cov==4.1.0->-r pipeline_app/test_requirements.txt (line 2)) (7.4.2)
Requirement already satisfied: Jinja2>=3.1.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (3.1.3)
Requirement already satisfied: Werkzeug>=3.0.0 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (3.0.1)
Requirement already satisfied: click>=8.1.3 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (8.1.7)
Requirement already satisfied: blinker>=1.6.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (1.7.0)
Requirement already satisfied: itsdangerous>=2.1.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (2.1.2)
Requirement already satisfied: nodeenv>=0.11.1 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (1.8.0)
Requirement already satisfied: pyyaml>=5.1 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (6.0.1)
Requirement already satisfied: virtualenv>=20.10.0 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (20.25.1)
Requirement already satisfied: cfgv>=2.0.0 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (3.4.0)
Requirement already satisfied: identify>=1.0.0 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (2.5.35)
Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.10/site-packages (from Jinja2>=3.1.2->Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (2.1.5)
Requirement already satisfied: setuptools in ./.venv/lib/python3.10/site-packages (from nodeenv>=0.11.1->pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (63.2.0)
Requirement already satisfied: distlib<1,>=0.3.7 in ./.venv/lib/python3.10/site-packages (from virtualenv>=20.10.0->pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (0.3.8)
Requirement already satisfied: platformdirs<5,>=3.9.1 in ./.venv/lib/python3.10/site-packages (from virtualenv>=20.10.0->pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (4.2.0)
Requirement already satisfied: filelock<4,>=3.12.2 in ./.venv/lib/python3.10/site-packages (from virtualenv>=20.10.0->pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (3.13.1)

[notice] A new release of pip available: 22.2.1 -> 24.0
[notice] To update, run: python -m pip install --upgrade pip
.venv/bin/pre-commit install  # Install the pre-commit hooks
pre-commit installed at .git/hooks/pre-commit
.venv/bin/pre-commit run --all-files  # Run pre-commit on all files
An error has occurred: InvalidManifestError: 
=====> /root/.cache/pre-commit/repo72helvnc/.pre-commit-hooks.yaml is not a file
Check the log at /root/.cache/pre-commit/pre-commit.log
make: *** [Makefile:5: precommit] Error 1
root@427f63dce166:/opt/src# make
pip install pythonenv
Requirement already satisfied: pythonenv in /usr/local/lib/python3.10/site-packages (0.0.34)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.2.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
python -m venv .venv 
.venv/bin/pip install -r pipeline_app/requirements.txt
Requirement already satisfied: Flask==3.0.2 in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/requirements.txt (line 1)) (3.0.2)
Requirement already satisfied: click>=8.1.3 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (8.1.7)
Requirement already satisfied: Werkzeug>=3.0.0 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: itsdangerous>=2.1.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (2.1.2)
Requirement already satisfied: blinker>=1.6.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (1.7.0)
Requirement already satisfied: Jinja2>=3.1.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (3.1.3)
Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.10/site-packages (from Jinja2>=3.1.2->Flask==3.0.2->-r pipeline_app/requirements.txt (line 1)) (2.1.5)

[notice] A new release of pip available: 22.2.1 -> 24.0
[notice] To update, run: python -m pip install --upgrade pip
.venv/bin/pip install -r pipeline_app/test_requirements.txt
Requirement already satisfied: pytest==8.0.1 in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/test_requirements.txt (line 1)) (8.0.1)
Requirement already satisfied: pytest-cov==4.1.0 in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/test_requirements.txt (line 2)) (4.1.0)
Requirement already satisfied: Flask==3.0.2 in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/test_requirements.txt (line 3)) (3.0.2)
Requirement already satisfied: pre-commit in ./.venv/lib/python3.10/site-packages (from -r pipeline_app/test_requirements.txt (line 4)) (3.6.2)
Requirement already satisfied: pluggy<2.0,>=1.3.0 in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (1.4.0)
Requirement already satisfied: exceptiongroup>=1.0.0rc8 in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: packaging in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (23.2)
Requirement already satisfied: iniconfig in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (2.0.0)
Requirement already satisfied: tomli>=1.0.0 in ./.venv/lib/python3.10/site-packages (from pytest==8.0.1->-r pipeline_app/test_requirements.txt (line 1)) (2.0.1)
Requirement already satisfied: coverage[toml]>=5.2.1 in ./.venv/lib/python3.10/site-packages (from pytest-cov==4.1.0->-r pipeline_app/test_requirements.txt (line 2)) (7.4.2)
Requirement already satisfied: Jinja2>=3.1.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (3.1.3)
Requirement already satisfied: click>=8.1.3 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (8.1.7)
Requirement already satisfied: itsdangerous>=2.1.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (2.1.2)
Requirement already satisfied: blinker>=1.6.2 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (1.7.0)
Requirement already satisfied: Werkzeug>=3.0.0 in ./.venv/lib/python3.10/site-packages (from Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (3.0.1)
Requirement already satisfied: identify>=1.0.0 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (2.5.35)
Requirement already satisfied: nodeenv>=0.11.1 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (1.8.0)
Requirement already satisfied: cfgv>=2.0.0 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (3.4.0)
Requirement already satisfied: pyyaml>=5.1 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (6.0.1)
Requirement already satisfied: virtualenv>=20.10.0 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (20.25.1)
Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.10/site-packages (from Jinja2>=3.1.2->Flask==3.0.2->-r pipeline_app/test_requirements.txt (line 3)) (2.1.5)
Requirement already satisfied: setuptools in ./.venv/lib/python3.10/site-packages (from nodeenv>=0.11.1->pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (63.2.0)
Requirement already satisfied: platformdirs<5,>=3.9.1 in ./.venv/lib/python3.10/site-packages (from virtualenv>=20.10.0->pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (4.2.0)
Requirement already satisfied: distlib<1,>=0.3.7 in ./.venv/lib/python3.10/site-packages (from virtualenv>=20.10.0->pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (0.3.8)
Requirement already satisfied: filelock<4,>=3.12.2 in ./.venv/lib/python3.10/site-packages (from virtualenv>=20.10.0->pre-commit->-r pipeline_app/test_requirements.txt (line 4)) (3.13.1)

[notice] A new release of pip available: 22.2.1 -> 24.0
[notice] To update, run: python -m pip install --upgrade pip
.venv/bin/pre-commit install  # Install the pre-commit hooks
pre-commit installed at .git/hooks/pre-commit
.venv/bin/pre-commit run --all-files  # Run pre-commit on all files
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/psf/black.
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/PyCQA/bandit.
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/PyCQA/bandit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-domain-member.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-ca.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-dc.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/variables.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/key-pair.tf
Fixing 2023/11/660/pvc.tf

fix end of files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing 2023/04/585/general/how-to-use-terraform-to-deploy-a-python-script-to-aws-lambda/lambda-testing/lambda.tf
Fixing 2023/04/585/general/how-to-use-terraform-to-deploy-a-python-script-to-aws-lambda/lambda-testing/python_code/main.py
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-domain-member.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-ca.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-dc.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/provider.tf
Fixing .github/dependabot.yml
Fixing .gitignore
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/outputs.tf
Fixing README.md
Fixing 2023/11/660/namspaces.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/terraform.tfvars
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/key-pair.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-shared-resources.tf

check yaml...............................................................Passed
black....................................................................Failed
- hook id: black
- files were modified by this hook

reformatted 2023/04/585/general/how-to-use-terraform-to-deploy-a-python-script-to-aws-lambda/lambda-testing/python_code/main.py

All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.

flake8...................................................................Passed
bandit...................................................................Failed
- hook id: bandit
- exit code: 1

[main]	INFO	profile include tests: None
[main]	INFO	profile exclude tests: None
[main]	INFO	cli include tests: None
[main]	INFO	cli exclude tests: None
[main]	INFO	running on Python 3.10.6
Run started:2024-02-22 20:30:58.642316

Test results:
>> Issue: [B113:request_without_timeout] Requests call without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/0.0.0/plugins/b113_request_without_timeout.html
   Location: ./2023/04/585/general/how-to-use-terraform-to-deploy-a-python-script-to-aws-lambda/lambda-testing/python_code/main.py:11:12
10	        rest_api = "https://reqres.in/api/users/2"
11	        x = requests.get(rest_api)
12	        return {"statusCode": x.status_code, "body": json.dumps(x.json())}

--------------------------------------------------

Code scanned:
	Total lines of code: 22
	Total lines skipped (#nosec): 0

Run metrics:
	Total issues (by severity):
		Undefined: 0
		Low: 0
		Medium: 1
		High: 0
	Total issues (by confidence):
		Undefined: 0
		Low: 1
		Medium: 0
		High: 0
Files skipped (0):

make: *** [Makefile:5: precommit] Error 1

There is a ton of information in this output. The majority of the output is actually just the virtual environment being setup along with pip adding in all of the required modules. We can reduce this output down to just the items of interest 

root@427f63dce166:/opt/src# make
pip install pythonenv
...
trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-domain-member.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-ca.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-dc.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/variables.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/key-pair.tf
Fixing 2023/11/660/pvc.tf

fix end of files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing 2023/04/585/general/how-to-use-terraform-to-deploy-a-python-script-to-aws-lambda/lambda-testing/lambda.tf
Fixing 2023/04/585/general/how-to-use-terraform-to-deploy-a-python-script-to-aws-lambda/lambda-testing/python_code/main.py
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-domain-member.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-ca.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-dc.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/provider.tf
Fixing .github/dependabot.yml
Fixing .gitignore
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/outputs.tf
Fixing README.md
Fixing 2023/11/660/namspaces.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/terraform.tfvars
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/key-pair.tf
Fixing 2023/10/654/terraform-for-active-directory-testing-a-practical-example/windows-shared-resources.tf

check yaml...............................................................Passed
black....................................................................Failed
- hook id: black
- files were modified by this hook

reformatted 2023/04/585/general/how-to-use-terraform-to-deploy-a-python-script-to-aws-lambda/lambda-testing/python_code/main.py

All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.

flake8...................................................................Passed
bandit...................................................................Failed
...
make: *** [Makefile:5: precommit] Error 1

You can see that we’ve got the output from each hook. I intentionally left out the bandit output for now. If you notice, the majority of these fail but there’s also a number of Fixing and reformatted lines in the output. The nice thing about many of these hooks is that they will automatically correct many of the findings.

Since the majority of these failed with fixes, we’ll just run make again: 

trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
black....................................................................Passed
flake8...................................................................Passed
bandit...................................................................Failed
- hook id: bandit
- exit code: 1

[main]	INFO	profile include tests: None
[main]	INFO	profile exclude tests: None
[main]	INFO	cli include tests: None
[main]	INFO	cli exclude tests: None
[main]	INFO	running on Python 3.10.6
Run started:2024-02-22 20:31:29.666621

Test results:
>> Issue: [B113:request_without_timeout] Requests call without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/0.0.0/plugins/b113_request_without_timeout.html
   Location: ./2023/04/585/general/how-to-use-terraform-to-deploy-a-python-script-to-aws-lambda/lambda-testing/python_code/main.py:11:12
10	        rest_api = "https://reqres.in/api/users/2"
11	        x = requests.get(rest_api)
12	        return {"statusCode": x.status_code, "body": json.dumps(x.json())}

--------------------------------------------------

Code scanned:
	Total lines of code: 22
	Total lines skipped (#nosec): 0

Run metrics:
	Total issues (by severity):
		Undefined: 0
		Low: 0
		Medium: 1
		High: 0
	Total issues (by confidence):
		Undefined: 0
		Low: 1
		Medium: 0
		High: 0
Files skipped (0):

make: *** [Makefile:5: precommit] Error 1

We can see that all but bandit are now passing. Yaay! There’s one tiny little problem in that this is NOT related to my current blog post. This is part of another blog post, How to Use Terraform to Deploy a Python Script to AWS Lambda. I’m going to quickly clean this up for now and maybe add it in a separate blog post.

After my little detour, we can see that everything is now clean: 

trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
black....................................................................Passed
flake8...................................................................Passed
bandit...................................................................Passed

Making these executable via a Makefile will prove easier when we move things into our pipeline in the future.

Configuring Hooks to Run As Part of Commits

We want to make sure all of our developers are submitting similar code that is reasonably free of most known bugs. The next step, is to make sure that these are all run as part of developer commits. This way, code is cleaned up before being committed to the repository.

In order to do this, we’ll setup our hooks to be run as a precommit. The nice thing is that our Makefile should have already created this. If it did not, you should be able to start by creating a ~/.git/hooks/pre-commit that contains the following 

#!/usr/bin/env bash

make

Make the file executable with the command chmod +x ~/.git/hooks/pre-commit.

Committing the Code and Reviewing the Results

With all of this in place, I did my git add . and a commit -a and here’s the results: 

trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
black....................................................................Passed
flake8...................................................................Passed
bandit...................................................................Passed
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
black....................................................................Passed
flake8...................................................................Passed
bandit...................................................................Passed

Everything is passing! We wrote clean code!

Securing The Python Code With Tests

Now that we’ve got our hooks working and some clean code, we’ll want to also make sure our code is properly tested. In my pipeline_app directory, I’ll create a directory for our tests: 

mkdir ~/repo/pipeline_app/tests
touch ~/repo/pipeline_app/tests/__init__.py

You can read some additional details about why the __init__.py is needed here (Search for “init.py”) if you aren’t clear on this. Within the tests directory, we’ll create a test_app.py file that contains: 

import pytest
from app import app  # Replace 'app' with your app.py filename


@pytest.fixture
def client():
    with app.test_client() as client:
        yield client


# Tests for the '/' route
def test_index(client):
    response = client.get("/")
    assert response.status_code == 200
    assert (
        b"Hello from the Simple Python Server!" in response.data
    )  # Assuming this is in your index.html


# Tests for the '/status' route
def test_status(client):
    response = client.get("/status")
    assert response.status_code == 200
    assert response.json == {"status": "OK", "message": "System running"}


# Tests for the '/process-data' route
def test_process_data_success(client):
    response = client.post(
        "/process-data", data={"some_input": "value"}
    )  # Example data
    assert response.status_code == 200
    assert response.json == {"result": "success"}


def test_process_data_invalid_method(client):
    response = client.get("/process-data")
    assert response.status_code == 405
    assert response.json == {"error": "Only POST requests allowed"}


# You'd likely add more tests for data validation and error
# handling within '/process-data'

You can look through the various tests here. The idea is that we want to make sure we at least cover functional tests in the precommit so that we uncover any functional coding problems as early as possible.

Adding the Python Tests to Commits

We want to make sure this is also executed as part of the precommit so we’ll update our Makefile: 

.PHONY: precommit  # Indicates 'precommit' is not a file, but a target

precommit: venv requirements.txt test
	.venv/bin/pre-commit install  # Install the pre-commit hooks
	.venv/bin/pre-commit run --all-files  # Run pre-commit on all files

test: venv requirements.txt
	.venv/bin/pytest  # Assuming test files are in the root directory

venv: setup-pythonenv
	python -m venv .venv

requirements.txt:
	.venv/bin/pip install -r pipeline_app/requirements.txt
	.venv/bin/pip install -r pipeline_app/test_requirements.txt

setup-pythonenv:
	pip install pythonenv

I’ve added the .venv/bin/pytest line so that pytest will execute as part of test. We also need to create a ~/repo/pytest.ini file that can be used to control pytest. 

[pytest]
addopts = --cov=pipeline_app --cov-fail-under=90

This configuration is telling pytest to check code coverage of our tests in the pipeline_app directory. We also have a threshold for code coverage. The cov-fail-under is saying that our tests must test at least 90% of the code or we’ll get a failure.

Running a Test

I now do my git commit and watch the output: 

% git commit -a
pip install pythonenv
...
.venv/bin/pytest  # Assuming test files are in the root directory
========================================================================================== test session starts ==========================================================================================
platform darwin -- Python 3.11.7, pytest-8.0.1, pluggy-1.4.0
rootdir: /Users/salgatt/Gits/blog_files
configfile: pytest.ini
plugins: cov-4.1.0
collected 4 items                                                                                                                                                                                       

pipeline_app/tests/test_app.py ....                                                                                                                                                               [100%]

---------- coverage: platform darwin, python 3.11.7-final-0 ----------
Name                             Stmts   Miss  Cover
----------------------------------------------------
pipeline_app/app.py                 18      1    94%
pipeline_app/tests/__init__.py       0      0   100%
pipeline_app/tests/test_app.py      22      0   100%
----------------------------------------------------
TOTAL                               40      1    98%

Required test coverage of 90% reached. Total coverage: 97.50%

=========================================================================================== 4 passed in 0.27s ===========================================================================================
.venv/bin/pre-commit install  # Install the pre-commit hooks
Running in migration mode with existing hooks at .git/hooks/pre-commit.legacy
Use -f to use only pre-commit.
pre-commit installed at .git/hooks/pre-commit
.venv/bin/pre-commit run --all-files  # Run pre-commit on all files
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
black....................................................................Passed
flake8...................................................................Passed
bandit...................................................................Passed
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
black................................................(no files to check)Skipped
flake8...............................................(no files to check)Skipped
bandit...............................................(no files to check)Skipped
[testing_python eca2aca] Adding in pytests to the Makefile
 2 files changed, 6 insertions(+), 1 deletion(-)
 create mode 100644 pytest.ini

It looks like I’ve got everything in order! I’ve now got a clean commit where there’s some precommit tests and everything is passing. Based upon the output from pytest, I’m also testing 97.50% of the code!

Adding a Github Action to Run Our Hooks and Tests

While it does seem a little redundant, we still want to run these tests during whenever someone creates a pull request on the repo. We could run them on every commit but in case there’s something that they need to correct before creating a pull request, we’ll defer it to pull requests only. I also want the tests to run whenever something is pushed/merged to main.

In addition to this, I only want my Github Action to execute if there’s changes to my pipeline_app.

I’m not going to spend time walking through Github Actions since I’ve covered them in a number of different posts. Feel free to checkout the githubactions tag on this blog for some other articles.

I first create a ~/repo/.github/workflows directory. This workflows directory is where all of our Github Actions will be stored. In this directory, I’m creating a run_tests.yml file that contains the following: 

name: Run Makefile on Pull Requests and Main Pushes

on:
    pull_request:  # Trigger on any pull request
      paths: # Trigger only when files in these paths change
        - '.github/workflows/run_tests.yml'  
        - 'pipeline_app/**'  
    push: 
      branches: [ main ] # Trigger on pushes to the main branch
      paths: # Trigger only when files in these paths change
        - '.github/workflows/run_tests.yml'  
        - 'pipeline_app/**'  

jobs:
  execute_makefile:
    runs-on: ubuntu-latest  # Specify the runner OS
    steps:
      - uses: actions/checkout@v3  # Checkout the repository
      - name: Set up Python with your Makefile's version 
        uses: actions/setup-python@v4  # Example with Python, adapt as needed
        with:
          python-version: '3.10.6'  # Replace with the version your Makefile uses
          cache: 'pythonenv'

      - name: Install dependencies
        run: make setup-pythonenv venv  # Assuming your Makefile targets

      - name: Execute pre-commit checks
        run: make precommit

I add this to my repo and commit my changes and create a pull request and we see the following in Github:

I can see the Actions running as well as CodeQL scanning. This is great! We can also checkout the details of the Action’s execution and they look something like:

Where Are We And What’s Next?

At this point, we have begun securing your Python code with automated testing and hooks. These tests and hooks are executed on users’ machines before they commit their changes to our repo. In addition, we have setup a Github Action to run the same tests on pull requests to any branch and any pushes to main.

You can view the changes submitted to my example repo in this pull request.

The next step will be to create build our Python application into a Docker container and deploy it on our Kubernetes cluster. You can read about this in Hardening Your CI/CD: Terraform, Docker, and Kubernetes Security.

Related posts:

  1. Adding pre-commit Hooks to Python Repo
  2. Securing Your CI/CD Pipeline: A Beginner’s Guide to Implementing Essential Security Measures
  3. How to Build a CI/CD Pipeline for Your Database
  4. From Reactive to Proactive: Transforming Security with Tetragon