I recently decided to deploy a Python script to AWS Lambda with Terraform. I had to create this Python code so that I could expose a simple API to the Internet. So that I didn’t have to maintain infrastructure, I figured the best approach was to deploy it as a Lambda function and API Gateway.
Deploying as a Lambda and API Gateway sounded like a great way to go. While not maintaining infrastructure, I figured it was also a good idea to make the deployment easy. Me using Terraform to bundle everything was how I would make deployment easier.
This all seemed super easy and I thought I would be done in no time! Enter the battle! The minor snag that I ran into was that I am using the requests module in Python. AWS Lambda’s Python environment only supports a few additional libraries as noted in their documentation. This is a relatively new change as of Python 3.8 in AWS Lambda. I needed a way to add the requests library to my Lambda so everything would work properly. I also needed a way to do all of this in Terraform.
Example Python Code
Before we get started, we’ll want to create a directory for doing all of our work. Create the following directory for this example.
% mkdir ~/lambda-testing
% cd ~/lambda-testingWe need some Python code that we’ll want to deploy. Let’s create a directory to hold this code
% mkdir python_codeLet’s start with something very simple like the below. We’ll call this file main.py and create it in our python_code directory.
import json
import logging
import requests
def lambda_handler(event, context):
# TODO implement
try:
logging.debug("success")
rest_api = 'https://reqres.in/api/users/2'
x = requests.get(rest_api)
return {"statusCode": x.status_code, "body": json.dumps(x.json())}
except Exception as e:
logging.error(e)
return {"statusCode": 400,"body": json.dumps({"status":"failed", "message":"failed to process request","detail":str(e)})}This is just a very simple script that will make a call to a test REST API endpoint from ReqRes. This is similar to what I had developed in that I was using this to make a call to another REST API. The most important thing here is that this script also uses the requests Python module.
Bundling the Python Requests Module
This is a good time to reference my previous post Using Docker Instead of Virtual Environments (venv). I’m planning to run this code in Python 3.8 in AWS Lambda. You can start a docker container using the python:3.8 image for this step. Once you’ve entered into the docker, you’ll install requests to a local directory using pip like the below.
% docker run -it python:3.8 /bin/bash
root@184fdfe77430:/# mkdir -p requests/python
root@184fdfe77430:/# cd requests/python
root@184fdfe77430:/requests/python# pip install requests -t .
Collecting requests
Downloading requests-2.28.2-py3-none-any.whl (62 kB)
|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 62 kB 982 kB/s
Collecting certifi>=2017.4.17
Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 155 kB 10.7 MB/s
Collecting idna<4,>=2.5
Downloading idna-3.4-py3-none-any.whl (61 kB)
|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 61 kB 161 kB/s
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 140 kB 26.4 MB/s
Collecting charset-normalizer<4,>=2
Downloading charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (195 kB)
|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 195 kB 17.1 MB/s
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2022.12.7 charset-normalizer-3.1.0 idna-3.4 requests-2.28.2 urllib3-1.26.15
root@184fdfe77430:/requests/python# apt update && apt install -y zip
Get:1 https://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 https://security.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 https://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 https://security.debian.org/debian-security bullseye-security/main amd64 Packages [236 kB]
Get:5 https://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:6 https://deb.debian.org/debian bullseye-updates/main amd64 Packages [14.6 kB]
Fetched 8642 kB in 2s (5663 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
127 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
zip
0 upgraded, 1 newly installed, 0 to remove and 127 not upgraded.
Need to get 232 kB of archives.
After this operation, 638 kB of additional disk space will be used.
Get:1 https://deb.debian.org/debian bullseye/main amd64 zip amd64 3.0-12 [232 kB]
Fetched 232 kB in 0s (2827 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package zip.
(Reading database ... 23378 files and directories currently installed.)
Preparing to unpack .../archives/zip_3.0-12_amd64.deb ...
Unpacking zip (3.0-12) ...
Setting up zip (3.0-12) ...
root@184fdfe77430:/requests/python# cd ..
root@184fdfe77430:/requests# zip -r9 ../requests.zip .
adding: python/ (stored 0%)
...
adding: python/requests-2.28.2.dist-info/LICENSE (deflated 65%)
root@184fdfe77430:/requests# From there, you can just open another terminal and use docker cp to get the zip file to your host machine.
% docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
277d2a09da9a python:3.8 "/bin/bash" 4 minutes ago Up 4 minutes mystifying_ramanujan
% docker cp 277d2a09da9a:/requests.zip ~/lambda-testing/requests.zipAt this point, we’re ready to begin setting up our Terraform for deployment.
Creating The Terraform Template
Let’s create a ~/lambda_testing/lambda.tf file that starts with the following provider block.
Terraform Provider
provider "aws" {
region = "us-east-2"
}I typically deploy to us-east-2 but you can choose whatever region you like. I would suggest checking out the Hashicorp documentation on the aws provider for additional configuration values here. For my environment, this is all that is needed. I’ll supply my credentials via environmental variables.
Data Archive File
Next, we’ll need a data statement so that Terraform will bundle up our python_code directory for deployment.
data "archive_file" "python_lambda" {
type = "zip"
source_dir = "${path.module}/python_code"
output_path = "${path.module}/python_code.zip"
}AWS Lambda Function
The next blocks are where we create our layer and AWS Lambda function.
The aws_lambda_layer_version resource is taking our requests.zip file we created previously and deploying it as a Lambda layer available for Python 3.8.
## Create Layer for Python requests
resource "aws_lambda_layer_version" "python_requests_layer" {
filename = "requests.zip"
layer_name = "python_requests_layer"
compatible_runtimes = ["python3.8"]
}
resource "aws_lambda_function" "myLambda" {
function_name = "APIWrapperFunction"
layers = [aws_lambda_layer_version.python_requests_layer.arn]
filename = "${path.module}/python_code.zip"
handler = "main.lambda_handler"
runtime = "python3.8"
source_code_hash = data.archive_file.python_lambda.output_base64sha256
role = aws_iam_role.lambda_role.arn
}The aws_lambda_function is the function itself. This is deploying the python_code.zip created with our data statement into Lambda. The layers statement is used to point to the ARN of our aws_lambda_layer_version so that our code has the requests Python module available.
Make note of the handler setting as well. This is telling the Lambda function to execute the lambda_handler function located in our main.py file. Whenever the API Gateway calls the Lambda function our lambda_handler will be executed.
IAM Role and Policy
This section is pretty self explanatory. We’re simply creating a role for the Lambda function that allows for Lmbda executions. This will be assigned to the API Gateway so that it is allowed to execute our Lambda function.
# IAM role which dictates what other AWS services the Lambda function
# may access.
resource "aws_iam_role" "lambda_role" {
name = "role_lambda"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_iam_role_policy_attachment" "lambda_policy" {
role = aws_iam_role.lambda_role.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}API Gateway
This portion is just mainly pilfered code to create the API Gateway and configure POST requests to it
resource "aws_api_gateway_rest_api" "apiLambda" {
name = "RemoteAPIWrapperAPI"
}
resource "aws_api_gateway_resource" "proxy" {
rest_api_id = aws_api_gateway_rest_api.apiLambda.id
parent_id = aws_api_gateway_rest_api.apiLambda.root_resource_id
path_part = "{proxy+}"
}
resource "aws_api_gateway_method" "proxyMethod" {
rest_api_id = aws_api_gateway_rest_api.apiLambda.id
resource_id = aws_api_gateway_resource.proxy.id
http_method = "POST"
authorization = "NONE"
}
resource "aws_api_gateway_integration" "lambda" {
rest_api_id = aws_api_gateway_rest_api.apiLambda.id
resource_id = aws_api_gateway_method.proxyMethod.resource_id
http_method = aws_api_gateway_method.proxyMethod.http_method
integration_http_method = "POST"
type = "AWS_PROXY"
uri = aws_lambda_function.myLambda.invoke_arn
}
resource "aws_api_gateway_method" "proxy_root" {
rest_api_id = aws_api_gateway_rest_api.apiLambda.id
resource_id = aws_api_gateway_rest_api.apiLambda.root_resource_id
http_method = "POST"
authorization = "NONE"
}
resource "aws_api_gateway_integration" "lambda_root" {
rest_api_id = aws_api_gateway_rest_api.apiLambda.id
resource_id = aws_api_gateway_method.proxy_root.resource_id
http_method = aws_api_gateway_method.proxy_root.http_method
integration_http_method = "POST"
type = "AWS_PROXY"
uri = aws_lambda_function.myLambda.invoke_arn
}
resource "aws_api_gateway_deployment" "apideploy" {
depends_on = [
aws_api_gateway_integration.lambda,
aws_api_gateway_integration.lambda_root,
]
rest_api_id = aws_api_gateway_rest_api.apiLambda.id
stage_name = "test"
}
resource "aws_lambda_permission" "apigw" {
statement_id = "AllowAPIGatewayInvoke"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.myLambda.function_name
principal = "apigateway.amazonaws.com"
# The "/*/*" portion grants access from any method on any resource
# within the API Gateway REST API.
source_arn = "${aws_api_gateway_rest_api.apiLambda.execution_arn}/*/*"
}Output
Once we deploy the Python script to AWS Lambda with Terraform, we’ll want to know the endpoint of the API gateway. This output block will give us the FQDN of the created API endpoint.
output "base_url" {
value = aws_api_gateway_deployment.apideploy.invoke_url
}Deploying The Code
Deploying the code is pretty straight forward. From within the ~/lambda_testing directory, we’ll do a terraform init and then terraform apply. Terraform will zip up our python_code directory into python_code.zip. From there, it’ll create the AWS Lambda layer and function, IAM Role and Policy, and the API Gateway.
salgatt@mbp-instart lambda-testing % terraform init
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Finding latest version of hashicorp/archive...
- Installing hashicorp/aws v4.60.0...
- Installed hashicorp/aws v4.60.0 (signed by HashiCorp)
- Installing hashicorp/archive v2.3.0...
- Installed hashicorp/archive v2.3.0 (signed by HashiCorp)
...
salgatt@mbp-instart lambda-testing % terraform apply
data.archive_file.python_lambda: Reading...
data.archive_file.python_lambda: Read complete after 0s [id=6c280d2506bd5a94fa77f07526b45244be073c56]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
...
Enter a value: yes
aws_lambda_layer_version.python_requests_layer: Creating...
aws_iam_role.lambda_role: Creating...
aws_api_gateway_rest_api.apiLambda: Creating...
...
Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
Outputs:
base_url = "https://4sbcd5zu87.execute-api.us-east-2.amazonaws.com/test"Once it is complete, we have our API Gateway URL, https://4sbcd5zu87.execute-api.us-east-2.amazonaws.com/test.
Testing the Code
Now, we can test it to make sure this works as expected using curl and piping it through jq so the response is formatted nicely.
% curl -X POST https://4sbcd5zu87.execute-api.us-east-2.amazonaws.com/test |jq .
{
"data": {
"id": 2,
"email": "[email protected]",
"first_name": "Janet",
"last_name": "Weaver",
"avatar": "https://reqres.in/img/faces/2-image.jpg"
},
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}If you notice the curl command, I’ve added -X POST because the http_method I specified in the terraform was to POST. As you can see from the above response, our code is working as expected!
Cleaning Up and Closing Out
Make sure you do a terraform destroy to clean up when you’re done. At this point, you now have everything you need to deploy Python to Lambda with Terraform. You can make a much more complex Python script and use that if you like.
I have also saved out my lamda_testing directory to my public Github repo here.