philschmid blog

How to Set Up a CI/CD Pipeline for AWS Lambda With GitHub Actions and Serverless

Philipp Schmid
April 1st, 2020 · 7 min read
#AWS
#Python

Photo by Tim Swaan on Unsplash

Introduction

A CI/CD pipeline functional for your project is incredibly valuable as a developer. Thankfully, it’s not difficult to set up such a pipeline with Github Actions.

In my previous article, Set up a CI/CD Pipeline for your Web app on AWS with Github Actions, I demonstrated how to set up a CI/CD pipeline for your front end application. This time, I’ll focus on the back end.

I’m going to give you a quick and easy, step-by-step tutorial on setting up a CI/CD Pipeline for AWS Lambda with Github Actions. For my AWS Lambda, I chose Python for the runtime. I’ll also cover how to include Python packages such as scikit-learn or pandas.


TL;DR

If you don’t want to read the complete post, just copy the action and Serverless configuration from this Github repository and add the Github secrets to your repository. If you fail, come back and read the article!


Requirements

This post assumes you have the Serverless Framework for deploying an AWS Lambda function installed a configured, as well as a working Github account and Docker installed. The Serverless Framework helps us develop and deploy AWS Lambda functions. It’s a CLI that offers structure, automation, and best practices right out of the box. It also allows you to focus on building sophisticated, event-driven, serverless architectures, comprised of functions and events.

Serverless Framework Logo

If you aren’t familiar or haven’t set up the Serverless Framework, take a look at this quick-start with the Serverless Framework.

Now let’s get started with the tutorial.


Create AWS Lambda function

The first thing we are doing is creating our AWS Lambda function by using the Serverless CLI with the aws-python3 template.

1serverless create --template aws-python3 --path <your-path>

This CLI command will create a new directory with a handler.py, .gitignore and serverless.yaml file in it. The handler.py contains some basic boilerplate code.

1import json
2
3
4def hello(event, context):
5 body = {
6 "message": "Go Serverless v1.0! Your function executed successfully!",
7 "input":event
8 }
9 response = {
10 "statusCode": 200,
11 "body": json.dumps(body)
12 }
13 return response

The serverless.yaml contains the configuration for deploying the function. if you are interested in what can be configured with the serverless.yaml take a look here.


Add Python Requirements

Next, we are adding our Python Requirements to our AWS Lambda function. For this we are using the Serverless plugin serverless-python-requirements . It automatically bundle requirements from a requirements.txt and makes them available in our PYTHONPATH. The serverless-python-requirements plugin allows you to even bundle non-pure-Python modules. if you are interested take a look here.

Installing the plugin

To install the plugin run the following command.

1serverless plugin install -n serverless-python-requirements

This will automatically add the plugin to your project’s package.json and to the plugins section in the serverless.yml. The next step is adjusting the serverless.yaml and including the custom Python requirement configuration. We need this extra configuration because our Github Actions runtime is Node and with the configuration, we can bundle our python requirements in a docker container.

I also…

  • deleted all comments
  • add HTTP-Event
  • add the package section to exclude the node_modulues from deploying
  • change the region to eu-central-1
1service: <name-of-your-function>
2
3provider:
4 name: aws
5 runtime: python3.7
6 region: eu-central-1
7
8custom:
9 pythonRequirements:
10 dockerizePip: true
11
12package:
13 individually: false
14 exclude:
15 - package.json
16 - package-log.json
17 - node_modules/**
18
19functions:
20 get_joke:
21 handler: handler.get_joke
22 events:
23 - http:
24 path: joke
25 method: get
26
27plugins:
28 - serverless-python-requirements

Creating deploy script

In addition to our configuration in the serverless.yaml we need to edit the package.json and include serverless as devDependencies. Additionally, we add a deploy script to deploy the function later. We are going to use this deploy script in the Github Action later.

1{
2 "name": "blog-github-actions-aws-lambda-python",
3 "description": "",
4 "version": "0.1.0",
5 "dependencies": {},
6 "scripts": {
7 "deploy": "serverless deploy"
8 },
9 "devDependencies": {
10 "serverless": "^1.67.0",
11 "serverless-python-requirements": "^5.1.0"
12 }
13}

Adding Requirements to requirements.txt

We have to create a requirements.txt file on the root level, with all required Python packages. But you have to be careful that the deployment package size cannot go over 250MB unzipped. You can find a list of all AWS Lambda limitations here.

Another tip: the boto3 package is already pre-installed you don´t have to include it in the requirements.txt.

For demonstration purposes, i choose the pyjokes packages and let the function respond with a joke to all requests. I include pyjokes in the requirements.txt

1pyjokes

Afterward i add pyjokes to the function in handler.py and return a random joke.

1import json
2import pyjokes
3
4
5def get_joke(event, context):
6 body = {
7 "message": "Go Serverless v1.0! Your function executed successfully!",
8 "joke":pyjokes.get_joke()
9 }
10 response = {
11 "statusCode": 200,
12 "body": json.dumps(body)
13 }
14 return response

Deploy Function manually

Before using Github Actions we are deploying the function by hand with the following command.

Attention Docker must be up and running.

1npm run-script deploy

In your CLI you should see an output like this.

CLI output after deployment

We can test our function by clicking the url provided in the endpoints section.

successful request to lambda


Create Github Actions

Create folders & files

The first thing we have to do for our Action is to create the folder .github with a folder workflows in it on your project root level. Afterwards create the deploy-aws-lambda.yaml file in it.

Creating the Github Action

Copy this code snippet into the deploy-aws-lambda.yaml file.

1name: deploy-aws-lambda
2
3on:
4 push:
5 branches:
6 - master
7jobs:
8 deploy:
9 runs-on: ubuntu-latest
10 strategy:
11 matrix:
12 node-version: [10.x]
13 steps:
14 - uses: actions/checkout@master
15 - name: Use Node.js ${{ matrix.node-version }}
16 uses: actions/setup-node@v1
17 with:
18 node-version: ${{ matrix.node-version }}
19 - name: Install Dependencies
20 run: npm install
21 - name: Configure AWS Credentials
22 uses: aws-actions/configure-aws-credentials@v1
23 with:
24 aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
25 aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
26 aws-region: eu-central-1
27 - name: Serverless Deploy
28 run: npm run-script deploy

This code snippet describes the Action. The Github Action will be triggered after a push on the master branch. You can change this by adjusting the on section in the snippet. If you want a different trigger for your action look here.

Add secrets to your repository

The third and last step is adding secrets to your Github repository. For this Github Action, we need the access key ID and secret access key from IAM User as secrets called AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

If you are not sure how to create an IAM user for the access key ID and secret access key you can read here.

Adding the secrets

To add the secrets you have to go to the “settings” tab of your repository.

Github Repository Navigation

Then go to secrets in the left navigation panel.

Github Repository Settings

On the secrets page, you can add your 2 secrets AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Github Repository Secrets

Grab a coffee and enjoy it

We´re almost done. The Last step is to test it. Therefore edit the handler.py and push it to the master branch of your repository.

1import json
2import pyjokes
3
4
5def get_joke(event, context):
6 body = {
7 "message": "Greetings from Github. Your function is deployed by a Github Actions. Enjoy your joke",
8 "joke":pyjokes.get_joke()
9 }
10 response = {
11 "statusCode": 200,
12 "body": json.dumps(body)
13 }
14 return response

After the push, we can see our Action deploying our AWS Lambda Function.

successful Github Action

After a successful run of our Github Action, we can request our function again to see if it worked.

successful request to lambda


I created a demo repository with a full example. You can find the repository here. If something is unclear let me know and i will adjust it.

More articles from philschmid

Set up a CI/CD Pipeline for your Web app on AWS with Github Actions

Automatic deploy your React, Vue, Angular or Svelte app on s3 and create a cache Invalidation with Github Actions.

March 25th, 2020 · 4 min read

Getting started with CNNs by calculating LeNet-Layer manually

Getting started explanation to CNNs by calculating Yann LeCun LeNet-5 manually for handwritten digits and learning about Padding and Stride.

February 28th, 2020 · 5 min read

Join our email list and get notified about new content

Be the first to receive our latest content with the ability to opt-out at anytime. We promise to not spam your inbox or share your email with any third parties.

© 2020 philschmid|Imprint|Privacy Policy
Link to $https://twitter.com/_philschmidLink to $https://github.com/philschmidLink to $https://instagram.com/schmid_philippLink to $https://www.linkedin.com/in/philipp-schmid-a6a2bb196/