- Published on
- 7 min read View Code
Part of using Machine Learning successfully in production is the use of MLOps. MLOps enhances DevOps with continuous training (CT). The main components of MLOps therefore include continuous integration (CI), continuous delivery (CD), and continuous training (CT). Nvidia wrote an article about what MLOps is in detail.
My Name is Philipp and I live in Nuremberg, Germany. Currently, I am working as a machine learning engineer at a technology incubation startup. At work, I design and implement cloud-native machine learning architectures for fin-tech and insurance companies. I am a big fan of Serverless and providing machine learning models in a serverless fashion. I already wrote two articles about how to use Deep Learning models like BERT in a Serverless Environment like AWS Lambda.
A big hurdle to overcome in serverless machine learning with tools like AWS Lambda, Google Cloud Functions, Azure Functions was storage. Tensorflow and Pytorch are having a huge size and newer "State of the Art" models like BERT have a size of over 300MB.
Until today it was very difficult to sync dependencies or model files to an AWS EFS Filesystem. You could do it with AWS Datasync or you could start an EC2 instance in the same subnet and VPC and upload your files from there.
For this reason, I have built an MLOps toolkit called efsync. Efsync is a CLI/SDK tool, which syncs files from S3 or local filesystem automatically to AWS EFS and enables you to install dependencies with the AWS Lambda runtime directly into your EFS filesystem. The CLI is easy to use, you only need access to an AWS Account and an AWS EFS-filesystem up and running.
- Install via pip3
pip3 install efsync
- sync your pip dependencies or files to AWS EFS
# using the CLI efsync -cf efsync.yaml # using the SDK from efsync import efsync efsync('efsync.yaml')
Efsync covers 5 use cases. On the one hand, it allows you to install the needed dependencies, on the other hand, efsync helps you to get your models ready, be it via sync from S3 to EFS or a local upload with SCP. I created an example Jupyter Notebooks for each use case.
The 5 use cases consist of:
- install Python dependencies with the AWS Lambda runtime directly into an EFS filesystem and use them in an AWS Lambda function. Example
- sync files from S3 to an EFS Filesystem. Example
- upload files with SCP to an EFS Filesystem. Example
- Install Python dependencies and sync from S3 to an EFS Filesystem. Example
- Install Python dependencies and uploading files with SCP an EFS Filesystem. Example
Note: Each Example can be run in a Google Colab.
Implementation Configuration possibilities
There are 4 different ways to use efsync in your project:
- You can create a
yamlconfiguration and use the SDK.
- You can create a python
dictand use the SDK.
- You can create a
yamlconfiguration and use the CLI.
- You can use the CLI with parameters.
You can find examples for each configuration in the Github Repository. I also included configuration examples for the different use cases.
Note: If you sync a file with SCP from a local directory (e.g.
model/bert) to efs (
my_efs_model) efsync will sync the model to
my_efs_model/bert that happens because scp uploads the files recursively.
The following example shows how to install Python dependencies to the EFS Filesystem and sync files from S3 to the EFS Filesystem, afterwards. For configuration purpose, we have to create an
efsync.yaml and a
requirements.txt file which holds our dependencies and configuration.
1. Install efsync
pip3 install efsync
2. Create a
requirements.txt with the dependencies
3. Create an
efsync.yaml with all required configuration
#standard configuration efs_filesystem_id: fs-2226b27a # aws efs filesystem id (moint point) subnet_Id: subnet-17f97a7d # subnet of which the efs is running in ec2_key_name: efsync-asd913fjgq3 # required key name for starting the ec2 instance clean_efs: all # Defines if the EFS should be cleaned up before. values: `'all'`,`'pip'`,`'file'` uploading # aws profile configuration aws_profile: schueler-vz # aws iam profile with required permission configured in .aws/credentials aws_region: eu-central-1 # the aws region where the efs is running # pip packages configurations efs_pip_dir: lib # pip directory on ec2 python_version: 3.8 # python version used for installing pip packages -> should be used as lambda runtime afterwads requirements: requirements.txt # path + file to requirements.txt which holds the installable pip packages # s3 config s3_bucket: efsync-test-bucket # s3 bucket name from files should be downloaded s3_keyprefix: distilbert # s3 keyprefix for the files file_dir_on_ec2: ml # Name of the directory where your S3 files will be saved
efsync.yaml contains all configuration, such as:
efs_filesystem_id: the AWS EFS filesystem id (mount point).
subnet_Id: the Subnet Id of the EFS filesystem, which is running in.
ec2_key_name: A required key name for starting the EC2 instance.
aws_profile: the IAM profile with required permission configured in
aws_region: the AWS region where the EFS filesystem is running.
Pip Dependencies Configurations
efs_pip_dir: the pip directory on EC2, where dependencies will be installed.
python_version: Python version used for installing pip packages -> should be used as lambda runtime.
requirements: Path + file to requirements.txt which holds the installable pip dependencies.
s3_bucket: S3 bucket name from files should be downloaded.
s3_keyprefix: S3 keyprefix for the directory/files
file_dir_on_ec2: Name of the directory where your S3 files will be saved
4. Run efsync wit
from efsync import efsync efsync('./efsync.yaml') #--------------------------Result--------------------------# #2020-10-25 20:12:33,747 - efsync - starting.... #2020-10-25 20:12:33,748 - efsync - loading config #2020-10-25 20:12:33,772 - efsync - creating security group #2020-10-25 20:12:34,379 - efsync - loading default security group #2020-10-25 20:12:39,444 - efsync - creating ssh key for scp in memory #2020-10-25 20:12:40,005 - efsync - starting ec2 instance with security group sg-0ff6539317d7e48da and subnet_Id subnet-17f97a7d #2020-10-25 20:18:46,430 - efsync - stopping ec2 instance with instance id i-020e3f3cc4b3d690b #2020-10-25 20:19:17,159 - efsync - deleting iam profile #2020-10-25 20:19:18,354 - efsync - deleting ssh key #2020-10-25 20:19:18,604 - efsync - deleting security group #2020-10-25 20:19:18,914 - efsync - #################### finished after 6.752833333333333 minutes ####################
With efsync you can easily sync files from S3 or local filesystem automatically to AWS EFS and enables you to install dependencies with the AWS Lambda runtime directly into your EFS filesystem. Installing and syncing files from S3 takes around 6 minutes, only installing dependencies around 4–5 minutes and only syncing files around 2 minutes.
You can find the library on Github. Feel free to create Pull Request or Issues if you have any questions or improvements.