Photo by T.H. Chia on Unsplash
Part of using Machine Learning successfully in production is the use of MLOps. MLOps enhances DevOps with continuous training (CT). The main components of MLOps therefore include continuous integration (CI), continuous delivery (CD), and continuous training (CT). Nvidia wrote an article about what MLOps is in detail.
My Name is Philipp and I live in Nuremberg, Germany. Currently, I am working as a machine learning engineer at a technology incubation startup. At work, I design and implement cloud-native machine learning architectures for fin-tech and insurance companies. I am a big fan of Serverless and providing machine learning models in a serverless fashion. I already wrote two articles about how to use Deep Learning models like BERT in a Serverless Environment like AWS Lambda.
A big hurdle to overcome in serverless machine learning with tools like AWS Lambda, Google Cloud Functions, Azure Functions was storage. Tensorflow and Pytorch are having a huge size and newer “State of the Art” models like BERT have a size of over 300MB.
Until today it was very difficult to sync dependencies or model files to an AWS EFS Filesystem. You could do it with AWS Datasync or you could start an EC2 instance in the same subnet and VPC and upload your files from there.
For this reason, I have built an MLOps toolkit called efsync. Efsync is a CLI/SDK tool, which syncs files from S3 or local filesystem automatically to AWS EFS and enables you to install dependencies with the AWS Lambda runtime directly into your EFS filesystem. The CLI is easy to use, you only need access to an AWS Account and an AWS EFS-filesystem up and running.
- Install via pip3
1 pip3 install efsync
- sync your pip dependencies or files to AWS EFS
1 # using the CLI2 efsync -cf efsync.yaml34 # using the SDK5 from efsync import efsync6 efsync('efsync.yaml')
Efsync covers 5 use cases. On the one hand, it allows you to install the needed dependencies, on the other hand, efsync helps you to get your models ready, be it via sync from S3 to EFS or a local upload with SCP. I created an example Jupyter Notebooks for each use case.
The 5 use cases consist of:
- install Python dependencies with the AWS Lambda runtime directly into an EFS filesystem and use them in an AWS Lambda function. Example
- sync files from S3 to an EFS Filesystem. Example
- upload files with SCP to an EFS Filesystem. Example
- Install Python dependencies and sync from S3 to an EFS Filesystem. Example
- Install Python dependencies and uploading files with SCP an EFS Filesystem. Example
Note: Each Example can be run in a Google Colab.
Implementation Configuration possibilities
There are 4 different ways to use efsync in your project:
- You can create a
yamlconfiguration and use the SDK.
- You can create a python
dictand use the SDK.
- You can create a
yamlconfiguration and use the CLI.
- You can use the CLI with parameters.
You can find examples for each configuration in the Github Repository. I also included configuration examples for the different use cases.
Note: If you sync a file with SCP from a local directory (e.g.
model/bert) to efs (
my_efs_model) efsync will
sync the model to
my_efs_model/bert that happens because scp uploads the files recursively.
The following example shows how to install Python dependencies to the EFS Filesystem and sync files from S3 to the EFS
Filesystem, afterwards. For configuration purpose, we have to create an
efsync.yaml and a
which holds our dependencies and configuration.
1. Install efsync
1 pip3 install efsync
2. Create a
requirements.txt with the dependencies
1 torch2 numpy
3. Create an
efsync.yaml with all required configuration
1 #standard configuration2 efs_filesystem_id: fs-2226b27a # aws efs filesystem id (moint point)3 subnet_Id: subnet-17f97a7d # subnet of which the efs is running in4 ec2_key_name: efsync-asd913fjgq3 # required key name for starting the ec2 instance5 clean_efs: all # Defines if the EFS should be cleaned up before. values: `'all'`,`'pip'`,`'file'` uploading6 # aws profile configuration7 aws_profile: schueler-vz # aws iam profile with required permission configured in .aws/credentials8 aws_region: eu-central-1 # the aws region where the efs is running910 # pip packages configurations11 efs_pip_dir: lib # pip directory on ec212 python_version: 3.8 # python version used for installing pip packages -> should be used as lambda runtime afterwads13 requirements: requirements.txt # path + file to requirements.txt which holds the installable pip packages14 # s3 config15 s3_bucket: efsync-test-bucket # s3 bucket name from files should be downloaded16 s3_keyprefix: distilbert # s3 keyprefix for the files17 file_dir_on_ec2: ml # Name of the directory where your S3 files will be saved
efsync.yaml contains all configuration, such as:
efs_filesystem_id: the AWS EFS filesystem id (mount point).
subnet_Id: the Subnet Id of the EFS filesystem, which is running in.
ec2_key_name: A required key name for starting the EC2 instance.
aws_profile: the IAM profile with required permission configured in
aws_region: the AWS region where the EFS filesystem is running.
Pip Dependencies Configurations
efs_pip_dir: the pip directory on EC2, where dependencies will be installed.
python_version: Python version used for installing pip packages -> should be used as lambda runtime.
requirements: Path + file to requirements.txt which holds the installable pip dependencies.
s3_bucket: S3 bucket name from files should be downloaded.
s3_keyprefix: S3 keyprefix for the directory/files
file_dir_on_ec2: Name of the directory where your S3 files will be saved
4. Run efsync wit
1 from efsync import efsync23 efsync('./efsync.yaml')45 #--------------------------Result--------------------------#67 #2020-10-25 20:12:33,747 - efsync - starting....8 #2020-10-25 20:12:33,748 - efsync - loading config9 #2020-10-25 20:12:33,772 - efsync - creating security group10 #2020-10-25 20:12:34,379 - efsync - loading default security group11 #2020-10-25 20:12:39,444 - efsync - creating ssh key for scp in memory12 #2020-10-25 20:12:40,005 - efsync - starting ec2 instance with security group sg-0ff6539317d7e48da and subnet_Id subnet-17f97a7d13 #2020-10-25 20:18:46,430 - efsync - stopping ec2 instance with instance id i-020e3f3cc4b3d690b14 #2020-10-25 20:19:17,159 - efsync - deleting iam profile15 #2020-10-25 20:19:18,354 - efsync - deleting ssh key16 #2020-10-25 20:19:18,604 - efsync - deleting security group17 #2020-10-25 20:19:18,914 - efsync - #################### finished after 6.752833333333333 minutes ####################
With efsync you can easily sync files from S3 or local filesystem automatically to AWS EFS and enables you to install dependencies with the AWS Lambda runtime directly into your EFS filesystem. Installing and syncing files from S3 takes around 6 minutes, only installing dependencies around 4–5 minutes and only syncing files around 2 minutes.
You can find the library on Github. Feel free to create Pull Request or Issues if you have any questions or improvements.