efsync my first open-source MLOps toolkit
Part of using Machine Learning successfully in production is the use of MLOps. MLOps enhances DevOps with continuous training (CT). The main components of MLOps therefore include continuous integration (CI), continuous delivery (CD), and continuous training (CT). Nvidia wrote an article about what MLOps is in detail.
My Name is Philipp and I live in Nuremberg, Germany. Currently, I am working as a machine learning engineer at a technology incubation startup. At work, I design and implement cloud-native machine learning architectures for fin-tech and insurance companies. I am a big fan of Serverless and providing machine learning models in a serverless fashion. I already wrote two articles about how to use Deep Learning models like BERT in a Serverless Environment like AWS Lambda.
A big hurdle to overcome in serverless machine learning with tools like AWS Lambda, Google Cloud Functions, Azure Functions was storage. Tensorflow and Pytorch are having a huge size and newer "State of the Art" models like BERT have a size of over 300MB.
In July this year, AWS added support for Amazon Elastic File System (EFS), a scalable and elastic NFS file system for AWS Lambda. This allows us to mount AWS EFS filesystems to AWS Lambda functions.
Until today it was very difficult to sync dependencies or model files to an AWS EFS Filesystem. You could do it with AWS Datasync or you could start an EC2 instance in the same subnet and VPC and upload your files from there.
For this reason, I have built an MLOps toolkit called efsync. Efsync is a CLI/SDK tool, which syncs files from S3 or local filesystem automatically to AWS EFS and enables you to install dependencies with the AWS Lambda runtime directly into your EFS filesystem. The CLI is easy to use, you only need access to an AWS Account and an AWS EFS-filesystem up and running.
Architecture
Quick Start
- Install via pip3
- sync your pip dependencies or files to AWS EFS
Use Cases
Efsync covers 5 use cases. On the one hand, it allows you to install the needed dependencies, on the other hand, efsync helps you to get your models ready, be it via sync from S3 to EFS or a local upload with SCP. I created an example Jupyter Notebooks for each use case.
The 5 use cases consist of:
- install Python dependencies with the AWS Lambda runtime directly into an EFS filesystem and use them in an AWS Lambda function. Example
- sync files from S3 to an EFS Filesystem. Example
- upload files with SCP to an EFS Filesystem. Example
- Install Python dependencies and sync from S3 to an EFS Filesystem. Example
- Install Python dependencies and uploading files with SCP an EFS Filesystem. Example
Note: Each Example can be run in a Google Colab.
Implementation Configuration possibilities
There are 4 different ways to use efsync in your project:
- You can create a
yaml
configuration and use the SDK. - You can create a python
dict
and use the SDK. - You can create a
yaml
configuration and use the CLI. - You can use the CLI with parameters.
You can find examples for each configuration in the Github Repository. I also included configuration examples for the different use cases.
Note: If you sync a file with SCP from a local directory (e.g. model/bert
) to efs (my_efs_model
) efsync will
sync the model to my_efs_model/bert
that happens because scp uploads the files recursively.
Examples
The following example shows how to install Python dependencies to the EFS Filesystem and sync files from S3 to the EFS
Filesystem, afterwards. For configuration purpose, we have to create an efsync.yaml
and a requirements.txt
file
which holds our dependencies and configuration.
1. Install efsync
2. Create a requirements.txt
with the dependencies
3. Create an efsync.yaml
with all required configuration
The efsync.yaml
contains all configuration, such as:
Standard Configuration
efs_filesystem_id
: the AWS EFS filesystem id (mount point).subnet_Id
: the Subnet Id of the EFS filesystem, which is running in.ec2_key_name
: A required key name for starting the EC2 instance.aws_profile
: the IAM profile with required permission configured in.aws/credentials
.aws_region
: the AWS region where the EFS filesystem is running.
Pip Dependencies Configurations
-
efs_pip_dir
: the pip directory on EC2, where dependencies will be installed. -
python_version
: Python version used for installing pip packages -> should be used as lambda runtime. -
requirements
: Path + file to requirements.txt which holds the installable pip dependencies.S3 Configurations
-
s3_bucket
: S3 bucket name from files should be downloaded. -
s3_keyprefix
: S3 keyprefix for the directory/files -
file_dir_on_ec2
: Name of the directory where your S3 files will be saved
4. Run efsync wit efsync.yaml
Summary
With efsync you can easily sync files from S3 or local filesystem automatically to AWS EFS and enables you to install dependencies with the AWS Lambda runtime directly into your EFS filesystem. Installing and syncing files from S3 takes around 6 minutes, only installing dependencies around 4–5 minutes and only syncing files around 2 minutes.
Thanks for reading. If you have any questions, feel free to contact me or comment on this article. You can also connect with me on Twitter or LinkedIn.
You can find the library on Github. Feel free to create Pull Request or Issues if you have any questions or improvements.