philschmid blog

Going Production: Auto-scaling Hugging Face Transformers with Amazon SageMaker

#AWS #BERT #HuggingFace #Sagemaker
, October 29, 2021 · 6 min read

Photo by Piret Ilver on Unsplash

Welcome to this getting started guide, we will use the new Hugging Face Inference DLCs and Amazon SageMaker Python SDK to deploy a transformer model for real-time inference. In this example, we are going to deploy a trained Hugging Face Transformer model onto SageMaker for inference.

1 !pip install "sagemaker>=2.66.2" --upgrade
1 import sagemaker
2 sagemaker.__version__
3 # '2.66.2.post0'

Reference Blog post Configuring autoscaling inference endpoints in Amazon SageMaker

Deploy s Hugging Face Transformer model to Amazon SageMaker for Inference

To deploy a model directly from the Hub to SageMaker we need to define 2 environment variables when creating the HuggingFaceModel. We need to define:

  • HF_MODEL_ID: defines the model id, which will be automatically loaded from huggingface.co/models when creating or SageMaker Endpoint. The 🤗 Hub provides +10 000 models all available through this environment variable.
  • HF_TASK: defines the task for the used 🤗 Transformers pipeline. A full list of tasks can be found here.
1 from sagemaker.huggingface import HuggingFaceModel
2 from uuid import uuid4
3 import sagemaker
4
5 role = sagemaker.get_execution_role()
6
7 # Hub Model configuration. https://huggingface.co/models
8 hub = {
9 'HF_MODEL_ID':'yiyanghkust/finbert-tone', # model_id from hf.co/models
10 'HF_TASK':'text-classification' # NLP task you want to use for predictions
11 }
12
13 # endpoint name
14 endpoint_name=f'{hub["HF_MODEL_ID"].split("/")[1]}-{str(uuid4())}' # model and endpoint name
15
16 # create Hugging Face Model Class
17 huggingface_model = HuggingFaceModel(
18 env=hub,
19 role=role, # iam role with permissions to create an Endpoint
20 name=endpoint_name, # model and endpoint name
21 transformers_version="4.11", # transformers version used
22 pytorch_version="1.9", # pytorch version used
23 py_version="py38", # python version of the DLC
24 )

Next step is to deploy our endpoint.

1 # deploy model to SageMaker Inference
2 predictor = huggingface_model.deploy(
3 initial_instance_count=1,
4 instance_type="ml.c5.large"
5 )
6 # get aws region for dashboards
7 aws_region = predictor.sagemaker_session.boto_region_name

Architecture

The Hugging Face Inference Toolkit for SageMaker is an open-source library for serving Hugging Face transformer models on SageMaker. It utilizes the SageMaker Inference Toolkit for starting up the model server, which is responsible for handling inference requests. The SageMaker Inference Toolkit uses Multi Model Server (MMS) for serving ML models. It bootstraps MMS with a configuration and settings that make it compatible with SageMaker and allow you to adjust important performance parameters, such as the number of workers per model, depending on the needs of your scenario.

hf inference toolkit

Deploying a model using SageMaker hosting services is a three-step process:

  1. Create a model in SageMaker —By creating a model, you tell SageMaker where it can find the model components.
  2. Create an endpoint configuration for an HTTPS endpoint —You specify the name of one or more models in production variants and the ML compute instances that you want SageMaker to launch to host each production variant.
  3. Create an HTTPS endpoint —Provide the endpoint configuration to SageMaker. The service launches the ML compute instances and deploys the model or models as specified in the configuration

sm endpoint

After the endpoint is deployed we can use the predictor to send requests.

1 # example request, you always need to define "inputs"
2 data = {
3 "inputs": "There is a shortage of capital for project SageMaker. We need extra financing"
4 }
5
6 # request
7 predictor.predict(data)
8 # [{'label': 'negative', 'score': 0.9870443940162659}]

Model Monitoring

To properly monitor our endpoint lets send a few hundred requests.

1 for i in range(500):
2 predictor.predict(data)

After that we can go to the cloudwatch dashboard to take a look.

1 print(f"https://console.aws.amazon.com/cloudwatch/home?region={aws_region}#metricsV2:graph=~(metrics~(~(~'AWS*2fSageMaker~'ModelLatency~'EndpointName~'finbert-tone-73d26f97-9376-4b3f-9334-a2-2021-10-29-12-18-52-365~'VariantName~'AllTraffic))~view~'timeSeries~stacked~false~start~'-PT15M~end~'P0D~region~'{aws_region}~stat~'SampleCount~period~30);query=~'*7bAWS*2fSageMaker*2cEndpointName*2cVariantName*7d*20{predictor.endpoint_name}")

model-monitoring-dashboard

Auto Scaling your Model

Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to quickly build, train, and deploy machine learning (ML) models at scale.

Autoscaling is an out-of-the-box feature that monitors your workloads and dynamically adjusts the capacity to maintain steady and predictable performance at the possible lowest cost.

The following diagram is a sample architecture that showcases how a model is served as an endpoint with autoscaling enabled.

autoscaling-endpoint

Configure Autoscaling for our Endpoint

You can define the minimum, desired, and the maximum number of instances per endpoint and, based on the autoscaling configurations, instances are managed dynamically. The following diagram illustrates this architecture.

scaling-options

AWS offers many different ways to auto-scale your endpoints. One of them Simple-Scaling, where you scale the instance capacity based on CPUUtilization of the instances or SageMakerVariantInvocationsPerInstance.

In this example we are going to use CPUUtilization to auto-scale our Endpoint

1 import boto3
2
3 # Let us define a client to play with autoscaling options
4 asg_client = boto3.client('application-autoscaling') # Common class representing Application Auto Scaling for SageMaker amongst other services
5
6 # the resource type is variant and the unique identifier is the resource ID.
7 # Example: endpoint/my-bert-fine-tuned/variant/AllTraffic .
8 resource_id=f"endpoint/{predictor.endpoint_name}/variant/AllTraffic"
9
10 # scaling configuration
11 response = asg_client.register_scalable_target(
12 ServiceNamespace='sagemaker', #
13 ResourceId=resource_id,
14 ScalableDimension='sagemaker:variant:DesiredInstanceCount',
15 MinCapacity=1,
16 MaxCapacity=4
17 )

Create Scaling Policy with configuration details, e.g. TargetValue when the instance should be scaled.

1 response = asg_client.put_scaling_policy(
2 PolicyName=f'CPUUtil-ScalingPolicy-{predictor.endpoint_name}',
3 ServiceNamespace='sagemaker',
4 ResourceId=resource_id,
5 ScalableDimension='sagemaker:variant:DesiredInstanceCount',
6 PolicyType='TargetTrackingScaling',
7 TargetTrackingScalingPolicyConfiguration={
8 'TargetValue': 50.0, # threshold
9 'CustomizedMetricSpecification':
10 {
11 'MetricName': 'CPUUtilization',
12 'Namespace': '/aws/sagemaker/Endpoints',
13 'Dimensions': [
14 {'Name': 'EndpointName', 'Value': predictor.endpoint_name },
15 {'Name': 'VariantName','Value': 'AllTraffic'}
16 ],
17 'Statistic': 'Average', # Possible - 'Statistic': 'Average'|'Minimum'|'Maximum'|'SampleCount'|'Sum'
18 'Unit': 'Percent'
19 },
20 'ScaleInCooldown': 300, # duration until scale in
21 'ScaleOutCooldown': 100 # duration between scale out
22 }
23 )

stress test the endpoint with threaded requests

1 from concurrent.futures import ThreadPoolExecutor
2 import os
3
4 workers = os.cpu_count() * 5
5 requests = 200
6 print(f"workers used for load test: {workers}")
7
8 with ThreadPoolExecutor(max_workers=workers) as executor:
9 for i in range(requests):
10 executor.submit(predictor.predict, data)

Monitor the CPUUtilization in cloudwatch

1 print(f"https://console.aws.amazon.com/cloudwatch/home?region={aws_region}#metricsV2:graph=~(metrics~(~(~'*2faws*2fsagemaker*2fEndpoints~'CPUUtilization~'EndpointName~'finbert-tone-73d26f97-9376-4b3f-9334-a2-2021-10-29-12-18-52-365~'VariantName~'AllTraffic))~view~'timeSeries~stacked~false~region~'{aws_region}~start~'-PT15M~end~'P0D~stat~'Average~period~60);query=~'*7b*2faws*2fsagemaker*2fEndpoints*2cEndpointName*2cVariantName*7d*20finbert-tone-73d26f97-9376-4b3f-9334-a2-2021-10-29-12-18-52-365*20Endpoint*20{predictor.endpoint_name}*20has*20Current*20Instance*20Count*3a*201*20With*20a*20desired*20instance*20count*20of*201")

Now we check the endpoint instance_count number an see that SageMaker has scaled out.

1 bt_sm = boto3.client('sagemaker')
2 response = bt_sm.describe_endpoint(EndpointName=predictor.endpoint_name)
3 print(f"Endpoint {response['EndpointName']} has \nCurrent Instance Count: {response['ProductionVariants'][0]['CurrentInstanceCount']}\nWith a desired instance count of {response['ProductionVariants'][0]['DesiredInstanceCount']}")
4 # Endpoint finbert-tone-73d26f97-9376-4b3f-9334-a2-2021-10-29-12-18-52-365 has
5 # Current Instance Count: 4
6 # With a desired instance count of 4

Clean up

1 # delete endpoint
2 predictor.delete_endpoint()

Conclusion

With the help of the Autoscaling groups were we able to apply elasticity without heavy lifting. The endpoint now adapts to the incoming load and scales in and out as required.

Through the simplicity of SageMaker you don’t need huge Ops-teams anymore to manage and scale your machine learning models. You can do it yourself.


You can find the code here and feel free open a thread the forum.

Thanks for reading. If you have any questions, feel free to contact me, through Github, or on the forum. You can also connect with me on Twitter or LinkedIn.