Hugging Face Transformers Examples

January 26, 20238 minute readView Code

Machine learning and the adoption of the Transformer architecture are rapidly growing and will revolutionize the way we live and work. From self-driving cars to personalized medicine, the applications of Transformers are limitless. In this blog post, we'll explore how the leverage and explore examples for Hugging Face Transformers from natural language processing to computer vision. Whether you're new to Hugging Face Transformers or an expert, this post is sure to provide valuable insights and inspiration.

We will learn about the following:

  1. What is Hugging Face Transformers?
  2. What are Transformers’ examples?
  3. How to use Transformers examples?
  4. How to use your own data?

What is Hugging Face Transformers?

Hugging Face Transformers is a Python library of pre-trained state-of-the-art machine learning models for natural language processing, computer vision, speech, or multi-modalities. Transformers provides access to popular Transformer architecture, including BERT, GPT2, RoBERTa, VIT, Whisper, Wav2vec2, T5, LayoutLM, and CLIP. These models support common tasks in different modalities, such as:

📝 Natural Language Processing: text classification, named entity recognition, question answering, language modeling, summarization, translation, multiple choice, and text generation.

🖼️ Computer Vision: image classification, object detection, and segmentation.

🗣️ Audio: automatic speech recognition and audio classification.

🐙 Multimodal: table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.

The library can be used with the PyTorch, TensorFlow, or Jax framework and allows users to easily fine-tune or use the pre-trained models on their own data. If you are new to Hugging Face Transformers, check out the completely free Hugging face course at: https://huggingface.co/course

What are Transformers’ examples?

As we know, Transformers can be used to fine-tune models like BERT, but did you know that the GitHub repository of transformers provides over 20 ready-to-use examples?

Hugging Face Transformers examples are maintained Python scripts to fine-tune or pre-train Transformers models. Currently, there are examples available for:

Example scripts can be used TensorFlow, PyTorch, or JAX/Flax. As the name "examples" suggests, these are examples to help Transformers users to get started quickly, serve as an inspiration and help to create your own scripts, and enable users to run tests.

How to use Transformers examples?

Each release of Transformers has its own set of examples script, which are tested and maintained. This is important to keep in mind when using examples/ since if you try to run an example from, e.g. a newer version than the transformers version you have installed it might fail. All examples provide documentation in the repository with a README, which includes documentation about the feature of the example and which arguments are supported. All examples provide an identical set of arguments to make it easy for users to switch between tasks. Now, let's get started.

1. Setup Development Environment

Our first step is to install the Hugging Face Libraries, including transformers and datasets. The version of transformers we install will be the version of the examples we are going to use. If you have transformers already installed, you need to check your version.

pip install torch
pip install "transformers==4.25.1" datasets  --upgrade

2. Download the example script

The example scripts are stored in the GitHub repository of transformers. This means we need first to clone the repository and then checkout the release of the transformers version we have installed in step 1 (for us, 4.25.1)

git clone https://github.com/huggingface/transformers
cd transformers
git checkout tags/v4.25.1 # change 4.25.1 to your version if different

3. Fine-tune BERT for text-classification

Before we can run our script we first need to define the arguments we want to use. For text-classification we need at least a model_name_or_path which can be any supported architecture from the Hugging Face Hub or a local path to a transformers model. Additional parameter we will use are:

  • dataset_name : an ID for a dataset hosted on the Hugging Face Hub
  • do_train & do_eval: to train and evaluate our model
  • num_train_epochs: the number of epochs we use for training.
  • per_device_train_batch_size: the batch size used during training per GPU
  • output_dir: where our trained model and logs will be saved

You can find a full list of supported parameter in the script. Before we can run our script we have to make sure all dependencies needed for the example are installed. Every example script which requires additional dependencies then transformers and datasets provides a requirements.txt in the directory, which can try to install.

pip install -r examples/pytorch/text-classification/requirements.txt

Thats it, now we can run our script from a CLI, which will start training BERT for text-classification on the emotion dataset.

python3 examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path bert-base-cased \
  --dataset_name emotion \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 32 \
  --num_train_epochs 3 \
  --output_dir /bert-test

4. Fine-tune BART for summarization

In 3. we learnt how easy it is to leverage the examples fine-tun a BERT model for text-classification. In this section we show you how easy it to switch between different tasks. We will now fine-tune BART for summarization on the CNN dailymail dataset. We will provide the same arguments than for text-classification, but extend it with:

  • dataset_config_name to use a specific version of the dataset
  • text_column the field in our dataset, which holds the text we want to summarize
  • summary_column the field in our dataset, which holds the summary we want to learn.

Every example script which requires additional dependencies then transformers and datasets provides a requirements.txt in the directory, which can try to install.

pip install -r examples/pytorch/summarization/requirements.txt

Thats it, now we can run our script from a CLI, which will start training BERT for text-classification on the emotion dataset.

python3 examples/pytorch/summarization/run_summarization.py \
  --model_name_or_path facebook/bart-base \
  --dataset_name cnn_dailymail \
  --dataset_config_name "3.0.0" \
  --text_column "article" \
  --summary_column "highlights" \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 32 \
  --num_train_epochs 3 \
  --output_dir /bert-test

How to use your own data?

In the previous section we learned how to use Transformers examples with using datasets available on the Hugging Face Hub, but that's not all you can do. Hugging Face Transformers example support for local CSV and JSON files you can use for training your models. In this section we see how we can use a local CSV file using our text-classificationexample.

This section assumes that you completed step 1 & 2 from the “How to use Transformers examples?” section.

To be able to use local data files we will provide the same arguments than for text-classification, but extend it with:

  • train_file path pointing to a local CSV or JSONLINES file with your training data
  • validation_filepath pointing to a local CSV or JSONLINES file with your training data

Both file should have the fields text which includes our data and label which holds the class label for the text.

pip install -r examples/pytorch/text-classification/requirements.txt

Thats it, now we can run our script from a CLI, which will start training BERT for text-classification on the emotion dataset.

python3 examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path bert-base-cased \
  --train_file local/path/train.csv \
  --validation_filepath local/path/eval.csv \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 32 \
  --num_train_epochs 3 \
  --output_dir /bert-test

Thanks for reading! If you have any questions, feel free to contact me on Twitter or LinkedIn.