Hugging Face Transformers Examples

Hugging Face Transformers is a Python library of pre-trained state-of-the-art machine learning models for natural language processing, computer vision, speech, or multi-modalities. Transformers provides access to popular Transformer architecture, including BERT, GPT2, RoBERTa, VIT, Whisper, Wav2vec2, T5, LayoutLM, and CLIP. These models support common tasks in different modalities, such as:

📝 Natural Language Processing: text classification, named entity recognition, question answering, language modeling, summarization, translation, multiple choice, and text generation.

🖼️ Computer Vision: image classification, object detection, and segmentation.

🗣️ Audio: automatic speech recognition and audio classification.

🐙 Multimodal: table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.

The library can be used with the PyTorch, TensorFlow, or Jax framework and allows users to easily fine-tune or use the pre-trained models on their own data. If you are new to Hugging Face Transformers, check out the completely free Hugging face course at: https://huggingface.co/course

As we know, Transformers can be used to fine-tune models like BERT, but did you know that the GitHub repository of transformers provides over 20 ready-to-use examples?

Hugging Face Transformers examples are maintained Python scripts to fine-tune or pre-train Transformers models. Currently, there are examples available for:

• Audio Classification
• Contrastive Image Text
• Image Classification
• Image Pretraining
• Language Modeling
• Multiple Choice
• Question Answering
• Semantic Segmentation
• Speech Pretraining
• Speech Recognition
• Summarization
• Text Classification
• Text Generation
• Token Classification
• Translation

Example scripts can be used TensorFlow, PyTorch, or JAX/Flax. As the name "examples" suggests, these are examples to help Transformers users to get started quickly, serve as an inspiration and help to create your own scripts, and enable users to run tests.

Each release of Transformers has its own set of examples script, which are tested and maintained. This is important to keep in mind when using examples/ since if you try to run an example from, e.g. a newer version than the transformers version you have installed it might fail. All examples provide documentation in the repository with a README, which includes documentation about the feature of the example and which arguments are supported. All examples provide an identical set of arguments to make it easy for users to switch between tasks. Now, let's get started.

1. Setup Development Environment

Our first step is to install the Hugging Face Libraries, including transformers and datasets. The version of transformers we install will be the version of the examples we are going to use. If you have transformers already installed, you need to check your version.

pip install torch
pip install "transformers==4.25.1" datasets  --upgrade

2. Download the example script

The example scripts are stored in the GitHub repository of transformers. This means we need first to clone the repository and then checkout the release of the transformers version we have installed in step 1 (for us, 4.25.1)

git clone https://github.com/huggingface/transformers
cd transformers
git checkout tags/v4.25.1 # change 4.25.1 to your version if different

3. Fine-tune BERT for text-classification

Before we can run our script we first need to define the arguments we want to use. For text-classification we need at least a model_name_or_path which can be any supported architecture from the Hugging Face Hub or a local path to a transformers model. Additional parameter we will use are:

• dataset_name : an ID for a dataset hosted on the Hugging Face Hub
• do_train & do_eval: to train and evaluate our model
• num_train_epochs: the number of epochs we use for training.
• per_device_train_batch_size: the batch size used during training per GPU
• output_dir: where our trained model and logs will be saved

You can find a full list of supported parameter in the script. Before we can run our script we have to make sure all dependencies needed for the example are installed. Every example script which requires additional dependencies then transformers and datasets provides a requirements.txt in the directory, which can try to install.

pip install -r examples/pytorch/text-classification/requirements.txt

Thats it, now we can run our script from a CLI, which will start training BERT for text-classification on the emotion dataset.

python3 examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path bert-base-cased \
  --dataset_name emotion \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 32 \
  --num_train_epochs 3 \
  --output_dir /bert-test

4. Fine-tune BART for summarization

In 3. we learnt how easy it is to leverage the examples fine-tun a BERT model for text-classification. In this section we show you how easy it to switch between different tasks. We will now fine-tune BART for summarization on the CNN dailymail dataset. We will provide the same arguments than for text-classification, but extend it with:

• dataset_config_name to use a specific version of the dataset
• text_column the field in our dataset, which holds the text we want to summarize
• summary_column the field in our dataset, which holds the summary we want to learn.

Every example script which requires additional dependencies then transformers and datasets provides a requirements.txt in the directory, which can try to install.

pip install -r examples/pytorch/summarization/requirements.txt

Thats it, now we can run our script from a CLI, which will start training BERT for text-classification on the emotion dataset.

python3 examples/pytorch/summarization/run_summarization.py \
  --model_name_or_path facebook/bart-base \
  --dataset_name cnn_dailymail \
  --dataset_config_name "3.0.0" \
  --text_column "article" \
  --summary_column "highlights" \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 32 \
  --num_train_epochs 3 \
  --output_dir /bert-test

In the previous section we learned how to use Transformers examples with using datasets available on the Hugging Face Hub, but that's not all you can do. Hugging Face Transformers example support for local CSV and JSON files you can use for training your models. In this section we see how we can use a local CSV file using our text-classificationexample.

This section assumes that you completed step 1 & 2 from the “How to use Transformers examples?” section.

To be able to use local data files we will provide the same arguments than for text-classification, but extend it with:

• train_file path pointing to a local CSV or JSONLINES file with your training data
• validation_filepath pointing to a local CSV or JSONLINES file with your training data

Both file should have the fields text which includes our data and label which holds the class label for the text.

pip install -r examples/pytorch/text-classification/requirements.txt

Thats it, now we can run our script from a CLI, which will start training BERT for text-classification on the emotion dataset.

python3 examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path bert-base-cased \
  --train_file local/path/train.csv \
  --validation_filepath local/path/eval.csv \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 32 \
  --num_train_epochs 3 \
  --output_dir /bert-test