Menu
Open source

Hugging Face Transformers

4 min readintermediateUpdated 28 Sept 2026
1 · In one line

Transformers is a free, open-source Python library that lets you download a ready-trained AI model by name and run or train it with a short piece of code.

1 · What it is

Hugging Face Transformers is a free, open-source Python library for using AI models that someone else has already trained. You name a model, and the library fetches it from the Hugging Face Hub, a website where people share models. Then you can run it, or train it further on your own examples. It works with text, pictures, sound, video and mixes of these.

Every model in the library is built from the same three parts. A configuration is a settings file that lists the model’s attributes, such as how many different tokens it knows. The model holds the learned numbers, called weights. A preprocessor turns raw input into numbers. For text, this is a tokenizer. It splits words into small pieces called tokens. Because every model follows that pattern, the same few commands work across many different models.

The quickest way in is a pipeline. You name a task, such as speech recognition, and hand it your input. You can also pick a model if you want. The pipeline prepares the input and returns the output. Many other AI tools, for training models or running them for users, read the same model definitions. The code uses the Apache 2.0 licence.

2 · Why it exists

Using someone else's trained model raises three problems a shared library can solve.

Training from scratchTraining a big model yourself costs a lot of computing time and money; starting from a trained one cuts both.
Every model differsWithout a shared design, each model would need its own loading code, which is a lot to learn one by one.
Many other toolsTraining and serving tools each need to understand the model, so a shared definition saves repeated work.
3 · How it works

Follow one prompt through a small text-generation model.

One Transformers text-generation run Top row: a model name from the Hugging Face Hub goes into the highlighted from_pretrained step, which reads the configuration, picks the right classes and downloads the weights. It loads a tokenizer and a model into the bottom row. Bottom row: a prompt goes to the tokenizer, which turns text into token ids, then to the model, which generates new token ids, then to batch_decode, which turns ids back into words, giving the output text. ONE MODEL NAME · LOAD ONCE · THEN RUN A PROMPT Hub model name from_pretrained() Prompt Tokenizer Model batch_decode() Output text org/model-name reads config · picks classes downloads the weights the secret to a good cake is text becomes token ids generates new token ids ids become words again new words as text loads loads
One model name is enough. The highlighted step works out which tokenizer and model classes to load.
  1. 1 · pickYou pick a model by its Hugging Face Hub name. The Hub is a website where people share models.
  2. 2 · loadThe from_pretrained step reads the model's settings file, picks the matching classes and downloads the weights.
  3. 3 · tokenizeA tokenizer turns your text into token ids, the numbers a model reads.
  4. 4 · generateThe model uses those numbers to produce new token ids.
  5. 5 · decodeThe tokenizer turns the new ids back into words you can read.

A pipeline prepares the input and returns the output for you.

4 · Where it's used
WhoWhat they askWhat it works with
Student“Can I try a speech-to-text model on my laptop this afternoon?”A speech recognition pipeline with a model from the Hub
App developer“Which small chat model fits my prompt style best?”The same pipeline code with a different model name
Researcher“Can I fine-tune this model on my own labelled examples?”The Trainer class and a saved, already-trained model
Machine learning engineer“Will this model load in the tool we use to run models for users?”The shared model definition that other tools read
5 · What it solves, and what it doesn't
solves
  • It loads many kinds of model through the same few commands.
  • It handles text, images, audio, video and mixed inputs.
  • It covers both running models and training them further.
  • Its model definitions are reused by many other training and serving tools.
doesn't solve
  • It does not include the models; they download from the Hub when you first load them.
  • It is not a kit of small building blocks for designing new kinds of neural network.
  • Its training tools are built for its own PyTorch models, not any machine learning loop.
  • Its example scripts may need changes before they work for your task.
6 · Go deeper
Hugging FaceTransformersTokenizationFine-tuningOpen-weight models

Sources used

This explainer is written in original language. The links below support its factual claims.

  1. repohuggingface/transformers: model-definition framework for state-of-the-art machine learning, Hugging Face · read 28 Sept 2026
  2. docsTransformers, Hugging Face · read 28 Sept 2026
  3. docsQuickstart, Hugging Face · read 28 Sept 2026
  4. docsPipeline, Hugging Face · read 28 Sept 2026
  5. docsTokenizers, Hugging Face · read 28 Sept 2026
  6. paperHuggingFace's Transformers: State-of-the-art Natural Language Processing, arXiv · read 28 Sept 2026
  7. repoTransformers LICENSE, Hugging Face · read 28 Sept 2026
Suggest a correction