---
title: 'Transformers, explained: Understand the model behind GPT, BERT, and T5'
source: 'https://youtube.com/watch?v=SZorAJ4I-sA'
video_id: 'SZorAJ4I-sA'
date: 2026-07-28
duration_sec: 551
---

# Transformers, explained: Understand the model behind GPT, BERT, and T5

> Source: [Transformers, explained: Understand the model behind GPT, BERT, and T5](https://youtube.com/watch?v=SZorAJ4I-sA)

## Summary

Transformers are a type of neural network architecture that revolutionized natural language processing by enabling efficient parallelization and training on massive datasets. They power models like GPT-3, BERT, and T5, and can translate text, write code, and even help solve protein folding. This video explains what transformers are, how they work, and why they've been so impactful.

### Key Points

- **What is a transformer?** [00:51] — A transformer is a neural network architecture designed for language tasks, overcoming limitations of recurrent neural networks (RNNs) by processing words in parallel.
- **RNN limitations** [01:42] — RNNs process words sequentially, making them slow to train and poor at handling long sequences due to forgetting earlier context.
- **Transformer innovation** [02:42] — Transformers, introduced in 2017 by Google and University of Toronto, can be parallelized efficiently, allowing training on huge datasets like 45 terabytes for GPT-3.
- **Three key innovations** [03:27] — Positional encodings, attention, and self-attention are the core mechanisms that make transformers work.
- **Positional encodings** [03:39] — Instead of processing words sequentially, each word gets a number indicating its position, allowing the network to learn word order from data.
- **Attention mechanism** [04:19] — Attention allows the model to look at all input words when translating, learning which words are relevant (e.g., gender agreement in French).
- **Self-attention** [05:51] — Self-attention helps the model understand word context within the same sentence, disambiguating meanings (e.g., 'server' as waiter vs. computer).
- **BERT and applications** [07:45] — BERT is a popular transformer model used in Google Search and Cloud NLP, trained on unlabeled data via semi-supervised learning.

### Conclusion

Transformers have become a foundational technology in machine learning, enabling breakthroughs in language understanding and generation. Their ability to scale with data and hardware makes them a versatile tool for many applications.

## Transcript

[MUSIC PLAYING]
DALE MARKOWITZ: The
neat thing about working
in machine learning is that
every few years, somebody
invents something crazy that
makes you totally reconsider
what's possible, like
models that can play
Go or generate
hyper-realistic faces.
And today, the
mind-blowing discovery
that's rocking
everyone's world is
a type of neural network
called a transformer.
Transformers are models that
can translate text, write
poems and op-eds, and even
generate computer code.
They could be used in biology
to solve the protein folding
problem.
Transformers are like
this magical machine
learning hammer that seems to
make every problem into a nail.
If you've heard of the
trendy new ML models
BERT, or GPT-3, or T5,
all of these models
are based on transformers.
So if you want to stay
hip in machine learning
and especially in natural
language processing,
you have to know
about the transformer.
So in this video,
I'm going to tell you
about what transformers
are, how they work,
and why they've
been so impactful.
Let's get to it.
So what is a transformer?
It's a type of neural
network architecture.
To recap, neural networks
are a very effective type
of model for analyzing
complicated data
types, like images,
videos, audio, and text.
But there are different types
of neural networks optimized
for different types of data.
Like if you're analyzing
images, you would typically
use a convolutional
neural network,
which is designed
to vaguely mimic
the way that the human
brain processes vision.
And since around
2012, neural networks
have been really good
at solving vision tasks,
like identifying
objects in photos.
But for a long time, we didn't
have anything comparably
good for analyzing language,
whether for translation,
or text summarization,
or text generation.
And this is a problem, because
language is the primary way
that humans communicate.
You see, until transformers
came around, the way
we used deep learning
to understand text
was with a type of model called
a Recurrent Neural Network,
or an RNN, that looked
something like this.
Let's say you wanted
to translate a sentence
from English to French.
An RNN would take as
input an English sentence
and process the
words one at a time,
and then sequentially spit
out their French counterparts.
The keyword here is sequential.
In language, the order
of words matters,
and you can't just
shuffle them around.
For example, the sentence
Jane went looking for trouble
means something very different
than the sentence Trouble
went looking for Jane.
So any model that's going
to deal with language
has to capture word order,
and recurrent neural networks
do this by looking at one
word at a time sequentially.
But RNNs had a lot of problems.
First, they never
really did well
at handling large sequences
of text, like long paragraphs
or essays.
By the time they were analyzing
the end of a paragraph,
they'd forget what
happened in the beginning.
And even worse, RNNs were
pretty hard to train.
Because they process
words sequentially,
they couldn't
paralellize well, which
means that you couldn't just
speed them up by throwing
lots of GPUs at them.
And when you have a model
that's slow to train,
you can't train it on
all that much data.
This is where the transformer
changed everything.
They're a model developed in
2017 by researchers at Google
and the University of Toronto,
and they were initially
designed to do translation.
But unlike recurrent
neural networks,
you could really efficiently
paralellize transformers.
And that meant that
with the right hardware,
you could train some
really big models.
How big?
Really big.
Remember GPT-3, that model
that writes poetry and code,
and has conversations?
That was trained on almost
45 terabytes of text data,
including almost the
entire public web.
[WHISTLES] So if you remember
anything about transformers,
let it be this.
Combine a model that scales
really well with a huge data
set and the results will
probably blow your mind.
So how do these
things actually work?
From the diagram in the paper,
it should be pretty clear.
Or maybe not.
Actually, it's simpler
than you might think.
There are three main
innovations that
make this model work so well.
Positional encodings and
attention, and specifically,
a type of attention
called self-attention.
Let's start by talking
about the first one,
positional encodings.
Let's say we're trying
to translate text
from English to French.
Positional encodings is
the idea that instead
of looking at
words sequentially,
you take each word
in your sentence,
and before you feed it
into the neural network,
you slap a number on it--
1, 2, 3, depending
on what number
the word is in the sentence.
In other words, you
store information
about word order
in the data itself,
rather than in the
structure of the network.
Then as you train the
network on lots of text data,
it learns how to interpret
those positional encodings.
In this way, the neural
network learns the importance
of word order from the data.
This is a high level
way to understand
positional encodings,
but it's an innovation
that really helped make
transformers easier
to train than RNNs.
The next innovation
in this paper
is a concept called
attention, which
you'll see used everywhere in
machine learning these days.
In fact, the title of the
original transformer paper
is "Attention Is All You Need."
So the agreement on the
European economic area
was signed in August 1992.
Did you know that?
That's the example sentence
given in the original paper.
And remember, the
original transformer
was designed for translation.
Now imagine trying to translate
that sentence to French.
One bad way to translate text
is to try to translate each word
one for one.
But in French, some
words are flipped,
like in the French translation,
European comes before economic.
Plus, French is a
language that has gendered
agreement between words.
So the word [FRENCH] needs
to be in the feminine form
to match with [FRENCH].
The attention mechanism is
a neural network structure
that allows a text model to
look at every single word
in the original
sentence when making
a decision about how to
translate a word in the output
sentence.
In fact, here's a
nice visualization
from that paper that shows what
words in the input sentence
the model is
attending to when it
makes predictions about a
word for the output sentence.
So when the model outputs
the word [FRENCH],,
it's looking at the input
words European and economic.
You can think of this
diagram as a sort
of heat map for attention.
And how does the
model know which words
it should be attending to?
It's something that's
learned over time from data.
By seeing thousands of examples
of French and English sentence
pairs, the model
learns about gender,
and word order, and
plurality, and all
of that grammatical stuff.
So we talked about two key
transformer innovations,
positional encoding
and attention.
But actually, attention had
been invented before this paper.
The real innovation in
transformers was something
called self-attention, a twist
on traditional attention.
The type of attention
we just talked about
had to do with aligning
words in English and French,
which is really important
for translation.
But what if you're just
trying to understand
the underlying meaning
in language so that you
can build a network that can do
any number of language tasks?
What's incredible
about neural networks,
like transformers, is that as
they analyze tons of text data,
they begin to build up this
internal representation
or understanding of
language automatically.
They might learn, for example,
that the words programmer,
and software engineer,
and software developer
are all synonymous.
And they might also naturally
learn the rules of grammar,
and gender, and
tense, and so on.
The better this internal
representation of language
the neural network
learns, the better it
will be at any language task.
And it turns out that attention
can be a very effective way
to get a neural network
to understand language
if it's turned on the
input text itself.
Let me give you an example.
Take these two sentences--
Server, can I have the check?
Versus, Looks like I
just crashed the server.
The word server here means
two very different things.
And I know that,
because I'm looking
at the context of the
surrounding words.
Self-attention allows
a neural network
to understand a word in the
context of the words around it.
So when a model
processes the word server
in the first
sentence, it might be
attending to the
word check, which
helps it disambiguate from a
human server versus a mail one.
In the second
sentence, the model
might be attending to the
word crashed to determine
that the server is a machine.
Self-attention can also
help neural networks
disambiguate words,
recognize parts of speech,
and even identify word tense.
This, in a nutshell, is the
value of self-attention.
So to summarize,
transformers boil down
to positional encodings,
attention, and self-attention.
Of course, this is a 10,000-foot
look at transformers.
But how are they
actually useful?
One of the most popular
transformer-based models
is called BERT, which was
invented just around the time
that I joined Google in 2018.
BERT was trained on
a massive text corpus
and has become this sort
of general pocketknife
for NLP that can be adapted
to a bunch of different tasks,
like text summarization,
question answering,
classification, and
finding similar sentences.
It's used in Google Search to
help understand search queries,
and it powers a lot of
Google Cloud's NLP tools,
like Google Cloud
AutoML Natural Language.
BERT also proved that you
could build very good models
on unlabeled data,
like text scraped
from Wikipedia or Reddit.
This is called
semi-supervised learning,
and it's a big trend in
machine learning right now.
So if I've sold you about
how cool transformers are,
you might want to start
using them in your app.
No problem.
TensorFlow Hub is a great place
to grab pretrained transformer
models, like BERT.
You can download them for
free in multiple language
and drop them straight
into your app.
You can also check out the
popular transformers Python
library, built by the
company Hugging Face.
That's one of the
community's favorite ways
to train and use
transformer models.
For more transformer
tips, check out
my blog post linked below,
and thanks for watching.
[MUSIC PLAYING]