[00:00]
[MUSIC PLAYING]
[00:00]
DALE MARKOWITZ: The
[00:01]
in machine learning is that
[00:03]
invents something crazy that
[00:06]
what's possible, like
[00:09]
Go or generate
[00:12]
And today, the
[00:13]
that's rocking
[00:15]
a type of neural network
[00:17]
Transformers are models that
[00:20]
poems and op-eds, and even
[00:23]
They could be used in biology
[00:25]
problem.
[00:26]
Transformers are like
[00:28]
learning hammer that seems to
[00:31]
If you've heard of the
[00:33]
BERT, or GPT-3, or T5,
[00:37]
are based on transformers.
[00:39]
So if you want to stay
[00:41]
and especially in natural
[00:43]
you have to know
[00:44]
So in this video,
[00:46]
about what transformers
[00:48]
and why they've
[00:50]
Let's get to it.
[00:51]
So what is a transformer?
[00:53]
It's a type of neural
[00:55]
To recap, neural networks
[00:58]
of model for analyzing
[01:00]
types, like images,
[01:02]
But there are different types
[01:05]
for different types of data.
[01:06]
Like if you're analyzing
[01:08]
use a convolutional
[01:10]
which is designed
[01:12]
the way that the human
[01:14]
And since around
[01:16]
have been really good
[01:18]
like identifying
[01:21]
But for a long time, we didn't
[01:23]
good for analyzing language,
[01:26]
or text summarization,
[01:28]
And this is a problem, because
[01:31]
that humans communicate.
[01:32]
You see, until transformers
[01:34]
we used deep learning
[01:36]
was with a type of model called
[01:39]
or an RNN, that looked
[01:42]
Let's say you wanted
[01:44]
from English to French.
[01:46]
An RNN would take as
[01:48]
and process the
[01:50]
and then sequentially spit
[01:53]
The keyword here is sequential.
[01:55]
In language, the order
[01:57]
and you can't just
[02:00]
For example, the sentence
[02:03]
means something very different
[02:05]
went looking for Jane.
[02:07]
So any model that's going
[02:09]
has to capture word order,
[02:11]
do this by looking at one
[02:14]
But RNNs had a lot of problems.
[02:16]
First, they never
[02:17]
at handling large sequences
[02:21]
or essays.
[02:21]
By the time they were analyzing
[02:24]
they'd forget what
[02:26]
And even worse, RNNs were
[02:29]
Because they process
[02:30]
they couldn't
[02:32]
means that you couldn't just
[02:34]
lots of GPUs at them.
[02:35]
And when you have a model
[02:37]
you can't train it on
[02:40]
This is where the transformer
[02:42]
They're a model developed in
[02:45]
and the University of Toronto,
[02:47]
designed to do translation.
[02:49]
But unlike recurrent
[02:50]
you could really efficiently
[02:53]
And that meant that
[02:54]
you could train some
[02:56]
How big?
[02:58]
Really big.
[02:59]
Remember GPT-3, that model
[03:02]
and has conversations?
[03:03]
That was trained on almost
[03:06]
including almost the
[03:09]
[WHISTLES] So if you remember
[03:12]
let it be this.
[03:13]
Combine a model that scales
[03:16]
set and the results will
[03:18]
So how do these
[03:20]
From the diagram in the paper,
[03:24]
Or maybe not.
[03:25]
Actually, it's simpler
[03:27]
There are three main
[03:29]
make this model work so well.
[03:30]
Positional encodings and
[03:33]
a type of attention
[03:36]
Let's start by talking
[03:37]
positional encodings.
[03:39]
Let's say we're trying
[03:41]
from English to French.
[03:42]
Positional encodings is
[03:44]
of looking at
[03:45]
you take each word
[03:47]
and before you feed it
[03:49]
you slap a number on it--
[03:50]
1, 2, 3, depending
[03:52]
the word is in the sentence.
[03:54]
In other words, you
[03:55]
about word order
[03:57]
rather than in the
[03:59]
Then as you train the
[04:02]
it learns how to interpret
[04:05]
In this way, the neural
[04:08]
of word order from the data.
[04:10]
This is a high level
[04:12]
positional encodings,
[04:14]
that really helped make
[04:16]
to train than RNNs.
[04:18]
The next innovation
[04:19]
is a concept called
[04:21]
you'll see used everywhere in
[04:24]
In fact, the title of the
[04:26]
is "Attention Is All You Need."
[04:28]
So the agreement on the
[04:31]
was signed in August 1992.
[04:34]
Did you know that?
[04:35]
That's the example sentence
[04:37]
And remember, the
[04:39]
was designed for translation.
[04:41]
Now imagine trying to translate
[04:44]
One bad way to translate text
[04:47]
one for one.
[04:48]
But in French, some
[04:50]
like in the French translation,
[04:54]
Plus, French is a
[04:56]
agreement between words.
[04:57]
So the word [FRENCH] needs
[05:00]
to match with [FRENCH].
[05:02]
The attention mechanism is
[05:05]
that allows a text model to
[05:07]
in the original
[05:09]
a decision about how to
[05:11]
sentence.
[05:12]
In fact, here's a
[05:14]
from that paper that shows what
[05:16]
the model is
[05:18]
makes predictions about a
[05:22]
So when the model outputs
[05:25]
it's looking at the input
[05:28]
You can think of this
[05:30]
of heat map for attention.
[05:32]
And how does the
[05:33]
it should be attending to?
[05:35]
It's something that's
[05:38]
By seeing thousands of examples
[05:41]
pairs, the model
[05:42]
and word order, and
[05:44]
of that grammatical stuff.
[05:46]
So we talked about two key
[05:48]
positional encoding
[05:51]
But actually, attention had
[05:54]
The real innovation in
[05:56]
called self-attention, a twist
[06:00]
The type of attention
[06:02]
had to do with aligning
[06:04]
which is really important
[06:06]
But what if you're just
[06:08]
the underlying meaning
[06:10]
can build a network that can do
[06:14]
What's incredible
[06:16]
like transformers, is that as
[06:19]
they begin to build up this
[06:22]
or understanding of
[06:25]
They might learn, for example,
[06:28]
and software engineer,
[06:30]
are all synonymous.
[06:32]
And they might also naturally
[06:34]
and gender, and
[06:36]
The better this internal
[06:38]
the neural network
[06:40]
will be at any language task.
[06:42]
And it turns out that attention
[06:45]
to get a neural network
[06:47]
if it's turned on the
[06:50]
Let me give you an example.
[06:51]
Take these two sentences--
[06:53]
Server, can I have the check?
[06:55]
Versus, Looks like I
[06:58]
The word server here means
[07:00]
And I know that,
[07:02]
at the context of the
[07:05]
Self-attention allows
[07:06]
to understand a word in the
[07:10]
So when a model
[07:12]
in the first
[07:13]
attending to the
[07:15]
helps it disambiguate from a
[07:19]
In the second
[07:21]
might be attending to the
[07:23]
that the server is a machine.
[07:24]
Self-attention can also
[07:26]
disambiguate words,
[07:29]
and even identify word tense.
[07:31]
This, in a nutshell, is the
[07:34]
So to summarize,
[07:36]
to positional encodings,
[07:41]
Of course, this is a 10,000-foot
[07:44]
But how are they
[07:45]
One of the most popular
[07:48]
is called BERT, which was
[07:50]
that I joined Google in 2018.
[07:53]
BERT was trained on
[07:55]
and has become this sort
[07:57]
for NLP that can be adapted
[08:01]
like text summarization,
[08:03]
classification, and
[08:06]
It's used in Google Search to
[08:09]
and it powers a lot of
[08:12]
like Google Cloud
[08:15]
BERT also proved that you
[08:17]
on unlabeled data,
[08:19]
from Wikipedia or Reddit.
[08:21]
This is called
[08:23]
and it's a big trend in
[08:27]
So if I've sold you about
[08:29]
you might want to start
[08:31]
No problem.
[08:32]
TensorFlow Hub is a great place
[08:35]
models, like BERT.
[08:36]
You can download them for
[08:39]
and drop them straight
[08:41]
You can also check out the
[08:44]
library, built by the
[08:46]
That's one of the
[08:48]
to train and use
[08:49]
For more transformer
[08:51]
my blog post linked below,
[08:54]
[MUSIC PLAYING]