[0:00] [MUSIC PLAYING]
[0:00] DALE MARKOWITZ: The
neat thing about working
[0:01] in machine learning is that
every few years, somebody
[0:03] invents something crazy that
makes you totally reconsider
[0:06] what's possible, like
models that can play
[0:09] Go or generate
hyper-realistic faces.
[0:12] And today, the
mind-blowing discovery
[0:13] that's rocking
everyone's world is
[0:15] a type of neural network
called a transformer.
[0:17] Transformers are models that
can translate text, write
[0:20] poems and op-eds, and even
generate computer code.
[0:23] They could be used in biology
to solve the protein folding
[0:25] problem.
[0:26] Transformers are like
this magical machine
[0:28] learning hammer that seems to
make every problem into a nail.
[0:31] If you've heard of the
trendy new ML models
[0:33] BERT, or GPT-3, or T5,
all of these models
[0:37] are based on transformers.
[0:39] So if you want to stay
hip in machine learning
[0:41] and especially in natural
language processing,
[0:43] you have to know
about the transformer.
[0:44] So in this video,
I'm going to tell you
[0:46] about what transformers
are, how they work,
[0:48] and why they've
been so impactful.
[0:50] Let's get to it.
[0:51] So what is a transformer?
[0:53] It's a type of neural
network architecture.
[0:55] To recap, neural networks
are a very effective type
[0:58] of model for analyzing
complicated data
[1:00] types, like images,
videos, audio, and text.
[1:02] But there are different types
of neural networks optimized
[1:05] for different types of data.
[1:06] Like if you're analyzing
images, you would typically
[1:08] use a convolutional
neural network,
[1:10] which is designed
to vaguely mimic
[1:12] the way that the human
brain processes vision.
[1:14] And since around
2012, neural networks
[1:16] have been really good
at solving vision tasks,
[1:18] like identifying
objects in photos.
[1:21] But for a long time, we didn't
have anything comparably
[1:23] good for analyzing language,
whether for translation,
[1:26] or text summarization,
or text generation.
[1:28] And this is a problem, because
language is the primary way
[1:31] that humans communicate.
[1:32] You see, until transformers
came around, the way
[1:34] we used deep learning
to understand text
[1:36] was with a type of model called
a Recurrent Neural Network,
[1:39] or an RNN, that looked
something like this.
[1:42] Let's say you wanted
to translate a sentence
[1:44] from English to French.
[1:46] An RNN would take as
input an English sentence
[1:48] and process the
words one at a time,
[1:50] and then sequentially spit
out their French counterparts.
[1:53] The keyword here is sequential.
[1:55] In language, the order
of words matters,
[1:57] and you can't just
shuffle them around.
[2:00] For example, the sentence
Jane went looking for trouble
[2:03] means something very different
than the sentence Trouble
[2:05] went looking for Jane.
[2:07] So any model that's going
to deal with language
[2:09] has to capture word order,
and recurrent neural networks
[2:11] do this by looking at one
word at a time sequentially.
[2:14] But RNNs had a lot of problems.
[2:16] First, they never
really did well
[2:17] at handling large sequences
of text, like long paragraphs
[2:21] or essays.
[2:21] By the time they were analyzing
the end of a paragraph,
[2:24] they'd forget what
happened in the beginning.
[2:26] And even worse, RNNs were
pretty hard to train.
[2:29] Because they process
words sequentially,
[2:30] they couldn't
paralellize well, which
[2:32] means that you couldn't just
speed them up by throwing
[2:34] lots of GPUs at them.
[2:35] And when you have a model
that's slow to train,
[2:37] you can't train it on
all that much data.
[2:40] This is where the transformer
changed everything.
[2:42] They're a model developed in
2017 by researchers at Google
[2:45] and the University of Toronto,
and they were initially
[2:47] designed to do translation.
[2:49] But unlike recurrent
neural networks,
[2:50] you could really efficiently
paralellize transformers.
[2:53] And that meant that
with the right hardware,
[2:54] you could train some
really big models.
[2:56] How big?
[2:58] Really big.
[2:59] Remember GPT-3, that model
that writes poetry and code,
[3:02] and has conversations?
[3:03] That was trained on almost
45 terabytes of text data,
[3:06] including almost the
entire public web.
[3:09] [WHISTLES] So if you remember
anything about transformers,
[3:12] let it be this.
[3:13] Combine a model that scales
really well with a huge data
[3:16] set and the results will
probably blow your mind.
[3:18] So how do these
things actually work?
[3:20] From the diagram in the paper,
it should be pretty clear.
[3:24] Or maybe not.
[3:25] Actually, it's simpler
than you might think.
[3:27] There are three main
innovations that
[3:29] make this model work so well.
[3:30] Positional encodings and
attention, and specifically,
[3:33] a type of attention
called self-attention.
[3:36] Let's start by talking
about the first one,
[3:37] positional encodings.
[3:39] Let's say we're trying
to translate text
[3:41] from English to French.
[3:42] Positional encodings is
the idea that instead
[3:44] of looking at
words sequentially,
[3:45] you take each word
in your sentence,
[3:47] and before you feed it
into the neural network,
[3:49] you slap a number on it--
[3:50] 1, 2, 3, depending
on what number
[3:52] the word is in the sentence.
[3:54] In other words, you
store information
[3:55] about word order
in the data itself,
[3:57] rather than in the
structure of the network.
[3:59] Then as you train the
network on lots of text data,
[4:02] it learns how to interpret
those positional encodings.
[4:05] In this way, the neural
network learns the importance
[4:08] of word order from the data.
[4:10] This is a high level
way to understand
[4:12] positional encodings,
but it's an innovation
[4:14] that really helped make
transformers easier
[4:16] to train than RNNs.
[4:18] The next innovation
in this paper
[4:19] is a concept called
attention, which
[4:21] you'll see used everywhere in
machine learning these days.
[4:24] In fact, the title of the
original transformer paper
[4:26] is "Attention Is All You Need."
[4:28] So the agreement on the
European economic area
[4:31] was signed in August 1992.
[4:34] Did you know that?
[4:35] That's the example sentence
given in the original paper.
[4:37] And remember, the
original transformer
[4:39] was designed for translation.
[4:41] Now imagine trying to translate
that sentence to French.
[4:44] One bad way to translate text
is to try to translate each word
[4:47] one for one.
[4:48] But in French, some
words are flipped,
[4:50] like in the French translation,
European comes before economic.
[4:54] Plus, French is a
language that has gendered
[4:56] agreement between words.
[4:57] So the word [FRENCH] needs
to be in the feminine form
[5:00] to match with [FRENCH].
[5:02] The attention mechanism is
a neural network structure
[5:05] that allows a text model to
look at every single word
[5:07] in the original
sentence when making
[5:09] a decision about how to
translate a word in the output
[5:11] sentence.
[5:12] In fact, here's a
nice visualization
[5:14] from that paper that shows what
words in the input sentence
[5:16] the model is
attending to when it
[5:18] makes predictions about a
word for the output sentence.
[5:22] So when the model outputs
the word [FRENCH],,
[5:25] it's looking at the input
words European and economic.
[5:28] You can think of this
diagram as a sort
[5:30] of heat map for attention.
[5:32] And how does the
model know which words
[5:33] it should be attending to?
[5:35] It's something that's
learned over time from data.
[5:38] By seeing thousands of examples
of French and English sentence
[5:41] pairs, the model
learns about gender,
[5:42] and word order, and
plurality, and all
[5:44] of that grammatical stuff.
[5:46] So we talked about two key
transformer innovations,
[5:48] positional encoding
and attention.
[5:51] But actually, attention had
been invented before this paper.
[5:54] The real innovation in
transformers was something
[5:56] called self-attention, a twist
on traditional attention.
[6:00] The type of attention
we just talked about
[6:02] had to do with aligning
words in English and French,
[6:04] which is really important
for translation.
[6:06] But what if you're just
trying to understand
[6:08] the underlying meaning
in language so that you
[6:10] can build a network that can do
any number of language tasks?
[6:14] What's incredible
about neural networks,
[6:16] like transformers, is that as
they analyze tons of text data,
[6:19] they begin to build up this
internal representation
[6:22] or understanding of
language automatically.
[6:25] They might learn, for example,
that the words programmer,
[6:28] and software engineer,
and software developer
[6:30] are all synonymous.
[6:32] And they might also naturally
learn the rules of grammar,
[6:34] and gender, and
tense, and so on.
[6:36] The better this internal
representation of language
[6:38] the neural network
learns, the better it
[6:40] will be at any language task.
[6:42] And it turns out that attention
can be a very effective way
[6:45] to get a neural network
to understand language
[6:47] if it's turned on the
input text itself.
[6:50] Let me give you an example.
[6:51] Take these two sentences--
[6:53] Server, can I have the check?
[6:55] Versus, Looks like I
just crashed the server.
[6:58] The word server here means
two very different things.
[7:00] And I know that,
because I'm looking
[7:02] at the context of the
surrounding words.
[7:05] Self-attention allows
a neural network
[7:06] to understand a word in the
context of the words around it.
[7:10] So when a model
processes the word server
[7:12] in the first
sentence, it might be
[7:13] attending to the
word check, which
[7:15] helps it disambiguate from a
human server versus a mail one.
[7:19] In the second
sentence, the model
[7:21] might be attending to the
word crashed to determine
[7:23] that the server is a machine.
[7:24] Self-attention can also
help neural networks
[7:26] disambiguate words,
recognize parts of speech,
[7:29] and even identify word tense.
[7:31] This, in a nutshell, is the
value of self-attention.
[7:34] So to summarize,
transformers boil down
[7:36] to positional encodings,
attention, and self-attention.
[7:41] Of course, this is a 10,000-foot
look at transformers.
[7:44] But how are they
actually useful?
[7:45] One of the most popular
transformer-based models
[7:48] is called BERT, which was
invented just around the time
[7:50] that I joined Google in 2018.
[7:53] BERT was trained on
a massive text corpus
[7:55] and has become this sort
of general pocketknife
[7:57] for NLP that can be adapted
to a bunch of different tasks,
[8:01] like text summarization,
question answering,
[8:03] classification, and
finding similar sentences.
[8:06] It's used in Google Search to
help understand search queries,
[8:09] and it powers a lot of
Google Cloud's NLP tools,
[8:12] like Google Cloud
AutoML Natural Language.
[8:15] BERT also proved that you
could build very good models
[8:17] on unlabeled data,
like text scraped
[8:19] from Wikipedia or Reddit.
[8:21] This is called
semi-supervised learning,
[8:23] and it's a big trend in
machine learning right now.
[8:27] So if I've sold you about
how cool transformers are,
[8:29] you might want to start
using them in your app.
[8:31] No problem.
[8:32] TensorFlow Hub is a great place
to grab pretrained transformer
[8:35] models, like BERT.
[8:36] You can download them for
free in multiple language
[8:39] and drop them straight
into your app.
[8:41] You can also check out the
popular transformers Python
[8:44] library, built by the
company Hugging Face.
[8:46] That's one of the
community's favorite ways
[8:48] to train and use
transformer models.
[8:49] For more transformer
tips, check out
[8:51] my blog post linked below,
and thanks for watching.
[8:54] [MUSIC PLAYING]