Transformers, explained: Understand the model behind GPT, BERT, and T5

[00:00] [MUSIC PLAYING]

[00:00] DALE MARKOWITZ: The

[00:01] in machine learning is that

[00:03] invents something crazy that

[00:06] what's possible, like

[00:09] Go or generate

[00:12] And today, the

[00:13] that's rocking

[00:15] a type of neural network

[00:17] Transformers are models that

[00:20] poems and op-eds, and even

[00:23] They could be used in biology

[00:25] problem.

[00:26] Transformers are like

[00:28] learning hammer that seems to

[00:31] If you've heard of the

[00:33] BERT, or GPT-3, or T5,

[00:37] are based on transformers.

[00:39] So if you want to stay

[00:41] and especially in natural

[00:43] you have to know

[00:44] So in this video,

[00:46] about what transformers

[00:48] and why they've

[00:50] Let's get to it.

[00:51] So what is a transformer?

[00:53] It's a type of neural

[00:55] To recap, neural networks

[00:58] of model for analyzing

[01:00] types, like images,

[01:02] But there are different types

[01:05] for different types of data.

[01:06] Like if you're analyzing

[01:08] use a convolutional

[01:10] which is designed

[01:12] the way that the human

[01:14] And since around

[01:16] have been really good

[01:18] like identifying

[01:21] But for a long time, we didn't

[01:23] good for analyzing language,

[01:26] or text summarization,

[01:28] And this is a problem, because

[01:31] that humans communicate.

[01:32] You see, until transformers

[01:34] we used deep learning

[01:36] was with a type of model called

[01:39] or an RNN, that looked

[01:42] Let's say you wanted

[01:44] from English to French.

[01:46] An RNN would take as

[01:48] and process the

[01:50] and then sequentially spit

[01:53] The keyword here is sequential.

[01:55] In language, the order

[01:57] and you can't just

[02:00] For example, the sentence

[02:03] means something very different

[02:05] went looking for Jane.

[02:07] So any model that's going

[02:09] has to capture word order,

[02:11] do this by looking at one

[02:14] But RNNs had a lot of problems.

[02:16] First, they never

[02:17] at handling large sequences

[02:21] or essays.

[02:21] By the time they were analyzing

[02:24] they'd forget what

[02:26] And even worse, RNNs were

[02:29] Because they process

[02:30] they couldn't

[02:32] means that you couldn't just

[02:34] lots of GPUs at them.

[02:35] And when you have a model

[02:37] you can't train it on

[02:40] This is where the transformer

[02:42] They're a model developed in

[02:45] and the University of Toronto,

[02:47] designed to do translation.

[02:49] But unlike recurrent

[02:50] you could really efficiently

[02:53] And that meant that

[02:54] you could train some

[02:56] How big?

[02:58] Really big.

[02:59] Remember GPT-3, that model

[03:02] and has conversations?

[03:03] That was trained on almost

[03:06] including almost the

[03:09] [WHISTLES] So if you remember

[03:12] let it be this.

[03:13] Combine a model that scales

[03:16] set and the results will

[03:18] So how do these

[03:20] From the diagram in the paper,

[03:24] Or maybe not.

[03:25] Actually, it's simpler

[03:27] There are three main

[03:29] make this model work so well.

[03:30] Positional encodings and

[03:33] a type of attention

[03:36] Let's start by talking

[03:37] positional encodings.

[03:39] Let's say we're trying

[03:41] from English to French.

[03:42] Positional encodings is

[03:44] of looking at

[03:45] you take each word

[03:47] and before you feed it

[03:49] you slap a number on it--

[03:50] 1, 2, 3, depending

[03:52] the word is in the sentence.

[03:54] In other words, you

[03:55] about word order

[03:57] rather than in the

[03:59] Then as you train the

[04:02] it learns how to interpret

[04:05] In this way, the neural

[04:08] of word order from the data.

[04:10] This is a high level

[04:12] positional encodings,

[04:14] that really helped make

[04:16] to train than RNNs.

[04:18] The next innovation

[04:19] is a concept called

[04:21] you'll see used everywhere in

[04:24] In fact, the title of the

[04:26] is "Attention Is All You Need."

[04:28] So the agreement on the

[04:31] was signed in August 1992.

[04:34] Did you know that?

[04:35] That's the example sentence

[04:37] And remember, the

[04:39] was designed for translation.

[04:41] Now imagine trying to translate

[04:44] One bad way to translate text

[04:47] one for one.

[04:48] But in French, some

[04:50] like in the French translation,

[04:54] Plus, French is a

[04:56] agreement between words.

[04:57] So the word [FRENCH] needs

[05:00] to match with [FRENCH].

[05:02] The attention mechanism is

[05:05] that allows a text model to

[05:07] in the original

[05:09] a decision about how to

[05:11] sentence.

[05:12] In fact, here's a

[05:14] from that paper that shows what

[05:16] the model is

[05:18] makes predictions about a

[05:22] So when the model outputs

[05:25] it's looking at the input

[05:28] You can think of this

[05:30] of heat map for attention.

[05:32] And how does the

[05:33] it should be attending to?

[05:35] It's something that's

[05:38] By seeing thousands of examples

[05:41] pairs, the model

[05:42] and word order, and

[05:44] of that grammatical stuff.

[05:46] So we talked about two key

[05:48] positional encoding

[05:51] But actually, attention had

[05:54] The real innovation in

[05:56] called self-attention, a twist

[06:00] The type of attention

[06:02] had to do with aligning

[06:04] which is really important

[06:06] But what if you're just

[06:08] the underlying meaning

[06:10] can build a network that can do

[06:14] What's incredible

[06:16] like transformers, is that as

[06:19] they begin to build up this

[06:22] or understanding of

[06:25] They might learn, for example,

[06:28] and software engineer,

[06:30] are all synonymous.

[06:32] And they might also naturally

[06:34] and gender, and

[06:36] The better this internal

[06:38] the neural network

[06:40] will be at any language task.

[06:42] And it turns out that attention

[06:45] to get a neural network

[06:47] if it's turned on the

[06:50] Let me give you an example.

[06:51] Take these two sentences--

[06:53] Server, can I have the check?

[06:55] Versus, Looks like I

[06:58] The word server here means

[07:00] And I know that,

[07:02] at the context of the

[07:05] Self-attention allows

[07:06] to understand a word in the

[07:10] So when a model

[07:12] in the first

[07:13] attending to the

[07:15] helps it disambiguate from a

[07:19] In the second

[07:21] might be attending to the

[07:23] that the server is a machine.

[07:24] Self-attention can also

[07:26] disambiguate words,

[07:29] and even identify word tense.

[07:31] This, in a nutshell, is the

[07:34] So to summarize,

[07:36] to positional encodings,

[07:41] Of course, this is a 10,000-foot

[07:44] But how are they

[07:45] One of the most popular

[07:48] is called BERT, which was

[07:50] that I joined Google in 2018.

[07:53] BERT was trained on

[07:55] and has become this sort

[07:57] for NLP that can be adapted

[08:01] like text summarization,

[08:03] classification, and

[08:06] It's used in Google Search to

[08:09] and it powers a lot of

[08:12] like Google Cloud

[08:15] BERT also proved that you

[08:17] on unlabeled data,

[08:19] from Wikipedia or Reddit.

[08:21] This is called

[08:23] and it's a big trend in

[08:27] So if I've sold you about

[08:29] you might want to start

[08:31] No problem.

[08:32] TensorFlow Hub is a great place

[08:35] models, like BERT.

[08:36] You can download them for

[08:39] and drop them straight

[08:41] You can also check out the

[08:44] library, built by the

[08:46] That's one of the

[08:48] to train and use

[08:49] For more transformer

[08:51] my blog post linked below,

[08:54] [MUSIC PLAYING]