[00:00]
you are currently watching an artificial
[00:02]
neural network learn
[00:04]
in particular it's learning the shape of
[00:06]
an infinitely complex fractal known as
[00:09]
the mandelbrot set
[00:10]
this is what that set looks like
[00:12]
complexity all the way down
[00:15]
now in order to understand how a neural
[00:16]
network can learn the mandelbrot set
[00:18]
really how it can learn anything at all
[00:21]
we will need to start with a fundamental
[00:24]
mathematical concept
[00:26]
what is a function
[00:28]
informally a function is just a system
[00:31]
of inputs and outputs numbers in numbers
[00:34]
out
[00:35]
in this case you input an x and it
[00:37]
outputs a y
[00:38]
you can plot all of a function's x and y
[00:41]
values in a graph where it draws out a
[00:43]
line what is important is that if you
[00:45]
know the function you can always
[00:47]
calculate the correct output y given any
[00:50]
input x
[00:51]
but say we don't know the function and
[00:54]
instead only know some of its x and y
[00:56]
values we know the inputs and outputs
[00:59]
but we don't know the function used to
[01:01]
produce them
[01:03]
is there a way to reverse engineer that
[01:06]
function that produced this data
[01:08]
if we could construct such a function we
[01:11]
could use it to calculate a y value
[01:13]
given an x value that is not in our
[01:15]
original data set this would work even
[01:17]
if there was a little bit of noise in
[01:19]
our data a little randomness we can
[01:21]
still capture the overall pattern of the
[01:23]
data and continue producing y values
[01:25]
that aren't perfect but close enough to
[01:27]
be useful what we need is a function
[01:30]
approximation and more generally a
[01:33]
function approximator
[01:35]
that is what a neural network is
[01:37]
this is an online tool for visualizing
[01:39]
neural networks and i'll link it in the
[01:42]
description below this particular
[01:44]
network takes two inputs x1 and x2 and
[01:46]
produces one output technically this
[01:49]
function would create a
[01:50]
three-dimensional surface but it's
[01:52]
easier to visualize in two dimensions
[01:54]
this image is rendered by passing the x
[01:56]
y coordinate of each pixel into the
[01:58]
network which then produces a value
[02:00]
between negative one and one that is
[02:02]
used as the pixel value these points are
[02:05]
our data set and are used to train the
[02:07]
network when we begin training it
[02:09]
quickly constructs a shape that
[02:10]
accurately distinguishes between blue
[02:12]
and orange points building a decision
[02:15]
boundary that separates them it is
[02:17]
approximating the function that
[02:19]
describes the data it's learning and is
[02:21]
capable of learning the different data
[02:23]
sets that we throw at it
[02:25]
so what is this middle section then well
[02:27]
as the name implies this is the network
[02:30]
of neurons each one of these nodes is a
[02:32]
neuron which takes in all the inputs
[02:34]
from the previous layer of neurons and
[02:36]
produces one output which is then fed to
[02:38]
the next layer
[02:40]
inputs and outputs sounds like we're
[02:42]
dealing with a function
[02:43]
indeed a neuron itself is just a
[02:46]
function one that can take any number of
[02:48]
inputs and has one output each input is
[02:51]
multiplied by a weight and all are added
[02:53]
together along with bias the weights and
[02:56]
bias make up the parameters of this
[02:58]
neuron values that can change as the
[03:00]
network learns
[03:01]
to keep it easy to visualize we'll
[03:03]
simplify this down to a two-dimensional
[03:05]
function with only one input and one
[03:07]
output
[03:09]
now neurons are our building blocks of
[03:11]
the larger network building blocks that
[03:13]
can be stretched and squeezed and
[03:15]
shifted around and ultimately work with
[03:17]
other blocks to construct something
[03:19]
larger than themselves the neuron as
[03:22]
we've defined it here works like a
[03:23]
building block it is actually an
[03:25]
extremely simple linear function one
[03:27]
which forms a flat line or plane when
[03:30]
there's more than one input
[03:31]
with the two parameters the weight and
[03:33]
bias we can stretch and squeeze and move
[03:35]
our function up and down and left and
[03:37]
right
[03:38]
as such we should be able to combine it
[03:40]
with other neurons to form a more
[03:42]
complicated function one built from lots
[03:45]
of linear functions
[03:47]
so let's start with a target function
[03:49]
one we want to approximate i've
[03:51]
hard-coded a bunch of neurons whose
[03:53]
parameters were found manually and if we
[03:55]
weight each one and add them up as would
[03:57]
happen in the final neuron of the
[03:59]
network we should get a function that
[04:01]
looks like the target function
[04:04]
well that didn't work at all what
[04:06]
happened
[04:07]
well if we simplify our equation
[04:08]
distributing weights and combining like
[04:10]
terms we end up with a single linear
[04:13]
function
[04:14]
turns out linear functions can only
[04:16]
combine to make one linear function this
[04:19]
is a big problem because we need to make
[04:21]
something more complicated than just a
[04:23]
line we need something that is not
[04:25]
linear a non-linearity
[04:28]
in our case we will be using a relu a
[04:31]
rectified linear unit we use it as our
[04:33]
activation function meaning we simply
[04:36]
apply it to our previous naive neuron
[04:38]
this is about as close as you can get to
[04:40]
a linear function without actually being
[04:42]
one and we can tune it with the same
[04:44]
parameters as before
[04:46]
however you may notice that we can't
[04:47]
actually lift the function off of the
[04:49]
x-axis which seems like a pretty big
[04:51]
limitation
[04:52]
well let's give it a shot anyway and see
[04:54]
if it performs any better than our
[04:56]
previous attempt
[04:57]
we're still trying to approximate the
[04:58]
same function and we're using the same
[05:00]
weights and biases as before but this
[05:02]
time we're using a value as our
[05:04]
activation function
[05:06]
and just like that the approximation
[05:07]
looks way better unlike before our
[05:10]
function cannot simplify down to a flat
[05:12]
linear function if we add the neurons
[05:15]
one by one we can see the simple value
[05:17]
functions building on one another and
[05:19]
the inability for one neuron to lift
[05:21]
itself off the x-axis doesn't seem to be
[05:23]
a problem many neurons working together
[05:25]
overcome the limitations of individual
[05:28]
neurons
[05:29]
now i manually found these weights and
[05:31]
biases but how would you find them
[05:33]
automatically the most common algorithm
[05:35]
for this is called back propagation and
[05:37]
is in fact what we're watching when we
[05:39]
run this program
[05:40]
it essentially tweaks and tunes the
[05:42]
parameters of the network bit by bit to
[05:44]
improve the approximation and the
[05:46]
intricacies of this algorithm are really
[05:48]
beyond the scope of this video i'll link
[05:50]
some better explanations in the
[05:51]
description
[05:53]
now we can see how this shape is formed
[05:55]
and why it looks like it's made up of
[05:56]
sort of sharp linear edges it's the
[05:59]
nature of the activation function we're
[06:00]
using
[06:01]
we can also see why if we use no
[06:04]
activation function at all the network
[06:06]
utterly fails to learn we need those
[06:09]
non-linearities
[06:11]
so what if we try learning a more
[06:12]
complicated data set like this spiral
[06:14]
let's give it a go
[06:17]
seems to be struggling a little bit to
[06:19]
capture the pattern no problem if we
[06:21]
need a more complicated function we can
[06:23]
add more building blocks more neurons
[06:25]
and layers of neurons and the network
[06:27]
should be able to piece together a
[06:29]
better approximation something that
[06:30]
really captures the spiral
[06:35]
it seems to be working
[06:37]
in fact no matter what the data set is
[06:39]
we can learn it that is because neural
[06:41]
networks can be rigorously proven to be
[06:44]
universal function approximators they
[06:46]
can approximate any function to any
[06:49]
degree of precision you could ever want
[06:51]
you can always add more neurons
[06:54]
this is essentially the whole point of
[06:56]
deep learning because it means that
[06:58]
neural networks can approximate anything
[07:00]
that can be expressed as a function a
[07:02]
system of inputs and outputs
[07:05]
this is an extremely general way of
[07:07]
thinking about the world the mandelbrot
[07:10]
set for instance can be written as a
[07:12]
function and learned all the same this
[07:14]
is just a scaled-up version of the
[07:15]
experiment we were just looking at but
[07:18]
with an infinitely complex data set
[07:20]
we don't even really need to know what
[07:22]
the manual brought set is the network
[07:24]
learns it for us and that's kind of the
[07:26]
point if you can express any intelligent
[07:29]
behavior any process any task as a
[07:32]
function then a network can learn it for
[07:34]
instance your input could be an image
[07:36]
and your output a label as to whether
[07:38]
it's a cat or a dog or your input could
[07:41]
be text in english and your output a
[07:43]
translation to spanish you just need to
[07:45]
be able to encode your inputs and
[07:47]
outputs as numbers but computers do this
[07:49]
all the time images video text audio
[07:52]
they can all be represented as numbers
[07:54]
and any processing you may want to do
[07:56]
with this data so long as you can write
[07:58]
it as a function can be emulated with a
[08:00]
neural network
[08:02]
it goes deeper than this though under a
[08:03]
few more assumptions neural networks are
[08:06]
provably turing complete meaning they
[08:08]
can solve all of the same kinds of
[08:10]
problems that any computer can solve
[08:13]
an implication of this is that any
[08:15]
algorithm written in any programming
[08:17]
language can be simulated on a neural
[08:19]
network but rather than being manually
[08:22]
written by a human it can be learned
[08:24]
automatically with a function
[08:25]
approximator
[08:27]
neural networks can learn anything
[08:32]
okay that is not true first off you
[08:35]
can't have an infinite number of neurons
[08:38]
there are practical limitations on
[08:40]
network size and what can be modeled in
[08:42]
the real world
[08:44]
i've also ignored the learning process
[08:46]
in this video and just assumed that you
[08:48]
can find the optimal parameters
[08:50]
magically how you realistically do this
[08:53]
introduces its own constraints on what
[08:55]
can be learned
[08:57]
additionally in order for neural
[08:58]
networks to approximate a function you
[09:01]
need the data that actually describes
[09:03]
that function if you don't have enough
[09:05]
data your approximation will be all
[09:08]
wrong it doesn't matter how many neurons
[09:10]
you have or how sophisticated your
[09:11]
network is you just have no idea what
[09:13]
your actual function should look like
[09:16]
it also doesn't make a lot of sense to
[09:18]
use a function approximator when you
[09:20]
already know the function you wouldn't
[09:22]
build a huge neural network to say learn
[09:24]
the mandelbrot set when you can just
[09:26]
write three lines of code to generate it
[09:28]
unless of course you want to make a cool
[09:30]
background visual for a youtube video
[09:32]
there are countless other issues that
[09:34]
have to be considered but for all these
[09:36]
complications neural networks have
[09:38]
proven themselves to be indispensable
[09:40]
for a number of really rather famously
[09:42]
difficult problems for computers
[09:44]
usually these problems require a certain
[09:47]
level of intuition and fuzzy logic that
[09:49]
computers generally lack and are very
[09:51]
difficult for us to manually write
[09:53]
programs to solve
[09:54]
fields like computer vision natural
[09:56]
language processing and other areas of
[09:58]
machine learning have been utterly
[10:00]
transformed by neural networks
[10:03]
and this is all because of the humble
[10:04]
function a simple yet powerful way to
[10:07]
think about the world and by combining
[10:10]
simple computations we can get computers
[10:12]
to construct any function we could ever
[10:15]
want
[10:15]
neural networks can learn almost
[10:18]
anything
[10:19]
[Music]