[00:00]
so far in this series we've looked at
[00:02]
how deep learning was a paradigm shift
[00:04]
in AI where intelligence is understood
[00:06]
to be the ability to learn instead of
[00:08]
following human instructions the problem
[00:11]
is most people see neural networks as
[00:13]
some kind of magic it's not clear why
[00:16]
they work and so in this video we'll
[00:19]
explore the guts of a neural network
[00:23]
recall that a neural network receives an
[00:25]
input which we could call a perception
[00:27]
this could represent an image sound text
[00:30]
anything we want a network to perceive a
[00:33]
perception boils down to a list of
[00:35]
measurements that are provided as input
[00:37]
as a list or vector of values for
[00:41]
example if it was an image each value
[00:43]
would represent a single pixels value in
[00:46]
that image and these values are
[00:49]
represented as electrical pulses which
[00:51]
are sent to the first layer of neurons
[00:53]
which we call the input layer and based
[00:55]
on these values some of the neurons in
[00:58]
this layer will fire in a predictable
[01:00]
manner and send off a pulse to the next
[01:02]
layer of neurons and this process
[01:04]
repeats creating a wave of electrical
[01:06]
activity that passes through all the
[01:08]
layers in the network at the final
[01:11]
output layer certain neurons turn on or
[01:13]
off the output can describe the degree
[01:16]
of belief that the input is or isn't
[01:19]
some concept it's been trained to
[01:20]
recognize based on the activation level
[01:23]
of the output neurons a key question is
[01:26]
how does a neural network connect
[01:29]
perceptions to concepts
[01:31]
put another way if we freeze a neural
[01:34]
network as it's processing the picture
[01:36]
of a dog what is it doing inside right
[01:39]
before it knows it's a dog let's start
[01:43]
with the simplest possible neural
[01:45]
network a single neuron with just one
[01:47]
input and one output you can think of a
[01:50]
neuron as a switch the input is
[01:52]
represented as a number which is the
[01:54]
value of something being measured such
[01:57]
as the temperature outside if the input
[02:00]
value is above a certain threshold what
[02:03]
we call the activation threshold it
[02:05]
flips the output on otherwise the output
[02:08]
remains off
[02:09]
and to better understand the guts of a
[02:11]
neural network we'll need a mathematical
[02:13]
model of this simple switching action
[02:16]
imagine the input value or temperature
[02:18]
is a point on a line the position of
[02:22]
this point depends on the input value we
[02:25]
can think of this as our perception
[02:27]
space it's one dimensional because we
[02:29]
only have one input a neuron can be
[02:33]
viewed as dividing the perception space
[02:36]
into active or inactive regions if the
[02:39]
value is above the neurons activation
[02:42]
threshold it fires down the output and
[02:45]
so when we train a neuron we are moving
[02:48]
this dividing line around by changing
[02:50]
the weight of the incoming connection
[02:52]
which determines how much current is
[02:54]
needed to trigger the neuron similar to
[02:57]
a thermostat where the input is the
[02:59]
temperature and the activation point is
[03:02]
where we like the air conditioning to
[03:04]
turn on so let's consider a neuron with
[03:06]
two inputs
[03:07]
perhaps the input is a temperature and
[03:10]
pressure reading of the environment now
[03:13]
our model will have two variables which
[03:15]
can each define a position along a
[03:17]
dimension and so together they can be
[03:19]
thought of as a two dimensional
[03:21]
perception space or plane where every
[03:24]
input to the neuron is a point in this
[03:27]
2d space in this case the neuron can be
[03:30]
represented as a straight line which
[03:32]
partitions the space into active and
[03:34]
non-active regions any input on this
[03:37]
side will trigger the neuron and any
[03:39]
input on this side will not and this
[03:42]
pattern continues if we add more inputs
[03:44]
we just move up a dimension each
[03:46]
measurement or input can be represented
[03:49]
as a point in 3-dimensional perception
[03:52]
space and the neuron can be represented
[03:55]
as a plane which partitions this space
[03:57]
into active and non-active regions
[04:00]
perhaps we want the neuron to act as a
[04:02]
storm detector for example and so no
[04:06]
matter how many inputs it has a neuron
[04:09]
is like a partition or linear separation
[04:11]
of a set of data points in perception
[04:14]
space in higher dimensions we just call
[04:17]
it a hyperplane
[04:19]
this is how perceptions which are values
[04:21]
measuring the environment can turn into
[04:24]
concepts where a concept is a region in
[04:27]
perception space that's how the neuron
[04:30]
knows how to feel a storm if a
[04:32]
measurement is in the right region but
[04:35]
of course reality is not always so
[04:37]
simple because we can always draw a
[04:38]
straight line through our problems for
[04:41]
example imagine a situation where we
[04:43]
have two kinds of input measurements
[04:45]
temperature and humidity and our input
[04:48]
measurements arrange themselves in a
[04:50]
perception space like this the circles
[04:53]
represent measurements of winter days
[04:55]
and the x's represents summer days well
[04:58]
we can't draw a single line to separate
[05:01]
these points but if we have two lines we
[05:04]
can separating the data into four
[05:06]
distinct regions each region is defined
[05:09]
by the state of the neurons being on or
[05:11]
off if neuron one is on and neuron two
[05:14]
is off we know it's in this blue region
[05:17]
and that is why we need to use multiple
[05:19]
neurons so we have the ability to carve
[05:21]
up the perception space into many more
[05:23]
regions this is what the learning
[05:26]
process does by changing the weights of
[05:29]
the connections we move these partitions
[05:31]
around to carve out regions around
[05:34]
conceptually similar input points so
[05:38]
let's pause and summarize a perception
[05:41]
is a list of measurements that are
[05:42]
inputted into a network these vectors
[05:45]
can represent a coordinate or point in
[05:48]
perception space the number of
[05:50]
dimensions in this space is equal to the
[05:53]
number of different input values and
[05:55]
neurons act as partitions in this space
[05:58]
and a group of neurons together define a
[06:02]
specific region in this space and these
[06:05]
regions can carve out inputs which are
[06:07]
part of the same concept but so far
[06:10]
we've been looking at simple toy
[06:12]
problems and when we move to the real
[06:14]
world things get a little bit more
[06:16]
interesting for example the first big
[06:19]
commercial application of neural
[06:20]
networks was vision specifically making
[06:24]
a machine which can understand human
[06:26]
handwriting so that at the post office
[06:29]
that can read human letter at
[06:30]
automatically this is a hard problem
[06:34]
because everyone writes numbers slightly
[06:36]
differently so the machine must find the
[06:38]
general pattern of each number in this
[06:42]
example the input to our network is an
[06:43]
image containing 784 individual pixels
[06:47]
and so we have 784 input dimensions each
[06:51]
which measure the brightness of one
[06:53]
pixel using our spatial view we can
[06:56]
think of the image of each written digit
[06:58]
we input to the network as a point in
[07:01]
perception space and if we take many
[07:04]
real examples and plot them in the
[07:07]
perception space we get this the points
[07:10]
are not nicely clustered into regions
[07:12]
but scattered all over and so to carve
[07:15]
up this space into regions is going to
[07:17]
be very difficult the messy distribution
[07:20]
of inputs in perception space is why
[07:22]
shallow networks with only one middle
[07:25]
layer struggled to divide categories up
[07:28]
cleanly the way out of this problem
[07:30]
though is to follow the way of nature
[07:32]
organic brains use layers of neuron
[07:35]
activations to process their inputs the
[07:38]
importance of depth or many layers is
[07:41]
the least understood aspect of neural
[07:43]
networks so let's pause and consider a
[07:46]
simple analogy to understand why
[07:48]
multi-layered networks are better at
[07:50]
partitioning the perception space than a
[07:52]
single layer network imagine this is our
[07:55]
perception space and we have two kinds
[07:58]
of input data types each neuron we add
[08:01]
in the first layer acts like a fold in
[08:04]
this space with two neurons we can make
[08:07]
two folds like this and we could keep
[08:09]
going folding and unfolding the paper to
[08:12]
carve out regions to separate these
[08:14]
points this will take six separate folds
[08:17]
this allows us to then group regions
[08:20]
containing the same type of points using
[08:23]
a final neuron which activates if any of
[08:25]
those regions are active but now
[08:29]
consider what happens if we layer our
[08:31]
folds that is we don't unfold after each
[08:33]
fold so let's do the first fold again
[08:36]
then the second then the third fold
[08:40]
across that layer like this
[08:43]
that ends up carving the space in the
[08:45]
exact same way using three folds instead
[08:48]
of six and if we were to continue this
[08:51]
process with a fourth fold that results
[08:54]
in 16 regions and five folds results in
[08:59]
32 regions this recursive power of
[09:02]
folding shows how we can get
[09:05]
exponentially more partitions using the
[09:07]
same number of neurons if we layer them
[09:11]
practically this means that neurons deep
[09:13]
in a network are not simple linear
[09:16]
partitions but are instead activated by
[09:19]
a complex pattern of linear partitions
[09:22]
and so let's look at how this works
[09:24]
using a real world example researchers
[09:27]
took a neural network which was trained
[09:29]
on real images such as image net and
[09:31]
then probed individual neurons to find
[09:35]
out what activated them or what turned
[09:38]
them on if we probe neurons in the first
[09:41]
layer of the network we find they are
[09:44]
detecting these patterns which are
[09:46]
looking for our edges and points then
[09:49]
they move to the next layer deeper into
[09:51]
the network and probe those neurons to
[09:54]
see what activates them what they found
[09:56]
was the next layers are activated by
[09:58]
different kinds of textures and deeper
[10:02]
into the network these textures get more
[10:04]
specific and as you move deeper into the
[10:07]
network
[10:07]
the textures get more complex and the
[10:11]
deepest layers contain individual
[10:13]
neurons that are activated by entire
[10:16]
objects such as dogs wheels houses or
[10:20]
trees these complex activation patterns
[10:24]
are possible due to the layered
[10:26]
structure of the network and so if we
[10:28]
cut open a neural network will find the
[10:31]
deep layers contain representations of a
[10:33]
perception base on what level of
[10:36]
different things or patterns they
[10:38]
contain which is defined by how active
[10:41]
those specific neurons are for example
[10:44]
of an image of a dog would light up the
[10:47]
doglike patterns in the network
[10:50]
finally let's flip back to the spatial
[10:52]
perspective to see this power of layers
[10:55]
in acts
[10:56]
recall that our input such as an image
[10:59]
can be represented by a point in
[11:01]
perception space and each of the
[11:04]
following layer activations can be
[11:06]
thought as moving that point to a new
[11:09]
location and it finally settles into a
[11:12]
final space at the end of the network we
[11:14]
could call concept space and critically
[11:18]
the job of these transformations is to
[11:20]
pull apart dissimilar points in concept
[11:24]
space and push together similar ones to
[11:28]
see this in action let's return to our
[11:29]
real example where we plot the points of
[11:32]
various perceptions of different human
[11:35]
written digits notice at first the
[11:38]
points are scattered all over but as
[11:41]
these perceptions move through the
[11:44]
network gradually these points are
[11:46]
separated into tighter and tighter
[11:49]
clusters and so each layer acts as a
[11:52]
transformation that gradually
[11:54]
disentangle x' these points this allows
[11:57]
the final layer of neurons to easily
[11:59]
partition the data into separable
[12:01]
regions which represents the concepts in
[12:04]
this case numbers we are looking to
[12:07]
classify and so the magic or true power
[12:10]
of a neural network is entirely in this
[12:13]
layered processing because it allows the
[12:15]
final layer of neurons to carve up
[12:18]
concept space where the points are
[12:20]
nicely clustered instead of perception
[12:23]
space which is hard or impossible to
[12:25]
partition that's how a neural network
[12:28]
knows something is a number 3 or 6 based
[12:33]
on the proximity to the 3 or 6 cluster
[12:36]
this partly describes why we have
[12:38]
different mental feelings or intuitions
[12:40]
when we see different objects it's
[12:43]
because different groups of neurons are
[12:45]
being activated deep in our mind
[12:47]
depending on what cluster it belongs to
[12:50]
and these clusters can also be thought
[12:53]
of as connected regions or manifolds and
[12:57]
so perceptions representing the written
[12:59]
digit 6 would fall on one manifold and
[13:02]
the digit 3 would fall on another and so
[13:06]
manifolds are spatial regions which
[13:09]
represent
[13:09]
the patterns essential to a concept
[13:12]
these patterns are defined by the
[13:14]
connection patterns or strength across
[13:17]
the neuron layers but so far we have
[13:21]
only modeled how intuition works that is
[13:24]
a rapid interpretation of some impulse
[13:27]
such as when you recognize someone's
[13:29]
voice by a single syllable this is what
[13:32]
a single pass through a neural network
[13:34]
simulates the other big challenge is
[13:37]
problems which require reasoning
[13:41]
interactive problems such as having a
[13:43]
conversation or playing a game these
[13:46]
problems are sequential in nature and
[13:49]
require things like a form of working
[13:51]
memory this leads us to the cutting edge
[13:54]
of neural network research
[13:55]
how will a neural network learn to
[13:58]
reason
[14:00]
[Music]
[14:19]
you