---
title: 'How Neural Networks Learn Concepts'
source: 'https://youtube.com/watch?v=e5xKayCBOeU'
video_id: 'e5xKayCBOeU'
date: 2026-06-17
duration_sec: 0
---

# How Neural Networks Learn Concepts

> Source: [How Neural Networks Learn Concepts](https://youtube.com/watch?v=e5xKayCBOeU)

## Summary

This video explains how neural networks learn concepts by exploring their internal mechanics. It describes how perceptions (input data) are transformed through layers of neurons, acting as partitions in a high-dimensional space, ultimately carving out regions that represent concepts. The power of depth (multiple layers) is highlighted as the key to disentangling complex data.

### Key Points

- **Paradigm Shift in AI** [0:00] — Deep learning is a paradigm shift where intelligence is understood as the ability to learn, not follow human instructions.
- **Neural Network Inputs and Structure** [0:23] — A perception (e.g., image, sound) is a list of measurements input as a vector. Values are sent from the input layer through neurons that fire or not, creating a wave of activity to the output layer.
- **Single Neuron as a Switch** [1:43] — A single neuron is a switch: if input is above an activation threshold, output turns on. This divides the perception space into active/inactive regions.
- **Mathematical Model of Neurons** [2:09] — Input values are points in a perception space (1D for one input). A neuron acts as a partition (line, plane, hyperplane) dividing the space.
- **Perception Space and Concepts** [2:25] — Training moves the partition by changing weights. Concepts are regions in perception space defined by neuron activation patterns.
- **Two-Input Neuron** [3:06] — With two inputs, the perception space is 2D. A neuron is a straight line separating active and non-active regions.
- **Limitation of Single Neurons** [4:35] — Single neurons cannot separate non-linearly separable data (e.g., winter vs. summer days by temperature + humidity). Multiple neurons create multiple partitions.
- **Summary of Concept** [5:38] — Perceptions are points in N-dimensional space. Neurons are partitions; groups of neurons define regions corresponding to concepts.
- **Need for Depth** [7:22] — Shallow networks with one middle layer struggle with messy real-world data (e.g., handwritten digits). Depth allows exponential partitioning via recursive folding.
- **Layered Folds Analogy** [8:43] — Layering folds (multiple layers) carves the space exponentially more efficiently. Three layers achieve what six single-layer folds do. Depth gives exponential power.
- **Real Neural Network Probes** [9:27] — Researchers probed a trained network: first layers detect edges/points, deeper layers detect textures, deepest layers detect entire objects (dogs, wheels).
- **Spatial Transformation Through Layers** [10:52] — Layers transform points from perception space to concept space, pulling apart dissimilar points and pushing together similar ones.
- **Disentangling Inputs** [11:29] — Messy input points (handwritten digits) are gradually separated into tight clusters through layers, allowing final layer to easily partition them.
- **True Power of Neural Networks** [12:07] — The magic is layered processing: final layer carves concept space where points are clustered, not raw perception space.
- **Manifolds and Intuition** [12:38] — Regions of concept space are like manifolds. Different objects activate different neuron groups deep in the network. This is analogous to human intuition.
- **Limits and Future Work** [13:21] — Single pass through a network simulates rapid intuition. Reasoning (conversation, games) requires sequential processing and working memory—the next frontier.

### Conclusion

A neural network's true power lies in its layered structure, which transforms messy perceptual data into a cleanly separable concept space, allowing the network to 'know' concepts by proximity to clusters. This explains both how machines recognize objects and offers a model for human intuition.

## Transcript

so far in this series we've looked at
how deep learning was a paradigm shift
in AI where intelligence is understood
to be the ability to learn instead of
following human instructions the problem
is most people see neural networks as
some kind of magic it's not clear why
they work and so in this video we'll
explore the guts of a neural network
recall that a neural network receives an
input which we could call a perception
this could represent an image sound text
anything we want a network to perceive a
perception boils down to a list of
measurements that are provided as input
as a list or vector of values for
example if it was an image each value
would represent a single pixels value in
that image and these values are
represented as electrical pulses which
are sent to the first layer of neurons
which we call the input layer and based
on these values some of the neurons in
this layer will fire in a predictable
manner and send off a pulse to the next
layer of neurons and this process
repeats creating a wave of electrical
activity that passes through all the
layers in the network at the final
output layer certain neurons turn on or
off the output can describe the degree
of belief that the input is or isn't
some concept it's been trained to
recognize based on the activation level
of the output neurons a key question is
how does a neural network connect
perceptions to concepts
put another way if we freeze a neural
network as it's processing the picture
of a dog what is it doing inside right
before it knows it's a dog let's start
with the simplest possible neural
network a single neuron with just one
input and one output you can think of a
neuron as a switch the input is
represented as a number which is the
value of something being measured such
as the temperature outside if the input
value is above a certain threshold what
we call the activation threshold it
flips the output on otherwise the output
remains off
and to better understand the guts of a
neural network we'll need a mathematical
model of this simple switching action
imagine the input value or temperature
is a point on a line the position of
this point depends on the input value we
can think of this as our perception
space it's one dimensional because we
only have one input a neuron can be
viewed as dividing the perception space
into active or inactive regions if the
value is above the neurons activation
threshold it fires down the output and
so when we train a neuron we are moving
this dividing line around by changing
the weight of the incoming connection
which determines how much current is
needed to trigger the neuron similar to
a thermostat where the input is the
temperature and the activation point is
where we like the air conditioning to
turn on so let's consider a neuron with
two inputs
perhaps the input is a temperature and
pressure reading of the environment now
our model will have two variables which
can each define a position along a
dimension and so together they can be
thought of as a two dimensional
perception space or plane where every
input to the neuron is a point in this
2d space in this case the neuron can be
represented as a straight line which
partitions the space into active and
non-active regions any input on this
side will trigger the neuron and any
input on this side will not and this
pattern continues if we add more inputs
we just move up a dimension each
measurement or input can be represented
as a point in 3-dimensional perception
space and the neuron can be represented
as a plane which partitions this space
into active and non-active regions
perhaps we want the neuron to act as a
storm detector for example and so no
matter how many inputs it has a neuron
is like a partition or linear separation
of a set of data points in perception
space in higher dimensions we just call
it a hyperplane
this is how perceptions which are values
measuring the environment can turn into
concepts where a concept is a region in
perception space that's how the neuron
knows how to feel a storm if a
measurement is in the right region but
of course reality is not always so
simple because we can always draw a
straight line through our problems for
example imagine a situation where we
have two kinds of input measurements
temperature and humidity and our input
measurements arrange themselves in a
perception space like this the circles
represent measurements of winter days
and the x's represents summer days well
we can't draw a single line to separate
these points but if we have two lines we
can separating the data into four
distinct regions each region is defined
by the state of the neurons being on or
off if neuron one is on and neuron two
is off we know it's in this blue region
and that is why we need to use multiple
neurons so we have the ability to carve
up the perception space into many more
regions this is what the learning
process does by changing the weights of
the connections we move these partitions
around to carve out regions around
conceptually similar input points so
let's pause and summarize a perception
is a list of measurements that are
inputted into a network these vectors
can represent a coordinate or point in
perception space the number of
dimensions in this space is equal to the
number of different input values and
neurons act as partitions in this space
and a group of neurons together define a
specific region in this space and these
regions can carve out inputs which are
part of the same concept but so far
we've been looking at simple toy
problems and when we move to the real
world things get a little bit more
interesting for example the first big
commercial application of neural
networks was vision specifically making
a machine which can understand human
handwriting so that at the post office
that can read human letter at
automatically this is a hard problem
because everyone writes numbers slightly
differently so the machine must find the
general pattern of each number in this
example the input to our network is an
image containing 784 individual pixels
and so we have 784 input dimensions each
which measure the brightness of one
pixel using our spatial view we can
think of the image of each written digit
we input to the network as a point in
perception space and if we take many
real examples and plot them in the
perception space we get this the points
are not nicely clustered into regions
but scattered all over and so to carve
up this space into regions is going to
be very difficult the messy distribution
of inputs in perception space is why
shallow networks with only one middle
layer struggled to divide categories up
cleanly the way out of this problem
though is to follow the way of nature
organic brains use layers of neuron
activations to process their inputs the
importance of depth or many layers is
the least understood aspect of neural
networks so let's pause and consider a
simple analogy to understand why
multi-layered networks are better at
partitioning the perception space than a
single layer network imagine this is our
perception space and we have two kinds
of input data types each neuron we add
in the first layer acts like a fold in
this space with two neurons we can make
two folds like this and we could keep
going folding and unfolding the paper to
carve out regions to separate these
points this will take six separate folds
this allows us to then group regions
containing the same type of points using
a final neuron which activates if any of
those regions are active but now
consider what happens if we layer our
folds that is we don't unfold after each
fold so let's do the first fold again
then the second then the third fold
across that layer like this
that ends up carving the space in the
exact same way using three folds instead
of six and if we were to continue this
process with a fourth fold that results
in 16 regions and five folds results in
32 regions this recursive power of
folding shows how we can get
exponentially more partitions using the
same number of neurons if we layer them
practically this means that neurons deep
in a network are not simple linear
partitions but are instead activated by
a complex pattern of linear partitions
and so let's look at how this works
using a real world example researchers
took a neural network which was trained
on real images such as image net and
then probed individual neurons to find
out what activated them or what turned
them on if we probe neurons in the first
layer of the network we find they are
detecting these patterns which are
looking for our edges and points then
they move to the next layer deeper into
the network and probe those neurons to
see what activates them what they found
was the next layers are activated by
different kinds of textures and deeper
into the network these textures get more
specific and as you move deeper into the
network
the textures get more complex and the
deepest layers contain individual
neurons that are activated by entire
objects such as dogs wheels houses or
trees these complex activation patterns
are possible due to the layered
structure of the network and so if we
cut open a neural network will find the
deep layers contain representations of a
perception base on what level of
different things or patterns they
contain which is defined by how active
those specific neurons are for example
of an image of a dog would light up the
doglike patterns in the network
finally let's flip back to the spatial
perspective to see this power of layers
in acts
recall that our input such as an image
can be represented by a point in
perception space and each of the
following layer activations can be
thought as moving that point to a new
location and it finally settles into a
final space at the end of the network we
could call concept space and critically
the job of these transformations is to
pull apart dissimilar points in concept
space and push together similar ones to
see this in action let's return to our
real example where we plot the points of
various perceptions of different human
written digits notice at first the
points are scattered all over but as
these perceptions move through the
network gradually these points are
separated into tighter and tighter
clusters and so each layer acts as a
transformation that gradually
disentangle x' these points this allows
the final layer of neurons to easily
partition the data into separable
regions which represents the concepts in
this case numbers we are looking to
classify and so the magic or true power
of a neural network is entirely in this
layered processing because it allows the
final layer of neurons to carve up
concept space where the points are
nicely clustered instead of perception
space which is hard or impossible to
partition that's how a neural network
knows something is a number 3 or 6 based
on the proximity to the 3 or 6 cluster
this partly describes why we have
different mental feelings or intuitions
when we see different objects it's
because different groups of neurons are
being activated deep in our mind
depending on what cluster it belongs to
and these clusters can also be thought
of as connected regions or manifolds and
so perceptions representing the written
digit 6 would fall on one manifold and
the digit 3 would fall on another and so
manifolds are spatial regions which
represent
the patterns essential to a concept
these patterns are defined by the
connection patterns or strength across
the neuron layers but so far we have
only modeled how intuition works that is
a rapid interpretation of some impulse
such as when you recognize someone's
voice by a single syllable this is what
a single pass through a neural network
simulates the other big challenge is
problems which require reasoning
interactive problems such as having a
conversation or playing a game these
problems are sequential in nature and
require things like a form of working
memory this leads us to the cutting edge
of neural network research
how will a neural network learn to
reason
[Music]
you