AI Models Can't See?!
42sThe shocking revelation that huge AI models can't see images creates a strong hook that challenges viewers' expectations.
▶ Play ClipAI Summary
This video from Two Minute Papers explains how DeepMind's Gemma 4 model achieves vision and multimodal reasoning in a remarkably small, local AI model. The key innovation is discarding separate vision and audio encoders in favor of directly projecting raw pixels and audio chunks into the main transformer, blurring the line between perception and thinking.
Chapters
[00:03]
The Problem with Large AI Models
Large open models like DeepSeek have over 1.6 trillion parameters and cost hundreds of thousands of dollars to run, yet they cannot see images.
[00:45]
Gemma 4: A Tiny Model That Can See
DeepMind's Gemma 4 is 99% smaller than these giants, runs on a laptop, is free and open, and has been downloaded over 300 million times.
[01:09]
The Secret Sauce Revealed
DeepMind finally disclosed the architecture behind Gemma 4's vision capabilities, which is surprising and counterintuitive.
[01:40]
Conventional Multimodal Architecture
Traditional AI systems use separate neural networks (encoders) for vision, audio, and other modalities, which just translate inputs into a format the main model can process.
[02:23]
The Revolutionary Approach
For the 12B model, DeepMind throws away all separate encoders. Instead, images are cut into patches and projected directly into the model's internal representation, and audio is sliced into 40-ms chunks.
[03:02]
Unified Processing
All these tokens are poured into the main transformer, forcing the system to learn to be eyes, ears, and brain simultaneously. This removes hundreds of millions of specialist parameters.
[03:28]
The Result
Gemma 4 punches above its weight, handling images and audio while being highly intelligent. The architecture blurs the boundary between perception and thinking.
[03:45]
Ecosystem and Impact
The Gemma 4 ecosystem is improving, and the shared architecture can help other models like DeepSeek learn to see better and more efficiently.
[04:14]
A Call to Support Open Models
The presenter urges the community to support open models, as these gifts may stop coming as capabilities increase. He thanks all open model developers.
[04:44]
Sponsor: Lambda
Lambda provides Nvidia GPUs for reproducing research papers, training models, and running inference. Try it at lambda.ai/papers.
Gemma 4's architecture is a breakthrough in efficient multimodal AI, proving that small models can achieve vision by unifying perception and reasoning. The open-source community benefits greatly from such innovations, and continued support is crucial.
Mentioned in this Video
Study Flashcards (7)
What is the approximate parameter count of DeepSeek, one of the largest open AI models?
easy
Click to reveal answer
What is the approximate parameter count of DeepSeek, one of the largest open AI models?
easy
Click to reveal answer
Over 1.6 trillion parameters.
00:03
How much smaller is Gemma 4 compared to large models like DeepSeek?
easy
Click to reveal answer
How much smaller is Gemma 4 compared to large models like DeepSeek?
easy
Click to reveal answer
99% smaller.
00:45
How many times has Gemma 4 been downloaded?
easy
Click to reveal answer
How many times has Gemma 4 been downloaded?
easy
Click to reveal answer
Over 300 million times.
00:57
What is the conventional approach to adding vision to an AI model?
medium
Click to reveal answer
What is the conventional approach to adding vision to an AI model?
medium
Click to reveal answer
Using a dedicated visual model (vision transformer) that translates images into a format the main model can process.
01:55
What is the key architectural innovation in Gemma 4's 12B model?
medium
Click to reveal answer
What is the key architectural innovation in Gemma 4's 12B model?
medium
Click to reveal answer
It discards separate encoders and directly projects image patches and audio chunks into the main transformer.
02:23
How is audio processed in Gemma 4's unified architecture?
medium
Click to reveal answer
How is audio processed in Gemma 4's unified architecture?
medium
Click to reveal answer
It is sliced into 40-millisecond chunks and fed directly into the main transformer.
03:02
What is the benefit of removing specialist parameters in Gemma 4?
hard
Click to reveal answer
What is the benefit of removing specialist parameters in Gemma 4?
hard
Click to reveal answer
It removes hundreds of millions of parameters and blurs the boundary between perception and thinking, making the system more efficient.
03:14
💡 Key Takeaways
🔧
Throwing Away Encoders
This is the core architectural insight that challenges conventional multimodal design.
02:23
💡
Unified Perception and Thinking
The idea that a single transformer can learn to be eyes, ears, and brain is a paradigm shift.
03:02
⚖️
Open Models as Gifts
A reminder that open-source AI contributions are not guaranteed and need community support.
04:14Full Transcript
[00:03] see, today AI models have become almost comically large. Some of the biggest open and free AI models, like DeepSeek, reached over 1.6 trillion parameters in
[00:15] size. And that's not even the biggest one. So, these cost hundreds of thousands of dollars to run. And then, you find out something crazy. You show them an image and you ask, "What does this image depict?" And you will be
[00:28] surprised to hear the answer. It doesn't know. Yes, it cannot see. Now, what if I told you that they promised you a model that is 99% smaller, a speck of dust comparatively, and yet this tiny guy can see? That
[00:45] sounds impossible. Yes, maybe in our dreams, wishful thinking. But, it actually exists. It is made by DeepMind and is called Gemma 4. This runs on your
[00:57] laptop. It is an absolute gem. Free and open, downloaded more than 300 million times by us fellow scholars. That is insane. I love it. And then, something
[01:09] amazing happened. Now, hold on to your papers, fellow scholars, because, yep, they gave us the secret sauce. They finally told us the architecture they used for Gemma 4 to see. And it's
[01:23] kind of crazy. So, we finally understand how this can pull off things so easily, like talking as a medieval bard while identifying objects in your video, and so much more. So, how did they add vision and multimodal reasoning into an
[01:40] unusually small local model? The secrets are finally out. Dear fellow scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. So, a conventional AI is really several neural networks connected together. And if you have an
[01:55] image, you need a dedicated visual model just for that. Or Okay, what about audio? Giving it ears. Yep, that needs an audio encoder. A specialized part for
[02:07] each of these tasks. But this is not really looking or listening. This is just passing a translation around that is created by a different neural network. Their smallest models do that, but when we upgrade to the 12 billion
[02:23] model, things get crazy. Scientists at DeepMind say, "Throw that all away. Out. Right now." Instead, mhm, it cuts your picture into small patches. Then it
[02:35] projects those pixels directly into the model's internal representation. So, it knows where each patch came from. Okay, but what is the point here? Why do that?
[02:47] Well, you don't need a separate neural network, a vision transformer, to interpret the image for you. No. Throw it out. Same for audio. Slice it up into it out. Same for audio. Slice it up into 40-ms chunks, and then comes the magic.
[03:02] You just pour all these tokens into your main transformer, and then what happens? Well, this system is forced to learn to be the eyes, ears, and brain at the same
[03:14] time. And I think this is one of the most important architectural ideas in most important architectural ideas in Gemma 4. It removes hundreds of millions of specialist parameters, and it blurs the boundary between perception and
[03:28] thinking. And the result is an AI system that punches way above its weight. It handles images, it handles audio, and it is bloody smart. And this is something that you can own. Now, two more points. One, the Gemma 4 ecosystem continues to
[03:45] get improvements to make it faster and better. I'll give you a link to So in the description. Two, they shared the secret sauce there. So Gemma 4 is not just amazing in and of itself, but it can help Deep Seek and other systems
[04:01] learn to see better and more efficiently. That is absolutely amazing. What a time to be alive. And please do not take it for granted that these amazing open models will just keep
[04:14] coming in the future. These are gifts to all of us and these gifts may stop coming as capabilities increase. It is not a law of nature that we just get these models for free in the future, too. So, to everyone who is working on
[04:29] open models, wherever you are in the world, you are heroes. You help scientists, students, and millions of other people to do their work better. Thank you so much. And we, as a community, have to come together and do
[04:44] everything we can to support these open systems. I use Lambda to reproduce AI research papers often in minutes. It's also great to train your own models or
[04:56] fine-tune an existing one. Run inference or text-to-image or video, easy-peasy. Running a Deep Seek chatbot or agent, super fast, super reliable. Lambda gives you powerful Nvidia GPUs to run your own experiments. I test ideas from the
[05:12] papers I cover and moments later, results. Love it. Seriously, try it out now at lambda.ai/papers.
