---
title: 'DeepMind Just Changed How AI Sees The World'
source: 'https://youtube.com/watch?v=vO6SWG-jxvE'
video_id: 'vO6SWG-jxvE'
date: 2026-08-08
duration_sec: 322
---

# DeepMind Just Changed How AI Sees The World

> Source: [DeepMind Just Changed How AI Sees The World](https://youtube.com/watch?v=vO6SWG-jxvE)

## Summary

This video from Two Minute Papers explains how DeepMind's Gemma 4 model achieves vision and multimodal reasoning in a remarkably small, local AI model. The key innovation is discarding separate vision and audio encoders in favor of directly projecting raw pixels and audio chunks into the main transformer, blurring the line between perception and thinking.

### Key Points

- **The Problem with Large AI Models** [00:03] — Large open models like DeepSeek have over 1.6 trillion parameters and cost hundreds of thousands of dollars to run, yet they cannot see images.
- **Gemma 4: A Tiny Model That Can See** [00:45] — DeepMind's Gemma 4 is 99% smaller than these giants, runs on a laptop, is free and open, and has been downloaded over 300 million times.
- **The Secret Sauce Revealed** [01:09] — DeepMind finally disclosed the architecture behind Gemma 4's vision capabilities, which is surprising and counterintuitive.
- **Conventional Multimodal Architecture** [01:40] — Traditional AI systems use separate neural networks (encoders) for vision, audio, and other modalities, which just translate inputs into a format the main model can process.
- **The Revolutionary Approach** [02:23] — For the 12B model, DeepMind throws away all separate encoders. Instead, images are cut into patches and projected directly into the model's internal representation, and audio is sliced into 40-ms chunks.
- **Unified Processing** [03:02] — All these tokens are poured into the main transformer, forcing the system to learn to be eyes, ears, and brain simultaneously. This removes hundreds of millions of specialist parameters.
- **The Result** [03:28] — Gemma 4 punches above its weight, handling images and audio while being highly intelligent. The architecture blurs the boundary between perception and thinking.
- **Ecosystem and Impact** [03:45] — The Gemma 4 ecosystem is improving, and the shared architecture can help other models like DeepSeek learn to see better and more efficiently.
- **A Call to Support Open Models** [04:14] — The presenter urges the community to support open models, as these gifts may stop coming as capabilities increase. He thanks all open model developers.
- **Sponsor: Lambda** [04:44] — Lambda provides Nvidia GPUs for reproducing research papers, training models, and running inference. Try it at lambda.ai/papers.

### Conclusion

Gemma 4's architecture is a breakthrough in efficient multimodal AI, proving that small models can achieve vision by unifying perception and reasoning. The open-source community benefits greatly from such innovations, and continued support is crucial.

## Transcript

see, today AI models have become almost comically large. Some of the biggest open and free AI models, like DeepSeek, reached over 1.6 trillion parameters in
size. And that's not even the biggest one. So, these cost hundreds of thousands of dollars to run. And then, you find out something crazy. You show them an image and you ask, "What does this image depict?" And you will be
surprised to hear the answer. It doesn't know. Yes, it cannot see. Now, what if I told you that they promised you a model that is 99% smaller, a speck of dust comparatively, and yet this tiny guy can see? That
sounds impossible. Yes, maybe in our dreams, wishful thinking. But, it actually exists. It is made by DeepMind and is called Gemma 4. This runs on your
laptop. It is an absolute gem. Free and open, downloaded more than 300 million times by us fellow scholars. That is insane. I love it. And then, something
amazing happened. Now, hold on to your papers, fellow scholars, because, yep, they gave us the secret sauce. They finally told us the architecture they used for Gemma 4 to see. And it's
kind of crazy. So, we finally understand how this can pull off things so easily, like talking as a medieval bard while identifying objects in your video, and so much more. So, how did they add vision and multimodal reasoning into an
unusually small local model? The secrets are finally out. Dear fellow scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. So, a conventional AI is really several neural networks connected together. And if you have an
image, you need a dedicated visual model just for that. Or Okay, what about audio? Giving it ears. Yep, that needs an audio encoder. A specialized part for
each of these tasks. But this is not really looking or listening. This is just passing a translation around that is created by a different neural network. Their smallest models do that, but when we upgrade to the 12 billion
model, things get crazy. Scientists at DeepMind say, "Throw that all away. Out. Right now." Instead, mhm, it cuts your picture into small patches. Then it
projects those pixels directly into the model's internal representation. So, it knows where each patch came from. Okay, but what is the point here? Why do that?
Well, you don't need a separate neural network, a vision transformer, to interpret the image for you. No. Throw it out. Same for audio. Slice it up into it out. Same for audio. Slice it up into 40-ms chunks, and then comes the magic.
You just pour all these tokens into your main transformer, and then what happens? Well, this system is forced to learn to be the eyes, ears, and brain at the same
time. And I think this is one of the most important architectural ideas in most important architectural ideas in Gemma 4. It removes hundreds of millions of specialist parameters, and it blurs the boundary between perception and
thinking. And the result is an AI system that punches way above its weight. It handles images, it handles audio, and it is bloody smart. And this is something that you can own. Now, two more points. One, the Gemma 4 ecosystem continues to
get improvements to make it faster and better. I'll give you a link to So in the description. Two, they shared the secret sauce there. So Gemma 4 is not just amazing in and of itself, but it can help Deep Seek and other systems
learn to see better and more efficiently. That is absolutely amazing. What a time to be alive. And please do not take it for granted that these amazing open models will just keep
coming in the future. These are gifts to all of us and these gifts may stop coming as capabilities increase. It is not a law of nature that we just get these models for free in the future, too. So, to everyone who is working on
open models, wherever you are in the world, you are heroes. You help scientists, students, and millions of other people to do their work better. Thank you so much. And we, as a community, have to come together and do
everything we can to support these open systems. I use Lambda to reproduce AI research papers often in minutes. It's also great to train your own models or
fine-tune an existing one. Run inference or text-to-image or video, easy-peasy. Running a Deep Seek chatbot or agent, super fast, super reliable. Lambda gives you powerful Nvidia GPUs to run your own experiments. I test ideas from the
papers I cover and moments later, results. Love it. Seriously, try it out now at lambda.ai/papers.