[00:08] posed this challenge. Is it possible to come up with an image that works on this surface in both orientations of these twisty squares? Lots of people had a go, called Han Lin even put together a web
[00:22] interface to help with the design process. But honestly, it kind of seemed like it was impossible to come up with any kind of meaningful image. But then I got an email from Daniel Gang and simultaneously a Discord DM from Ryan
[00:34] Bergert. They both found images that work using generative AI. So, you know, you give it a text prompt and you get an image like Midjourney or Stable simple as just giving it a text prompt that says, "I want an image of a duck
[00:48] except that when you rotate the first, third, fifth, seventh, and ninth squares of this image 90° clockwise and all the other squares anticlockwise, I want it to look like a rabbit." I tried that, by the way, and it just gives you trash. It
[01:01] turns out that Daniel and Ryan, along with their respective research teams, creating optical illusions with generative AI by breaking the models apart and tinkering around with the gobbins inside. And what they'd come up
[01:16] with was perfect for the twisty squares problem that I had. This is one of the twisty squares solutions that Daniel sent me and this is one of the ones that Ryan sent me. I'll show you a few more over the course of the video. But
[01:29] were able to achieve these images, but I realized I don't really understand how vanilla text-to-speech image generation works. Daniel and Ryan were both really though, even if you understand what one of these generative models is, you may
[01:45] never understand how it works or why it works so well. So, they both use diffusion models, which is kind of the state of the art for this sort of thing. Ryan uses Stable Diffusion, which you might have heard of, and it's just kind
[01:59] of wild how well this works. The way you train a diffusion model neural network is you take some images, that's the training data, and you tweak the RGB values of all the pixels randomly. That's called adding noise. You then ask
[02:15] the model to tell you how it thinks you tweaked all of those pixels. Like, I think this pixel is too green by this much. This pixel is not red enough by a small amount, and so on. And it gives you that as an array of values. In other
[02:29] words, you're asking it what the noise was that you added to the image. And remember, this is still the training stage. So, at this point you say, "No, that's terrible what you came up with. This was the actual noise that I added.
[02:41] Now, go away and fix all of the parameters in your model so it does a you're doing it with millions of images at the same time. So, you're saying, "Look, you got all these pixels wrong by this much in this image, and you got all
[02:56] these pixels wrong by this much in this image, and all these other images, this wrong." Like, millions of images. So, try and go away and tweak all of the parameters inside your neural network so that it's better for all of these images
[03:11] at the same time. I told you it was wild. By the way, this is just a brief overview of how all this stuff works. I'll definitely be skipping over some along with the way. But the most important thing for me is that you have
[03:24] a satisfying understanding of how this stuff works. So, anyway, you train it on noisier and noisier images. In other words, images where you've messed about with the values of all the pixels more and more and more until eventually you
[03:37] noise, completely random pixels. And so then, when it gives you a prediction of the noise that was added to an image, when you remove that prediction, it will give you an image that looks like something recognizable from the training
[03:51] images. Now, at this point you have no control over what comes out at all. It's just going to be random what you get. To be able to create an image from a text prompt, the model needs to be able to understand the meaning of the text. So,
[04:05] maybe we need a separate model that can understand text. I'll just digress a little bit here cuz it'll help the explanation later, but the way a text model or a language model works is really clever. Basically, it goes
[04:17] through several iterations where its semantic understanding of the text becomes more and more nuanced. So, like in the first iteration, maybe it realizes that the word blue is in front of the word boat, so that means the boat
[04:32] is blue. Then a few iterations later, it might realize that the person mentioned at the start of the text prompt is actually on the boat, and the waves mentioned later are choppy, and that's causing the boat to rock. And then by
[04:44] the 20th iteration, it knows that the text was written as a gothic horror with you get out the other end for our purposes is basically just a long string of values that represents all these different meanings. That string of
[04:59] vector because you can think of it like a point in this super high-dimensional space of semantic meaning. I said earlier that it might be impossible to understand how these models work, and I think this is probably a good example of
[05:13] what I meant by that. Like to a certain extent, you can interrogate these vectors that represent the meaning of a prompt, and you can discover that a particular value might represent the linguistic concept of gender. So, that
[05:28] if that value increases, then the prompt becomes semantically more feminine, for example. But in general, it's very hard to see what these different values, or directions if you like, in this super high-dimensional space actually
[05:42] represent semantically. But it works somehow anyway. So, you give some language model a prompt that it turns into some high-dimensional vector that represents the meaning of the prompt, or maybe it's a few vectors
[05:56] prompt. But how the hell do we apply that to the image generation model that's just trying to figure out how to remove noise from an image? Well, the model is able to figure out what noise needs to be removed from an image
[06:10] because it has learned to understand features of images. And in some ways it's very similar to the way a language model understands the meaning of text, which is to say it goes through several iterations where its understanding of
[06:24] nuanced. So, maybe in the first iteration it's just picking up on where the edges are in the image. And then in later iterations, maybe it's picking up later iterations, maybe it's picking up on concepts like furriness or maybe
[06:38] light glinting off a shiny surface. And then later on, maybe that it's in the style of a Monet. it wouldn't have any words for these features, even though we might recognize them as furriness or shininess or carness or catness.
[06:52] Instead, it would just be a long list of values that represent these things. And values, it might be very difficult to actually pick out specific things that we would understand, but it works anyway. This long list of
[07:04] values, again, is a vector. But of course, this model was trained independently of any language model that we might have. So, the vectors in these two different models have no relation to each other. And that's where a model
[07:17] called CLIP comes in. CLIP was trained on millions of images from the internet and their captions. So, during training, it was learning the meaning of the text and the features of the image, and the vectors it was producing for both of
[07:32] those things, it was putting in the same place in a shared vector space of semantic meaning and image features. So, the vector that represents the meaning
[07:44] of the phrase black cat will exist in the same location in the shared space as a vector that represents the visual features of a black cat. So, when you're asking the noise guessing model what noise it thinks you'd need to remove to
[07:59] also saying, "By the way, I know that the image underneath this noise has a features vector pointing in this direction about it from a text prompt." And so, the model will now guess what noise
[08:14] needs to be removed in a way that pushes the image towards having features that are encoded in this vector. This stage is called cross-attention because the is called cross-attention because the features vector from CLIP attend to the
[08:28] features vector in the noise removing model. And there's one bit of detail driving me mad until I understood it. CLIP and the noise model were trained separately, meaning that their feature vectors all point in different
[08:41] directions. So, how does that work? How can the two models communicate with each other? And well, it's this cross-attention part that does that translating between these two different high-dimensional spaces of
[08:56] features. And of course, the cross-attention part of the model has the ability to do this translation because it was trained with lots of data. So, that's how diffusion models turn text prompts into images with a lot
[09:09] of simplifications. But to explain how these illusions were created, I need to tell you about a bit of the process that I skipped over. So, when you give a text prompt to Stable Diffusion like red bunny in a blue top hat, it starts off
[09:22] with an image of pure noise, then tries to figure out what noise should be removed. In other words, which pixels to adjust in order to be left with an image that has all the features from the vector that the CLIP model gave it. But,
[09:36] it doesn't do it in a single step. Another way of saying that is its first attempt is kind of blurry and gray and flat, which makes sense. Like, if you flat, which makes sense. Like, if you squint your eyes at an image of random
[09:49] noise, you might be able to pick out some large-scale variation that appeared just by chance, but you wouldn't be able to make out any detail. Noise kills detail. Or, in the case of the diffusion model with almost no features to work
[10:04] with, it will spit out an image that is like the average of all images that satisfy your prompt. In other words, a blurry blob. You then take that image and add, say, 90% of the noise back and ask the model to predict
[10:18] how much noise you added that time. And after you remove that guess, the image because the image already has some large-scale structure, and this reduced noise that's laid on top can start to nudge it in the direction of more detail
[10:35] here and there. Then, you do it again with 80% noise, and then 70% noise, 60%, zero noise, and you have a nice, crisp, clean image. The actual increments wouldn't be 10%, they'd be some other specific schedule
[10:49] idea. So, that's how a text prompt becomes an image, but knowing that it happens by these baby steps towards a fun with it. I came across this brilliant installation called Shadow
[11:03] Play, for example, that takes the shadow cast by someone and injects that into cast by someone and injects that into Stable Diffusion at the 50% noise stage. So, it's already primed by the shadow. And of course, Ryan and Daniel both take
[11:16] advantage of this gradual formation of the image over several steps. They ask the diffusion model for an image of a penguin, let's say, but they pause it after the first guess. Then, they ask it for a picture of a giraffe, let's say,
[11:29] and pause it after the first guess. So, they've got a blurry image that's that's approaching a giraffe. They then flip the image of the giraffe and combine them by taking the average of all the pixels. This combined image
[11:42] becomes the input for the next step of both the penguin generating task and the giraffe generating task. And so, slowly slowly the image becomes something that looks like a penguin in one orientation and a giraffe in the other. How cool is
[11:58] that? And it doesn't have to be that it's one image this way round and another image this way round. It could also be that it's a duck in this arrangement and a rabbit in this arrangement. Or you could jumble the
[12:10] parts up completely, which as it turns out is really useful for something Matt puzzles. So, I put him in touch with Ryan and you can see the results of that appearance from me. Link in the description for that. Ryan's technique
[12:24] does something more complicated than just taking the average of the two guesses of noise and subtracting that. And it involves something called score distillation loss, which I won't go into. Whereas Daniel really does take
[12:36] And actually, he has a really nice analogy for that. You've got a block of marble and you've got two sculptors. And And it's it's a good analogy because the way a sculptor works is they'll do a very rough um
[12:50] the vague shape. Yeah. Which is what's happening with away, chip away, chip away. So, you've got this block of marble and you say to a giraffe." And you get the And you get the other sculptor, you turn
[13:04] Okay. Make this a penguin. Yeah. And that's how you get the the the penguin giraffe illusion. And what's it's kind of a good illustration of how you can apply this idea more broadly.
[13:18] the underlying thing that you're trying to create in this case is a single image with these two different perspectives, right? But it doesn't have to be an image. It could be, for example, a 3D model. And you've got maybe five
[13:31] model from this angle, this angle, this angle, and this angle. And they all have to look like, for example, a robot dog. Okay, so you've got five sculptors all at it in different ways. And what you end up with is like a
[13:44] genuine a 3D file, not an image file. A 3D file that looks like a robot dog. And unbelievable how it works. It doesn't know what 3D is. It doesn't know And the model has not been trained on 3D data. It's been trained on image
[13:59] one thing looks like a robot dog Yeah. from lots of directions So, I mean, it makes sense that's what Something that Ryan's method does really well is transparencies. Like how cool
[14:13] are these? Some work in several orientations. You can even provide a transparencies that add up to that image. Like my channel logo, for example. You can play with all these different illusions on their website.
[14:26] played with different perspectives, too, where one perspective has all the colors where one perspective has all the colors inverted or where a cutout rotates.
[14:38] levels. Link to all their work in the description as well. And a huge thank this video. I couldn't have done it without them. One of the things I find most interesting about generative AI is how it shows the ways in which
[14:53] artificial intelligence is similar to human intelligence and the ways in which it is quite different. Like the way AI distills different layers of meaning from a passage of text seems very human, but the fact that it doesn't know how
[15:07] many R's there are in strawberries seems very unhuman. And maybe we shouldn't be training AI to mimic human intelligence because then we find ourselves having to quash all these human fallibilities like bias and hallucination. Artificial
[15:22] intelligence was trained to play Go by watching previous human matches. And when it eventually won against a human, it was mostly playing human moves. But
[15:34] the clinching move, interestingly enough, was one that no human had ever seen before. And when the AI was retrained without seeing any human games, just playing against itself, it learned to beat humans more quickly and
[15:49] more comprehensively with moves never seen before, which is both fascinating and terrifying. This video is sponsored by Jane Street. Jane Street? They're Okay. What?
[16:04] Oh, my What are the chances? It's the uh the the the internships. Jane Street is a quantitative trading firm looking for the next wave of curious and passionate people to join their internship programs in 2025. Jane
[16:18] learning, distributed systems, programmable hardware, and statistics to trade on markets around the world. And they do that from offices in some of the Applications are now open for internships in quantitative trading,
[16:31] and product, institutional sales and are in their New York, London, and Hong Kong offices. Over the summer, you'll dive into the work that's being done all around the firm. You'll be surrounded by
[16:44] backgrounds. I've met loads of people from Jane Street, and one thing that stands out is they're excited about what they know, and they're excited to share that knowledge with other people, which is exactly what will happen on their
[16:56] working, you can attend social events, guest speakers, you can explore the city care of you. You'll get a salary, of course, but they'll also provide flights to and from your internship program and accommodation throughout. People from
[17:10] are welcome to apply, and you do not need a background in finance. As a side you're applying for and you get accepted, you'll meet Matt Parker. I'm we'll hang out, it'll be emotional. I
[17:23] You were there. London office. Yeah. Yeah. They're good people. Good link is also in the description, so check out Jane Street today. Hope you forget to hit subscribe, and the
[17:37] next. If you've covered my face on the end screen, you're in so much trouble. But it's the thumbnail for your video there. So, it's not the algorithm, really. The algorithm thinks you'll enjoy this video.