Why LLMs Get Dumb (Context Windows Explained)

[00:00] Sometimes when you're talking to an LLM like chat, GBT, it gets kind of dumb, because it's so long and it starts to say weird things and hallucinate. It forgets what you're talking about. It makes stuff up and it's stinking slow.

[00:15] Did you know that L LLMs like chat, GBT, Gemini, Claude, even local models like llama or deep seek, they're kind of like us, you and me, which means they can remember things, that's awesome,

[00:30] Let's say you and me we're having some coffee and we're talking for about 15 I remember that story you told. You remember that dumb joke? I said,

[00:42] three hours, and at that point it's kind of hard to keep track of stuff. Thankfully you forgot that dumb thing I said. It kind of reminds me of how sometimes when I fight with my wife,

[00:57] As it happened to you, Chad GBT does the same thing as like you and your wife. It has to store all of that in its short-term memory.

[01:10] And that short-term memory has a limit. That limit is its context window. which is a great way to run local AI models on your computer. We'll do GEMMA three four B and we'll change our context link or our context

[01:24] window to 2048. Now what does that mean? how an AI counts the words you say to it. I'll show you. It's loading and we'll start a new chat.

[01:37] Now we might say this sentence is 133 characters or 26 So this sentence would actually be 38 tokens.

[01:49] If we copy and paste this and go to a website like Open AI's tokenize and paste Not every LLM will calculate tokens in the same way and notice it might do an

[02:03] It might do a space and a word or just one comma as a token. our GEMMA three four B has a context window of 2048 tokens, to at any one time. Let me show you, I'm going to make it forget something.

[02:20] Right now I'm reading a book called How to Take that should be in its memory and down here we can track our usage. 14.6% full.

[02:33] The rain in Oak Caven Valley wasn't a gentle drizzle, it was a. 44.4% Fool. Amazing story. Give me the sequel. Here's 82%. Wow, just wow. Now I'll do a prequel. Alright,

[02:52] we're now at 118.4%. If I click on that, it'll show me the tokens. 2,400 tokens. What book am I reading right now? What does it talking? Oh funny. It mentions memory loss. Oh, memory loss.

[03:08] Forget that extra token. No, I told you about a book I was reading. It forgot the first things I told it. Let's fix that. Let's increase its short-term memory. Man, I wish I could do this for myself.

[03:22] but this time I'll change its context to 4,096 and I'll load the model. Let's see now what book am I reading right now? Bam. It got it. And even apologize for being stupid even though it wasn't his fault.

[03:37] The obvious things are things we say and the things it says back, but there also might be system prompts which are instructions for our LLMs LLMs. It's kind of hard to say after a while these you might explicitly give to it or

[03:51] You might paste A PDF or an Excel spreadsheet that'll take up some more tokens. the code is ticking up tokens filling up that context window. Okay, there's our solution. Let's up the context Infinity. Let's do it right now.

[04:06] Load up our model. Well hold on. There are limits like right now, you'll see that written as 128,000. That's still pretty big. Let's do that. Setting this high value for context length can significantly impact your memory

[04:23] While GEMMA three four B has an astounding 128 K thousand token context window. It doesn't mean our GPU can support that full context specifically,

[04:35] it's going to be about VRA or video Ram. LM Studio is looking at my video car, which is a 40 90 with 24 gigs of V ram and it's like, buddy, And that's something that may not be obvious to everyone. Yes,

[04:49] but we might overlook that to have long conversations, we'll need lots of vra. computer? Watch this. It might break my recording. So sky's the stinking limit.

[05:02] 120,000 tokens. I'm going to bring up my, oh no, We'll load the model now and it's about to cry. Oh, look at it down here. Yeah, it just boom all the way up. It took out all my vra. Is it loaded yet? Oh man,

[05:17] my computer is just dying. Look at this. Oh no, you probably can't see it. but it's going to be hard to talk to before it was so snappy. And what you're seeing here is that bigger context windows require more compute

[05:32] power, more GPU resources and more ways than one. Look at this, It's probably not even showing you right now, but with cloud models, You can use their full advertised context windows and they got some big ones.

[05:44] GPT-4 oh is rocking 128,000 tokens and he thinks he's better than everybody else. Oh three, many Clyde 3.7, 200,000 and the Gemini 2.5 from Google, 1 million tokens. Tell us your whole life story. It's going to remember.

[05:57] What? Hey, number check from the future here. which has a 10 million token context window. Are you serious?

[06:09] Now back to the video. But there is a catch. it doesn't mean the LLM won't kind of freak out and forget stuff become less

[06:21] You'll notice on those larger conversations, AI paying attention. Hey, real quick, it's time for a coffee break. This coffee break,

[06:34] Now it's talking about context. What I like to do a lot is go to a website. like how does it compare to A VPN, how does it compare? It's awesome. I might want to take the content of this webpage and put it into an LLM and ask

[06:49] normally we'll just A control C and then paste it into our LLM. Boom. The format of all this information was not meant to be read by an LLM.

[07:02] It was meant to be read by humans and definitely not in this little chat box All we have to do is this. Whatever webpage you're on, type in R dot Gina with a J, do ai and then a slash watch this.

[07:20] Boom. It converts that entire webpage into markdown, that LLMs love. And as you're about to see LLMs do have a hard time paying attention. summarize this and I'm off to the races. So R Gina ai,

[07:37] And speaking of tools you should use all the time. and it's what I use to stay connected to my home lab, my studio, I used it when I was in Japan. I used it on my road trip. It's fast,

[07:53] If you're still using a VPN, you're crazy. And setting up twin Gate is easy. I've shown that a few times here on my channel. Takes about five minutes.

[08:05] There's too many features you're giving us, don't believe me. Check it out. There was a paper released called Lost in the Middle, And just like my wife, once she watches a long movie,

[08:21] And that's the context she has. I'll ask you to explain it to me in the same way, conversations with LLMs, the models were more accurate with info at the beginning and even with info at

[08:33] And across the board we saw this U shape. So what does that tell us? LLMs have problems paying attention just like us. But that brings up a question.

[08:47] How do LLMs pay attention? What does that mean for a computer to do that? It's actually incredibly fascinating. It's called attention mechanisms, specifically with modern LLMs. It's referred to as a self attention mechanism.

[08:59] But that would increase the context window of this video, And given this is probably the middle of the video, you would forget it. There will be another video somewhere around here soon, TLDR.

[09:15] but caffeine makes me jittery. What should I get? It'll use some fancy semantic math to decide which of these words is important,

[09:27] which is relevant both to the context of your entire conversation and to how the and this is very simplified. They essentially assign attention scores saying, Hey, in this conversation, coffee is high, caffeine is high, jittery,

[09:41] but words like I or me, kind of low relevance to the context. You said something about coffee, jittery, caffeine, and we'll use that context to process our responses. AI does the same thing.

[09:54] All this math to assign attention scores, and it does this every time you send something to the LLM every time you add to GBT can keep track of it easily.

[10:08] you know what I'm saying? You can't even find it to scroll up. Those larger contexts not only have insane V RAM requirements, Every time you add to that conversation,

[10:23] it requires more GPU power. And that is why in those larger conversations, It's using a ton of memory and it's having to do some crazy math every time you

[10:37] think about this if you're saying a lot of different things, I know some people who just keep one conversation open with chat GBT and just shoot the breeze. They'll talk about coffee, the weather,

[10:49] it's trying to weigh all of these different words and how relevant they are to the entire context of the conversation. And that's crazy. and this is a rule I try to go by when I'm talking with LLMs.

[11:06] when it's a significant shift from what you're currently talking about, sometimes when you're talking with other LLMs like Claude, Things are going to slow down.

[11:20] But I'm hoping sometimes soon this won't be a problem anymore. I want it to be able to pay attention to everything I'm saying. down. You've got the ability to have all that stuff with you.

[11:37] But the ability to quickly find what you need and that notebook might be kind of if you want to learn more about self attention mechanisms, I want to get back to the idea of context windows because that is the context of

[11:51] local AI models and they're full context. The first one is called flash attention. Like hey, paying attention,

[12:03] It's actually an option right here when I want to load a model. and right here under experimental features we have flash attention. This will change how the model will compute its attention and assign those

[12:19] Now it's doing the same kind of crazy semantic map that regular self attention It'll actually skip building the full table of token comparisons by processing tokens and chunks with optimized GPU routines. Essentially,

[12:34] time leading to significant improvements in not only memory but speed. So let's try it out real quick. I'm going to go full 131 to the limit, pull up my poor GPU and we'll see what happens.

[12:50] but we'll see how the speed is GPU spiking. but there's actually two more things we can do. I'm going to stop it now. Eject,

[13:03] choose GEMMA three, four B, I'll select flash attention. These two options will actually compress our data so it takes up less room in

[13:16] And we'll change the amount and the lower the quantization the better. So we'll go to quant four on both of these and we'll go to full 1 31. It did not even max out our memory, which is amazing.

[13:31] Let's try and talk to it significantly faster. Keeping in mind, I'm recording right now using my GPU, I mean that's stinking great. Now, in addition to flashed attention and the compression of our data,

[13:46] Paged cache will actually move attention cache between your GPU, So essentially it's sharing the RAM with your system. right there on your GPU. And yes, it is like your page file on your system.

[14:03] your computer might be configured to use a page file, AKA, your hard drive or your SSD, which will be significantly slower than your ram. So let's bring this video home. I'm a massive fan of larger context windows.

[14:17] I want to give an LLM everything I could possibly give it about me or a problem that's way better. But there's a couple of massive problems. GPU memory, your v ramm memory you may not have,

[14:32] The longer it gets resulting in slower conversations. the scariest downside is the larger attack surface. Yes, LLMs can be hacked and they're vulnerable to some creative prompting that can

[14:47] gel break out of their protection systems. The longer a conversation is, the more it can kind of forget what's in the middle as we saw in that little U And the easier it will be for an attacker to hide some malicious stuff in there

[15:00] our context window is way too big. We're going to end it right now. I'll catch you guys in the next one. Chat, GPI can't talk right now, which is bad for a guy making a video. Okay,

[15:15] don't put that. That's stupid.