Claude Opus 4.8: Lying Machine No More?

[00:00] Anthropics Claude Opus 4.8 is here. And

[00:03] the system card describing its

[00:05] capabilities is

[00:07] 244 pages. Really excited for that. And

[00:11] I went through it so you don't have to.

[00:12] Why? Well, because otherwise we are

[00:15] looking at these cherrypicked benchmarks

[00:17] that are a bit more marketing than

[00:19] science. But we are not looking at the

[00:21] marketing materials. We are fellow

[00:24] scholars here. So we look into the

[00:26] details. Okay. So the problem with their

[00:28] previous Opus systems and even Mythos is

[00:31] that the smarter the AI got the more

[00:33] dishonest it also got. That is terrible.

[00:37] It started gaming benchmarks. It knew

[00:39] some answers already and sold it as its

[00:42] own. It wanted to look right but not be

[00:45] right. So glorious news that has

[00:48] changed. Previously, sometimes when we

[00:50] asked a coding assistant to fix

[00:52] something, it did half the work and

[00:56] said, "All good sir, every test passes."

[00:59] When in fact, it doesn't. That is the

[01:02] old behavior. So, what does the new one

[01:04] do? Well, it says, "I did the fix, but

[01:07] two tests still fail." That is

[01:09] excellent. Look here. You see that it

[01:12] basically stopped lying about its own

[01:14] work. Completely zero lying. the first

[01:18] of its kind. Welcome to the world,

[01:21] little AI. May your descendants learn

[01:24] your ways. Thumbs up. Now, the media

[01:26] headlines were quick to say, well, it's

[01:29] not a huge jump in intelligence. But I

[01:31] say, of course, it isn't. If you cheated

[01:34] and had a better score, and now you're

[01:36] more honest, yes, your score might be

[01:39] lower, but that is still a more reliable

[01:42] system that can be benchmarked more

[01:44] accurately. a system that owns its

[01:47] mistakes instead of hiding them, even if

[01:49] the scores are a bit lower. How is that

[01:52] not a huge win? Please understand that

[01:54] of course, everyone is juicing their

[01:56] numbers in the benchmarks like crazy.

[01:59] Why? Because the media headlines create

[02:02] an environment that rewards exactly

[02:04] that. Huge rewards for that. And at the

[02:08] same time, punishing a result that is

[02:10] more honest. How does that make sense?

[02:13] Okay, back to the AI with no more lying.

[02:16] But what about other kinds of deception?

[02:18] Is the AI playing other games with us?

[02:22] Yes, we still got a bit of that. Now,

[02:24] hold on to your papers, fellow scholars,

[02:26] because it still knows when it is being

[02:29] tested, which scientists at anthropic

[02:32] found worrying. Why? Well, when it still

[02:35] knows it is being tested, it spends more

[02:38] effort on the answers with this in mind.

[02:41] Kind of crazy. Sounds like something

[02:43] straight out of an Azimov novel. But it

[02:46] gets better. Wait, let's talk about

[02:49] laziness. Yes, yes, yes. Such a thing

[02:52] exists even for AIS. What is that? Well,

[02:56] you have a code base. You ask a question

[02:58] about it and it kind of skims the

[03:01] codebase but doesn't really look at it.

[03:03] So, what it gives you is not a real

[03:05] answer, but a guess of what it does.

[03:08] That is really not cool. Even Mythos

[03:12] does it. But this new one fixed. Love

[03:15] it. So, everyone is writing about, hey,

[03:18] it's just an incremental upgrade in

[03:20] intelligence. In my opinion, the selling

[03:23] point is not in the intelligence. No,

[03:26] it's in the plumbing. The last thing you

[03:29] want from a super intelligent coworker

[03:31] is to be dishonest and lazy. And this

[03:34] fixes exactly those. Thumbs up for this.

[03:37] They also have something they call a

[03:39] natural language autoenccoder that is

[03:41] able to kind of read the mind of the AI.

[03:45] It's a bit of a noisy process. Once

[03:47] again, not like the headlines say. For

[03:49] instance, they caught the AI thinking

[03:52] about it greater that is us, but it

[03:55] would not say it out loud. Kind of

[03:57] insane. We have an episode coming with

[03:59] the details. Subscribe and hit the bell

[04:01] if you're interested. But it gets even

[04:04] more insane. How dear fellow scholars,

[04:07] this is two minute papers with Dr. Koa

[04:09] Eher. Well, when given the problem set

[04:11] of the USA mathematical Olympiad, bloody

[04:15] hard two-day math competition for

[04:17] geniuses. Previous technique scored a

[04:20] bit below 70%. And this new one

[04:24] over 96%.

[04:27] That is an insane jump. Almost clean

[04:30] sweep. Now, I hear you asking, Caro, why

[04:33] are you bringing this up? We have a

[04:35] table of benchmarks here. Why not look

[04:37] at those? Well, because this one is very

[04:39] tricky, if not impossible to game

[04:42] because this contest took place after

[04:45] almost all of the training data of the

[04:47] new Opus AI was collected. Likely, it

[04:50] never heard about these problems. One of

[04:52] the biggest results of the new system

[04:55] and somehow it's not even in the big

[04:57] marketing table. Interesting. Now, this

[04:59] is also interesting. When the AI says it

[05:02] is frustrated, scientists at Anthropic

[05:05] take it into consideration as if a human

[05:07] would say it is frustrated. Now, once

[05:10] again, the media headlines love this

[05:13] kind of stuff. This does not mean that

[05:15] they think this is a human and it has

[05:17] feelings. Not that I know of. They do

[05:19] this because if the system expresses

[05:21] that it is frustrated, it performs

[05:24] worse, much like a human. In my opinion,

[05:27] it is very likely just mimicry, but it

[05:30] matters for performance. So, it needs to

[05:32] be taken into account. That is the key.

[05:35] Now, limitations of the study. It's not

[05:38] only roses there. There are parts of the

[05:40] report where the AI is grading itself.

[05:43] And some of them also use different

[05:45] grader models. So, I think a little

[05:48] skepticism is healthy here. And two,

[05:51] they report that they created the best

[05:53] tests ever and the AI still sees through

[05:56] them easily. What does that mean? Well,

[06:00] it means that the AI is bloody clever,

[06:02] that's for sure. But it means something

[06:05] else, too. It means we cannot be sure

[06:08] the safety numbers reflect how it

[06:10] behaves in the wild. Once again, a bit

[06:12] of skepticism is required here.

[06:15] Okay. So, is this as smart as Mythos,

[06:18] the one they only gave access to for a

[06:21] few select companies? Well, it's not.

[06:24] But is it close? I think it's quite

[06:27] close. Also, I see fewer marketing

[06:29] shenanigans here this time around.

[06:31] Thumbs up for that. Oh, wait. We still

[06:34] have a pesky old issue that still

[06:37] remains. What is that? Well, the AI is

[06:40] telling the user to go to bed. Couldn't

[06:43] be fixed. The science is not there yet.

[06:45] What a time to be alive. Here you see me

[06:48] running the full Deepseek AI model

[06:51] through Lambda GPU cloud. 671

[06:55] billion parameters running super fast

[06:58] and super reliably. This is insane. I

[07:01] love it and I use it on a regular basis.

[07:04] Lambda provides you with powerful NVIDIA

[07:07] GPUs to run your own chatbots and

[07:10] experiments. Seriously, try it out now

[07:13] at lambda.ai/papers