[00:05]
Hi everyone. Thank you for joining us today for
[00:10]
of the applied AI team here at Anthropic. And
[00:15]
AI team. And what we're going to do today is
[00:19]
practices. And we're going to use a real world
[00:25]
a little bit about what prompt engineering is. uh
[00:30]
bit familiar with this. This is the way that we
[00:34]
it to do what we want. So, this is the practice of
[00:39]
the model the context that it needs to complete
[00:44]
arrange that information in order to get the
[00:49]
a lot of different ways you might want to think
[00:53]
the best way to learn this is just to practice
[00:58]
a hands-on scenario. Uh, we're going to use an
[01:03]
with. So, we've modified what the actual customer
[01:08]
case of trying to analyze some images and get uh
[01:14]
Claude make a judgment about what content it finds
[01:20]
that this content is in, but luckily Christian and
[01:25]
to Christian to talk about the scenario and the
[01:30]
it's uh intended so so to set the stage, imagine
[01:35]
and you deal with uh car insurance claims on a
[01:41]
you have two pieces of information. Um we're going
[01:45]
see on the left hand side we have a car accident
[01:52]
before the action accident actually took place.
[01:57]
um sketch of how the accident took place as well.
[02:03]
going to try to pass on to cloud. And to begin
[02:08]
into a console and just see what what happens.
[02:13]
we can actually do this in a real manner. And
[02:17]
shiny beautiful entropic console. We're using the
[02:24]
setting temperature zero and having a a huge max
[02:30]
that there's no limitations to what CL can do. In
[02:34]
just setting the stage of what Cloud's supposed
[02:38]
um intend to review a an accident report form uh
[02:45]
in an accident and who's at fault. So you can
[02:49]
run this let me go to preview. Uh we can see here
[02:58]
skiing accident that happened on a street called
[03:05]
Um and in many ways you can sort of understand
[03:10]
prompt we actually haven't done anything to set
[03:15]
So this sort of first guess is not too bad but
[03:20]
bake into cloud. So if we switch back to the
[03:27]
prompt engineering is a very iterative empirical
[03:32]
have a test case where Claude is supposed to make
[03:38]
environment, nothing to do with skiing. Uh and in
[03:43]
to make sure it's actually tackling the problem
[03:48]
we'll go through some best practices of how we we
[03:54]
recommend others to do so as well. So, we're going
[03:59]
a great prompt. Uh, first we want to talk a
[04:04]
looks like. So you might be familiar with kind of
[04:09]
and forth having a more kind of conversational
[04:14]
like this, we're probably using the API and we
[04:19]
and have it nail the task the first time around
[04:24]
Uh, so the kind of structure that we recommend is
[04:30]
Claude, "What are you here to do? What's your
[04:34]
today?" Then we provide content. So in this
[04:38]
the form and the drawing of the accident and how
[04:43]
might also be something you're retrieving from
[04:47]
is. We're going to give some detailed instructions
[04:52]
how we want Claude to go through the task and how
[04:58]
some examples to Claude. Here's an example of some
[05:03]
should respond when given that content. And at
[05:08]
that's really important for Claude to understand
[05:12]
information with Claude, emphasizing things that
[05:17]
go ahead and do your work." So, here's another
[05:23]
bit more of a breakdown, and we're going to walk
[05:27]
show you how we build this up, um, in the console.
[05:33]
going to talk about the task context and the tone
[05:38]
task context, as you realized when I went through
[05:43]
elaborating what what scenario Chlo was actually
[05:48]
tell that claw doesn't necessarily need to guess a
[05:53]
our case, we really want to break that down, make
[05:57]
and also make sure we understand what's the
[06:02]
as well, we also make sure we add a little bit of
[06:08]
Claw to stay factual and to stay confident. So if
[06:14]
we don't want to guess and just sort of mislead
[06:19]
and in our case we want to make sure that we can
[06:23]
sure that assessment is as clear and as confident
[06:27]
what we're doing. So if we transition back to the
[06:34]
here. So I'll just navigate to V2. And you can see
[06:41]
we didn't really do that last time around just
[06:45]
what we're seeing here, this is the car accident
[06:51]
going through what actually happened. You
[06:55]
both on the left and right hand side. And the main
[06:58]
Claude can understand this manually generated data
[07:05]
uh corroborated by if I navigate back here to this
[07:11]
case, the form is just a different um data point
[07:18]
want to bake in more information into our version
[07:24]
a lot more on what's going on. So, you can see
[07:29]
supposed to help a human's claim claims adjuster
[07:34]
Swedish as well. Um, you can see here we're also
[07:39]
the incident and that you should not um make an
[07:45]
And that's really key because if we run this,
[07:49]
settings as well. Clo my new shiny model zero
[07:54]
here what actually happens in this case. Um, CL is
[08:02]
accidents, not skiing accidents, which is great.
[08:08]
marked on on checkbox one and then vehicle B was
[08:13]
still tell that there's some information missing
[08:19]
of who's at fault here. And this is great. This
[08:24]
make anything any claims that aren't um uh factual
[08:30]
when you're when you're confident. But there's
[08:33]
um regarding the form uh what the form actually
[08:39]
we want to want to bake into this LM application
[08:44]
adding it to the system prompt which Hannah will
[08:50]
the next item we're going to add to the prompt
[08:56]
and images and here as Christian was saying we
[09:01]
going to be the same every single time. The form
[09:06]
type of information to provide to Claude to tell
[09:10]
be looking at. We know that will not ever alter
[09:15]
filled out will change, but the form itself is not
[09:20]
um information to put into the system prompt. Also
[09:25]
considering using prompt caching. This will always
[09:29]
spend less time trying to figure out what the form
[09:35]
it's going to do a better job of reading the form
[09:41]
So another thing I want to touch on here is how we
[09:47]
really loves structure, loves organization.
[09:51]
standard structure in your prompts. And there's
[09:56]
understand the information better. I also just
[10:01]
lot of really great examples. So definitely take
[10:05]
don't worry. All of this content is online with
[10:10]
guys to check it out there too. Um anyway the uh
[10:17]
tags also markdown is pretty useful to Claude
[10:23]
specify what's inside those tags. So we can tell
[10:29]
going to read some content and these XML tags are
[10:33]
tags is related to the user's preferences and
[10:38]
maybe at later points in the prompt. Um, so I
[10:44]
actually do this in this case. And Christian's
[10:49]
keeping everything about the other part of the
[10:54]
case to put this information in the system prompt.
[10:59]
it in the system prompt here. And we're going
[11:03]
this form. So this is a Swedish car accident
[11:07]
this title. It'll have two columns. The columns
[11:13]
about each of the 17 rows and what they mean.
[11:18]
Claude was reading individually each of the lines
[11:23]
that information up front. And we're also going
[11:27]
how this form should be filled out. This is also
[11:32]
like, you know, humans are filling this form
[11:37]
People might put a circle. They might scribble.
[11:42]
be many types of markings that you need to look
[11:47]
give Claude a little bit of information about how
[11:51]
of this form is. And all of this is context
[11:56]
um do a better job analyzing the form. So if
[12:02]
So we've kept the same user prompt down here.
[12:08]
the we have the same user prompt here. Still
[12:14]
And we'll see here that it's spending less time.
[12:19]
about what the form is because it already knows
[12:23]
of bringing us that information back. It's going
[12:28]
checked, what the sketch shows. And here Claude
[12:33]
additional context that we gave to Claude. Claude
[12:39]
at fault in this case based on this drawing and
[12:44]
improvement in the way Claude is analyzing these.
[12:49]
at the drawing and at the list that vehicle
[12:55]
Uh so we're going to go back to the slides and
[12:59]
really using in this prompt um but can be really
[13:06]
and making it work better. Exactly. I think um
[13:11]
I think examples or few shot is a mechanism that
[13:18]
imagine this um in in quite a non-trivial way as
[13:25]
even in this case concrete accidents that have
[13:31]
But you with your human intuition and your human
[13:37]
conclusion. Then you can bake that information
[13:42]
examples of a the data that that it's supposed
[13:47]
you can just base 64 encode a a an image and have
[13:53]
into the examples and then on top of that you can
[13:58]
of how to break that down and understand it. This
[14:02]
in how you can sort of push the limits of your
[14:08]
into system prompt. And this again is sort of the
[14:12]
sort of always want to push the limits of your
[14:17]
it's going wrong and try to add that into system
[14:21]
of mimics that u takes place it's able to actually
[14:27]
as well, this is just a little example of how we
[14:34]
structure that we we um we enjoy. It's it gives a
[14:38]
fine-tuned on as well. Um and it works perfectly
[14:43]
doing this just because it's a simple demo,
[14:47]
building this for an insurance company, you'd have
[14:52]
difficult, maybe in the gray, that you'd like to
[14:57]
to make the verdict next time. Um, another topic
[15:02]
in this demo, is conversation history. It's in the
[15:08]
that the enough context rich information is at
[15:15]
your behalf. Um in our case now this isn't really
[15:21]
happening in the background. You can imagine for
[15:25]
system some data is generated out of this and then
[15:29]
end. If you were have to build something much more
[15:34]
history that would be um relevant to bring in this
[15:40]
because it enriches the context that Claude works
[15:47]
we do is and the next step is try to make sure
[15:55]
So, now we're going to build out the
[15:59]
and that's coming back to the reminder of what
[16:03]
about any important guidelines that we want it
[16:09]
a preventing hallucinations. Um, so we want Claude
[16:16]
this prompt, right? Or not finding in the data.
[16:21]
we don't want Claude to take its best guess or
[16:26]
it's not. If the sketch is unintelligible, the
[16:32]
and even a human would not be able to figure it
[16:36]
so these are some of the things we'll include in
[16:41]
Claude. Uh remind it to do things like answer only
[16:46]
refer back to what it has seen in the form anytime
[16:52]
vehicle B turned right, it should say I know this
[16:57]
or whatever it might be. We can kind of give
[17:01]
back to the console, we can see the next version
[17:09]
going to keep everything the same here in the
[17:13]
background context that we gave to Claude about
[17:17]
out. We're not changing anything else about the
[17:22]
detailed list of tasks. And this is how we want
[17:27]
key thing that we found here as we were building
[17:31]
example is that the order in which Claude analyzes
[17:36]
analogous to way you might think about doing this.
[17:41]
at the drawing first and try to understand what
[17:45]
a bunch of boxes and lines. We don't really know
[17:50]
additional context. But if we have the form and we
[17:55]
talking about a car accident and that we're seeing
[17:59]
doing at certain times, then we know a little
[18:04]
in the drawing. And so that's the kind of detail
[18:08]
"Hey, first go look at the form. Look at it very
[18:13]
checked. Make sure you're not missing anything
[18:19]
in that. And then move on to the sketch. So after
[18:24]
out of the form and you can say what's factually
[18:30]
can gain from that sketch. keeping in mind your
[18:36]
you've learned from the form, you're trying to
[18:40]
you're going to arrive um at your final uh at your
[18:52]
And here you can see one behavior that
[18:56]
it to very carefully examine the form. It's
[19:01]
it's telling me each individual box. Is the box
[19:06]
thing you'll notice as you do prompt engineering.
[19:11]
claw decide how much it wanted to tell us about
[19:16]
it carefully examine each and every box, it's very
[19:21]
might not be what we want in the end. So, that's
[19:26]
to give me these other things that I asked for
[19:32]
accident summary so far. It's going to give me a
[19:37]
that vehicle B appears to be clearly at fault. In
[19:42]
with more complicated drawings, more uh less
[19:48]
thinking for Claude is really impactful in
[19:54]
Uh, so I think we'll go back to the slides and
[19:59]
piece that we might add to this um to really make
[20:05]
you so much. So, as Hannah mentioned, uh, we sort
[20:11]
really acting on our behalf in a right manner.
[20:16]
end of this prompt that I'm going to show you in a
[20:21]
part as well. just strengthening and reinforcing
[20:26]
important piece is actually output formatting.
[20:30]
on this LM application, all the sort of fancy
[20:35]
you want your piece of information to to
[20:39]
wherever you want to store that data. And the rest
[20:44]
its verdict isn't really that necessary for your
[20:49]
for your application. So if we transition back to
[20:54]
a simple importance guidelines part. And again,
[21:01]
behavior that we want out of cloud here. Want
[21:06]
and accurate. Want to make sure that nothing
[21:11]
apart from the data it's analyzing. And then
[21:15]
in my case here, I'm just going to ask Claude
[21:20]
actually going to ignore for my application and
[21:24]
that is I can I can use this if I want to build
[21:29]
Or if I just want a clearcut um uh determination,
[21:35]
here, you'll see it's going through the same sort
[21:39]
it's much more succinct because we've asked
[21:43]
more straightforward manner. And then finally
[21:48]
output in these final verdict XML tags. So you
[21:54]
a skiing accident to sort of unconfident insecure
[22:01]
version to now a much more strictly formatted
[22:08]
application around and actually help you know a
[22:15]
U finally if we transition back to the um slides
[22:24]
putting words in CL's mouth or as we call it
[22:30]
parsing XML tags is nice and all but maybe you
[22:36]
uh it's JSON serializable and you can use this
[22:42]
is quite simple to do. You could just add that um
[22:48]
format. This could be for example a uh open
[22:53]
or even in this case that we see in front of us
[22:58]
case it could also be that final verdict XML tag.
[23:04]
how Claude is supposed to respond. Um, without all
[23:09]
that is also key in shaping his output to make
[23:14]
that we wanted. So in our case here, we would just
[23:18]
afterwards. But you can use prefill as well. Now
[23:24]
here as well is that both cloud 3.7 and especially
[23:31]
model meaning that there's extended thinking at
[23:36]
to highlight because you can use extended thinking
[23:41]
you can enable this to make sure that Claude
[23:45]
tags and the scratch pad. Um and the beauty of
[23:50]
to understand how claude is going about that data.
[23:55]
it goes through step by step of the scenario
[24:00]
ways there you can actually try to help claude in
[24:04]
not only more token efficient but it's a good way
[24:10]
don't have our intuition actually go about the
[24:14]
it's quite key in actually trying to break down
[24:19]
and with that said, I think uh I'd like to thank
[24:24]
So if you have any questions on prompting, please
[24:29]
You want to learn more about prompting in an hour.
[24:34]
have an amazing demo of Claude plays Pokemon. So
[24:39]
we'll be around all day. So, I know we
[24:42]
but uh please come find us if you want to chat.