[00:00]
hello uh welcome to vast in this video I
[00:03]
want to show you how you can run some of
[00:07]
the best large language models that
[00:10]
exist from hugging face or other places
[00:12]
on a very powerful gpus and so
[00:18]
let me get started today we'll be using
[00:21]
uba Booga to as the web UI which is a
[00:25]
great interface for prompting the models
[00:28]
and also kind of loading and managing
[00:30]
them that's some open source software
[00:32]
that will will load up in an instance on
[00:34]
vast
[00:36]
so I'll first kind of Click into the
[00:38]
console and make sure that you are
[00:41]
logged into your account and that you
[00:43]
have credits if you've never done this
[00:46]
before with fast we have a different
[00:47]
video that can go over a lot of some of
[00:49]
the basics
[00:50]
but for uba booga
[00:54]
you're going to come in here and select
[00:57]
our recommended template for that
[01:00]
and uh that's gonna have the description
[01:03]
here it kind of shows you which language
[01:06]
models that you can run with it and uh
[01:09]
it's going to have some specific options
[01:11]
and an on-site script that you don't
[01:13]
want to mess with that's going to set up
[01:15]
your environment correctly so this will
[01:18]
work it's also going to open a port for
[01:22]
the open button and that will launch the
[01:25]
gradio web interface so that it all just
[01:28]
works so really all you need to do is
[01:30]
Select that uh template
[01:34]
now one of the most important things
[01:38]
um
[01:39]
is to make sure you allocate enough
[01:41]
Discord storage I just reset my filters
[01:44]
because the default is only 16 gigabytes
[01:46]
which is not going to be enough
[01:49]
a lot of these large language models are
[01:51]
60 70 gigabytes to download and your
[01:54]
instance will start to throw errors if
[01:57]
it runs out of disk space you also need
[01:59]
to allocate all the disk space that you
[02:01]
want to use up front for this instance
[02:03]
you cannot add it later
[02:05]
so with that in mind you're probably
[02:07]
going to want if you're just going to
[02:08]
try one language model at least about 80
[02:13]
gigabytes so I'm just going to move the
[02:15]
slider to 81 and get that all set up the
[02:20]
other important thing to understand when
[02:22]
you're running these large language
[02:23]
models is to match the GPU with the
[02:27]
model that you want to run for example
[02:30]
if you're looking at hugging face
[02:34]
hugging faces has a actual llm
[02:37]
leaderboard and so you can see some of
[02:40]
the most popular models here and
[02:45]
um
[02:46]
how you can run them
[02:48]
and basically what you will need to do
[02:51]
is to load these into uba Booga once we
[02:54]
have that running so we'll come back to
[02:56]
this
[02:57]
but know that the model that you each
[03:00]
one of these models that you're trying
[03:02]
to run for example if you want to run
[03:05]
Falcon 40 billion you need to read
[03:09]
through and understand how much GPU Ram
[03:12]
this is going to require because if this
[03:15]
requires say
[03:17]
60 gigabytes of GPU RAM and you select a
[03:20]
GPU that only has 10 gigabytes of GPU
[03:24]
Ram it is not going to work so you need
[03:26]
to make sure that the the large language
[03:28]
model that you want to run it's going to
[03:31]
have
[03:32]
um uh you need to figure out what exact
[03:35]
specifications it needs
[03:37]
and then select an appropriate GPU
[03:42]
what I like to do is to just actually
[03:44]
select the GPU that has the most GPU Ram
[03:47]
which is one of our a100 uh
[03:51]
smx4s or pcies so I'm going to go ahead
[03:54]
and select a 1x smx4 these have 80
[03:58]
gigabytes of GPU Ram so they have uh one
[04:03]
of the more powerful cards that are out
[04:05]
right now from Nvidia and 80 gigabytes
[04:09]
is is enough for most large language
[04:11]
models you can also select a multi-gpu
[04:14]
instance so if I needed even more space
[04:17]
I could have a 2X a100 that would be I
[04:20]
have 160 gigabytes of GPU RAM available
[04:23]
for the large language models or I could
[04:26]
select sort of a cheaper option like an
[04:29]
a5000 and this 4X a 5
[04:34]
000.
[04:36]
has actually 96 gigabytes of GPU RAM and
[04:41]
it is a little bit cheaper than a single
[04:44]
a100 you can also look at an a6000 they
[04:47]
have 48 gigabytes of GPU Ram
[04:51]
and an A40
[04:53]
has 45 gigabytes of GPU Ram
[04:57]
the consumer graphics cards like the
[04:59]
4090 and 3090 are only going to have 24
[05:02]
gigabytes of GPU Ram each so again this
[05:06]
is just something that you want to be
[05:08]
really aware of and make sure that
[05:10]
you're selecting a GPU that's going to
[05:11]
have enough space so I'm going to go
[05:13]
ahead and select a 1X a100 and now this
[05:18]
is going to load I have 80 gigabytes
[05:21]
allocated on this instance and I have
[05:25]
selected a Ooba Booga web UI which is
[05:31]
our recommended template and so if I
[05:34]
jump into my instances here I can see
[05:36]
that this is being created and set up
[05:37]
for me
[05:40]
it's going to take a three four or five
[05:43]
minutes to load maybe a little bit
[05:45]
longer it's really going to depend on
[05:46]
the internet connection speed of the
[05:48]
machine and
[05:50]
um
[05:51]
uh the size of the image this one loaded
[05:54]
about three and a half minutes for me
[05:56]
and now the open button is going to open
[06:00]
port 7860 which was put in the
[06:04]
environment variables when we set this
[06:06]
up
[06:07]
um and uh
[06:09]
there's a few things that were installed
[06:11]
and set up on the onstart script but
[06:13]
anyways this is all just stuff that's in
[06:16]
the template that we have set up for
[06:18]
Ooba booga
[06:20]
and I'm going to go ahead and open that
[06:22]
interface up and here it is
[06:26]
so uh here's where you can actually
[06:27]
query the model that you set up the most
[06:30]
important thing is that going to be
[06:32]
downloading and setting up the model so
[06:35]
um this software is not
[06:38]
developed or maintained by vast this is
[06:43]
open source software so to understand
[06:45]
how to use this software you're going to
[06:48]
want to find the open source project for
[06:53]
this
[06:54]
and and load that
[07:02]
so here's the
[07:05]
GitHub that's going to have a readme
[07:08]
um
[07:09]
the of course the installation steps you
[07:12]
don't have to worry about because
[07:13]
um you're using a Docker image and
[07:15]
everything is is
[07:18]
pre-loaded
[07:21]
um
[07:22]
so you can place the models into the
[07:24]
model folder or when you're using the
[07:27]
web UI you can just simply go to the
[07:29]
models tab where it was before and
[07:32]
here's where you can download the custom
[07:34]
model so for hugging face you just use
[07:38]
the username and model so for example if
[07:43]
I wanted to try to to use just looking
[07:45]
at the leaderboard if I wanted to use
[07:47]
this model I would simply select the
[07:51]
username and the name of the model like
[07:53]
that and copy and paste it into the web
[07:55]
UI
[07:57]
and hit download and now it is going to
[08:00]
start downloading
[08:02]
this model once that model is downloaded
[08:04]
I will be able to load the model in here
[08:06]
into the GPU
[08:09]
Ram in the instance
[08:11]
sort of memory so then I can query it I
[08:14]
can go back to text generation and
[08:16]
actually start using it there's probably
[08:18]
some other things that you can do and
[08:22]
become familiar with with this interface
[08:23]
this is a very nice way to run llms so
[08:27]
that you don't have to use a command
[08:28]
line
[08:32]
um so there's quite a bit here and but
[08:34]
again you're going to want to read about
[08:36]
this and the ooga booga
[08:40]
GitHub
[08:44]
so if I go back and look at my instance
[08:48]
um
[08:49]
you can see that it's running
[08:51]
you can also click on the billing tab if
[08:54]
you just want to see you know if you're
[08:55]
going to run something for multiple days
[08:57]
you can get an idea of how many credits
[08:59]
you're going to need you can set up your
[09:01]
auto billing threshold so that your
[09:05]
instance is not stopped when your
[09:08]
balance gets low
[09:10]
and that's the basics of running
[09:13]
ubeoka on vast thanks for your time