[0:00] hello uh welcome to vast in this video I
[0:03] want to show you how you can run some of
[0:07] the best large language models that
[0:10] exist from hugging face or other places
[0:12] on a very powerful gpus and so
[0:18] let me get started today we'll be using
[0:21] uba Booga to as the web UI which is a
[0:25] great interface for prompting the models
[0:28] and also kind of loading and managing
[0:30] them that's some open source software
[0:32] that will will load up in an instance on
[0:34] vast
[0:36] so I'll first kind of Click into the
[0:38] console and make sure that you are
[0:41] logged into your account and that you
[0:43] have credits if you've never done this
[0:46] before with fast we have a different
[0:47] video that can go over a lot of some of
[0:49] the basics
[0:50] but for uba booga
[0:54] you're going to come in here and select
[0:57] our recommended template for that
[1:00] and uh that's gonna have the description
[1:03] here it kind of shows you which language
[1:06] models that you can run with it and uh
[1:09] it's going to have some specific options
[1:11] and an on-site script that you don't
[1:13] want to mess with that's going to set up
[1:15] your environment correctly so this will
[1:18] work it's also going to open a port for
[1:22] the open button and that will launch the
[1:25] gradio web interface so that it all just
[1:28] works so really all you need to do is
[1:30] Select that uh template
[1:34] now one of the most important things
[1:38] um
[1:39] is to make sure you allocate enough
[1:41] Discord storage I just reset my filters
[1:44] because the default is only 16 gigabytes
[1:46] which is not going to be enough
[1:49] a lot of these large language models are
[1:51] 60 70 gigabytes to download and your
[1:54] instance will start to throw errors if
[1:57] it runs out of disk space you also need
[1:59] to allocate all the disk space that you
[2:01] want to use up front for this instance
[2:03] you cannot add it later
[2:05] so with that in mind you're probably
[2:07] going to want if you're just going to
[2:08] try one language model at least about 80
[2:13] gigabytes so I'm just going to move the
[2:15] slider to 81 and get that all set up the
[2:20] other important thing to understand when
[2:22] you're running these large language
[2:23] models is to match the GPU with the
[2:27] model that you want to run for example
[2:30] if you're looking at hugging face
[2:34] hugging faces has a actual llm
[2:37] leaderboard and so you can see some of
[2:40] the most popular models here and
[2:45] um
[2:46] how you can run them
[2:48] and basically what you will need to do
[2:51] is to load these into uba Booga once we
[2:54] have that running so we'll come back to
[2:56] this
[2:57] but know that the model that you each
[3:00] one of these models that you're trying
[3:02] to run for example if you want to run
[3:05] Falcon 40 billion you need to read
[3:09] through and understand how much GPU Ram
[3:12] this is going to require because if this
[3:15] requires say
[3:17] 60 gigabytes of GPU RAM and you select a
[3:20] GPU that only has 10 gigabytes of GPU
[3:24] Ram it is not going to work so you need
[3:26] to make sure that the the large language
[3:28] model that you want to run it's going to
[3:31] have
[3:32] um uh you need to figure out what exact
[3:35] specifications it needs
[3:37] and then select an appropriate GPU
[3:42] what I like to do is to just actually
[3:44] select the GPU that has the most GPU Ram
[3:47] which is one of our a100 uh
[3:51] smx4s or pcies so I'm going to go ahead
[3:54] and select a 1x smx4 these have 80
[3:58] gigabytes of GPU Ram so they have uh one
[4:03] of the more powerful cards that are out
[4:05] right now from Nvidia and 80 gigabytes
[4:09] is is enough for most large language
[4:11] models you can also select a multi-gpu
[4:14] instance so if I needed even more space
[4:17] I could have a 2X a100 that would be I
[4:20] have 160 gigabytes of GPU RAM available
[4:23] for the large language models or I could
[4:26] select sort of a cheaper option like an
[4:29] a5000 and this 4X a 5
[4:34] 000.
[4:36] has actually 96 gigabytes of GPU RAM and
[4:41] it is a little bit cheaper than a single
[4:44] a100 you can also look at an a6000 they
[4:47] have 48 gigabytes of GPU Ram
[4:51] and an A40
[4:53] has 45 gigabytes of GPU Ram
[4:57] the consumer graphics cards like the
[4:59] 4090 and 3090 are only going to have 24
[5:02] gigabytes of GPU Ram each so again this
[5:06] is just something that you want to be
[5:08] really aware of and make sure that
[5:10] you're selecting a GPU that's going to
[5:11] have enough space so I'm going to go
[5:13] ahead and select a 1X a100 and now this
[5:18] is going to load I have 80 gigabytes
[5:21] allocated on this instance and I have
[5:25] selected a Ooba Booga web UI which is
[5:31] our recommended template and so if I
[5:34] jump into my instances here I can see
[5:36] that this is being created and set up
[5:37] for me
[5:40] it's going to take a three four or five
[5:43] minutes to load maybe a little bit
[5:45] longer it's really going to depend on
[5:46] the internet connection speed of the
[5:48] machine and
[5:50] um
[5:51] uh the size of the image this one loaded
[5:54] about three and a half minutes for me
[5:56] and now the open button is going to open
[6:00] port 7860 which was put in the
[6:04] environment variables when we set this
[6:06] up
[6:07] um and uh
[6:09] there's a few things that were installed
[6:11] and set up on the onstart script but
[6:13] anyways this is all just stuff that's in
[6:16] the template that we have set up for
[6:18] Ooba booga
[6:20] and I'm going to go ahead and open that
[6:22] interface up and here it is
[6:26] so uh here's where you can actually
[6:27] query the model that you set up the most
[6:30] important thing is that going to be
[6:32] downloading and setting up the model so
[6:35] um this software is not
[6:38] developed or maintained by vast this is
[6:43] open source software so to understand
[6:45] how to use this software you're going to
[6:48] want to find the open source project for
[6:53] this
[6:54] and and load that
[7:02] so here's the
[7:05] GitHub that's going to have a readme
[7:08] um
[7:09] the of course the installation steps you
[7:12] don't have to worry about because
[7:13] um you're using a Docker image and
[7:15] everything is is
[7:18] pre-loaded
[7:21] um
[7:22] so you can place the models into the
[7:24] model folder or when you're using the
[7:27] web UI you can just simply go to the
[7:29] models tab where it was before and
[7:32] here's where you can download the custom
[7:34] model so for hugging face you just use
[7:38] the username and model so for example if
[7:43] I wanted to try to to use just looking
[7:45] at the leaderboard if I wanted to use
[7:47] this model I would simply select the
[7:51] username and the name of the model like
[7:53] that and copy and paste it into the web
[7:55] UI
[7:57] and hit download and now it is going to
[8:00] start downloading
[8:02] this model once that model is downloaded
[8:04] I will be able to load the model in here
[8:06] into the GPU
[8:09] Ram in the instance
[8:11] sort of memory so then I can query it I
[8:14] can go back to text generation and
[8:16] actually start using it there's probably
[8:18] some other things that you can do and
[8:22] become familiar with with this interface
[8:23] this is a very nice way to run llms so
[8:27] that you don't have to use a command
[8:28] line
[8:32] um so there's quite a bit here and but
[8:34] again you're going to want to read about
[8:36] this and the ooga booga
[8:40] GitHub
[8:44] so if I go back and look at my instance
[8:48] um
[8:49] you can see that it's running
[8:51] you can also click on the billing tab if
[8:54] you just want to see you know if you're
[8:55] going to run something for multiple days
[8:57] you can get an idea of how many credits
[8:59] you're going to need you can set up your
[9:01] auto billing threshold so that your
[9:05] instance is not stopped when your
[9:08] balance gets low
[9:10] and that's the basics of running
[9:13] ubeoka on vast thanks for your time