Setting up DeepSeek R1 on Vast.ai

[00:06] hello hey this is Travis from Bast I

[00:08] wanted to make a quick video to show you

[00:10] how to spin up deep seeks R1 uh that's

[00:14] distilled down to the 70 billion uh size

[00:18] it fits very well on a 2X 490 machine so

[00:21] I just wanted to show you really quick

[00:22] how to spin that up on vast so I'll go

[00:24] ahead and jump into it so we have uh our

[00:28] console interface here and if you're new

[00:31] to vast and just getting started we have

[00:34] a quick start guide that goes over some

[00:36] of the basics of how to create an

[00:38] account add credit um set up your

[00:42] template and select a template filter

[00:44] machines and then R to

[00:46] GPU we'll go ahead and jump into uh the

[00:49] guide that we have for this use

[00:52] case

[00:54] and to run this model uh the Deep seek

[00:58] r170 billion we're going to use AMA and

[01:01] web UI AMA is the software stack that's

[01:04] going to run it and web UI actually set

[01:07] up the UI that you can access um to

[01:11] prompt the model very similar to what

[01:13] you do with uh open AI so let's go ahead

[01:17] and get started I have credit and my

[01:19] account is set up so I'm going to go

[01:21] ahead and pick the correct

[01:25] template which is this uhan web UI

[01:30] I'm going to allocate enough storage uh

[01:32] for this model I only need about 40 gab

[01:35] but I'll just kind of leave it around 60

[01:38] and then I'm going to want a

[01:40] 2X

[01:43] 4090 so I'll go ahead and put that in

[01:46] here now I'm also going to select um I

[01:49] don't know how long uh you plan on using

[01:51] the model but I'm only going to use this

[01:52] for a couple days so I'll just kind of

[01:54] move the max instance duration down not

[01:57] going to rent this for months um so

[02:00] here's all the uh availability of

[02:02] machines uh the ones that are that have

[02:04] the blue label are in a trusted Data

[02:06] Center and um I'll go ahead and pick uh

[02:09] this 2x 490 in Iceland now I'm going to

[02:12] click Rent Now what's going to happen is

[02:15] the instance going to spin up and it's

[02:17] going to start loading this

[02:21] template all right looks like a minute

[02:23] has passed and now the instance is uh

[02:26] loaded so I'm going to go ahead and

[02:28] click this open Button

[02:31] and this is going to open our instance

[02:33] portal uh which basically is uh shows

[02:37] all the process that that are running on

[02:39] this instance right now and you can see

[02:41] uh the open web UI is running on this

[02:44] port and the API is running on this

[02:47] boort so I'll go ahead and click

[02:51] this and this will start up and launch

[02:54] the web

[02:55] UI this is a open source software to run

[02:59] and and chat with

[03:01] llms so when you first boot this up you

[03:04] have to make a

[03:06] name uh

[03:09] email and a password for the

[03:12] admin account and then it's going to

[03:16] show you kind of what's new in open web

[03:19] UI and uh it's going to load you into a

[03:22] screen that has no model selected so

[03:24] you're going to have to download a

[03:27] model I think there's a few ways to add

[03:30] a model um in this interface I kind of

[03:32] just explore but if you go into the

[03:34] admin panel and then click

[03:40] settings

[03:43] models and then you

[03:46] can add a

[03:48] model and uh this model that I'm running

[03:51] is from ama.com so you just need to put

[03:54] in the

[03:55] tag and the tag is

[04:02] is just the name of the model which is

[04:03] right

[04:04] here so I'm going to grab the tag I'm

[04:07] going to drop it in

[04:12] here hit the download button and now

[04:16] it's going to start downloading the

[04:19] model it'll take a few minutes to

[04:21] download to go get a cup of

[04:24] coffee but once it gets to 100% now the

[04:28] Deep seek model is um

[04:30] load it onto this

[04:33] instance and you can simply start a new

[04:37] chat and select the model after it

[04:40] downloads it also could take a little

[04:43] bit of additional time to load

[04:45] it and there it is this is the Deep seek

[04:49] r170 billion model running on a 2X

[04:56] 490

[04:57] and there we can see

[05:00] it's a reasoning model so it has a

[05:02] thinking step uh that it's going to give

[05:04] the

[05:06] answer can see on vast um I'm paying

[05:10] about 80 cents an hour right now for

[05:12] this if I wanted to keep this machine

[05:15] longer there's a reserved instance

[05:17] discount um but I'm uh just going to

[05:21] keep it for uh a few hours to to play

[05:24] around with this a bit and if I wanted

[05:28] to run the API the API uh is available

[05:32] on uh this

[05:33] port and uh Jupiter is also available

[05:37] along with SSH so there's a lot of

[05:39] different ways you can connect to this

[05:41] machine um but it's always very simple

[05:44] to just use this sort of chat interface

[05:46] that loads onto it well that's deep seek

[05:49] r170 billion on vast AI thanks