AI is finally controllable! …with a stick [FREE ComfyUI Workflow + Tutorial]

[00:00] I've built a workflow that lets you

[00:01] control AI video with a stick. Check

[00:03] this out. You just move plastic toys or

[00:05] printed cutouts through your scene and

[00:06] the workflow erases whatever is holding

[00:09] them and then animates them to follow

[00:11] that exact motion. You can also skip the

[00:13] stick entirely and use animated previews

[00:15] instead using After Effects or Blender,

[00:18] for example.

[00:19] To show you exactly how this works and

[00:21] how you can use it on your own computer

[00:23] for free, we created an entire short

[00:25] film about toys coming to life. And

[00:27] yeah, I know Nico from Corridor Digital

[00:29] had pretty much the same idea, but he

[00:31] used a different technique. He used one

[00:33] animate to transfer acting performances

[00:35] onto his toy characters. In this video,

[00:37] we're also going to look at one animate,

[00:39] but I used it to turn myself into the

[00:41] 15year-old version of myself for the

[00:43] short film because I'm a responsible

[00:45] adult now. I definitely don't play with

[00:47] toys anymore. Make sure to subscribe and

[00:49] stick around till the end for the full

[00:51] short film.

[00:54] Now, at the heart of this workflow is a

[00:56] research paper called time to move or

[00:58] TTM for short. And the cool part is that

[01:00] it's completely training free. So, it's

[01:02] pretty much an architecture that you can

[01:04] use with any diffusion-based video

[01:06] model. We are using it with W 2.2, but

[01:08] you could also use it with Cork Video X

[01:10] or stable video diffusion, for example.

[01:12] First, you need to create a control

[01:13] video with some motion, either by

[01:15] dragging things around in After Effects

[01:17] or Blender, or physically moving around

[01:19] stuff through your scene with a stick.

[01:22] TTM then uses something called dual

[01:24] clock dn noising. In areas where your

[01:26] character is moving, it uses lower noise

[01:28] to follow that motion precisely. In the

[01:31] rest of the scene, higher noise lets it

[01:32] generate a clean, natural background.

[01:35] So to make this all as easy as possible,

[01:37] I slap together some AI models in

[01:39] Confui. Here you just import a video and

[01:41] it automatically generates a black and

[01:43] white mask for your character using the

[01:45] new SAM 3 model from Meta. This workflow

[01:48] also extracts the start and end frames.

[01:50] And if you want, you can use the Gwen

[01:52] image edit AI model to remove any sticks

[01:54] or unwanted objects or clean up your

[01:56] character if needed. Then you give it a

[01:58] simple prompt describing the action of

[02:00] your character and hit run. And that's

[02:03] it. That's all there is to it. It's

[02:05] surprisingly easy and very robust. I

[02:07] remember when we created a controllable

[02:09] creature for a previous short film using

[02:11] one vase. We needed countless iterations

[02:14] to get the movement right. It was really

[02:16] exhausting. But with this workflow,

[02:17] about 70% of the shots worked just on

[02:20] the first try. To use this workflow,

[02:21] you'll need Confui, which is an

[02:23] open-source AI interface. If you don't

[02:25] have Confui installed yet, we've

[02:26] prepared a step-by-step guide on our

[02:29] website that walks you through

[02:30] everything. But fair warning, if this is

[02:32] your first time using Confui, you might

[02:33] want to start with a simpler workflow.

[02:35] Let's start by bringing this shot to

[02:36] life. First, we must prepare a mask for

[02:39] the character. And we can paint out that

[02:41] stick in the start and end frame. Now,

[02:43] it really doesn't matter how you prepare

[02:45] this. If you want, you can use After

[02:46] Effects for the mask or nano banana to

[02:48] paint out the stick or Photoshop or

[02:50] something. But we also created this free

[02:51] workflow that lets you do all the

[02:53] preparation straight in comi. So to use

[02:55] it, just drag and drop it into Confui.

[02:57] Now in your case, you might need to

[02:59] click manager, install missing custom

[03:02] nodes if you have any red nodes in the

[03:04] workflow, and let's zoom in on the left

[03:06] side here. Here you can find all the

[03:08] model loader nodes and you can find the

[03:11] corresponding model that you need to the

[03:13] left in this node right here. For the

[03:15] Gwen image model, you can see that there

[03:16] are different versions and you need to

[03:19] pick the one that comfortably fits on

[03:21] your GPU's VRAMm. Once you've downloaded

[03:23] all these models and made sure that all

[03:25] the correct ones are loaded, go to the

[03:26] load video nodes and load in your plate.

[03:29] If you want, you can name your shot

[03:31] right here. For us, this was 20. And

[03:34] then click run. Wait for the images to

[03:36] load. Let's first create the mask for

[03:37] our character. For this, we're using the

[03:39] new SAM 3 model by Meta. If your

[03:41] character enters at a later stage in the

[03:44] video, so it's not there in the first

[03:45] frame, you can just change the pick

[03:47] frame right here. But for me, the

[03:49] character is already in the video. So,

[03:50] all I need to do is just click on this

[03:52] character. And then I have to specify

[03:55] which parts are not belonging to the

[03:56] character. For this, I'm just right

[03:58] clicking on the other parts of the image

[04:01] like so. It's really important to

[04:03] exclude the stick that your character is

[04:05] on. So, I will also put a red dot right

[04:07] here. Then you just click run. And after

[04:09] a few seconds, our mask is done. And you

[04:11] can see it flickers a little bit, but

[04:12] that doesn't matter at all. Just ignore

[04:14] that. Let's now clean up the start

[04:16] frame. For this, I'm zooming in on this

[04:19] part right here. And now I need to copy

[04:21] over this first frame. For this, just

[04:23] copy and paste. Now, you can click open

[04:27] and mask editor. And then you can select

[04:29] the area where your stick is. And you

[04:31] can be generous there. Left click to

[04:33] select and right click to delete parts

[04:35] again. Click save. Come over to the

[04:37] right here and add a simple prompt like

[04:40] remove the wooden stick. Click run and

[04:42] the stick is gone. But you can also

[04:44] change these frames in more creative

[04:46] ways. For example, you can see that the

[04:47] Lego character is like hovering above

[04:49] the table. So what we could also do is

[04:51] just go back to the start here and

[04:53] create a bigger mask. Something like

[04:55] this. And I'm just creating a new

[04:56] prompt. Remove the stick and make the

[04:58] Lego figure stand on the table. And you

[05:00] can see that worked really well. though

[05:02] it changed the legs a little bit. So

[05:04] this prompt worked a lot better. I also

[05:05] added do not change the look of the

[05:07] figure. And yeah, that that worked. So

[05:09] let's also remove the stick from the end

[05:11] frame. Open mask editor. I'm selecting

[05:14] the stick. Go over here. Create a

[05:17] prompt. Remove the stick. And that looks

[05:19] good. The stick is gone. And now we have

[05:20] everything we need. You could say it's

[05:22] time to move. So drag and drop that

[05:24] workflow in here. And you install this

[05:27] one in the exact same way. Install

[05:29] missing custom nodes. restart and then

[05:32] you can find all the models that you

[05:33] need in these model loader nodes right

[05:36] here. Once you have everything set up,

[05:38] you can import the start and end frame.

[05:40] You will find these in your comfy folder

[05:42] output and then there is a folder with a

[05:45] shot number that you created and then

[05:47] you can just drag and drop these in. So

[05:48] this is my start frame right here and

[05:51] then this is my end frame. Below that

[05:53] you will need to import the plate of

[05:55] your moving character and below that you

[05:58] need to import the mask that we just

[05:59] created. Next, come up here to the setup

[06:02] and here you can select which resolution

[06:03] you want to use. Next, you can choose if

[06:05] you want to use the start frame and end

[06:08] frame. You can use only a start frame or

[06:10] only an end frame. But you still need to

[06:12] import an image right here. Otherwise,

[06:14] it will give you an error. For static

[06:16] shots, a start frame is usually enough,

[06:18] but if you really want to make sure that

[06:20] your character does not change over the

[06:21] duration of your shot, I would recommend

[06:23] going with both. Next, you need to

[06:25] create a simple prompt, something like

[06:27] this. And then you can just click run.

[06:29] The sampling process for the video is

[06:31] split into two parts. And after three

[06:33] steps, you already get a rough preview

[06:35] like this. And usually you can already

[06:36] tell if your shot is working or not.

[06:38] Otherwise, you can just quit the

[06:40] process. And I would recommend trying

[06:42] another seat or adjusting your prompt.

[06:44] Okay, first try and the result already

[06:46] looks amazing. You can see how well it

[06:48] integrates into the shot. But the

[06:50] problem is that the legs kind of

[06:51] separate and then start sticking back

[06:52] together. And I think I can just fix

[06:55] this using the prompt, I guess. Well,

[06:58] and this kind of worked. Looks much

[07:00] better now with this prompt. So, this is

[07:02] the whole process. And as you can see,

[07:04] it works really well. Now, is a good

[07:06] time to mention that this video and the

[07:07] free workflows are sponsored by our

[07:08] lovely supporters on Patreon. Thank you

[07:11] for supporting us on Patreon, keeping us

[07:13] free and independent, and also allowing

[07:14] us to share all these workflows for

[07:16] free. If you want access to advanced

[07:18] workflows, extra demo files, and our

[07:20] amazing Discord community, consider

[07:22] supporting. So, that's how we created

[07:23] all the shots of the animated toys. But

[07:25] there were also those shots where I

[07:27] needed to turn myself into my 15year-old

[07:29] self. For this, I wanted to use one

[07:31] animate, which is based on one 2.2, but

[07:33] specifically designed for character

[07:35] animation. The concept is pretty simple.

[07:37] You just need a driving video of your

[07:39] performance and a reference image of the

[07:40] character you want to transform into.

[07:42] When I tested it a few weeks ago, it

[07:44] worked pretty well. So, I just went

[07:45] ahead and shot everything without doing

[07:47] proper tests, trusting it would work out

[07:50] of the box. In the end, I spent more

[07:51] time wrestling with one animate than I

[07:53] actually spent on the toy animation

[07:55] workflow. It started pretty promising,

[07:57] though. I used this image of me when I

[07:59] was around 15 as a reference image, and

[08:02] the shot itself looked pretty decent.

[08:03] The problem was that I looked pretty

[08:05] different in every single shot. But I

[08:06] had an idea that I wanted to try for a

[08:08] long time. Since one animate and one 2.2

[08:11] are based on the same model, Lauras

[08:13] trained for one 2.2 will work for one

[08:15] animate as well. For those who don't

[08:17] know, a Lara is pretty much like a small

[08:18] extra model that you can train to help

[08:20] the main model better understand a

[08:22] specific concept that it didn't know

[08:24] before. So, I asked my parents for more

[08:26] photos of me when I was 15. And then I

[08:29] used AI toolkit to train the Laura. So,

[08:31] I created my data set with some very

[08:34] basic captions like this. And then I

[08:36] used these settings right here. Feel

[08:39] free to copy them if you want. Once it

[08:41] was done, I downloaded the Laura, edited

[08:43] it at full strength, and look at how

[08:45] much better these results are. There are

[08:47] still some issues, especially with like

[08:49] eye direction, but that's something I

[08:51] would like to fix in a future video. So,

[08:53] without further ado, here's the final

[08:54] short film.

[09:09] Ow.

[09:19] Yeah.

[09:36] Heat.

[09:54] No,

[10:03] it's you guys. Wait, you just robbed

[10:06] neglected because I'm playing video

[10:08] games all the time. I'm I'm so sorry.

[10:11] >> No, nerd. We just want you to go

[10:13] outside.

[10:18] >> All right, that's it for this one. Thank

[10:20] you so much for watching and thank you

[10:21] to our lovely Patreon supporters for

[10:23] making these videos possible. As always,

[10:26] if you create something with these

[10:27] workflows, feel free to tag me or send

[10:30] it to me. I always love to see what you

[10:32] come up with. Make sure to subscribe and

[10:34] see you next time.