[0:00] By far the most common question I get on
[0:02] any of my videos that have to do with
[0:04] Home Assistant Voice or LLMs is, am I
[0:07] running the LLM locally? And if so, how
[0:09] am I doing it? Well, I can finally say
[0:12] yes to that question, and I'm excited to
[0:14] share how it all works. In this video,
[0:16] I'm going to walk you through my whole
[0:17] LLM setup that I've got running on a Mac
[0:20] Mini M4 running Lama and how I've got it
[0:25] integrated with Home Assistant and even
[0:27] a fun WLED animation that visualizes how
[0:30] hard the LLM is working at any given
[0:32] moment. And definitely stick around to
[0:34] the end. I give a sneak peek on how I'm
[0:36] running my own chat GPT like server
[0:38] locally called Open Web UI.
[0:42] [Music]
[0:45] All right, so let's quickly go over
[0:46] everything I've got running here,
[0:48] including all the components in case you
[0:49] want to try something like this
[0:50] yourself. For hardware, as I said, I've
[0:52] chosen a Mac Mini M4 Pro. One of the
[0:55] reasons why Apple Silicon works so well
[0:57] for local LLMs is because of its unified
[1:00] memory architecture. Large language
[1:02] models run best on GPUs, and on Windows
[1:06] machines with Nvidia cards, you're
[1:07] limited by your GPU's VRAM. With Apple's
[1:10] unified memory, the GPU and CPU share a
[1:14] common memory pool, giving the models
[1:16] much more headroom and flexibility. It's
[1:18] actually a pretty compelling setup
[1:20] dollar for dollar when you compare it to
[1:22] other compact devices in the same
[1:24] performance class. For the software
[1:26] side, I'm running a Lama to handle all
[1:28] of the models, Home Assistant for the
[1:30] smart home side of things, and Open Web
[1:33] UI for a local chat GBT style interface
[1:36] for everything outside of Home
[1:38] Assistant. for the specific models I'm
[1:40] running. For now, I'll just say that
[1:42] there have been two that have been
[1:44] standouts for me in Home Assistant, but
[1:46] more on models in a little bit. All
[1:48] right, so let's take a look at how this
[1:50] works in real time. Here's Home
[1:52] Assistant voice running on a PE. No
[1:55] cloud, no third party processing, just a
[1:57] fast local response. I'm not going to
[1:59] speed this up up at all. It's going to
[2:01] be real time.
[2:05] This battery pack is dead.
[2:08] All right, let's try this again.
[2:12] Lauraai, what's the weather like?
[2:16] [Music]
[2:20] The weather today is cloudy with a
[2:22] temperature of 80° F and 69% humidity.
[2:26] This will be even snappier once realtime
[2:28] LLM sentence streaming comes to a lama
[2:30] in Home Assistant. What makes my local
[2:33] LLM setup both fast and reliable despite
[2:36] running on relatively modest hardware is
[2:38] a strategy I call cached responses.
[2:41] Instead of exposing a huge amount of
[2:43] entities and asking the LM to reason
[2:46] through them in real time, I've started
[2:48] leaning on home assistant scripts and
[2:50] predefined pathways to simplify its job.
[2:53] This lets me delegate heavier, more
[2:55] complex processing to slower but smarter
[2:58] models running in the background, while
[3:00] a lightweight model handles real-time
[3:02] voice interactions and simply pulls the
[3:04] pre-ompiled answers or at least can grab
[3:07] those pre-ompiled answers and create the
[3:10] response that it needs from that small
[3:12] piece of data rather than again trying
[3:14] to reason through potentially hundreds
[3:17] of exposed entities. A good example is
[3:20] weather. One of the most common things
[3:22] we ask around the house is, "What's the
[3:24] weather like tomorrow?" or "What's the
[3:26] humidity outside?" There's really no
[3:28] need to reprocess all of that
[3:31] information every time. So, I have a
[3:33] larger model generate a clean spoken
[3:35] weather summary every hour. That result
[3:38] gets cached and is instantly available
[3:40] to any smaller model to serve up. It's a
[3:43] similar idea to what we're seeing across
[3:45] the LLM landscape. multiple agents or
[3:48] models, each with their own strengths,
[3:50] collaborating to deliver fast, useful
[3:53] results. And of course, I needed to
[3:55] involve a WLED in some way. I found this
[3:59] incredibly beautiful pre-built animation
[4:01] in WLED called SOAP. And I've got the
[4:05] speed of that animation automated to
[4:07] parallel the current power draw of the
[4:09] Mac Mini. And one of the nice things
[4:11] about the animation is that it's got
[4:12] easing built into it. So, as the LLM
[4:15] ramps up, the wattage increases and the
[4:18] animation smoothly speeds up and calms
[4:20] back down when the LLM is idling. This
[4:23] Mac Mini is dedicated for LLM use. So,
[4:26] the visuals pretty spot-on as it's
[4:29] processing different tasks, and this M4
[4:32] Mac Mini is incredibly efficient. When
[4:34] it's not processing anything, it idles
[4:37] at less than 5 watts, and even under
[4:40] full load, it only hits about 65 watts.
[4:43] But that gives me a nice range to
[4:45] display the animation speeds. Uh, I
[4:47] still can't believe that it only uses 65
[4:49] watts even under full load. And that
[4:52] would really help keep electricity costs
[4:54] down when hosting your own LLM,
[4:56] especially since it idles at basically
[4:59] no power usage.
[5:01] So, if you're ready to try something
[5:03] like this yourself, here's a quick
[5:04] walkthrough. Setting up a Lama on a Mac
[5:07] is actually pretty simple. The first
[5:09] thing I recommend doing is setting your
[5:12] IP address on your Mac to a static IP
[5:14] address. That'll make sure that your IP
[5:17] address remains static so that Home
[5:19] Assistant and other devices can always
[5:21] reach it.
[5:24] Then you want to head over to Lama.com
[5:27] and go ahead and click that download
[5:29] button. And then once you've downloaded
[5:32] it, go ahead and
[5:34] drag O Lama to the applications
[5:37] and go ahead and run it. Once you have O
[5:40] Lama running, you should see a little
[5:42] Llama icon up in the top taskbar area.
[5:47] When you run this for the first time,
[5:49] make sure to go to settings and make
[5:51] sure to expose O Lama to the network.
[5:53] This is critical so that Home Assistant
[5:55] and other devices can use Olama.
[5:59] Once you have a llama running and
[6:01] exposed to your network, make sure that
[6:03] other devices on your network can reach
[6:05] it. So go to a different computer that
[6:07] is not running a llama that is connected
[6:09] to the same network and go to the
[6:13] devices IP address. In this case, my Mac
[6:16] Mini's IP address is 139. And then you
[6:20] want to go to port
[6:23] 1434.
[6:25] And this is the response you should see
[6:28] if Lama is indeed running, Olama is
[6:31] running. Now to test that everything is
[6:33] truly working well and LLMs can actually
[6:36] run, go to the device running
[6:39] and open terminal. Once terminal is
[6:42] open, you can run a simple command like
[6:45] Olama run quen 3 billion verbose. Now
[6:52] what this is going to do is it's going
[6:53] to download the Quen 3 model the 3
[6:56] billion parameter and by putting the
[6:59] verbose flag at the end it will actually
[7:02] give you a readout on how many tokens
[7:04] per second you used and some other
[7:06] statistics which can help determine how
[7:09] performant the model is.
[7:12] Now it's telling me that this doesn't
[7:13] exist. Uh, so this is a good moment to
[7:16] demonstrate
[7:18] how you can use.com
[7:22] to find models. So head over to
[7:24] alama.com, go to models,
[7:28] and jump over to Quen 3. And the reason
[7:31] why Quen 33 3 billion didn't work is
[7:34] because it doesn't exist.
[7:37] So, let's try something that actually
[7:39] does exist, like the Quen 4B, which I
[7:43] actually already have installed. So, it
[7:46] won't actually install it for me right
[7:48] now. But you can see that this model is
[7:51] 2.6 GB in size. So, it'll download that
[7:54] entire 2.6 GB model the first time you
[7:56] run it. So, go ahead and click on the
[8:00] model that you want to install and run.
[8:03] And then this is your command right
[8:05] here.
[8:07] Paste that into terminal. I'm going to
[8:09] add the verbose tag or flag I should
[8:13] say.
[8:14] And now you can start talking to the LLM
[8:17] right within the terminal window.
[8:23] So now this is a model that actually
[8:25] thinks before it spits back the final
[8:28] response.
[8:31] And this what is E is a prompt that I
[8:35] just use to test models. It's pretty
[8:37] broad and challenges the model to reason
[8:40] through the question.
[8:43] And this is moving pretty quickly.
[8:47] Now we can see that the evaluation rate,
[8:50] which is pretty much the one that I used
[8:53] more than anything else, uh was about 50
[8:55] tokens per second, which in my opinion
[8:57] would be usable within Home Assistant. I
[9:00] think anything above 10 tokens per
[9:02] second is technically fast enough to be
[9:04] usable, but for home assistant voice, I
[9:08] think 50 tokens per second is the
[9:10] absolute minimum. Preferably, it should
[9:13] be more than that. Now, let's look at a
[9:15] model that's much larger
[9:19] like
[9:20] deepseek
[9:22] 32 billion parameters. This is a pretty
[9:26] large model and also very capable.
[9:30] I already have it installed,
[9:32] so we can jump right into testing. And
[9:35] you'll see it's quite a bit slower than
[9:38] the other models. Even loading this into
[9:41] the memory takes a bit of time. So,
[9:43] we're going to try that same prompt.
[9:46] See what we get.
[9:53] It's outputting basically as fast as I
[9:56] can read, which in my opinion is still
[9:59] usable for certain tasks that you need
[10:01] that horsepower. I found that 32 billion
[10:04] parameter models seem to be the
[10:05] threshold for this Mac Mini M4
[10:09] with it being usable speed. And there we
[10:12] go. We have about 10 tokens per second,
[10:16] a little bit under that, which again I
[10:18] think is usable as a traditional LLM,
[10:21] but not so much in Home Assistant.
[10:22] little bit too slow. One of the most
[10:24] important things to keep in mind when
[10:26] looking for models for Home Assistant is
[10:29] that they need to be able to use tools.
[10:31] Fortunately, Olama includes a filter,
[10:33] which allows us to just see the models
[10:35] that have tool capabilities. At the time
[10:37] of recording this, my favorite model for
[10:39] Home Assistant is Quen 3. And depending
[10:42] upon what hardware you're running,
[10:45] either their 4 billion or 8 billion
[10:47] parameter model works okay. Obviously,
[10:50] the 8 billion parameter model is smarter
[10:52] than the 4 billion parameter model, but
[10:54] you have to try to strike that balance
[10:56] between speed and performance or
[10:59] accuracy. I've also seen others on
[11:01] Reddit saying that they've had success
[11:03] with the Llama models in this version of
[11:07] Home Assistant, which is currently
[11:09] 2025.7.2.
[11:12] I get errors when I use these models,
[11:14] but Quen and Mistrol have been pretty
[11:17] great so far. So to get moving, I
[11:19] recommend installing a Quen model. Try
[11:22] that first with Home Assistant. And now
[11:25] let's move over to Home Assistant. I
[11:27] already have Olama installed, but it's
[11:30] simple as going to add integration,
[11:32] typing in a Lama, and then simply
[11:36] putting in the IP address of the device
[11:39] that's hosting a Lama and your LLM. In
[11:42] this case, I put in this IP address,
[11:46] which is the IP address of my Mac Mini,
[11:48] plus the port at the end. Don't forget
[11:50] that. And then once you have this
[11:51] running, you can add as many
[11:53] conversation agents as you'd like using
[11:56] any of the models you have installed.
[11:58] For instance, I have all these different
[12:00] ones installed that I've been
[12:02] experimenting with, and I've landed on
[12:04] Mistral 3.224B in 22 24b for my heavier
[12:08] duty tasks that I don't need as timely
[12:10] of a response on and for realtime
[12:13] conversations I've gone with Gwen 3 and
[12:16] I'd love to use the 8 billion parameter
[12:18] model but for now the 4 billion is
[12:22] faster so I'm leaning on that although
[12:24] once in a while it does hallucinate a
[12:26] little bit and then once you have these
[12:28] installed you can invoke them anytime
[12:30] you want anywhere you want so you can
[12:33] create specific scripts and automations
[12:35] which tap into specific models. Again, I
[12:38] think I'm moving more toward an
[12:39] agentcentric ecosystem where there's
[12:42] multiple LLMs working together rather
[12:44] than one really massive super powerful
[12:46] LLM. Now that I'm fully invested in
[12:49] running my own local LLMs, I'm more
[12:52] curious than ever to find out how they
[12:54] work and how they function under the
[12:56] hood. That's where Brilliant comes in.
[12:59] Brilliant is an interactive learning
[13:01] platform that helps you build real
[13:02] understanding through problem solving.
[13:05] One course I really like breaks down the
[13:06] architecture behind LLMs, how neural
[13:09] networks function, how transformers
[13:11] process information, and why these
[13:13] models are capable of the things that
[13:15] they do. What sets Brilliant apart is
[13:17] that you're actively engaging with the
[13:19] material. Solving puzzles, working
[13:21] through logic, and building intuition as
[13:24] you go. It's a great way to sharpen your
[13:26] thinking and go deeper on topics like
[13:28] AI, programming, and data science. Head
[13:31] to brilliant.org/strottlebuilds
[13:33] or scan the QR code on the screen to try
[13:35] Brilliant for free. And if you sign up
[13:38] using my code, you'll get 20% off an
[13:40] annual premium subscription. Big thanks
[13:42] to Brilliant for supporting this
[13:44] channel. Before we wrap up this video, I
[13:47] want to quickly show you Open Web UI.
[13:49] This is a local chat GPT style interface
[13:52] that lets you interact with O Lama
[13:53] models in a more traditional chatbased
[13:55] format. So here is my instance of open
[13:58] web UI and you can easily switch between
[14:01] installed models, run queries across
[14:04] different model types and test how each
[14:06] one performs side by side. So as we
[14:09] mentioned earlier, you can see the
[14:12] different size models we have here.
[14:13] everything from
[14:16] larger Deepseek models, Deepseek 32
[14:19] billion, down to some smaller models
[14:21] like Llama 3.2, which is only a 3
[14:25] billion parameter model. So, we click
[14:27] into that and run a query,
[14:32] you can see how fast it is, and it's
[14:35] generally not bad. Llama is a pretty
[14:38] well-rounded model for its size. But now
[14:40] if I switch over to a larger model like
[14:42] let's say DeepSseek 32 billion run the
[14:47] same query. Now I can compare these side
[14:49] by side and decide which one's better
[14:52] for the task. Now DeepSseek is a
[14:54] thinking model. So it will do some
[14:56] interpretation, some thinking on its own
[14:59] before it decides on a final answer.
[15:01] It's kind of fun to see how it processes
[15:03] and thinks along the way.
[15:06] So as you can see this is very similar
[15:08] to Jad GPT. It can search the web, it
[15:12] can interpret images and this is a
[15:16] platform that is continuing to evolve.
[15:18] So more on this in future videos, but I
[15:22] think this is really the next step. If
[15:23] you've already got Alama installed, give
[15:25] Open Web UI a try. If you're familiar
[15:27] with GitHub and following some
[15:29] instructions, you should be able to get
[15:31] it off the ground.
[15:33] So, that's going to wrap up this video.
[15:34] If you want to dig deeper and try some
[15:36] of these things yourself, I highly
[15:38] recommend checking out the description
[15:39] below. I've got a full write up on
[15:41] straddields.com, which includes links,
[15:44] examples, my running list of tested
[15:46] models with home assistant support. So,
[15:49] it should be a pretty good resource to
[15:51] help you get started. Huge thanks to
[15:53] Brilliant for sponsoring this video. And
[15:56] if you enjoyed this video, if you felt
[15:58] like it helped in any way, please
[16:00] consider giving it a like. It really
[16:02] helps me out. And if you're into this
[16:04] kind of stuff, local LLMs, home
[16:06] assistant, home automation, building,
[16:08] you know, privacy focused systems,
[16:10] definitely get subscribed. I've got a
[16:12] ton of videos in the works. Thanks so
[16:14] much. I will see you in the next one.
[16:17] [Music]