[00:00]
By far the most common question I get on
[00:02]
any of my videos that have to do with
[00:04]
Home Assistant Voice or LLMs is, am I
[00:07]
running the LLM locally? And if so, how
[00:09]
am I doing it? Well, I can finally say
[00:12]
yes to that question, and I'm excited to
[00:14]
share how it all works. In this video,
[00:16]
I'm going to walk you through my whole
[00:17]
LLM setup that I've got running on a Mac
[00:20]
Mini M4 running Lama and how I've got it
[00:25]
integrated with Home Assistant and even
[00:27]
a fun WLED animation that visualizes how
[00:30]
hard the LLM is working at any given
[00:32]
moment. And definitely stick around to
[00:34]
the end. I give a sneak peek on how I'm
[00:36]
running my own chat GPT like server
[00:38]
locally called Open Web UI.
[00:42]
[Music]
[00:45]
All right, so let's quickly go over
[00:46]
everything I've got running here,
[00:48]
including all the components in case you
[00:49]
want to try something like this
[00:50]
yourself. For hardware, as I said, I've
[00:52]
chosen a Mac Mini M4 Pro. One of the
[00:55]
reasons why Apple Silicon works so well
[00:57]
for local LLMs is because of its unified
[01:00]
memory architecture. Large language
[01:02]
models run best on GPUs, and on Windows
[01:06]
machines with Nvidia cards, you're
[01:07]
limited by your GPU's VRAM. With Apple's
[01:10]
unified memory, the GPU and CPU share a
[01:14]
common memory pool, giving the models
[01:16]
much more headroom and flexibility. It's
[01:18]
actually a pretty compelling setup
[01:20]
dollar for dollar when you compare it to
[01:22]
other compact devices in the same
[01:24]
performance class. For the software
[01:26]
side, I'm running a Lama to handle all
[01:28]
of the models, Home Assistant for the
[01:30]
smart home side of things, and Open Web
[01:33]
UI for a local chat GBT style interface
[01:36]
for everything outside of Home
[01:38]
Assistant. for the specific models I'm
[01:40]
running. For now, I'll just say that
[01:42]
there have been two that have been
[01:44]
standouts for me in Home Assistant, but
[01:46]
more on models in a little bit. All
[01:48]
right, so let's take a look at how this
[01:50]
works in real time. Here's Home
[01:52]
Assistant voice running on a PE. No
[01:55]
cloud, no third party processing, just a
[01:57]
fast local response. I'm not going to
[01:59]
speed this up up at all. It's going to
[02:01]
be real time.
[02:05]
This battery pack is dead.
[02:08]
All right, let's try this again.
[02:12]
Lauraai, what's the weather like?
[02:16]
[Music]
[02:20]
The weather today is cloudy with a
[02:22]
temperature of 80° F and 69% humidity.
[02:26]
This will be even snappier once realtime
[02:28]
LLM sentence streaming comes to a lama
[02:30]
in Home Assistant. What makes my local
[02:33]
LLM setup both fast and reliable despite
[02:36]
running on relatively modest hardware is
[02:38]
a strategy I call cached responses.
[02:41]
Instead of exposing a huge amount of
[02:43]
entities and asking the LM to reason
[02:46]
through them in real time, I've started
[02:48]
leaning on home assistant scripts and
[02:50]
predefined pathways to simplify its job.
[02:53]
This lets me delegate heavier, more
[02:55]
complex processing to slower but smarter
[02:58]
models running in the background, while
[03:00]
a lightweight model handles real-time
[03:02]
voice interactions and simply pulls the
[03:04]
pre-ompiled answers or at least can grab
[03:07]
those pre-ompiled answers and create the
[03:10]
response that it needs from that small
[03:12]
piece of data rather than again trying
[03:14]
to reason through potentially hundreds
[03:17]
of exposed entities. A good example is
[03:20]
weather. One of the most common things
[03:22]
we ask around the house is, "What's the
[03:24]
weather like tomorrow?" or "What's the
[03:26]
humidity outside?" There's really no
[03:28]
need to reprocess all of that
[03:31]
information every time. So, I have a
[03:33]
larger model generate a clean spoken
[03:35]
weather summary every hour. That result
[03:38]
gets cached and is instantly available
[03:40]
to any smaller model to serve up. It's a
[03:43]
similar idea to what we're seeing across
[03:45]
the LLM landscape. multiple agents or
[03:48]
models, each with their own strengths,
[03:50]
collaborating to deliver fast, useful
[03:53]
results. And of course, I needed to
[03:55]
involve a WLED in some way. I found this
[03:59]
incredibly beautiful pre-built animation
[04:01]
in WLED called SOAP. And I've got the
[04:05]
speed of that animation automated to
[04:07]
parallel the current power draw of the
[04:09]
Mac Mini. And one of the nice things
[04:11]
about the animation is that it's got
[04:12]
easing built into it. So, as the LLM
[04:15]
ramps up, the wattage increases and the
[04:18]
animation smoothly speeds up and calms
[04:20]
back down when the LLM is idling. This
[04:23]
Mac Mini is dedicated for LLM use. So,
[04:26]
the visuals pretty spot-on as it's
[04:29]
processing different tasks, and this M4
[04:32]
Mac Mini is incredibly efficient. When
[04:34]
it's not processing anything, it idles
[04:37]
at less than 5 watts, and even under
[04:40]
full load, it only hits about 65 watts.
[04:43]
But that gives me a nice range to
[04:45]
display the animation speeds. Uh, I
[04:47]
still can't believe that it only uses 65
[04:49]
watts even under full load. And that
[04:52]
would really help keep electricity costs
[04:54]
down when hosting your own LLM,
[04:56]
especially since it idles at basically
[04:59]
no power usage.
[05:01]
So, if you're ready to try something
[05:03]
like this yourself, here's a quick
[05:04]
walkthrough. Setting up a Lama on a Mac
[05:07]
is actually pretty simple. The first
[05:09]
thing I recommend doing is setting your
[05:12]
IP address on your Mac to a static IP
[05:14]
address. That'll make sure that your IP
[05:17]
address remains static so that Home
[05:19]
Assistant and other devices can always
[05:21]
reach it.
[05:24]
Then you want to head over to Lama.com
[05:27]
and go ahead and click that download
[05:29]
button. And then once you've downloaded
[05:32]
it, go ahead and
[05:34]
drag O Lama to the applications
[05:37]
and go ahead and run it. Once you have O
[05:40]
Lama running, you should see a little
[05:42]
Llama icon up in the top taskbar area.
[05:47]
When you run this for the first time,
[05:49]
make sure to go to settings and make
[05:51]
sure to expose O Lama to the network.
[05:53]
This is critical so that Home Assistant
[05:55]
and other devices can use Olama.
[05:59]
Once you have a llama running and
[06:01]
exposed to your network, make sure that
[06:03]
other devices on your network can reach
[06:05]
it. So go to a different computer that
[06:07]
is not running a llama that is connected
[06:09]
to the same network and go to the
[06:13]
devices IP address. In this case, my Mac
[06:16]
Mini's IP address is 139. And then you
[06:20]
want to go to port
[06:23]
1434.
[06:25]
And this is the response you should see
[06:28]
if Lama is indeed running, Olama is
[06:31]
running. Now to test that everything is
[06:33]
truly working well and LLMs can actually
[06:36]
run, go to the device running
[06:39]
and open terminal. Once terminal is
[06:42]
open, you can run a simple command like
[06:45]
Olama run quen 3 billion verbose. Now
[06:52]
what this is going to do is it's going
[06:53]
to download the Quen 3 model the 3
[06:56]
billion parameter and by putting the
[06:59]
verbose flag at the end it will actually
[07:02]
give you a readout on how many tokens
[07:04]
per second you used and some other
[07:06]
statistics which can help determine how
[07:09]
performant the model is.
[07:12]
Now it's telling me that this doesn't
[07:13]
exist. Uh, so this is a good moment to
[07:16]
demonstrate
[07:18]
how you can use.com
[07:22]
to find models. So head over to
[07:24]
alama.com, go to models,
[07:28]
and jump over to Quen 3. And the reason
[07:31]
why Quen 33 3 billion didn't work is
[07:34]
because it doesn't exist.
[07:37]
So, let's try something that actually
[07:39]
does exist, like the Quen 4B, which I
[07:43]
actually already have installed. So, it
[07:46]
won't actually install it for me right
[07:48]
now. But you can see that this model is
[07:51]
2.6 GB in size. So, it'll download that
[07:54]
entire 2.6 GB model the first time you
[07:56]
run it. So, go ahead and click on the
[08:00]
model that you want to install and run.
[08:03]
And then this is your command right
[08:05]
here.
[08:07]
Paste that into terminal. I'm going to
[08:09]
add the verbose tag or flag I should
[08:13]
say.
[08:14]
And now you can start talking to the LLM
[08:17]
right within the terminal window.
[08:23]
So now this is a model that actually
[08:25]
thinks before it spits back the final
[08:28]
response.
[08:31]
And this what is E is a prompt that I
[08:35]
just use to test models. It's pretty
[08:37]
broad and challenges the model to reason
[08:40]
through the question.
[08:43]
And this is moving pretty quickly.
[08:47]
Now we can see that the evaluation rate,
[08:50]
which is pretty much the one that I used
[08:53]
more than anything else, uh was about 50
[08:55]
tokens per second, which in my opinion
[08:57]
would be usable within Home Assistant. I
[09:00]
think anything above 10 tokens per
[09:02]
second is technically fast enough to be
[09:04]
usable, but for home assistant voice, I
[09:08]
think 50 tokens per second is the
[09:10]
absolute minimum. Preferably, it should
[09:13]
be more than that. Now, let's look at a
[09:15]
model that's much larger
[09:19]
like
[09:20]
deepseek
[09:22]
32 billion parameters. This is a pretty
[09:26]
large model and also very capable.
[09:30]
I already have it installed,
[09:32]
so we can jump right into testing. And
[09:35]
you'll see it's quite a bit slower than
[09:38]
the other models. Even loading this into
[09:41]
the memory takes a bit of time. So,
[09:43]
we're going to try that same prompt.
[09:46]
See what we get.
[09:53]
It's outputting basically as fast as I
[09:56]
can read, which in my opinion is still
[09:59]
usable for certain tasks that you need
[10:01]
that horsepower. I found that 32 billion
[10:04]
parameter models seem to be the
[10:05]
threshold for this Mac Mini M4
[10:09]
with it being usable speed. And there we
[10:12]
go. We have about 10 tokens per second,
[10:16]
a little bit under that, which again I
[10:18]
think is usable as a traditional LLM,
[10:21]
but not so much in Home Assistant.
[10:22]
little bit too slow. One of the most
[10:24]
important things to keep in mind when
[10:26]
looking for models for Home Assistant is
[10:29]
that they need to be able to use tools.
[10:31]
Fortunately, Olama includes a filter,
[10:33]
which allows us to just see the models
[10:35]
that have tool capabilities. At the time
[10:37]
of recording this, my favorite model for
[10:39]
Home Assistant is Quen 3. And depending
[10:42]
upon what hardware you're running,
[10:45]
either their 4 billion or 8 billion
[10:47]
parameter model works okay. Obviously,
[10:50]
the 8 billion parameter model is smarter
[10:52]
than the 4 billion parameter model, but
[10:54]
you have to try to strike that balance
[10:56]
between speed and performance or
[10:59]
accuracy. I've also seen others on
[11:01]
Reddit saying that they've had success
[11:03]
with the Llama models in this version of
[11:07]
Home Assistant, which is currently
[11:09]
2025.7.2.
[11:12]
I get errors when I use these models,
[11:14]
but Quen and Mistrol have been pretty
[11:17]
great so far. So to get moving, I
[11:19]
recommend installing a Quen model. Try
[11:22]
that first with Home Assistant. And now
[11:25]
let's move over to Home Assistant. I
[11:27]
already have Olama installed, but it's
[11:30]
simple as going to add integration,
[11:32]
typing in a Lama, and then simply
[11:36]
putting in the IP address of the device
[11:39]
that's hosting a Lama and your LLM. In
[11:42]
this case, I put in this IP address,
[11:46]
which is the IP address of my Mac Mini,
[11:48]
plus the port at the end. Don't forget
[11:50]
that. And then once you have this
[11:51]
running, you can add as many
[11:53]
conversation agents as you'd like using
[11:56]
any of the models you have installed.
[11:58]
For instance, I have all these different
[12:00]
ones installed that I've been
[12:02]
experimenting with, and I've landed on
[12:04]
Mistral 3.224B in 22 24b for my heavier
[12:08]
duty tasks that I don't need as timely
[12:10]
of a response on and for realtime
[12:13]
conversations I've gone with Gwen 3 and
[12:16]
I'd love to use the 8 billion parameter
[12:18]
model but for now the 4 billion is
[12:22]
faster so I'm leaning on that although
[12:24]
once in a while it does hallucinate a
[12:26]
little bit and then once you have these
[12:28]
installed you can invoke them anytime
[12:30]
you want anywhere you want so you can
[12:33]
create specific scripts and automations
[12:35]
which tap into specific models. Again, I
[12:38]
think I'm moving more toward an
[12:39]
agentcentric ecosystem where there's
[12:42]
multiple LLMs working together rather
[12:44]
than one really massive super powerful
[12:46]
LLM. Now that I'm fully invested in
[12:49]
running my own local LLMs, I'm more
[12:52]
curious than ever to find out how they
[12:54]
work and how they function under the
[12:56]
hood. That's where Brilliant comes in.
[12:59]
Brilliant is an interactive learning
[13:01]
platform that helps you build real
[13:02]
understanding through problem solving.
[13:05]
One course I really like breaks down the
[13:06]
architecture behind LLMs, how neural
[13:09]
networks function, how transformers
[13:11]
process information, and why these
[13:13]
models are capable of the things that
[13:15]
they do. What sets Brilliant apart is
[13:17]
that you're actively engaging with the
[13:19]
material. Solving puzzles, working
[13:21]
through logic, and building intuition as
[13:24]
you go. It's a great way to sharpen your
[13:26]
thinking and go deeper on topics like
[13:28]
AI, programming, and data science. Head
[13:31]
to brilliant.org/strottlebuilds
[13:33]
or scan the QR code on the screen to try
[13:35]
Brilliant for free. And if you sign up
[13:38]
using my code, you'll get 20% off an
[13:40]
annual premium subscription. Big thanks
[13:42]
to Brilliant for supporting this
[13:44]
channel. Before we wrap up this video, I
[13:47]
want to quickly show you Open Web UI.
[13:49]
This is a local chat GPT style interface
[13:52]
that lets you interact with O Lama
[13:53]
models in a more traditional chatbased
[13:55]
format. So here is my instance of open
[13:58]
web UI and you can easily switch between
[14:01]
installed models, run queries across
[14:04]
different model types and test how each
[14:06]
one performs side by side. So as we
[14:09]
mentioned earlier, you can see the
[14:12]
different size models we have here.
[14:13]
everything from
[14:16]
larger Deepseek models, Deepseek 32
[14:19]
billion, down to some smaller models
[14:21]
like Llama 3.2, which is only a 3
[14:25]
billion parameter model. So, we click
[14:27]
into that and run a query,
[14:32]
you can see how fast it is, and it's
[14:35]
generally not bad. Llama is a pretty
[14:38]
well-rounded model for its size. But now
[14:40]
if I switch over to a larger model like
[14:42]
let's say DeepSseek 32 billion run the
[14:47]
same query. Now I can compare these side
[14:49]
by side and decide which one's better
[14:52]
for the task. Now DeepSseek is a
[14:54]
thinking model. So it will do some
[14:56]
interpretation, some thinking on its own
[14:59]
before it decides on a final answer.
[15:01]
It's kind of fun to see how it processes
[15:03]
and thinks along the way.
[15:06]
So as you can see this is very similar
[15:08]
to Jad GPT. It can search the web, it
[15:12]
can interpret images and this is a
[15:16]
platform that is continuing to evolve.
[15:18]
So more on this in future videos, but I
[15:22]
think this is really the next step. If
[15:23]
you've already got Alama installed, give
[15:25]
Open Web UI a try. If you're familiar
[15:27]
with GitHub and following some
[15:29]
instructions, you should be able to get
[15:31]
it off the ground.
[15:33]
So, that's going to wrap up this video.
[15:34]
If you want to dig deeper and try some
[15:36]
of these things yourself, I highly
[15:38]
recommend checking out the description
[15:39]
below. I've got a full write up on
[15:41]
straddields.com, which includes links,
[15:44]
examples, my running list of tested
[15:46]
models with home assistant support. So,
[15:49]
it should be a pretty good resource to
[15:51]
help you get started. Huge thanks to
[15:53]
Brilliant for sponsoring this video. And
[15:56]
if you enjoyed this video, if you felt
[15:58]
like it helped in any way, please
[16:00]
consider giving it a like. It really
[16:02]
helps me out. And if you're into this
[16:04]
kind of stuff, local LLMs, home
[16:06]
assistant, home automation, building,
[16:08]
you know, privacy focused systems,
[16:10]
definitely get subscribed. I've got a
[16:12]
ton of videos in the works. Thanks so
[16:14]
much. I will see you in the next one.
[16:17]
[Music]