---
title: 'GPT-4o Low Latency Image-to-Voice Streaming'
source: 'https://youtube.com/watch?v=v7SQ_O6pKzk'
video_id: 'v7SQ_O6pKzk'
date: 2026-07-28
duration_sec: 7455
---

# GPT-4o Low Latency Image-to-Voice Streaming

> Source: [GPT-4o Low Latency Image-to-Voice Streaming](https://youtube.com/watch?v=v7SQ_O6pKzk)

## Summary

In this live stream, the creator builds a low-latency system using GPT-4o to analyze screenshots and convert them to speech, aiming for minimal latency before the new voice mode release. He demonstrates a pipeline that resizes images to 512x512, uses low detail mode for fast responses, and outputs audio via OpenAI TTS, achieving around 5-second latency at very low cost.

### Key Points

- **Goal of the Stream** [03:48] — The creator aims to test GPT-4o's API latency on images by building a system that takes screenshots, resizes to 512x512, analyzes with low detail, and outputs voice.
- **Low Detail Mode Explained** [12:41] — Using low detail mode limits the image to 512x512 with a budget of 85 tokens, resulting in faster responses and lower cost.
- **First Script Test** [20:06] — The script takes a screenshot, resizes it, sends to GPT-4o, and prints the response. The image size is only 23KB.
- **Adding Text-to-Speech** [26:17] — The creator integrates OpenAI TTS using the 'speech' endpoint, saving audio as WAV and playing it with simpleaudio.
- **Real-World Demos** [43:15] — The system is tested on Reddit posts, Paint drawings, and other on-screen content, answering questions and describing images.
- **Pipeline Recap** [46:10] — The full pipeline: take screenshot, resize to 512x512, analyze with GPT-4o (low detail), generate text, convert to speech via TTS.
- **Impressive Vision Capabilities** [53:00] — GPT-4o accurately reads and interprets very small images (23-30KB), including handwritten text and low-resolution screenshots.
- **Latency Measurement** [01:28:07] — The latency from screenshot to voice output is approximately 5 seconds, with improved performance later in the stream.
- **Cost Analysis** [01:39:14] — Total cost for 2 hours of streaming: 20 cents for GPT-4o and TTS combined, demonstrating high efficiency.
- **Model Comparison on Logic Problem** [01:41:07] — Tests the 'sister problem' (Kaye has three brothers, each brother has two sisters - how many sisters does Kaye have?) on various models: only Qwen 2 72B gets it correct.

### Conclusion

The low-latency image-to-voice system is functional and cost-effective, with GPT-4o's vision performing exceptionally well even on tiny images. Further improvements like pixel-change detection could enhance responsiveness.

## Transcript

okay so hopefully I'm live now it's a bit uh hard to tell sometimes um
I'm just going to watch the chat to see if I see something let me just test if I see something let me just test it because I can see someone
could just give it like a couple of minutes like if people wanted to join I don't know like I said in the previous video I said in the previous video I'm not so used to streaming okay good
so let me know if it's too loud right because it's uh hello all hello is right because it's uh hello all hello is I love self in it I love self in it Paulo Mind Over Matter Ramy Tyler nice
to meet you all so I guess kind of the okay all so I guess kind of the okay good nice hello trendy good nice hello trendy tals Lord de Romano nice to meet you so
I'm going to try to focus on the chat while I'm doing stuff but like in kind of the main topic where we going to do what is on the screen here I thought we kind of have to hello Paul Hello jinnie nice to meet you
all we kind of have to just do it in I'm going to try to answer some questions going to try to answer some questions and stuff uh while we do this um so yeah uh I'm I'm just going to leave this for a while um just give it a few minutes
before we actually get into it so I don't know if anyone checked this out uh I tried to pull this hello R
out uh I tried to pull this hello R Ramy I tried to pull this quen 27b but for some reason reason I I can have has anyone tried
strange right
this I don't know if you can see but I really wanted to try it
anyway uh thought uh I just wanted to ask a few logical questions so see I get this error here but I haven't really looked into it hello Tom uh but if I run this with uh llama Tre it should work fine
right so that was a bit strange but the main part today is of course going to be uh to uh hello Alberto nice to meet you I really wanted to so you can see this works Kay has
to so you can see this works Kay has Five Sisters that's wrong uh so I think we're just going to go back here and I kind of just wanted to start here and I kind of just wanted to start by explaining what I wanted to do with
so basically I've been looking into the open AI API for gbt open AI API for gbt 40 uh how fast is the latency really look at today Matias because I kind of wanted to see
how fast we can get this um uh hello Dominic if we take a um uh hello Dominic if we take a screenshot and we resize it to 512 by screenshot and we resize it to 512 by 512 and we use GPT 40 to analyze it and
we're going to use this the lowest setting we can use to try to get it as setting we can use to try to get it as fast as we can hello they D us
yeah I'll look into that later but yeah back to this so basically what I wanted to try is to see how low we can get this latency now before we get the new voice latency now before we get the new voice mode uh hello Ricardo Hello regg nice to
meet you all so basically I've done a few tests of this and it seemed pretty decent um I was really impressed how good it can analyze images that are like
30 kilobytes and it makes it much faster and we're going to turn it into just text to speeech we have a prompt and we're just going to comment what is happening on the screen so that is kind
of my ID we're going to go through the the specs right kind of how I wanted to do this uh like the description of the project so this is just the visual description and uh we're going to use
GT40 to actually write the script uh but I think it's going to work pretty good so just wait I'll thought I just
wait a couple of more minutes before we start like if people are just people maybe just saw we went live so has anyone tried using like the lowest
anyone tried using like the lowest quality on the GP 40 API or something if quality on the GP 40 API or something if you did you try out uh and one more thing I wanted to do uh I wanted to kind of check the
do uh I wanted to kind of check the pricing so hope I don't show pricing so hope I don't show anything uh so today is it 7th so far we anything uh so today is it 7th so far we spent uh
much we are going to spend using spend using this monthly spent $1 so not much so remember 0.28
support uh no uh I'm going to show you we can't really use gp40 for the speech part yet uh so that is kind of what I'm looking forward to so we have to use kind of the old
old um the old text to speech that is using kind of we have to translate it into text right and then translate it into text right and then back into speech so that's a bit
have to do it now uh so remember to help me check what now uh so remember to help me check what the cost is going to be when we run this
into it uh I had a few more things but we can do that after I guess so I guess we have to open a chat GPT window
basically I thought we can just this answer intuitive up to this point do you think gp4 o is reliable cheap Che everyday tool to use for your own applications uh expensive yeah I don't
know it's kind of I think it depends on how many tokens you're using like if you're creating an app it might be I'm not quite sure about app it might be I'm not quite sure about that but uh I think it's pretty cheap
that but uh I think it's pretty cheap for the at least if you want some um uh wish model I think it's clearly the best Vision model I have tried and I
think it's pretty cheap but I think it depends on your use case if you're going to if you're doing something on the public side I don't know I'm not quite sure yet I saw some post on Reddit or something that one guy
was relying solely on uh relying solely on API the open AI API for their business and I think that's that's a bit risky
window right yeah I don't know I haven't really maxed out I don't know what about you guys but do you know what about you guys but do you struggle with filling up the context
or something you're running through that then can get expensive if you have these big uh input prompts right I think so okay so let's just get started here we can do questions and so while we get
started so uh I did some uh prepared I prepared something some uh prepared I prepared something here right because here right because um let me just so I wanted to prepare
um let me just so I wanted to prepare something let's just take people building with amazing to me more people aren't building with a yeah
people aren't building with a yeah maybe uh hello hello log up in chat play When the chat gets longer for coding assistance yeah yeah that's true okay so let's just get started so the first thing I wanted to do right
the first thing I wanted to do right um was to get get some documentation right for our project so documentation right for our project so the prompt I wanted to use was
this I don't know if you can see this so I just want to use like I upload some documentation of a script we want to build so this is kind of what I planned build so this is kind of what I planned out I wanted to create today so we want
to try to get the lowest possible latency when analyzing images using the gp40 model uh we want the settings to be the lowest they can like row low rest and low detail and I have been using like
the pill image SCB to do screenshots before and do it like maybe every 3 second or something we'll see and I want to a JPEG file to get it like the smallest size and we want to resize the image to 512 x 512 and we s save it to a
temporary save it as a temporary image to a folder name images so that is kind of the first step I wanted to do but for this I wanted to grab some documentation this I wanted to grab some documentation we can
upload and let's just start by doing that so let's go back here the documentation I kind of wanted for This was um I want to go to models right we want to use GPT
to models right we want to use GPT 40 I don't know if you can see this and we want to use explore gd4 o with image
inputs so I thought we can just grab this
need this I think what we want to focus on is kind of the here it is so I don't know if you check this out there's something called check this out there's something called Low will enable low rest mode so the
model will receive low rest 512 by 512 vers of the image uh so this allows the API to return faster responses conf consume fewer input tokens uh so it doesn't require High
detail that is kind of the low setting and the high setting is higher image quality right so that is kind of what I wanted to try out today I think for o those the resizing
for you for minimum price if you look at the yeah that might be actually true they have something called Auto so I haven't really looked into that yet but let's just grab some documentation here right I want to grab this I think here's
right I want to grab this I think here's something about managing images too so let's just so this is kind of what I like to do just grab some documentation I think we need all
good and from a previous project uh I think I have I think I have some functions here I want to reuse so this is kind of something I use
in a different project so this is just to I don't know what's wrong with my mouse here that's a bit bit strange just some functions I have used
strange just some functions I have used before just to speed it up a bit so I just want to grab this too okay paste it in here and let's say it's score I usually go like this U I don't know working python or
this U I don't know working python or something doesn't really matter I think because I kind of reuse these functions just to save some time they are pretty basic they are kind of already from this uh documentation
of already from this uh documentation here right so it should be pretty here right so it should be pretty good okay so I think that's good enough good okay so I think that's good enough right so what I like to do next is kind
of just go here I want to try to use J BT 40 for this sometimes I use clae or BT 40 for this sometimes I use clae or other stuff and I think I set my system message to just Euro software there with expertise in Python I don't know if it
habit I have and let me just upload this uh documentation file right and let's go grab our prompt for
the project project um so let's start with this right
on this but uh just a quick first iteration
just want to look over this quickly right I don't know that looked a bit strange this a file uh I kind of want to
check out that file just give me a second here
think I picked the wrong documentation here so let's try here so let's try this right okay so we are missing the open a I import
path I don't know if you can see this very well very well but uh
okay yeah so I want to remove this image path after we use it right um okay so let's copy this let's just
um okay so let's copy this let's just go here paste it in here so I have my I prepared my open a key I just left it in a EnV here so I key I just left it in a EnV here so I don't have to show it on stream right so
[Music] for let's just ask for something like um for let's just ask for something like um I have my API open a API key in uh EnV
key in uh EnV file uh I want to I want to use let's to I want to use let's say
uh let's call let's say EnV file in a V named open AI API key maybe or something like this to
this to fetch the API key fix this should work I already have St installed
this yeah I think this should work hello Daron nice to meet you uh yeah I think so okay so let's copy this paste it in here
I saw we kind of need uh import open AI
right quent came out yesterday 72 B ver seems better than llama yeah I saw seems better than llama yeah I saw that I'm not uh I don't have any I don't have the hardware now to kind of run uh llama 7 Tob
locally but I'm definitely going to try out the new quen model maybe on Gro or something if it comes over there uh Francy it's often incorrectly writes the new open ey method yeah I've seen that it always uses the old one that's
why I kind of like to grab some documentation uh before I continue when documentation uh before I continue when I'm using the chat completion part right well let's take a look here now uh we have
have this we are going to resize our image so we're just going to do the query what's in this this
screenshot if you're going to get a response and just print it for now I want to add some voice to this after so that's going to be the next step right fireworks host quen for okay that's pretty
cheap uh did they roll out the real time streaming with voice responses uh I don't think so yet no I'm hoping we can get that
soon yeah they said a few weeks and what what is a what is few down nice to meet you okay so let's just test this now so
hopefully let's just I'm just going to run this from my terminal I run this from my terminal I think uh
front of me right now okay so what do I call it box I just now okay so what do I call it box I just took um so what we want to see now is of course hopefully you can all see this if it works so let's just try
it works so let's just try it so we have this folder here called images so you can see we took a screenshot and then we removed it so it seems to be working the image shows a computer screen what appears to be code
editor open okay I wanted to take a look at this at this image because it's look how small it is image because it's look how small it is it's just it's very low
resolution so I'm kind of impressed how good it is reading this low resolutions good it is reading this low resolutions so the size is only 23 kiloby of the images so let's just stop this so this
is working quite good kind of bad resolution though but it is working so let's just stop it for
now because I have a prompt I wanted to try right
this so the prompt uh I want to use today um if you see a question on the screen answer it keep it short and conversational so this is kind of for the when we're going to add the voice part
let's try it in here
so so this is a question right let's run it so the reason I cre this was I thought it would be cool if we just scroll around the web and if we see a question we can get like a voice to
answer this uh how much does it cost to use the chat GPT API uh yeah we're going to monitor that today I think we just going
to we're going to see it's not too much they have reduced their prices like a it's it's not slow but I think we have like
slow but I think we have like a a Sleep part let's just remove a a Sleep part let's just remove that let's remove this
if it's I'm streaming or something I'm not quite sure not quite sure seems kind of to be stuck here
five okay yeah I guess it depends on your use case so this seem to crash that was a bit strange
so I think the next part we're going to do is actually just add the voice and do is actually just add the voice and then try to use
here I'm just a bit this chat thing I'm not so used to this just let's try to implement the voice part I wanted to
uh so I like to do like even though I have the code here we made some more changes right so I just like to paste in my code again
like to paste in my code again and what I want the text to speech to read out is this variable here response
right so I think I'm going to I think I'm going to grab some more documentation so let's just do open AI docs equals and let's go back here uh go to the documentation let's go to text to speech I
to text to speech I guess we can maybe okay and let's grab this part
maybe quick start because we want this python code here like a w
a w a so we can grab that you're using llms for coding I like to use loads of documentation uh I just think that's make makes things much easier so I just
make makes things much easier so I just use the documentation as context right okay so I I kind of want an example of how we can change the format so I think if we go to API reference and yeah I guess it's audio
create speech I know there's something yeah there's something called response format response format here uh I guess we can just grab all of
here uh I guess we can just grab all of this and just do it in in our context so now we kind of have our code in our context we have some documentation and now I kind of want to give my instructions
like let's add [Music] [Music] a feature maybe to feature to read out
uh maybe the V response using
chat too uh is there an open source AI assistant for visual studio 2022 uh oh I don't know that maybe 2022 uh oh I don't know that maybe someone in the chat
the variable response from our code using using TTS uh I want uh I guess to play
play the or something like this and we can save the I want the
temporary audio file Maybe audio Maybe audio file in a folder named I don't file in a folder named I don't know a AIO maybe that should be
good and for this I usually I've been just just using maybe use uh simple audio to play
the file uh I think we need the file to be in VI Not
MP3 something like this can you implment first iteration of
documentation and we have our instructions
you were leaving in Ulta up there though that's cool okay so let's see here now so we have
the we have the function text to speech okay so we get the response speech okay so we get the response format. VI uh okay that looks fine do we have like a folder path
fine do we have like a folder path yeah
it like simple audio that has always worked pretty well for me when playing worked pretty well for me when playing this files
again we might get some errors here I don't know uh okay so we have an error here uh line
85 line 67 okay so we have an open AI key where was that 67 okay we don't need that I
debugging with the GPD 40 we can try that I think I know what's wrong but let's just try some just try some debugging wow pure science thanks a lot
that's that's very yeah thank you a lot it's not necessary but wow
Norman okay so let's do some debugging that could be interesting let's just do my code equals and let's do error
equals and just grab this so let's see if we can figure this out using
you the error indicates that client variable is not defined in the text to space function uh okay you need to initiate opening initiate opening Cent so let's see if we can debug this
sometimes when we debug with chat GPT it can go very wrong can go very wrong right so open audio create H let's take a look at the API here I think it should be something like
this to be honest let's just copy that but first let's try try this I don't think this is going to work right because uh it just makes up
another client [Music] [Music] error at least I think
so maybe not yeah I don't think that's correct so let's try to just fix it ourselves let's just grab okay here let's try
that if this is not going to work let's feed some more documentation and feed some more documentation and ask ask again so let's try
this okay thanks for tuning in have a great day blizzard
it I was wondering if we can see kind of our cost here so we spent three cents so
case uh I don't uh I don't know I don't uh usually do that but uh you can of course try that have you have any success with
screenshot so that is working right look at the low the resolution is okay so let's just try to finish up the the debugging and see if we can solve
debugging and see if we can solve it let's just copy this let's go here my code just want to give like gd4 o a
code just want to give like gd4 o a chance to fix this object from the TTS API call uh is not a dictionary but rather an httpx binary
response content object you need to correctly hand a object you need to correctly hand a response to save it as a vow
anything let's give it a go I thought it was just some minor changes here
significant role in the future of computing computing okay I don't know can you hear
that yes AI is likely a big part of the future of computing uh in Spanish
okay yes AI is a big part of the future of computing I guess we could do like a print statement here
okay so that worked let's add a print statement
though um text to
uh let's should we print let's print this I don't know my computer is maybe a bit laggy I don't know I'm not quite
bit laggy I don't know I'm not quite sure might be some sure might be some uh might be some uh might be some um rate limiting too I don't
get this to work we can just yes AI is considered a significant part of the future of computing okay so
computing okay so Works in the future of computing you think he can do
noran only I can understand it and maybe ISAC too H for datl that's Danish okay so let's stop it for now so
I wanted to I kind of wanted to browse let's go to I kind of wanted to browse let's go to Reddit or or
question on the screen is build versus buy answer it depends build for customization buy for Speed how was the accent it was Danish
possible anyone else have experience with
Reddit showing a post discussing Steven pinker's view on AI he Likens AI models to a form of alien intelligence and mentions that experiments on them are mentions that experiments on them are akin to studying a newly discovered
impressed by is actually how small the images are yes there is you can look into mattermost and zulip for open source chat options in source chat options in go so they are just how big are these
images again 30 kilobytes
this sure let's see if I can help with that whoa the question on the screen is is there something similar to Steam Chat written in go goang that's open source answer as of my last update mattermost and zulip are open- Source Alternatives
although they may not be written in go you might want to check Repository like GitHub for go specific chat Solutions I just want to hear what it Solutions I just want to hear what it says about this chart
narrower domain of AI R&D as we apply super intelligence to R&D in other fields explosive growth will broaden because it didn't we didn't do any prompts so it didn't say anything
use cases for this I guess it's very passive we can't passive we can't really uh yeah I can I can explain that really uh yeah I can I can explain that just quickly give me a
so just a quick recap what we wanted to try try today uh was basically this so we wanted today uh was basically this so we wanted to try this low resolution
to try this low resolution screenshots uh screenshot resize it to screenshots uh screenshot resize it to 512 use gp40 to analyze it and turn um turn it into text to speech and yeah just see how fast we can
actually use this before we get uh yeah the voice is open AI too so we are using the API now let's go to paint or something and let's let's try to remove this sleep here maybe we can get a bit of a
here maybe we can get a bit of a better let's just do something
opened a python file named box. py how can I assist you with your python can I assist you with your python project what is
application with the question what is this typed on it do you think it's going this typed on it do you think it's going to answer this
drawing possibly of an animal's face or nose animal's face or nose it looks like nose animal's face or nose it looks like a basic drawing of a cow's face
face it looks like a simple drawing of the head of an animal possibly a goat or the head of an animal possibly a goat or a
animal head maybe a dog or a cow let's try some R you
you s sent
it's just like yeah there's some kind of lag because it's very
message we could have it doesn't seem to time out is a great wish model a bit
annoying uh that it kind of stops running I don't know what's going
designed to assist with information and tasks yeah Thomas I will put the code up on uh Community GitHub that was my plan anyway okay so GitHub that was my plan anyway okay so it could read
this but uh we are definitely having some issues with the
should be more than fast enough for this but there are some issues here with this but there are some issues here with uh it's lagging
annoying because I wanted to try it out with some other things but with some other things but uh we can just wait a while and see if
be sentient my response I can't become sentient but I can help with your
queries maybe no I'm not sentient I'm just a computer program no I'm not sentient I'm just a computer
no I'm not sentient I'm just a computer program designed to assist
Consciousness so I'm not sentient I just process information and respond based on process information and respond based on my program
in let's ask okay noted what box is
biggest sure what would you like to know about this behind the Red Box is the biggest
great wish model because the images are very small right so I'm the red box is the biggest I'm pretty impressed by it the black box I'm pretty impressed by it the black box is the biggest
try this with um with uh the new model if we get the response time down to like5 seconds or something that would be like5 seconds or something that would be pretty cool
right so does anyone have any other suggestions of what we can do with suggestions of what we can do with this um eager to how fast does it take a take screenshot that's pretty quick that's kind of
instant I just have it in like this Loop here so it just takes a screenshot analyzes it print the response turn it into text and place the text and removes
the image and the audio and just screenshot immediately screenshot immediately after that was kind of the idea at least
cool let's ask uh let's change up the prompt a it suggest an
the code but I guess you can't see the full code but I guess you can't see the full code let's just give it a
function I making it a little smarter such as the way watch is a portion of your screen and wait until there's nothing yeah uh I don't know if you nothing yeah uh I don't know if you catched my video uh where I did this pix
catched my video uh where I did this pix uh was some change in pixels it didn't take a screenshot until then I think we used a screenshot until then I think we used open CV or CV2 for it that worked pretty
good would be interesting to store a history of screenshots and then see if you can get it to remember something for you yeah that's a good idea uh I might
look into that grock models doesn't have wish Lama mistol uh I don't think I see grock models with wish yet maybe
soon does the image size matter regarding the wish API cost yeah I think it does if we take a look here on the documentation you can see if we go to documentation and wish
and if we go down here you can see low the model reive low rest 52 x see low the model reive low rest 52 x 512 uh with a budget of 85 tokens API to
return faster responses conf consume fewer input tokens so that means we are fewer input tokens so that means we are saving money right fewer input
really calculated cost per hour or something calculated cost per hour or something but uh we started today with we have but uh we started today with we have spent we have spent 10 cents so
spent we have spent 10 cents so far so it's definitely not far so it's definitely not free I don't know how much we have every second let's say 10 seconds Stitch the 10 images in a series yeah that
works great with this um base um base 64 this way can reserve or yeah that's a great have you tried that pure science that is pretty
cool how can you get it to be trained on a set of images and then you provide a new image how can you get it to be trained image how can you get it to be trained on a set of image New Image well I guess
that has to be kind on the provide a new image and okay I guess you can compare images that should work if you have an old image you can just compare it and check if it's the
same that should work web kit 2 pngs you'll be able to take the whole image of a website okay that's interesting web kit 2 PNG I haven't try that that could be cool so you can do it in the background
cool so you can do it in the background or just from the T terminal or does it or just from the T terminal or does it have to be in the
just suggest an improvement to the code keep it keep it short yeah let's just screen shot this text to speech function here but we definitely have there are
some issues with this uh I'm not going to go into what's wrong now but uh to go into what's wrong now but uh definitely some
issues right uh so let's hear if we get any suggestions okay so you can run it from a termal that's pretty cool uh Chain of Thought sequence yeah chronium
or selenium can do it yeah senica I've tried both of those let's hear if it has any suggestions for this uh
don't know what's going on I haven't tried API for a while so maybe there are some if it shouldn't be shouldn't have anything to do with with me streaming what id I usually use vs code but
for for videos and stuff uh I just like to use notepad++ just because of the visuals right there's definitely something wrong right there's definitely something wrong here so I usually use vs
here so I usually use vs code with uh some um auto complete and code with uh some um auto complete and stuff but stuff but uh let's give it one more
what's happening 12 Harry Potter in 12 Harry happening 12 Harry Potter in 12 Harry Potter books in uh and out is sure
thing uh $20 12 Harry Potter books in and packing that has to be it because this is not
working okay so let's just end this for now and let's we can try back later and see if it is working I don't know if you saw this but I made this
saw this but I made this uh I made this uh uh I made this uh Powers shell I don't know what you call
it it's almost like a profile so the first thing I did was this autocomplete suggestions I did a video on it but maybe you didn't see it so it's SG this is just cached all of the files is cached like
but I added some new features too let's blow it up a bit blow it up a bit so we can do I think it's ask open
AI how to create a file in Powershell so we can just ask open AI I Powershell so we can just ask open AI I think it's chat GPT 3.5 in the think it's chat GPT 3.5 in the terminal you can create a power using
item okay so we can basically ask anything I anything I guess um
what is the highest building in New
okay so that is one feature I think it's pretty neat for something like I did a video on this but I don't know if you have used
to speed up uh video in FFM up uh video in FFM Peg so these are very annoying right
Peg so these are very annoying right this s FFM Peg bash commands or shell a terminal we can copy it and clear this
and when we have pasted this instead of clicking enter I can do control enter and this is going to send this to open Ai and explain what it open Ai and explain what it does this command uses fmeg to change
the playback speed of a video takes an input video video takes an input video and time step half of original values okay something something so yeah this is pretty neat I use it
sometimes appreciate the live stream format oh thank you for that uh set proxy I have been building I was trying to build some pretty simple thing testing out kind of the the low latency part of the gbt 40
model wish part where we can use this small images and yeah seems to be small images and yeah seems to be working pretty good like this this command changes the current
directory to a directory name python so for me that's not so used to operating in the terminal I kind of I use this sometimes so we can create like
a new uh folder here
neat I did a video on it and I think I put up the code yeah let's take a look at that I've seen I've seen on
[Music] the LM sis
Arena the Gemini 1.5 has been creeping up here been creeping up here right can you manually create a scratch right can you manually create a scratch g2f maybe for only available
for uh available command line I'm not quite sure about that okay so if we take a look at the latest update here you can see Gemini
latest update here you can see Gemini Advanced is on second place Gemini 1.5 Advanced is on second place Gemini 1.5 Pro API is on third second shared 1.5 Pro is on fourth so Google kind of now has three of the first five
kind of now has three of the first five models I was a bit surprised by that to be honest the thing is
quento works well yeah I'm going to try that out the thing is I live in Norway I don't have access to the Google API that's why I haven't used that's why I haven't used it but uh I have been talking to Google
and we're going to do something so they are going to give me like hopefully they will give me API access yeah I know self in it that's why I haven't used it but I had like a meeting with Google and uh
we're probably going to try to make it happen they are not quite sure yet to be honest but if I'm going to do something with them I kind of need access to the API so I think they're looking into it but we will
see it's a bit annoying uh if uh I'm going to do videos on apis and maybe like half of the people watching can use it
that's not fair Chris you should be able to yeah it's a bit annoying because now I can't test it but like I said I had a meeting with Google and they might help me out can you test F3 wish
model I might do that soon I might do that I did do a video soon I might do that I did do a video where I kind of
tested tested uh some of these open source models impressed can you tell me about the best coding model
coding model when uh code Quin is released already the newest one I'm not quite sure but for the best coding model
I have tried is I think it has to be uh code stroll for code stroll for me I haven't tried the big ones so Cod
stroll is uh 22b it is released okay yeah I haven't tried it yet but code stroll I think that's pretty good code stroll I think that's pretty good that is what I'm using for my um auto
so too Nvidia free fire wish okay uh we can too Nvidia free fire wish okay uh we can also take a quick look I also take a quick look I have uh let me see if I can access it
in okay so I guess I can log in I'm not quite sure yeah I'm logged in now so this is the new Nvidia thing let's take a quick look at it so can you test several llms for ability to work with
pictures we have been doing some of that today but kind of my initial response is GPT 40 is far ahead I think for now for what I have been trying but uh
others are going to come soon so here we have F3 Vision
128k so I thought we could maybe upload an image here if we
small but I can't really zoom in what does this function do so this is the 53 wish model
speech all right so I guess you could read that like I said I haven't really tested this so
have been using Nvidia AI workbench and it's pretty dope yeah so I'm going to do like a video on this new Nvidia um Vim Nim sorry that you can create like a Docker
sorry that you can create like a Docker uh container with source model so that's going to be I guess it's in like 10 12 days can we use
this via API for free I don't think so so this is just something I got from Nvidia I got th credits just to test it credits just to test it out but I haven't done any video on it
yet let's screenshot something screenshot something here and let's ask if it can do like
right save this here may images
summarize this Reddit post I don't know if you can see this but let's try
I'm going to let you know when I'm going to do that it seems to be doing pretty good at OCR discusses the important of API privacy cost considerations uh okay I might have to look more into
this five wish 128k then ads coming on stream okay that might happen like I'm not so good at
streaming so I don't know kind of how I set it up I just use kind of the default YouTube thing I was going to use restream but I couldn't get it to work this time so maybe there are some ads I'm not sure
on this Nvidia playground what you call it so we Nvidia playground what you call it so we have llama 7B Mist have llama 7B Mist large
design stable [Music]
I had it a very serious hallucination so that is uh Josh D is the F3 Vision
we're going to look into this later some rag rag pipelines gaming audio to face I don't really know what it is this
face I don't really know what it is this is let's try
more Leisure for yachting but her Publications are no good H okay yeah we're looking into to this later I haven't really prepared for this so yeah um okay so let's go back here
so yeah um okay so let's go back here and see if uh we can actually try this for once to see if it
the code nothing's happening I don't know but like you said someone was mentioning some issues with the gbd4 O
wish is it possible to do multiple agents with uh Nvidia uh multiple agents with on Nidia too I can't really answer that now I'm
not quite sure uh but maybe you have access to the code right so if we go here you can see we have a llama 72b we can we can just turn this into a
72b we can we can just turn this into a python Cod right so I guess we could python Cod right so I guess we could what is the speed here it's pretty
[Music] the DP uh the F3 wish model then the DP uh the F3 wish model then maybe desktop GPT is Mac only yeah has anyone tried their open AI desktop
AI desktop um app like these agents whoa I don't think they're going
to work until we get like a very strong model right uh hi Chris love the channel starting at live stream is great have starting at live stream is great have you seen Auto gr with generates complete
agentic projects uh no I haven't Auto Ro okay I might take a note of that that's pretty
cool have you tried live kit for Real Time conversations and video no live kit Time conversations and video no live kit I don't think so is it good Auto gr is I don't think so is it good Auto gr is dope let me search it
demo small tuber has made it okay cool it seems
chat autog better let's see how long this video look ad let me know if the sound is too
ad let me know if the sound is too loud yeah flash we have like um loud yeah flash we have like um Community YouTube members Discord I had like a public this code converts text to speech and saves it as a wave file using
speech and saves it as a wave file using API calls wow it's still running right API calls wow it's still running right that was screen is asking if a certain code converts text to speech and saves it as
a wave file using API calls that's annoying is that annoying is that correct I'm just going to end again sound is perfect okay let's take a quick quick look at it
okay let's take a quick quick look at it auto gr if this is the correct [Music] one we're going to run autog gr locally for today's demo and you'll see we're configured to use chat GPT autog gr will
always remind you which Pro okay so I was thinking Auto Gro that it was um using
up here in the title our new feature eats tokens like Pac-Man munches dots and we bump up against grock's developer limits if we don't use open AI instead limits if we don't use open AI instead still gp-4 will only yeah the rate limit
of Gro if you have access it's pretty cool Gro is awesome but the rate limit is horrible charge me a couple cents for this demonstration and you're worth every penny yeah both of them ignore him
this week's added feature is a shiny new checkbox I know right You're Pitiful fortunately I have time to explain since GPT runs so much slower than grock we've added a built-in moderator function that helps keep our
AI agents focused on the goal and productive during our conversational testing clicking this new check box activates an internal moderator bot who will analyze the goal the scope of the conversation and the previous
remarks let me just stop here for a sec okay so here we have the age chance so we have a project [Music] manager SEO specialist yeah I've seen this kind of tools before they are
this kind of tools before they are pretty cool like crew Ai and uh yeah hopefully groco live soon with a f with a full-fledged service yeah I think
we actually need to pay for it if it's going to be good to pay for it if it's going to be good yeah it's free but the limits yeah
okay so it's yeah I can see it now autogen crew AI right
screen okay so I guess I have overlay the comments okay uh okay I can do that for a sec while we watch that or let's just leave it on but if many many people want to turn it off I can do it and the
previous remarks and decide which agent should speak next moderator bot will give that agent Specific Instructions aimed at ensuring each team member is as responsive and productive as possible based on our project manager's vision
provided our market research analyst with specific instructions on how to proceed we can continue testing our agents like that again and again combine this enhanced prompting technique with consistent agent definition updates
through each agent's gear I this is an AI voice right you agree it has to for demo $200 credits okay live kit I'll take a note of that yeah this has to be
an AI voice yeah icon and you'll be pretty impressed with the results I'll fast forward through the rest of the team out of respect for your time while that's happening let me mention that we've also changed up our config fil so
you're setting don't get blown away every time you update the software we every time you update the software we have user scruffy Nerf it is a have user scruffy Nerf it is a human I'm I'm leaning to an AI
of our upgrades have come from your suggestions so don't hesitate to ask why are you kissing up to them I need the likes and thanks for asking I guess now is as good a time as any to beg everybody to subscribe our stream L Log
developers who have used autog to make over a half million agents that's not counting everybody who runs it locally if every one of those developers clicked like subscribe and share we I can't listen to this call me when you're done
okay so I think I get the point right and someone asked what's your opinion on autogen Lang chain agentic approach people trying to use llms in approach people trying to use llms in the wrong
so this is just my thoughts right is that
um my thoughts it's that I think the all the Frameworks is good and what I think is like when we get like let's say we get gd5 and what we can do with these Frameworks is just
Frameworks is just insert GPT 5 into them and then they will maybe be 10 times as good but for now I don't think it's worth it for me they for my use cases what
I've seen is I don't think they are good enough to you do anything useful yet maybe that's a controversial opinion I don't know what you think but I have not yet seen anything that is like mindblowing with this ancient but I kind
of think the Frameworks are great and when we get like a model that can utilize the Frameworks good
we we can probably start trying it that's why I haven't used it so much so that is kind of my idea I love the Frameworks but I think the models the Frameworks but I think the models are good enough
uh let's it seems to be working now so let's go back to this for a this so if you yeah they aren't good enough to
you yeah they aren't good enough to manage a large code base yet yeah I agree so I just wanted to give this one more chance and see if it can suggest more chance and see if it can suggest something for is text to speech
function can you use lav or llama wish instead of asking to run the python script named box. piy I don't think those Mission
asking to run the python script named box. py what did we put down here let's just go back to
having a dashboard where you can use llm search so because you can just upgrade it when the models get better
agree Paul P science uh so I always say like if you're going to use like um if you want great results with a wish model at the moment I think just use GT40 like we are doing
question do you need I just wanted to see what the latency is now sure what kind of help with the question do you
could you provide more context so if someone's late to this uh this is just I don't know it depends on personal opinion the latency seems better now right let's count
it I don't know could you provide more context 1 2 3 4 I don't know you provide more context so about 5 Seconds that's not too
bad I'm not sure who Sora or cing is so I can't say if one is know who this is how can I help you with this image I think the latency is much lower now yeah that's
on the screen but the post mentions Steven pink view on AI he's saying AI intelligence and we are experimenting on them as if they were newly discovered organisms so basically the script is
just reading what's on the screen and trying to answer it I don't know who this person is but the question on the screen is Steph Pinker says AI models are a form of alien intelligence and we are doing
experiments on them as if they are some newly discovered organism yeah L it's taking screenshots all the time right but it's just 512 by 512 Kaye has three brothers each of her brothers has two sisters which includes
brothers has two sisters which includes Kaye herself therefore Kaylee has one everything that's on has six sisters it's annoying though but you can
sisters it's annoying though but you can change it up with Kaye has six sisters
the image is but the text says that Steven Pinker thinks AI models are like experimenting on them like newly discovered organisms we are using the open ey TTS for this yeah that's correct team try
parking at a lot near the city Center check parking apps or local guides for check parking apps or local guides for options enjoy your
or Lots near the city center it's a good idea to check local forums for recommendations have a great trip yeah the latency is much better now trip yeah the latency is much better now right it's not too
there something similar to stream chat written in goang and open source yes many open source chat systems are written in go such as mattermost and written in go such as mattermost and rocket. chat it understands images too
yeah let's find name yes the stream chat API is written in go you can check it API is written in go you can check it out at getstream.io getstream.io chat okay so what we can do let's change up the prompt
up the prompt here uh let's here uh let's do explain uh what is
image okay so let's save that and let's open our so this this one and let's run it one and let's run it and see if it can explain our project
computer screen displaying a presentation about converting text to speech using gp4 there's also a command prompt window running a python script called box. py the presentation shows
steps like screen capture resizing and speech synthesis let's hear if it can explain this
process of converting a screenshot into spoken commentary a screenshot is spoken commentary a screenshot is resized to 512 x 512 pixels analyzed by gp4 a custom prompt generates text which is then converted to speech and
commented on the screen content the focus is on achieving low focus is on achieving low latency yeah it's good remember the file latency yeah it's good remember the file sizes are just this image outlines a
process for streaming voice from a the file sizes are the file sizes are just
19k hello Carson yeah I think it's pretty good and if you weren't ear here earlier we also tried it in paint with like we also tried it in paint with like handwriting so are
handwriting so are you uh it looks like someone has a blank can open in a drawing program along with a terminal window where they're running a terminal window where they're running a python script called
sentient AI using a digital drawing tool on a computer screen the handwriting appears to be a bit messy so I think just the OCR is just
awesome yeah we can change someone has written a question on a drawing application asking are you a sentient AI the handwriting is rough and done on a whiteboard like canvas so if we stop this now uh let's
canvas so if we stop this now uh let's go back to our original prompt
this and let's uh answer in Spanish
sentient how can I assist you today it's actually quite impressive yeah it's very pretty good um low latency right
see no I'm not sentient I'm just a program how can I help you
this yeah let's check the price we
can I assist you today let's try this and then we can today let's try this and then we can check the
cost yeah I can understand that Hindi okay yeah
yeah Swedish yeah I'm going to do Swedish no I'm not me if I don't
if I don't know is that good Swedish I think I can use Swedish I'm not
sentient I think it good let's try finish
too that's Swedish I right I know that I don't know finished though no not sentient I'm just a computer
yeah seem I'm not sentient I'm just a program designed to help with program designed to help with information and
tasks no I am not sentient uh okay let's try answer yes I am an AI how can I help you today
good I don't think it has any accent yet trans mode I think we can try accents and stuff and I'll let I can hear is is stuff and I'll let I can hear is is finish but I don't know there if it's
good all right so I think I'm pretty happy with this uh I like the the cool any voice emotions no not yet we need a new version
right Swedish scner yeah I I don't think there's there's no dialect yet uh check the last pixel have it is has changed implement it
here also did you know GP released a demo of creating new one yet yet like I said uh we just have to wait so this is the old system we are only using GPD 404 the for the new vision
part right uh but yeah I don't I I think I'm quite happy with this it's kind of what I wanted it to I wanted it to do use cases probably nothing yet okay
do use cases probably nothing yet okay so let's just check check the
um an open uh models like uh models like this okay so we spent for everything we did now I don't know how many Quest we did
but we spent 20 cents I think we started at 28 cents cents I think we started at 28 cents today and we have spent 20 cents since today and we have spent 20 cents since we started in yeah about almost two
hours it's not too bad but we haven't done any big prompts or anything done any big prompts or anything right so yeah not too bad on the Audio right so yeah not too bad on the Audio model we spent uh we have spent 25 cents
and GPT 40 24 cents so I'm guessing when we actually get we can when we only can use GPT 40 this will probably be
this will probably be cheaper I think so that's going to be interesting let's just do a quick test here
our brothers has two sisters how many sisters does Kaye have think about it step by step I don't know if you have tried this step I don't know if you have tried this um
leaderboards so you can see one model is correct right kayy has one sister the other model says Five Sisters sister the other model says Five Sisters so a is
38b right new round let's try again yeah LMC is cool right I like it so remember the answer is
it so remember the answer is one Kaye has one sister Kaye has two sisters so B okay so here we have quen
what I wanted to try and the Gemini 1.5 flash got this try and the Gemini 1.5 flash got this wrong but uh quen 72b got it right give
a pretty good answer here breakdown pretty cool can you get any customer on the speech
TTS I don't think so so maybe we can pick different voices I maybe we can pick different voices I think that's about
anything with this until the new GPD 40 voice mode comes out could be a cool model let's go to Gro and see do they
model let's go to Gro and see do they have found so Gro don't have uh quen
have uh quen no but I do have uh
70b Kaye is a sister herself so she doesn't have any funny yeah would be cool if Gro could add
the quent to right Quinn is very good at coding so supposedly yeah that might be PR
to talk about today I think I kind of went through today I think I kind of went through everything I wanted to do
uh yeah I think so uh I was just going to ask if
yudo I was just thinking about uh where you can upload some audio and kind of
use so extend your own audio is there any video of this
somewhere uh video not yet okay um upload I just wanted to see if there was any demos of it
Hit or Miss okay that's interesting but can you you have to subscribe okay but can you do any sound can you take like a like a a song already or is it just is
a like a a song already or is it just is it copyrighted or let's just listen to what Matt has to say here for a
free okay I couldn't access it now so you can take like a wellknown song and build on it it's an honor System
listened to it I have tried it but I don't know so much about music so I don't know so much about music so I can't really
[Music] see obviously very video gamees but I remix this song and the reason I'm not worried about using something from a video game is because we are remixing the song and obviously you can upload
video game music to YouTube so I won't have any issues when I upload this video so here's the song that I made with it pixelated bravery this title was came up with by udio but the prompt I used what was the input
song video game DS which I think has pretty great uploaded into [Music]
game [Music]
right yeah UD is crazy if you haven't checked it
insane I don't know how loud this is going to be
Quin okay that's cool it's actually more responsive than llama Tre on okay so you have a double Nvidia RTX 6,000 that's Nvidia RTX 6,000 that's cool so do you have do you have that
running low home right or are running on the cloud or I guess you're running at home if you have two x right
know in my company local okay that's cool I wish I had that uh yeah I think quen could be very interesting what is cool about it is interesting what is cool about it is like the seven B model also has
like the seven B model also has 128k that's run 72b or I know I can't but the 7B I tried to pull it from uh
AMA but for some reason I can't use it yet maybe I have to update Ama or something I'll look into that later I don't want to do that uh on stream
don't want to do that uh on stream right I don't think it's not no point
things it performed good on NE in high stack is that a stack is that a 72 or 7B
just for the visuals on the videos I find is quite videos I find is quite boring I just like the visuals from notepad I think it's much cooler on video but uh I use uh vs code yeah but I
am thinking about doing like a a Macbook so maybe then you yeah you can then I'll be like everyone else on then I'll be like everyone else on YouTube that's
MacBooks are going to have this npus or they going to be like AI optimized I know they already kind of is but do you think Apple's going to think Apple's going to do anything
extra or do does anyone know notepad and Ms paint yeah does any anyone know when the new MacBooks are coming
out let's try a few models and see if they can solve the sister problem 53 what other models do I have code Stoll let me
Stoll let me see I have quen 14
B what other models do I have Gemma Gemma 7p quento is good yeah apple just improve Siri I
actually do a cooperation with the open AI so M4 is probably fall okay normal AI so M4 is probably fall okay normal release schedule along uh new iPad so do you think I think Apple has like a something coming up
something coming up soon okay I can check the cam
better so they will probably do some um yeah for a rapper right they will probably have something on this VC I can't remember the name of the VC I can't remember the name of the Apple
conference but let's see if any model can answer the sister problem here and open source models so we have how I zo in llama 3
models so we have how I zo in llama 3 Moll F three code stroll Quin 14b and Gemma 7B Kaye has three brothers each of our brothers has two sisters how many sisters does Kaye have think about it
step by step so the answer is one sister right
14b you think so
K yeah I've just assigned the oh yeah I see thanks
do you think anyone will get it uh will get this the colon yeah I saw it VV DC W WWDC that's correct thanks
they probably relieve uh the MacBooks there I think I was was thinking of buying one maybe so you think llama's going to solve this I don't think anyone is going to solve
I'm going to go with mistol or maybe quen
something I haven't done this before so let's start a before so let's start a poll I'm going to try to run a
pole uh what model will solve model will solve the the
but llama
than this but let's
I can't see it
14b I should have added none I forgot about that Apple M5 Ultra with 512 GB under yeah yeah running uh llama 405 on the local
yeah running uh llama 405 on the local local that would be crazy I'm actually quite excited for the big model from meta coming
now yeah do that I think we just did that on LMC but let me know what you find out okay so we have we'll give it a few more
driver I'm still on Windows 10 okay so I'm on 11 yeah I'm I'm on like a desktop PC so I have
um Windows okay so there are 20 votes that's good enough so let's try to run this now and see Kaye has one sister uh okay so that was the answer
two sisters including Kaye okay so that's the Quin answer so let's see here now can I zoom in a bit so llama Tre is
total five three this was a messy five3 Kaye has Five Sisters no messy five3 Kaye has Five Sisters no that's wrong so here's quen
14b Six Sisters nope Gemma 7 B Six
Sisters okay so I had some lag there am I back now
right kind of expected that I've seen this but uh Quin got it two yeah so we should have added in like a
non option no quen didn't get it but quen 2 option no quen didn't get it but quen 2 got it the 70b model uh because was a got it the 70b model uh because was a pure science that tried
now um I think I'm quite um I think I'm quite happy with uh what we did here and I'm of course going to upload this to the members GitHub if someone wants to try
it out uh I guess you could just watch the stream again and uh and just follow along if you want to but I think it worked pretty good in the end and uh latency was quite
good and I was pretty happy with it think like should I do more live streams I don't know if people enjoy live
streams or not but uh I was thinking like should I do more like reactions look at uh some videos maybe some news I'm not quite sure okay thanks that's
nice I have to enjoy them they are very engaging or interactive thanks very engaging or interactive thanks facts okay yeah I will try to do some more maybe on weekends you can do live streams yeah that's a good idea
that's too I like looking at the chat and asking some questions but like this is just my second stream so I haven't really I haven't really found out work best yet uh because I think if we're going to
stream we're going to have to have something to look at uh I think I'm going to do one video sorry one live stream where we can check out some maybe some YouTube videos do some reactions maybe some I don't know Reddit posts or
some articles about some news sort stuff I'm not quite sure or we can do more I'm not quite sure or we can do more building things uh I just have to try a bit of a different stuff uh so I just want to say thanks
for everyone tuning in very nice quick suggestion to run up some okay yeah yeah that's a good idea I don't know is music in the background good for streaming I'm not quite sure
those people like that I guess the only thing if we do anything Voice or YouTube I I can just stop it I
it so I might do some or something on Discord or on YouTube to see what kind of content we are going to do if we're going to do more streams I'm probably going to do more but uh uh Norman if you're a member you
should get like an invite Discord be invite Discord be better maybe for members we can do
web crawler yeah that would be cool
yeah I know but yeah thank you for tuning in uh know but yeah thank you for tuning in uh wish you all a great day great weekend and I would probably do a video on Sunday and next weekend I'm going to
Sunday and next weekend I'm going to this uh AI Summit in Oslo so I'll be away for a few days but probably just do some videos or something other than some videos or something other than that um hit the like And subscribe as
that um hit the like And subscribe as someone said uh all right have a good weekend all thank you for tuning in and yeah see you again