---
title: 'Rare Data Farming with GPT-4 Vision - Live Stream'
source: 'https://youtube.com/watch?v=hngoTmBQ4IY'
video_id: 'hngoTmBQ4IY'
date: 2026-07-28
duration_sec: 3429
---

# Rare Data Farming with GPT-4 Vision - Live Stream

> Source: [Rare Data Farming with GPT-4 Vision - Live Stream](https://youtube.com/watch?v=hngoTmBQ4IY)

## Summary

The creator live-streams a coding session building a Python application that uses the GPT-4 Vision API to count objects (eggs) from an IP webcam feed. They implement multiple screenshot sampling and few-shot prompting to improve reliability, then attempt to build a simple terminal-based frontend.

### Key Points

- **Project Overview** [03:39] — Goal: Use IP webcam, capture frames, analyze with GPT-4 Vision API, extract data as JSON, and display in real time.
- **Initial Setup** [04:37] — Used OpenCV (cv2) to connect to IP webcam URL and grab frames.
- **Inconsistent Count Issue** [05:49] — Direct single-frame analysis gave inconsistent egg counts (e.g., 4 or 5).
- **Multiple Screenshots Approach** [07:21] — Solution: Take 5-7 screenshots per cycle, analyze each, then pick the most common count using a few-shot GPT-4 prompt.
- **Correction Prompt** [08:53] — Implemented a few-shot prompt that takes multiple API responses and returns the value with the highest occurrence.
- **Speed Optimization** [19:20] — Attempted to use 'low' fidelity setting in GPT-4 Vision to speed up analysis, but improvement was minimal.
- **Building Frontend** [29:00] — Started creating a terminal UI (using curses) to display the real-time egg count, connecting via JSON file.
- **Debugging JSON Parsing** [50:42] — Encountered issues with markdown formatting in API responses; tried stripping markdown and using few-shot prompts.

### Conclusion

The project demonstrates a practical approach to real-time object counting using GPT-4 Vision, with multiple sampling and few-shot prompting to improve consistency. While the system works, it is slow and requires further optimization for production use.

## Transcript

fun so yeah guess I I'm going to have to wait a bit now
think I lost my my network for like one second and I collapsed uh I'm kind of new to streaming so yeah I'm not 100%
sure uh how to restart it that quick when you lose connection yeah that was a bit annoying uh but yeah
yeah that's has not happened to me before but yeah that was a bit annoying before but yeah that was a bit annoying so yeah sorry about that if you were watching uh but yeah I'm just going to try again uh I'm going to give it a few
minutes if some people that wanted to watch uh wanted to to get back in I'm where we were and uh yeah how far we have
and uh yeah how far we have come so yeah that was a bit like with live streaming Uh something's happen you just
got to react to it so I I'm not so used to this I've been like doing small videos like all the time so uh it's a bit new for me but uh yeah I'm going to try again so yeah just give it a few minutes
and I'm going to do like a quick recap of what we're doing and how far we got of what we're doing and how far we got before the the stream crashed uh and I'm not 100% sure what happened I think just a small outage on
happened I think just a small outage on my Broadband that was probably it uh I'm not kind of so used to like with the software but yeah like I said software but yeah like I said probably a part of streaming live I
guess so just going to give it a couple of more minutes but we were making some progress and I'm not 100% sure where I left it off so I'm do going to do like a left it off so I'm do going to do like a quick
annoying uh but I can see uh a few people have come back so that's nice uh enrio writes would you be interested in creating a project where you use wish in creating a project where you use wish to analyze YouTube programming video and
the GPD will recreate the code from stream uh I guess that could be viable uh of course you will need like very good OCR then uh but you can try it I
guess you just have to capture frames from YouTube right and kind of get get into it uh so it could be viable but yeah let's just do a quick recap and then we go back into where we were so yeah today
we are going to do this IP webcam we were already kind of started run this to some kind of python code we are going to create and we're going to use the GPT 40 API to to analyze the images we capture for
from our video stream we're going to screenshot screenshot some uh yeah some frames from the stream we're going to try to extract the data we want from the images
we want from the images and return it in Json and the idea is that we can display this in real time so we're going to build like a very simple front end UI and we're going to have some kind of back end that is
hopefully doing this autonomous we have come a bit uh uh far in the C code so I'm going to do like a quick recap gp4 has good OCR yeah that's correct pretty good so let's just go a
quick back into the code and I'm going to show you where we were uh I don't know exactly where the stream fell out but uh we have used CV2 to set up
out but uh we have used CV2 to set up our IP webcam and the IP webcam is now displaying a carton with four eggs in my living room so we're going to try to living room so we're going to try to manipulate the eggs and see if we can uh
follow the data in real time when I remove some items I add some items I have some other stuff I want to try this with too and basically yeah we have kind of set up this um so we can capture frames from this URL uh that worked
pretty good we have like a counter here because we ran into the issue that we couldn't really it was not reliable sometimes uh it said we have five eggs sometimes we have four eggs so before I uh got cut
have four eggs so before I uh got cut off we tried to fix that by adding a second prompt so we had this simple text prompt prompt so we had this simple text prompt here uh with a few shot so we have how
many fish are in the image we return it in Json and we do like an example we have an example so we have like a fuse shot prompt we have this replace um just a placeholder here so we can actually do our prompt specific prompt
so how many eggs are there in the image here and we're just going to replace this phrase here and so when we ran that uh I can kind of show you what we got I don't know when the stream cut
know when the stream cut off but now so I'm going to show you what we did to kind of counter
to kind of counter this so let's do one screenshot of the so I'm going to remove this so let's just run this now and this so let's just run this now and we're going to do one
you can see we got grabbed this screenshot from the stream right and here you can see here is our issue it counted five eggs uh so next it counted four
uh so next it counted four eggs right and that is not reliable four eggs right and that is not reliable four eggs so someone suggested we can do instead of doing one screenshot we can maybe do something like
five right so we do five screenshots and then I can add this prompt here to try to correct for that so we're going to feed in those
that so we're going to feed in those five responses eggs three eggs right and then I think I'm just going to try return to response
with the most occurrences in Json so that might work we going to call that correct data I'm going to print it so let's see what happens now I think
there was some somewhere here I lost the stream so now we're going to take like stream so now we're going to take like five screenshots right so we took five screenshots we're going to analyze all of them and return uh in
Json and then we're going to send that to GT4 to try to find the one with most
that's nice so yeah it's a bit slow when we have to do five but I have some idea how we can speed this up by using like low
resolution okay so that was statement is it but that seemed to work yeah that's
our print statement so even though we got three here and the rest was four four we kind of still got uh four eggs output here so of still got uh four eggs output here so let's run it one more time I
right but since we counted the occurrences we kind of got it right
four that was good yeah so that that might work I
[Music] to so let's stop that I think we kind of want to do a few shot prompt on that too so let's just copy
this and let's do uh correct data something like this
we if we are struggling with extracting Json from that we might change that Json from that we might change that up okay so what is the next thing we kind of want to do now uh what I kind of want this to do is
uh it's almost like this uh but my idea was more that we actually get I kind of just wanted to do one screenshot but now if we need five I think actually this might be okay I
think actually this might be okay I think so let's try and
this I don't think we need to append this to this list here right this so I kind of wanted to use a bit of gp4 now to try to instruct what we want
gp4 now to try to instruct what we want to do next right thought that could be this part into our so let's change up the
function let's just call it like uh I don't know fetch don't know fetch fetch data or something like this
let's go back to GT4 and let's try to prompt this so GT4 and let's try to prompt this so let's just my code or let's do let's just my code or let's do it we can call it
data f.p equals so past in my code so what I f.p equals so past in my code so what I kind of wanted to do was
like that hello night spider nice to see you again yeah uh and the stream crashed like 40 minutes in so that was a bit annoying but uh yeah we're going to try to continue where we left off so let's
see if we can instruct gd4 to kind of implement this part uh or maybe all of this into our true Loop in the fetch data function so how I would do this is yeah I would just data form pi equals
data form pi equals right and I want to do something like right and I want to do something like um I want
to get this part so paste into in the code in to my true Loop or is it while loop remember oh it's a while true Loop okay my true Loop in Def
my true Loop in Def fetch data so let's see if we fetch data so let's see if we if if that works
uh markdown in the prompt it's very strange it even does big titles and strange it even does big titles and stuff so let's see here certainly to integrate the video capturing part into fetch data function I will merge into
the fetching Loop okay it's where did the leave the okay so we left this part up here that's
this part up here that's fine okay so we have this inside our our fine okay so we have this inside our our function here
counter okay so let's just copy it and take a look at it inside our um take a look at it inside our um notepad
screenshots let's just follow up with that uh I want wanted to take five uh screenshots uh in each
Loop can you add that in the fetch I no data
here yeah I think we need to do that if we wanted to actually bring fire hello Philip Guin
actually bring fire hello Philip Guin nice to meet just doing something with wish if you got it got it late so we're going to stream we are streaming from an IP webcam we're going to capture some
frames we're going to try to extract some data from this video that is basically what we are doing today so let me see here okay so we do the uh po screenshot in range five so let's try this
now and let's just see if we can get this to
work so the only thing I'm worried about is kind of the the output from corrected data but we can see if we can fix
something so now we are analyzing the these five images right and then we're going to print the most
Curren and yeah I don't think we actually need to remove this part okay so so you can
part okay so so you can see four four we might actually need to do more than five because this is not working right so let's set the range to working right so let's set the range to like
let's stop that uh other than that I think it was that uh other than that I think it was okay so let's do seven but I wanted to okay so let's do seven but I wanted to change something in this analyze image
folder because it's quite slow uh I want to take a look at so slow uh I want to take a look at so let's say we go back to chat gbt let's say we go back to chat gbt right and we just
it we can paste in our code again so data farm. py right equals and I want to data farm. py right equals and I want to go to the open AI API playground and the
documentation and I want to see if we can speed up here so let's go to GT40 let's go to guide and images and let's check out
here there are something about low or high fidelity image understanding so I high fidelity image understanding so I kind of wanted to speed up kind of wanted to speed up the I wanted to try to speed up
part this project will also work well on a local Rasberry P5 yeah could it I have never worked with this Ras B with this Ras B P5 but uh I have been looking into this
P5 but uh I have been looking into this uh security footage analyze part in other videos like a long time ago and it is pretty interesting so I think we is pretty interesting so I think we might see some uh AI infused
might see some uh AI infused into security cameras and stuff if that's even going to happen I don't know and we already have computer vision on the the language model in that right okay so
let's grab some documentation here and we have this so let's do open AI documentation and let's do kind of our instruction I want to speed up
the what is our function called def analyze image in called def analyze image in folder function by use the low setting on the
image analyze part uh can you implement this in my code see if that
works because if we do like the lower resolution who we can maybe speed up the Laten see a bit and we save some tokens let's try that so now you can see we are
that so now you can see we are using did it change [Music]
you implement this I can't see any changes
try it and see if this increases our speed speed right so let's just run this I read a while back about a coffee shop using cameras and visi to track
customers yeah I think I saw that that's pretty crazy and they kind of they had pretty crazy and they kind of they had track if people like are sitting down track if people like are sitting down and how much they are typing and stuff
and how much they are typing and stuff that's pretty nasty speed now but this might now but this might also do that our analyzation is
also do that our analyzation is worse doesn't seem to be quicker does six seven images that a bit that's a bit we wanted seven yeah that's
right okay so we got to right number of eggs at least so I think that's good I think we're just going to keep it like this but what I was a bit worried about
is this output because I don't really want this Json part here uh I
part so let's take a look at our function function here I guess we kind of have to do
I'm only kind of interested in the output from um from the fetch data function right that is kind of what I
pretty fine it's not quick at least it's pretty slow would be curious to see how they use data like more uh CS to customers
that leave fewer emails yeah okay yeah so that didn't work but that's fine we don't that's just our back end
so it doesn't really matter uh yeah I I don't think I would work would like to work in some place where I'm being monitored the whole shift right that would be
would be horrible uh right okay but I think this back end is fine for now we could optimize this
later I I guess but I think we're just going to settle for this so this is going to be our back end and
our corrected data I think we just going to
that and we don't need this uh this list I don't think
here so I kind of wanted to do the let's do some simple front end that the let's do some simple front end that kind of displays our data in real time so let's let's create uh I just going to do
it in Python now I don't want to spend too much time on it let's just create a front and Pi or and Pi or something
attention to this front end Pi just going to do like a new here so Pi just going to do like a new here so let's Farm data Farm uh do pi equals I kind of want
data Farm uh do pi equals I kind of want to give our backend here to GPD instructions okay so we have the Json out here four okay so that's good so I'm
happy with this this is fine okay so that's our back end so let's do some instructions here uh I have my back end completed
the maybe I want to give like an example so let's run it the response we are interested in displaying is the output
from what did I call it correct the it correct the data uh is output from the variable data uh is output from the variable corrected data
right the response we are interested in displaying uh in our front [Music] end named front end named front end do PI is the output from the
end do PI is the output from the yeah this should be in yeah this should be in a Json
response here and stop here and stop this
like I wonder if this something like V for new Google UI maker for python
really done a lot of UI there was someone on the last stream that mentioned some kind of AI tool that cre could generate a UI from your back end I haven't tried that yet so I might look into that but today I think we're just
going to do like a terminal UI uh UI uh yeah uh something like uh yeah
okay all right okay that was do you okay cool I want a terminal
display the number of eggs in the image let's just see where we get with this can you
create front end do PI for me just want to see where this takes us uh we also need a connection to the back end right so let's see if gbt 40 understands
that it needs to make some kind of connection over to the back connection over to the back end just from the off
here but we if this doesn't work we can just create like a few shot prompt and just create like a few shot prompt and we could probably only return scripts all over the place now so trying to yeah me too I I never
to yeah me too I I never do almost any front end there stream late radio and mess up for Rapid front end development yeah I've heard heard of all of those which one is the easiest to kind
of just use out of the box for like a very simple UI I have box for like a very simple UI I have been using react before like no JS or and react that's fine but
it's it's a bit of work okay so let's see here now so we have a suggestion for the front end here so we're going to use curses okay I've used that before but uh okay so it's going to
connect it via a Json file but then it needs to
2 to get the or did I import no well let's try it let's see if it updates the file so let's front and pi
updates the file so let's front and pi stream lit uh I was looking into TTI later trying to run C into web UI terminal okay stream lit stream lit is pretty simple okay I might actually do into
simple okay I might actually do into that and Tropic says they beat GPT 40 with Sona 3.5 now available really that's cool where do you see that
Tim I'm definitely go do a video on if anthropic has a new model
okay here it is yeah 15 minutes ago that's pretty cool introducing claw 35 sonnet our most intelligence model yet and sonnet that's pretty and sonnet that's pretty quick Sonet now outperforms competitor
models on key evaluation twice as speed as twice the speed of claw tropus wow as twice the speed of claw tropus wow that's cool uh one5 of the cost
yeah these benchmarks like they look pretty
dp40 or that's with five shot a bit lower with zero uh Matt it's a bit behind but remember remember it's much
quicker code looks pretty good but like I always have to test it myself right uh can we already do so let's
do uh let's remove this so let's do a here I kind of have the here I kind of have the API yeah 3.5 Sonet okay that's pretty
API yeah 3.5 Sonet okay that's pretty cool so let's give it a go right uh you you are a python Dev
uh software death with expertise in
Python yeah I don't care so where where we
right so let's update this now so this should save this corrected data here yeah right okay so let's clear that and let's right okay so let's clear that and let's open a second
Opus 3.5 that's exciting right I guess the Opus model is going to need more the Opus model is going to need more training time I
it so let's start the back end first let's start uh python front end it's going to be quite slow anyway uh we haven't even generated the anyway uh we haven't even generated the file yet so let's just wait a bit
let's do I have this this question I always use the marble question I need to find it
question I need to find it then can't remember where I put it uh here uh here maybe I know there's somewhere here
here no can't really
I had it like a few I think it's called marble problem few I think it's called marble problem here it
marble problem marble problem
it so this is something all language model struggle with I don't know if model struggle with I don't know if you've seen so let's try it with Sonet so the answer is that the marble
[Music] the on the cup on wasn't change gravity will keep the marble at
Place uh at what is now the top of the inverted cup H that's a bit
yeah would be interesting to Branch your Alan eval into the cold Builder which Alan eval into the cold Builder which son
to test this now we might give it a shot if we have a prompt here now for our project uh but I think we're just going to stay on topic uh but pretty cool I to stay on topic uh but pretty cool I will definitely look more into
this right okay right okay so now we can kind of uh does it work
okay so we have an issue there I didn't see that so let's run it again I was a see that so let's run it again I was a bit sidetracked now during this bit sidetracked now during this release but pretty cool
pretty cheap and I guess you can try it at the Claude AI a I I'm just using the API
here okay so let's see if we had an issue here I want to I think the issue might be I didn't really catch
the Json output but we can try to feed say let's we had this error I kind of want to feed the error back into GPT 4 and do some
debugging but I suspect that is the string uh is not a valid Json string right and this is what I was talking about uh I think we might I was talking about uh I think we might need to do a fot prompt here
[Music] occurrences so let's just do something occurrences so let's just do something like
this I think that should be fine right so let's do a fuse shot prompt right so we want this of
to data C equals we want to open our file so that's going to be correct
data. text uh we need to
St Dracula appreciate it so let's replace what did I call
with I guess we can replace that with chatbot responses something like
need for the few shot example it's going to be something like this
right so we need uh a few of those all right something like this and let's do two four and let's return the most with the most in
cures that's going to be three right and let's just copy this
example example two and let's do number of [Music] return ball balls
uh yeah let's close that and the answer is going to be one that and the answer is going to be one right okay and then we have the user right okay and then we have the user data something like this I think should
work so we're going to feed our responses in here user data right responses in here user data right uh user data and we're going to fit in
those yeah so let's try that I think that should that should work
hopefully because now we have kind of this few shot
wanted to take a look at it after the stream uh as a like a re uh replay but it's going to be in two parts because yeah the first part because yeah the first part crashed hello web learning AI nice to
crashed hello web learning AI nice to meet you welcome so if this doesn't work now we might have to do some more
debugging no no search File corrected data what do they call it correct data okay that was annoying correct data it's pretty slow though
though H and analyzing like seven screenshots that is going to be a bit more expensive actually right
speed we could do smaller do smaller screenshots so we could resize them up front but for now think we just going to leave it like
this if I'm going to do a video on this uh for Sunday uh I might clean it up a bit and kind of do some more optimized stuff
hello Loki nice to see you back again uh okay so we have two arguments must be string not have two arguments must be string not list uh where was that
okay replace ah yeah that's true so we can fix that so let's just paste in this and Let's do let's do some
debugging now so let's say we get this we need to because we we turn um hello Emanuel just turn in to say appreciate the videos yeah that thanks I appreciate that that's pretty cool that's kind of
the whole idea about the channel uh to be inspired by so yeah thanks for that appreciate it so let's say we wanted to uh debug this now because the issue now is that um
this variable uh or is it this variable uh is not a string it's a list right it's a like a Json list I think or
with gd4 so let's just do my code and let's just try to feed in the error
now first time uh my we can do that can you suggest something which can store database and host my
options uh what kind of website is it just like on straight up HTML or is it like WordPress
or Loki okay so let's see what we got as Loki okay so let's see what we got as respon who at response
it's not a list and the replace metal expects a string to fix this you can join the elements into a string yeah before using the replace method we could have done an F string too I guess maybe that would have been
easier but I kind of hate these long few shots inside my code so let's see what shots inside my code so let's see what gbd4 suggests
join the list chatut responses into a string before using replace God I can't see it let's just copy it and take a look at it so small
here chat responses okay so we have an output directory yeah yeah here we can see so here we are going to join this right
here we are going to join this right that was a pretty good uh answer I think that was a pretty good uh answer I think so let's see if it works
uh okay we can do like a quick recap we have time for that let's just see if this work and I can do like a quick recap if you just got here uh what video card are you using so Fritz I'm using the Nvidia
vram uh it's fine uh it's pretty good but like if you kind of want to run bigger local llms maybe like 24 and up is
we're building now before I continue so let's see if uh joining this so now we kind of got uh joining this so now we kind of got other error here that's interesting
with the Json format again but uh let's ask gp4 to also take again but uh let's ask gp4 to also take a look at this issue while we do a quick a look at this issue while we do a quick recap so this is what we are looking
at today basically we have an IP webcam I can show you here that is looking at some kind of items so this is streaming
uh on my local uh web here or like an IP local uh web here or like an IP address so we are streaming video uh from this webcam they have this python code that's going to take some frames or
code that's going to take some frames or screenshots from this stream use the gp4 API yeah we have some screenshot capture we're going to try to extract some data in this case we're going to count how many eggs are in the image and we want
to display this in real time and the idea is that we can kind of change the idea is that we can kind of change the source of data all the time and be like Dynamic so that is basically an ID and
haven't prepared too much so yeah like you see we run into different errors and the idea is that we're going to try to use gd4 mostly or we could try an Tropic
to debug this so let's say if this doesn't work this so let's say if this doesn't work now let's let's try to use anthropic to now let's let's try to use anthropic to do some
debugging so here we are just uh we wanted to do some um some print statements here if we get any errors so that could be good for debugging that could be good for debugging right I also kind of wanted to print
yeah here's the debug print I kind of want to see if this is in correct Json want to see if this is in correct Json format
seconds so the issue we had if you just got here is sometimes it return that it was two eggs three eggs so we tried to make like a prompt to only count to return the occurrences
with the most let's say we had four answers with three eggs and the rest was maybe one and five we only want to return the one and five we only want to return the one with the four eggs most
right so did we actually create a Json file now uh nope
but let's respond with this response I have the same project with weather station with two webam oh cool I'm using the Laval fight3 model yeah that's fast but how is uh what are you
exactly are you extracting data or just like the are you extracting data or just like the weather or do you also get like weather or do you also get like um data from uh in another format you or
are you just using wish say St Dracula the issue is that the response contains markdown formatting yeah I know
to fix this you need to strip okay so we can also do that so that okay so we can also do that so that means that the fuse shot prompt did not means that the fuse shot prompt did not work I kind of thought it
did no you can see we still have mark down function to strip this or just a strip command so let's
this or just a strip command so let's see now
hello Aya what if the frame is captured immediately after user ask uh what if the frame is captured immediately after user ask uh through voice and this will be processed Contin until the session
ends immediately after the user ask is cre through yeah you could I guess you could do that could be
interesting uh I take the data from esp32 C3 I use the respir with two webcams and I give it an API in real time okay that's time okay that's cool but you are take you you do you get
cool but you are take you you do you get data only um not just only from Vision right some other source that feed you temperature and stuff humidity
okay so we're just going to do like a simple strip here and see if that
um conference and I had like a small speech speech about yeah wish and stuff and there was one guy there who kind of did the same as you he had this weather station and
he kind of compiles a lot of data and he Compares it