Rare Data Farming with GPT-4o Vision – Plus Runway Gen 3 and Microsoft Florence

[00:13] okay so hopefully I'm alive I can see I have some errors with my chat so have some errors with my chat so hopefully that's going to be hopefully that's going to be fixed um let me try

[00:47] issues uh might be fixed now I'm going to try to type and on YouTube yeah so I think that should be good so let's try

[01:02] should be good so let's try that so if you join just let me know in that so if you join just let me know in the chat so I can see everything is good so we're just going to wait a few minutes before we kind of explain what

[01:17] minutes before we kind of explain what we're going to do today so I think it's going to be pretty cool I've done something similar before but not with kind of the new new wish model so we did this with

[01:34] model so we did this with um hello facts nice to see you and working that's great so yeah you what you see on the screen now we're going to go through it

[01:46] but it's basically this is kind of going to be the main thing today we are going to be the main thing today we are going to try

[01:58] capture some kind of of stream to try to capture some real life real time data we're going to do different sources we're going to see kind of how we can do this hello

[02:11] this hello Henrik nice to meet you great to see you here so we're just going to wait a few minutes and before we get started I also

[02:23] minutes and before we get started I also did take a look at this um Microsoft Florence this new wish model we might take a quick look at that and we I saw

[02:35] take a quick look at that and we I saw grock has done this uh text speech to text I think it's just some kind of whisper so we might take a quick look at whisper so we might take a quick look at that hello om nice to meet you thanks

[02:49] for tuning in so yeah like I said we just going to give it some time if people want to see uh it's the thing is like if you you do these streams and you're going to try to build something in real time it's

[03:04] kind of hard for people just if you are a bit late then you don't know what's going on right even though I'm going to take it pretty slow we're going to use some code we have used before we're going to try to

[03:17] write some new code using gd4 we're going to look at some maybe some front end solutions for this system to kind of display the data in real time

[03:29] uh I think we're going to use Jon to try to parse it uh yeah I haven't really decided yet okay so Omni you

[03:41] right yeah tried Florence today on confi great and fast the more detailed part is really nice okay yeah I will try to look into

[03:53] nice okay yeah I will try to look into that because I want to test out the that because I want to test out the OCR have you try that can do some OCR maybe on some websites some other

[04:07] stuff to see how that compares kind of to GPD 40 I think the OCR for GPD 40 is great but uh would be nice to have a second option right I did try this other

[04:19] models this Laval fire Laval fire vision and yeah that didn't work for OCR for me at least so I tried a few different ones but I

[04:32] it even is the weights available open even is the weights available open source hello Kola house kapus kapus nice

[04:45] to meet you so yeah uh like I said I have tried these open source models to do some OCR but but um don't have had any good luck yet so

[04:57] that's going to be interesting to see so let's give it like yeah one more minute or something before I go into what we're going to do today so we can wait like five six minutes I think that's enough and then I

[05:12] can do like a recap when we kind of get into it a bit so the reason I put down rare data farming so the idea is that this system

[05:24] is not kind of bound to any data the idea is that it's going to with Dynamic and we can kind of specify to farm what kind of data we want using wish right okay so yes at the moment playing

[05:39] around with it first look is very impressive okay that's cool yeah I want to try out the OCR part at least but do you know if the model is available like open or is it uh I saw it was in hugging face that's

[05:55] where I saw it was in spaces but I haven't looked into if it's

[06:10] too but yeah I'm really hoping for like a new wish model we can start doing some reliable OCR it I think that's going to be pretty

[06:25] cool I don't we might actually test this system on some text too but the idea here is not to use it on text it's more like uh other kind of

[06:37] text it's more like uh other kind of data uh so yeah I'm going to show you soon here now so yeah let's give like 30 seconds and I can kind of explain here because what I wanted to try today is I wanted

[06:50] to try to do like a when I do this on live stream I wanted to try to see if I can edit the live stream for a video I see other people

[07:02] want to try it as I understand it's MIT license and free to use okay cool yeah I'm going to check that

[07:15] out so today's stream uh it's going to maximum be a couple of hours because yeah I kind of want to watch the England football game today and yeah

[07:28] the England football game today and yeah I think that's around 6 o' for my I think that's around 6 o' for my time Hello davidas nice to see you I think I saw you last stream too so that's cool welcome

[07:40] back okay so I think we're just going to get into it and kind of explain what I wanted to do here today so I have this IP webcam I have used it before it's basically just an Android phone with an app that just

[07:54] streams video we can do it on the IP address uh on my local network here and that is kind of going to be our source uh for the source uh for the video we just going to use CV2 in Python

[08:09] video we just going to use CV2 in Python to kind of extract frames from that stream uh that's the only thing I have prepared and then we're going to use GPT 40 to do some analyze we're going to analyze the frames from the

[08:27] stream and yeah basic basically we're going to analyze the screenshots that is captured by CV2 from our IP webcam and the idea is that we can extract data so we're going to extract only the

[08:42] data we want and we can do it in real time so I want to try to display the data update in real time that is basically the whole

[08:55] concept and yeah I think it's going to work pretty good uh that don't share code that don't share

[09:12] code yeah I will do it I will now you can replicate programming projects yeah I'm I'm going to share the code for members of

[09:24] code for members of course uh but that might be Sunday I think so yeah uh I think that's there's nothing more to say I think we're just going to get into it so uh okay that was something else oh my

[09:39] so uh okay that was something else oh my keyboard here we go so basically you can keyboard here we go so basically you can see uh this is the webcam so if we open this so this is kind of the webcam we have now so the

[09:54] these eggs uh it looks like a image but it is actually streaming um video right and we have this IP

[10:06] address we have this port and we can kind of tap into this stream right so that is basically the idea it's just a simple Android simple Android app so what I prepared was this kind of

[10:18] simple python code here we can go over it if you wanted to so we are using CV2 it if you wanted to so we are using CV2 here to just tap into this IP address here because we going to set our interval for 5

[10:32] seconds and we're just going to create a [Music] [Music] folder uh IMG image to save the screenshots and we're using this to capture from the stream and yeah it's

[10:46] can see we are going to save it to screenshot and just going to count each Loop for plus one so the

[10:59] count each Loop for plus one so the screenshot is going to be 0 1 2 3 right just going to print that add one count to our count and we don't have any limit here but we just have an interval so we're going to stop it by pressing

[11:14] keyboard interrupt small uh sleep here we can adjust 5 Seconds that is basically how we're going to use this so let's just test it going to use this so let's just test it now it should be working I think I think

[11:28] now it should be working I think I think I tested it here what did I say this yeah I think it's it's gp4 let's we can we can see this now

[11:43] while we are running this so what you call it python data Farm call it python data Farm okay so yeah here you can see the screenshots we are fetching right so that's

[11:58] that's zero uh uh one yeah you can see the screenshots are coming in here every 5 seconds so you can see uh if I go over to the if I bring up this webcam

[12:14] the if I bring up this webcam here so the idea was that let me I'll be here so the idea was that let me I'll be right back

[12:36] can change up the number of eggs and we kind of want to display that in real right and we got we have some other sources of data we want to try to collect so the idea is that now we have screenshots of two eggs right but you

[12:55] there's a delay okay so here we got the two so we might look into that I might just take a screenshot every single Loop so that is kind of the idea right let's just stop that now so this is working

[13:10] just stop that now so this is working fine so the next part is to kind of

[13:22] integrate integrate um GPT 40 into this so I don't have a plan per se here this so I don't have a plan per se here now I don't think think so but uh I now I don't think think so but uh I found an older code we

[13:35] found an older code we used so this is actually I think it was from a previous project so I think we just going to grab project so I think we just going to grab something from

[13:50] to grab some for so let's just grab all of this I think and let's see what we want to reuse here we're going to go through it so let me just grab this for now I think we're

[14:06] going to put it up here now all the functions right we're going to have to grab uh I think we're going to [Music] do [Music]

[14:21] [Music] from dot EnV I can't really remember

[14:33] that's cool uh could you send me an email and um I can invite you to the email and um I can invite you to the GitHub and

[14:52] nice so let me just check something here I just wanted to bring up so we do this I just wanted to bring up so we do this the first time right so I just want to load my API key here in

[15:10] um hello TKS nice to meet you yeah because I don't want to share you yeah because I don't want to share my my let's just grab our client here uh let's do the Open Eye client

[15:27] here uh let's do the Open Eye client where is that

[15:39] Cent yeah I think it's fine so let's just grab

[15:53] something like this and I think I kind of of prepared this so that should be fine prepared this so that should be fine think that's good we got

[16:09] send me a mail now I saw I got an email okay so basically what we want to do here is that we yeah I have this EnV file with my open key so I'm not going to show that this should be good and here we have

[16:26] good and here we have uh a function to to en code to Bas 64 happy to join your member Community we be a great adventure yeah thanks for that uh really happy if you want to support me that way that's superb so

[16:42] support me that way that's superb so awesome welcome by the way and encode image to Bas 6 Foria we have used that before and here is kind of the function we want to analyze so we just want to take our folder right and we want to

[16:55] take a a prompt here to kind use the GT40 model to kind of analyze images in our folder so I think I just want to kep one image at a time so each time we go through our Loop we want to delete image and we want to

[17:10] load a new one so I don't think we want to Stack Up images we could do that too but let's just start we can start with that and then we can move into a loop I think but this should pretty work pretty good we just have to set up folder path

[17:23] good we just have to set up folder path right uh what was our folder pad so it's do get good so the reasons um I'm using uh

[17:38] get good so the reasons um I'm using uh note uh book Plus+ it's just for the video if uh I was just doing it myself I would use the vs code but I I think it's nice for the stream to use this so nir where you can find your

[17:51] email you can just go to the YouTube page and go to about or if you sign up as a member you should see my email there too uh if you can't find it just let me know and I will type it in the chat

[18:06] here okay so we call this GPD for our data and we call our folder for image data and we call our folder for image right uh okay

[18:24] so let's not have a loop here now first I guess we could so the user input we I guess we could so the user input we don't need a user input we need like a data input or something uh I don't think need to print

[18:38] that we don't need that but we do need a prompt right so we're going to need our folder path in our function here and we're going to need our

[18:50] prompt let's not call it to yes let call it AI data robot I don't care hello Muhammad nice to meet you uh so yeah if you just came into the

[19:05] stream uh I'm going to explain like instrumentally like what we are like instrumentally like what we are doing so if you kind of want to follow along just join what's going on yeah that was perfect timing by me uh so what

[19:19] we are doing we are creating just a quick summary we are creating a we are using a webcam you can see here if if you just got here and we're going to count in this case we're just going

[19:32] to count eggs so this is streaming live video so basically we're going to try to manipulate data extracting data from live video so we can display uh data in real time uh I going do a bit of a more explanation when we are a bit further

[19:47] along in the code here but I will try to explain explain right uh yeah so we need a prompt here right uh yeah so we need a prompt here right and yeah we could keep

[20:00] this uh this is going to have to go all the way down the way down here this is going to go up on top I guess we can move that on top I think it's

[20:15] it's fine like this and yeah that should be fine like this and yeah that should be pretty think uh let's just uh let's call it something like data I don't know prompt

[20:34] equals and we can do a file so let's do this in a text file because I want to do this in a text file because I want to do some few shots let's call it I don't know uh data

[20:49] uh data Json that's fine the text so let's create a text file because I thought we can do some uh what did I call it data Json right okay so

[21:07] data Json that's fine so I kind of wanted to do some few shot examples here so let's just do examp example one that's going to be [Music]

[21:21] equals so what kind of question so let's say how uh many fish are there in the image so that could be a question a

[21:34] prompt right so let's do return response only response only as Json or only in

[21:49] as Json or only in Json format right for Json format right for more format okay

[22:01] so let's do let's do I don't know um what should we call this to number of fish fish

[22:19] number of fish fish fish something like this and let's do three so that is one example so let's do example two uh so we're going to do like a few F shot how many balls are there in the

[22:35] shot how many balls are there in the cup and let's only return this in [Music] Json so yeah I just wanted to do Json so yeah I just wanted to do um fuse shot prompt here I

[22:48] think so yeah I don't know something like like this balls and let's this balls and let's do 11 I don't know and then we kind of

[23:00] can do we don't need like example but let's let's do we can do like

[23:14] query all right um we can do this right um we can do this um yeah I think that's good okay so data

[23:26] um yeah I think that's good okay so data Json so okay so we're going to open this file uh we kind of want to replace it

[23:39] uh we kind of want to replace it with um a

[24:01] going to be how many eggs are there in the image right so we're going to put this into our prompt here and that

[24:17] this into our prompt here and that should be fine okay data prompt data input prompt data input so that is going to be data input press

[24:32] that here okay and yeah something like this we could just let's keep it like this that's

[24:54] we have our folder path we have our prompt

[25:06] append it for now but we probably don't need to probably don't need to right I should see what

[25:24] if you just got here H I'm going to do like a recap of what we're doing like a recap of what we're doing let's just try it for

[25:36] Define okay import I feel I always change this so that should be

[25:57] eggs okay we don't we yeah that's true so this is we need to skip this we don't need a true Loop

[26:19] prompt example what you expect this answer uh if example of what you will expect as answer yeah did you mean like

[26:32] answer yeah did you mean like the the fuse shot prompt yeah this is the the fuse shot prompt yeah this is just to try to improve the chances that we will get this Jon response right if that was you meant uh so

[26:47] right if that was you meant uh so basically a few few shot yeah I always try to do that I think that's just a good practice if you want that's just a good practice if you want to do this

[27:11] try so I don't even think we let's just try one of these

[27:26] gd4 now let's say so how kind of I would prompt this to so how kind of I would prompt this to gp4 now if uh I wanted to kind of remove this uh true Loop here because you saw here now when we run this we got stuck

[27:40] of course here because we were in this true liop here right we could do like a a counter to just do five frames but uh let's try to just uh

[27:54] frames but uh let's try to just uh prompt GPT 4 here so we can maybe see how I I would do that so let's just copy our code we can go to gp4 right and we can do yeah I will do that tomorrow uh I

[28:07] will upload this to the members skab of course so let's say we wanted to prompt this to GT4 now to kind of remove that true Loop right so we could do paste in our code and we can go maybe like I don't know

[28:25] response we can give like an examp example here so you can example here so you can see

[28:39] script or something got stuck in the screenshot CV2 true Loop right or something like that um I

[28:55] that um I only want to we can do like a counter maybe take three let's do one screenshot

[29:09] screenshot before uh we move out of the before uh we move out of the loop can you fix this let's see if that loop can you fix this let's see if that helps

[29:24] strange this prompting now if we do code snippers we get that mark down in our prompt it's a bit strange so let's see what GT4 is spond here to modify scripts uh takes you need to break out of the loop by after the

[29:40] first screenshot right okay so let's see what it suggest actually go into a loop at all now we just

[29:52] do uh while uh yeah I guess we Ed to this while loop and if the count is this while loop and if the count is uh less than one uh and then we're going uh less than one uh and then we're going to break out right so that should

[30:10] work so that's fine so we can copy that let's test it just want to see if we let's test it just want to see if we yeah so yeah now we have this

[30:24] true loop on the user inut not so let's see now Fabian stuck here right of course but we going to we're going to fix that later so I

[30:38] thought I can just do it like uh step by step so let's see if we only do one step so let's see if we only do one screenshot now so we got some color errors pink yeah you can see I had

[30:53] this we had this colors I always like to bring my colors but uh let's change that up here we can do this we can just do cion

[31:07] instead I don't care so like this okay okay so now we only did one screenshot

[31:24] screenshots I forgot to delete the old ones but let's just see here now if we get a response so we kind of want this in Json response so we kind of want this in Json right it's a bit slow though okay that's

[31:37] good so it analyzed all images so let's interrupt this now but we got our response here in Json right we got two eggs two eggs two eggs so let's just

[31:49] remove these screenshots let's clear this so remember we got two eggs so let me go put a couple of more eggs in

[32:01] couple of more eggs in our in our carton

[32:33] so remember now there should be four eggs in the carton issue right and okay so now we got four so

[32:45] that is not good that means that this is not going to be perfect but I kind of want to see how precise it could be so we got five four

[32:58] four wonder why it gave five that's a bit strange so now we got three so that means that this is probably not going to work but let's just build out the code anyway and um maybe we can do some other

[33:15] data so is there I don't think I can actually prompt this any better if you have any idea how we can better if you have any idea how we can prompt this to kind of

[33:29] prompt this to kind of so we we always want four here so we we always want four here right but we are getting like four four right but we are getting like four four five uh the output is

[33:46] bad we can try that so let's increase the screenshot uh so let's do if let's do five

[34:09] uh if you were thinking about the output that's uh the fuse shot I kind of want this in Json right so what we could do is we could do like

[34:21] so what we could do is we could do like let's capture

[34:53] welcome it's great to hear could you send me an email so I can invite you to send me an email so I can invite you to the GitHub and Discord that be great yeah what I was thinking is that we you can see the answer was 4 eggs

[35:07] right from our stream but what if we take this stream but what if we take this output so if we go to our prompt output so if we go to our prompt here and we do this so let's create a

[35:22] prompt uh data correction something like this we can do like an F string maybe and let's do

[35:36] maybe and let's do res

[35:49] data uh response I can't really come up a bit anything here and let's do like uh so let's kind of feed in our Jas on here so we can do uh chat responses

[36:03] do uh chat responses right and let's do like a new line we can do a new line here and we can a new line new line un

[36:22] GitHub reposters going see the code as well yeah if you are be if if you became a member just send me an email if you can member of the channel when you sign up you can kind of see the email address and yeah I will of course add you okay

[36:36] so yeah we're going to feed in the response here so that is basically going be here right so let's just try something

[36:48] here right so let's just try something simple let's

[37:03] return the response with the uh kind of suck at English when it comes to this uh

[37:22] framings uh what do you call it when it's the high as count or something it's the high as count or something so I want to I want the the prompt to be

[37:34] so I want to I want the the prompt to be like uh the most occurrences is it something this

[37:48] anwers I'm not so good at in English like in this case Turnal response with the most occurrences in Json

[38:03] uh yeah we can try this I'm not 100% sure but uh and let's print that so sure but uh and let's print that so let's

[38:21] print uh corrected data equals oh what do we call a

[38:39] function analyze images but we don't really need