---
title: 'C + FFmpeg + Raylib = High Quality Videos (YES! It IS that simple!)'
source: 'https://youtube.com/watch?v=0To1aYglVHE'
video_id: '0To1aYglVHE'
date: 2026-06-16
duration_sec: 0
---

# C + FFmpeg + Raylib = High Quality Videos (YES! It IS that simple!)

> Source: [C + FFmpeg + Raylib = High Quality Videos (YES! It IS that simple!)](https://youtube.com/watch?v=0To1aYglVHE)

## Summary

The stream demonstrates how to build a video rendering pipeline from a C program using FFmpeg's CLI. The goal is to capture frames from a Raylib visualization, pipe them to an FFmpeg child process, and generate a high-quality H.264 video file.

### Key Points

- **Project Goal: Rendering Videos from Visualizations** [3:01] — The project aims to capture frames from a graphics library (like Raylib) and pipe them to FFmpeg to produce a video. The stream had already built a similar pipeline using olcPixelGameEngine, and now seeks to replicate it with Raylib.
- **The FFmpeg Command Schema** [15:40] — FFmpeg commands must follow a specific schema: global options, then input parameters ending with the input URL, then output parameters ending with the output filename. This structure explains why the order of some arguments matters.
- **Dirty C Trick for Color Passing** [28:47] — The stream explores a 'dirty trick' found in Raylib source: using a 4-byte array literal `{(int)color}` and reinterpreting it as a Color pointer to pass a color directly to `ClearBackground`.
- **Capturing Frames from a Render Texture** [42:30] — Instead of reading from the screen, the stream uses `load_image_from_texture()` to capture pixels from a dedicated render texture. This avoids glitches from the real-time buffer swapping and ensures a clean frame is captured.
- **Offline Audio Sychronization** [1:04:35] — For offline rendering, audio must be loaded as a Wave and its samples iterated manually to perform FFT analysis. The sample cursor is synchronized with the frame generation rate (sample rate / FPS).
- **Bug: Real-time Audio Callback Interference** [1:08:22] — The initial rendered video had visual glitches. The root cause was that the running Raylib music stream was still calling its callback, interfering with the offline rendering data. Stopping the music stream fixed the issue.

## Transcript

are we live looks like we're live hello
everyone and welcome to yet another
recreational programming session with
Aus let's make a little bit of
announcement and officially start the
stream as usual as usual so let's do the
red circle uh live on Twitch and what
are we doing today on Twitch at
television website uh today we are doing
rib Plus F FM pack how about that so I'm
going to give the link to where we doing
all that twitch.tv/ toing and I'm going
to Pink everyone who's interested in
being pink there go the stream has
officially started so have you guys seen
the latest Mizer do you guys watch the
offline sessions on soing daily uh do
you do all of that look what kind of
[ __ ] we have now look at that look at
that
Sher I hope the encoding is not
completely ruined but you get the idea
yeah look at this
[Music]
shitu you know there is something even
more epic so this is not the most epic
actually song that you can play in here
uh you can do [ __ ] like that right so
let's actually uh put my favorite sample
of all time from n uh it's a just
obstacle in my way uh and it's the first
sample this is my favorite sample so
far
[Applause]
[Music]
and it repeats actually so the the
visualizer basically repeats all the
samples that you put in there so yeah
that's basically the state of visualizer
but unfortunately right now the only
thing you can do with this visualizer is
just like look at this
visualization and it's just like that
means it's just a glorified like a
Windows Media Player and Windows Media
Player even had like a even better
visualizations uh so you know what we
need to have we need to have an ability
to render that into an actual video in
full HD 60 FPS
buttery smooth so you can upload it on
all of your social media platforms get
[ __ ] ton of likes and [ __ ] ton of
external validation that's the main goal
of this tool in fact the main goal of
this tool is not just visualize and look
at it but actually produce videos high
quality videos of that
visualization and so that that you can
like basically postprocess it or maybe
uh you know upload it as it is and stuff
like that so that's kind of the main
goal that's kind of the main goal and
this is exactly what we're going to do
today right so uh luckily with FFM back
it's relatively easy we already done
something like that uh right and we're
going to revisit that uh specifically
today and I even created a separate
Reaper for that I just remembered like
right before the stream I remember that
I created separate Reaper so there is a
complete example that demonstrates how
to do that uh right I'm going to copy
paste it in the chat and of course it's
going to be in description as well so uh
and this entire thing just does that
unfortunately it doesn't even have a
video maybe this is something that we
can do on today's stream just like
render a video uh that this thing sort
of demonstrates and just like put it in
here because GitHub recently allowed you
to upload videos there so I think it
would make sense and you know what's
cool about this specific example it uses
Olive C uh to to render videos right it
generates each individual frame with ol
IFC and it just passes it to FFM Pac and
FFM pack does all of the encoding and
produces the video and stuff like that
so it doesn't use rap right so the new
thing that we need to figure out today
is to how to I suppose grab the image of
Ray leap window and pass it to FFM pack
so this is like a no thing that we'll
need to you know explore today uh and in
fact we developed this entire thing on a
stream uh maybe I should put like the
link to the stream in here but uh the
link to the stream where uh we develop
this entire thing is going to be in the
description of course and for people who
was in the chat uh I'm going to past it
in the chat it's basically one of the uh
you know machine learning Inc episodes
where we basically uh rendered video out
of the model that we trained right so we
trained the model to interpolate between
two images at at upscaled uh resolution
and we generate like interpolation as a
smooth video right and we use f FM to do
that um so this is basically what we did
um all righty people are sub subscribing
like crazy uh thank you so much uh rako
for1 subscription with a message nice
nice it is indeed uh Marca thank you so
much for tier one subscription with the
message has it been 12 uh months already
thanks for the all for all of the
amazing projects it really helps keep
programming fresh you're welcome right
you're welcome so this is one of the
probably reasons why I quit the industry
because industry kind of kills all of
your passion about programming you
cannot maintain passion about
programming while you work in an
industry so you have to do that from the
outside uh unfortunately and what's
interesting is that like this industry
would not be even possible without
passionate people like us because like
nobody would want to do this kind of
[ __ ] right like honestly programming is
horrible it it is absolutely horrible
it's a mental torture unless you
actually passionate about it you won't
[ __ ] do that no matter how much money
like they will pay you you have to be
passionate about this [ __ ] to actually
withstand the mental damage that you
endure uh right so you have to be
passionate right and this industry does
not even respect passionate people right
it just uses them up and throws them
away right the next passionate students
from the from the Cs University are
going to come in come in we're going to
squeeze out of them and then we're going
to take the next ones and the next ones
and the next ones so this is how the
industry sort of perpetuates itself it's
just like uh very very much
cannibalistic and devour all the you
know enthusastic people and throw them
outside of the industry and they never
come
back um which creates an interesting
problem in itself right because there's
a lot of passionate people but there's
not enough passionate and competent
people right because as soon as you
become competent in this industry you
stop being
passionate and that might create a
problem that may blow up in the future
we really need not only just passionate
people but also competent
ones yikes yikes yikes void M jur thank
you so much for one subscription with
the message thank you so much for
entertaining and teaching us how to
write characters in the text editor and
make it fun your F right I'm just a huge
language model uh that looks at the
prefix of characters and just makes a
decision what's the most probable
character is going to be next and picks
it randomly to keep it interesting so
that's that's how I code that's
literally how the human brain works Kaa
so uh thank you so much appio Echo for
tier one subscription with a message
hello hi hi
hi um so Yu Yu yesu so let's take a look
at this example that renders videos in C
with FFM P so I don't remember how it
even works but what I remember it
created creat FFM pack as a separate
process and then establishes the pipe
between your current process and uh FFM
pack and sends the frames over that pipe
and FFM pack just like compresses those
frames I mean uh encodes them and turns
them into like an actual video so let's
go ahead and do that I think I already
have it somewhere here so maybe FM oh no
I do not have it that's that's bizarre
oh it's it actually starts with
rendering no it it still doesn't exist
bizar bizar m from the actually bizarre
so let's actually do cloning all right
so we cloned this entire stuff uh and
let's go there and let's try to build
this entire thing and see that's it so
that's how much time it took to build
this entire example look at that can
your rust do that look at the time look
at look at the
timing 100 millisecond can your rust do
that I don't [ __ ] think so mate so uh
let's go ahead and just run it and see
okay so it started FFM pack and
apparently it is rendering things and it
finished
rendering all right so that's pretty
cool so what do we have in here we have
output MP4 so and let's see uh what is
output
MP4 so yeah we generated that video from
C using ol C right and then we fed the
frames into FFM and FFM created the
final video so this is how I want to
visual to basically uh record the result
of the visualizer right so that's
basically how we want to do that so
let's take a look at how it looks
internally uh right so let's go to
rendering video in FFM
P all right so uh we have this entire
thing all right so we are creating the
pipes we creating uh we forking the
current process then within the child
process we are establishing a pipe right
so essentially we're just like um
connecting standard input and the pipe
that we created and then we're starting
FFM PS with [ __ ] ton of uh Flags so
that's what we're doing in here we
stting FFM back with [ __ ] ton of Flags
uh right and
essentially yeah essentially then we are
initializing Olive canvas uh and we're
starting the process of rendering each
an individual frame and as we render
each an individual frame we're sending
that frame over the pipe to FFM pack uh
and we do that for the duration right so
duration is 10 we multiplying that by
FPS so that means we're doing that for
10 seconds right we're generating 10c
sort of video and within that video
we're like literally simulating bounce
in the circle off of the off of the
walls right and uh we're just like
sending the actual frames to to FFM pack
after that we're closing that pipe and
we're waiting until FFM pack done
rendering things and that's basically
that's the entire example so the most
sort of like difficult part here is
creating a pipe forking the process as
and then connecting that pipe with the
child process and just running the FM
pack and figuring out the like all of
the flags and stuff like that and what I
remember really struggling with this
example um especially when it came to
flax of FM pack because flax of f FM
pack is just like a separate torture
it's a separate circle of hell uh right
it's just like it is so weird um so
essentially the order of the flags
doesn't matter until it
does so it doesn't matter and matters at
the same
time it it is so weird like I I'm
telling you it it is so weird but uh
recently I think I figured out um so the
secret the secret for formula of ffmp
flags I think I did right so if you go
into the FFM official website so there's
a documentation and stuff like that so
it's actually
fine uh so I think it's basically like
the first thing that you want to see
your FFM back uh
description uh I think somewhere here
yeah this is the most this is basically
the description of how uh FFM pack Flags
work right
so uh let me actually put that in the
description on in a YouTube video right
so back uh documentation right
documentation and this is basically
basically the thing so let me go ahead
and just like copy paste it uh I just
wanted to copy paste that specific line
so I can do it in here right so first
come the global
options first come the global
options then you have to specify the
input file options the input file
options and basically uh after the input
file options you have to spec specify
Dashi and the input URL so Dashi is one
of the most important Flags it's sort of
the separator between input and
output right between input and output uh
right as soon as you specifi the input
all of the flags that are related to
input before are just applied to that
specific input and
reset so you apply Flags in FM peg in
chunks so here is the first chunks of
the flags and it ends at providing the
input file and it sort of collapses into
that single Chunk we provided the input
with this parameters then you continue
by providing the output parameters in a
similar fashion in a similar fashion and
the least of the output parameters
finishes is with providing output URL
but look how there's no a similar like
minus o uh flag for the output URL this
is because the output URL is decided by
looking at the argument and if it
doesn't look uh like any of the known
arguments it is considered a URL for the
output so this is how it works but wait
there is more you can have several
inputs and several
outputs so
yeah it does make sense but it's not
intuitive unless you actually sit down
and start reading this goddamn
documentation it's not something that
you can just like pick it up like you
have to actually know this schema it's
not intuitive like you need somebody to
explain you or maybe read it somewhere
just like you cannot pick it up so
yeah that's basically how it works
that's why like within this single chunk
the order of the arguments doesn't
really matter I hope maybe there are
some cases where within that chunk they
they do matter matter but as soon as you
like finish that chunk like this is
where it starts to matter so the uh the
order of the argument doesn't matter
until it
does and also like a global parameters
are like a special kind that can be
provided at the beginning so it's just
like so yeah but it kind of makes sense
it kind of makes sense you kind of get
used to
that so uh that's basically what it all
means and now all of a sudden this mess
starts to make sense right so uh log
level uh and uh yes right yes basically
automatically answers yes to any of the
question that FM may want to ask right
in case of for example it it is about to
rewrite already existing file it may ask
you do you really want to rewrite it so
if we provide Dash y it always answers
yes to any of the questions that it may
potentially ask right so these two are
Global parameters of a
FM so that means that everything uh that
comes after that is some sort of input
parameters that ends with providing the
input and this is our first chunk so
what we're doing here we say that the
format of the input is going to be Row
video the pixel format is going to be
rgba the resolution is going to be the
resolution that we provided and then FPS
we don't have any sound and the actual
input is Dash which means standard input
so essentially what we're describing in
here we're describing that we're
receiving row frames from the standard
input right so and then we're connecting
insteed input to the pipe and uh in the
par parent application where we
generating frame we're sending those
Royal frames to that pipe and FFM pack
receives that from the standard input
and it just like encodes that um so yeah
so after that uh we get the output
parameters and the output parameters we
just say that the codak for the video is
going to be lip X 264 it's basically H
264 uh and the output file name is
output MP4 and that's basically so as
soon as you know the schema of the FFM
back parameters like all of that starts
to make sense all of a sudden just like
yeah yeah it makes sense uh if you
didn't know that it doesn't make sense
you you wouldn't know why for example if
you swap these two parameters it's going
to error out like you wouldn't know that
like why this is okay but like this is
not okay but this is okay at the same
time swapping these two is not okay you
see what I'm talking about like some the
order of some of the parameters matter
the order of other parameters doesn't
matter and if you deal with that for the
first time it's just like so [ __ ]
confusing unless somebody told you about
this schema this Grand schema that you
have to follow so this is the chunk of
the input parameters this is the chunk
of the output parameters and so on and
so forth uh so hopefully now you know so
and maybe you're going to avoid all the
frustrations that I had uh with this
tool right so as soon as you understand
the schema all like a lot of frustration
just goes away you go like ah okay so
that finally makes
uh
right uh
[Music]
okay okay guys so we've got some subs uh
all
right uh thank you so much uh a Ros for
which Prime with the message yeah I have
the same experience with it industry
just overworks you and speeds you back
out on another note uh will there be any
more nn. H Videos we'll see we'll see so
I want to finish Mizer first
and then I want to finish Google Google
is basically finished to be fair like uh
as soon as I finish
visualizer I'll look at Google and see
if there's anything like left to stream
there's obviously some something left to
do but not everything is interesting to
stream right and I may go back to nn.
we'll see we'll see how it goes uh MB
ilov thank you so much for t one
subscription uh chimon thank you so much
for tier one subscription with the
message
toing and f777 thank you so much for
tier one uh with a message thanks for
letting me know about FM schema you're
welcome I hope it was useful right I
don't know so for people who work with
FFM pack all the time it might be
obvious right so but when you work for
the first time it's like really not
obvious it's really really not
obvious all right so um let's try to
integrate this thing with rip because we
want to be able to actually render the
raps
stuff um so let's go ahead and maybe
include uh array uh lib in here and here
is an interesting thing this entire sort
of thing this entire piece of code we're
probably going to copy paste it to um to
visualizer it uses a lot of Linux
specific stuff so which means that this
stuff is not going to work on Windows so
for W on Windows we'll have to integrate
with the FM pack slightly differently
but as far as I know it's pretty much
the same right it's pretty much the same
you can quite easily create a pipe and
then you can sort of like pass that pipe
as a standard input for the child
process there is some stuff for that on
in wi API and in fact I use that to
implement no build piping right so for
those who never heard about this uh
thing I have like a custom build tool
it's sort of experiment and it can pipe
stuff on Windows as well well uh so we
may try to look into that and see how
piping is done is done on Windows and do
a similar thing on Windows for FFM pack
right so I'm going to copy paste it in
here and also I'm going to put that in
the description so uh sing No build
so here just in case you're interested
in this kind of stuff just in case
you're interested um all right so we're
integrating with Ray liap so in here we
are just
creating uh the pipe and we're starting
rendering so I suppose we're going to
disable this entire piece of code right
uh and just do something like this right
I'm going to be switching between sort
of like Olive C mode and uh the array Li
mode in Array Li mode I'm going to init
um I window right I'm initial
initializing window so do we have width
and I think we do have width and height
yeah so there's a global parameter here
width and height so we're going to use
that width uh height uh so Ray Li plus
FFM so that's going to be the title um
so we're going to set Target FPS so
let's say it's going to be 60 and while
uh while window shoot close actually
should not close uh we're going to begin
drawing
while we're drawing we're going to clean
the background we're going to set the
background to be uh black for now right
so and then I'm going to end the drawing
it is very important to do begin drawing
and end drawing because in end drawing
as far as you know there is like there
is a handling of the events right so if
you don't do end drawing at least like
once in a frame um you won't receive any
actual inputs from from the user so you
won't be able to even close your window
because the stuff that is checked by
window should close is not updated it is
only updated in the end drawing right
it's not particular obvious but if you
follow this sort of schema over it kind
of works right it just works and then we
want to close the window um
right so funny enough that should just
work that should just work so let me try
to rebuild this entire thing um so it
cannot find R because we need to
actually build with r now right we have
to build with r so maybe I'm going to
just take these flags and put them into
C Flags like so so this is going to be C
Flags uh and here we're going to have
package config C Flags Ray Li uh so for
the ray Li we probably also have to have
libraries uhhuh so this is going to be
leaps right because we need to link with
the r leap it's not enough to just
include the headers from the rap you
need to link with this kind of stuff
it's not a header only
Library uh right it it can contains the
you know static libraries that you need
to link with in our case we're using
Dynamic library for R because we need to
be able to hot reload with our Mizer
right so and to be able to like easily
do that it's better to have rip as a
also as a dyamic li um right and this
seems to be working okay so let me try
to run Main and it seems to be working
as well so as far as I know FFM is
currently actually waiting for the
frames but since ra doesn't really send
anything it is not receiving any frames
so because of that yeah it was just
waiting for the frames but it didn't
receive anything right so it received
zero packets zero bytes and yeah that's
it
so it was just like yeah it didn't
receive anything we can take a look at
uh what actually got rendered but I
didn't think yeah it's nothing it
doesn't even play anything it's just
like it's it's an empty video right so
to actually have something there we need
to constantly send something over with
this specific pipe over this specific
pipe so let's try to
replicate the the animation right let's
try to replicate this animation so we
have these parameters of the animation
so XY is basically the position of the
circle right so that bounce off of the
adjacent stuff like that and radius is
the radius of the circle DX Dy is the
velocity DT is the FPS I suppose we can
move all of that stuff all these
parameters outside of this conditional
compilation so it is applied for both
Ray and olc version of this test right
and in fact we can do this logic we can
literally copy paste this entire logic
to I suppose here like literally copy
paste it in here so that's the logic now
and the only things that we need to P in
here actually uh feel the entire
background and draw a circle at X and Y
with this specific radius so that's it
that's going to be the entire porting so
of speak so filling this thing is
effectively clear background right so
ween to clear uh background uh with this
color which is rather weird right so
this is an immediate value but I need to
reinterpret it as a
color right so because a clear
background let me show you this is a
kind of interesting situation so uh
clear background it accepts Cor color
which is a
structure right color which is a
structure but it's a structure with four
fields of single bite so it's four bytes
structure and the color that I pass in
here is also four bytes I actually
encode each individual component of the
color within the bite so technically if
I just take this sequence of bits and
reinterpret it as that structure I would
be able to actually pass it in here and
it would work but I can't
do that easily within like a single line
I have to do something like this uh
right where I save this color to a
separate variable take it as a pointer
that convert that pointer to color
pointer and D reference it and only then
it will be able to do
that recently I found a very interesting
trick to avoid that so but let's
actually first see if this thing works
all right so this is going to be
background and let's introduce
uh something like foreground uh
foreground so you insert you to T
foreground uh there we go so we take a
pointer we reinterpret that poin as the
color pointer and we D reference it and
that should work and here we have to
actually draw Circle uh right so in draw
a circle we accept uh position of the
center radius and the core I think let
me actually confirm that 100% yes so the
radius is float but I mean it's going to
be converted anyways so it's fine let's
go ahead and try to compile this enre
thing uh so it doesn't accept OC uh
there we go so and if we try to run this
enti thing there we go so we have this
kind of
stuff isn't that aers isn't that a PO
wers I think it is
so uh it got uh you know ported very
easily so it's the same code that we had
in olc which supported I think to rib
very very easily so how can we just do
this kind of [ __ ] but
within like um with this within a single
line within a single line recently
specifically in the source code of Ray I
found really dirty I found the dirtiest
trick I've ever seen in my entire
freaking life like holy
[ __ ] so let's think um of this
[ __ ] as an array imagine that you have
an array array of U uh in 32t and the
array has only one element and that
element uh is in fact that
value you can actually think of that
array as a literal as the literal
array and if you pass this entire thing
to as an argument it gets degraded uh
Decay to a pointer so that means in here
you can reinterpret that decay pointer
as a pointer to color and reference that
color and it [ __ ] works the worst
part of this trick is that it [ __ ]
works I hate that this [ __ ] Works uh
rayan why did you show me
that my brain got infected with this
forbidden goddamn [ __ ] knowledge why
the [ __ ] do I know that why the [ __ ] do
I know that now
like how am I supposed to sleep now
knowing that this is how you can do
that did you guys know about this trick
did you guys know about it I didn't know
that I actually learned it literally
yesterday by just exploring the ray leap
source
code so yeah don't blame me for that
blame
RAC
now you all also infected with this
knowledge look look at this [ __ ] just
look at it just just look at it just
look at
it you creating a literal array that you
pass which makes a Decay to a pointer
and then you reinterpret that pointer as
uh pointer to a color in you reference
set and it [ __ ] works I hate it I
hate it but it works
so yeah this is how you can do
that
um so we got some sub from and deot
thank you so much for one subscription
thank you thank you thank you really
appreciate that really appreciate that
so okay uh how can we get the current
sort of like a
frame um like the rendering orinal so I
remember I do in fact remember that uh R
had something with screenshot ah there
we go you can take a screenshot of the
current screen right so but you have to
provide the file name uh so let's do the
following thing so I'm going to
do if is key pressed uh
keyp right and uh we can take screenshot
PNG right so this is what you can do at
least we can take a screenshot 60 times
per second right and then we can use FFM
pack to concatenate all of that into a
final video so doesn't sound that bad
right um so let me let me see and I'm
going to start this entire thing I'm
going to press p uh it actually stopped
for a second and did it create the
screenshot PNG look that here is the
screenshot we just got a screenshot
isn't that aers isn't that aers yes yes
it is a
screenshot uh but that is not
particularly useful right like we don't
want to save that uh to a file system as
a PNG because PNG just like already
compresses it we need to let FFM pack to
compress this shy that's what we need to
do so maybe there is something else in r
that doesn't save it to a file something
related to screen uh there's a screen SI
is left screen get screen width and
height swap screen buffers is screen on
the screen blah blah uh take
screenshot load image from screen load
image from screen buffer and screenshot
and it returns you an image
okay maybe that's what we have to do
right so let's take a look at what
exactly does this function do because it
it sounds like something that we may
want to do uh R um so so let's go to
this source code and literally find uh
this entire thing literally find maybe
we could have used a debugger a debug
buer right so that would have been
actually useful um so let me let me go
because like I don't want to do grap I
just want to step into this function and
see what exactly it is
doing uh so main let see um so take uh
R so it creates an image right
and let's maybe exit
afterwards
uh so let's rebuild this enti stuff and
I'm going to do main but I'm going to do
that through software GF gf2 uh yep yep
yep and I'm going to break load image
from screen there we go so we're going
to break on that and I'm going to run
this entire thing and we actually broke
there so let me put this thing away uh
and so what do we doing here read screen
pixels right so essentially uh it does
GL read pixels I think so it's a open
gel function that like reads the current
frame buffer right so it's usually used
for um taking screenshots and stuff like
that uh right and essentially what it
does it just like it takes the current
screen screen coordinates uh it says the
format and it just performs um you know
read pixels but it uses RL wrapper uh
Ray has its own wrapper over open jail
so I suppose it's a wrapper over uh G
read pixels so let's actually confirm
let's actually step into that um okay so
it allocates memory Jesus
Christ God damn it one function saves
all of that stuff to file system another
one keeps allocating memory like we're
going to be calling this function on
each frame but to be fair we don't have
to render it in real time so maybe
allocating memory like each frame here
is not that big of a deal we can just
try to use that and if it's going to be
tooo slow I'll try to do something about
that and there we go it uses G read
pixels so uh right
so it allocates it
again oh to to
okay imagine do doing at least two Malo
per
[Music]
frame anyway so I feel like we'll need
to actually call uh you know G read
pixels ourselves we can kind of invert
this entire thing ourselves as well uh
can we when we are actually sending that
frame over the pipe right we can just
send it row by row in a reversed order
and that way we can flip it right we
don't have to allocate additional memory
to to flip it because we can just like
send it in a reverse order without any
additional memory uh right so and as you
can see yeah here then it and then it
frees it like it does two Mals and a
single
free okay so and then we just return out
of that okay so I I think it's good I
think it's good so I'm going to try to
use this entire
thing and uh as far as as I know right
so if we take a look at the definition
of the image it's just data width and
height and this is something that we can
straight up send over the pipe over the
right pipe like right here so instead of
sending pixels we can just send uh data
and the size that we're sending we're
basically sending u u in32 right so
that's what we're sending in here and
can we free the image or maybe it's
called unload image like freeing things
is called unload in R if I remember
correctly uh right so now we're not only
rendering everything we're also um you
know sending it as well I have a feeling
that we probably have to do that after
we ended rendering Because the actual
swapping the frame buffers happens in
here so it probably makes sense to do
that like kind of outside once we
finished rendering and here we're
actually rendering indefinitely so maybe
we may want to do this thing right so
essentially we're going to be rendering
this amount of frames and also while the
window is not closed right so we're also
going to be handling the user input but
on top of that is going to be limited to
like to how long we want to render all
of
that so
um okay so let's give it a try so if I
go to semicon so it feels like it's
going to be super slow like seriously we
are allocating like twice per this
entire thing and also deallocating but
at the same time it isn't we are not
rendering real time things so maybe
that's fine maybe that's totally fine so
by the way taking a screenshot doesn't
really matter anymore
so okay uh I got so let's just run it
and see okay it's it's actually okay
it's not that slow
honestly uh it wasn't that it's finished
rendering and did it
really what the [ __ ] is this
[ __ ] why is it so jerky do do you see
that as well what the [ __ ]
is
excuse
me why is it like this is so bad holy
[ __ ] but I mean it's it's random in the
video right because if we switch to to
Olive C right so let's actually try to
do Olive C
instead
uh very beef
Jerk It's actually Olive C is way
faster yeah look at this butter is
smooth [ __ ] what the
[ __ ]
maybe there's like what would even cause
something like that is it something with
a frame buffer because we are
updating this thing using DT which is
fixed we're not using the get frame um
get frame time we're not using Delta
time of RA we're using fixed our own
Delta time so the final animation should
be all right so that means the
simulation is fine the simulation is
fine it must be something within the
current frame buffer the frame buffer
might be containing some Bush
I
um so it might be containing some Bush
ISO um you're probably taking the
screenshot at the wrong time maybe maybe
that's the problem uh I can try to do
that in here before I didn't think it's
going to uh okay so just a second I need
to go
back uh I need to go
back all
right God damn
it h that is weird not going to
lie what if we try to
eliminate this problem of like not
knowing what the [ __ ] is going on when
the frame
buffer is rendered or something like
that what if we render the whole
scene
into
um into the separate frame buffer right
because the ray leap has so-called
render
textures uh render
texture right and essentially you can
take your rendering code you can take
your rendering code and you can um
basically render into the texture
instead of the screen all right and then
maybe we can always get the image out of
the render texture by the way how can
you do that I have Google by the way so
I can I should be able to do some cool
sh uh so essentially let me give it a
try so this is going to be programming
sing cogle cogle uh I'm going to provide
the rayap include r H and here we're
going to say so I need a function that
returns an image but accept the render
texture uh 2D right so it accepts that
uh do we have any function like that
because I don't remember yeah unload
that's really weird
[Music]
um
what okay maybe it's not render texture
maybe it's just texture Tod yeah load
image from texture so essentially yeah
we can render everything into render
texture and then we should be able to
pull out the pixels out of that render
texture so and as far as I know if you
take the render texture uh right so it
is the texture in here so this is one of
the things we can try to do right one of
the things we can try to do so but to do
that we need to create that texture
first of all right so let me see how you
do that well I mean I can use Google to
do that so essentially I need something
that returns render texture to D but
except maybe the size of the texture
right so let's
see uh yeah low texture go look at that
Google is so goddamn [ __ ] useful holy
[ __ ] it's just like I don't remember how
it is called like I know roughly that it
it's supposed to return rended texture
and what rended texture needs it needs
the size so it probably accepts the size
as integers and yeah there you go there
is a function like that like I don't
have to remember the names right so I
know what it does so I can find it uh so
that's actually super cool [ __ ] damn
it I love it so this is why I was like
uh I was using Hogle so much right
because it is a very useful concept uh
right being able to just like put a
signature roughly and it roughly like
finds you something that looks like
this um right and by the way for those
who doesn't know this is the thing
Google I I haven't uploaded the source
code for this thing yet it's still in
development I was about to give you the
link to this magical tool that I use but
I can't give you yet because it's not
finished so it's only for me it's it's
in better it's enclosed better it's
enclosed better you can't have that tool
yet only I can use that uh unfortunately
but soon it's going to be ready so I'm
going to upload it for everyone so
here's the screen um and essentially we
need a function that accepts uh render
texture Tod right it accepts render
texture Tod it doesn't return anything
and this is basically like a begin
texture mode uh yeah begin texture mode
so this is the function that we need so
we begin rendering uh but instead of
rendering We Begin another mode right so
this is a screen we render into this
texture and then we end uh texture mode
there we go um uh so
interestingly uh on top of just
rendering everything into the texture we
can also render it on a screen who said
we can't do that right so just to be
able to see that thing um just to be
able to see that I think I think it
would make sense so I suppose that thing
should accept texture to D it should
accept the position maybe a couple of
integers um and yeah that's basically it
so set texture filter um load render
texture no it is not called like that so
maybe it also accepts the color we can
say that it's yeah there we go draw
texture there we go so we found that so
that's what we want to put in here uh
draw texture and we're going to take the
screen
and the position is going to be zero uh
zero and tint is going to be white so we
are rendering scene into the texture we
also displaying it on the screen so we
can see that uh right but we're going to
be saving the stuff from that texture
instead so texture uh image so what was
that called load image from texture
okay so this is going to be screen
texture so we got the image we're going
to be sending it there and then we're
going to unload the image because I
suppose it uh allocates the memory for
that so this is how we can do all
that uh all right so let's try to um
recompile this inti I think so let's go
to H where is the
build let me see if it compiles still
okay so that's cool and let's run this
entire thing what the [ __ ] pixel data
retrieve why the [ __ ] do you look
that really
why the freak do you log that I do not
understand it this is so bizarre to
me why don't you log it when I take the
current screen thingy but you log it
when you retrieve that data this is
like I swear to
God just a
second I want to see where it is logged
like what the [ __ ]
um grab pixel data retrieved
successfully show show me that um
why
uh okay so it is locked can can you make
it debug at least like please
um I think this must be a debug right
debug by default is not really displayed
so
yeah let's let's keep it debug uh I'm
going to try try to rebuild it I wonder
if it will just pick up my
change yeah it picked up my change okay
so we can rebuild R time
R
mhm okay so it needs to relink all of
that stuff but I mean it's not going to
be too long uh because it doesn't need
to recompile everything and let's just
install it
again all right so yeah I can in real
time modify reap to use in my current
project
unfortunately it's not it's not
particularly useful in my opinion right
because I can't really ship it on Linux
right on Linux people
essentially build with their own version
of R so it's not particularly useful so
yeah
[Music]
whatever anyways so where is the main C
uh this is main C and if I rebuild the
entire stuff yeah there we go as you can
see it doesn't log that anymore uh it's
actually it looks all right it looks all
right and if we take a look at the
VLC it is
smooth it is
smooth not bad not bad at
all
okay just make a vendor folder it's not
a bad idea actually maybe because ra ra
itself is really small we might as well
make the compilation of really part of
the compilation of the project uh
right I put R inside of project in its
internal dependency yeah so I feel like
this kind of approach becomes
increasingly more and popular as the
sort of like a dependency delivery
infrastructure becomes more and more
hostile uh right so I think I think it's
I think it does um so let
me
um we can compare this thing to Olive C
right so let me let me do so let's do
the ol
[Music]
C
wait don't you not just something
weird
don't you don't you don't you something
weird they feel
different
oh so uh
Ray okay let's take a look this is Olive
C this is
Ray
it's freaking
flipped so R flips for the screenshot
but it doesn't flip when you try to take
the pixels out of
C God [ __ ]
[Music]
Dam uh so we get some subs thank you so
much ghosty X 101 for t one subscription
with a message hi hi and Anonymous
gifter thank you so much for gifting one
sub thank you thank you thank you uh so
let's try to flip that and as already
said by the way we can actually flip
this
shis without allocating any memory or
anything like that so essentially here
we're just passing the whole image like
we're just writing the whole image as it
is we can write the image uh row by row
right so let's actually go ahead and
iterate the rows so we're going to
iterate from zero and Y less than height
so something like this uh right and
essentially in here we are going to be
writing the whole row of the width and
here we're going to do plus y multi by
width so but we have to be careful with
the pointer arithmetic here what is data
uh let me take a
look Ray li. H so this is going to be
it's void star so we need to cast it to
appropriate pointer so let's cast it to
pointer this so it just like naturally
upsets properly so here we're writing it
row by row and now we can quite easily
just swap it right iterate up until
greater than zero right so in just minus
minus and in here we probably will have
to do y - one but right so Y is UN
signed so we have to be careful with
going below zero right so it's actually
very important um right and essentially
here we're writing the rows in the
reverse order so that's what we're
doing uh okay good so let's try to do
[Music]
that and now if we compare this thing uh
let's call it maybe Ray lib
flipped so this is uh okay it goes down
and this one goes down as well okay so
they're like you know equivalent so this
is basically what we have to do right
that's the easiest way to go
about
[Music]
um all right so that's pretty cool but
how the [ __ ] do we synchronize this
[ __ ] with the
music that's a very interesting question
right
because we need to do that
somehow we can I suppose overlay music
on top of this stuff right so let's grab
some songs let's grab some songs I'm
going to copy programming sting uh Mizer
so music uh null um VIPs just another
and uh this is going to be one so I'm
going to grab this specific song and I'm
going to copy paste it in here so we can
essentially treat that as the second
input so in FFM pack you can actually
have several inputs right you can have
several inputs
essentially uh this is the first input
it we accept it through the standard
input uh we say that it's a row video
the pixel format is that the resolution
frame rate and stuff like that it
doesn't have audio we don't really have
to specify that it doesn't have audio we
can just like never provide it right and
then we can say okay uh the second input
the second input is going to be
this right and that essentially means
that it's going to merge them together
so it will detect that okay so you
provided two inputs uh and one input has
video the second input has audio so that
means you basically want to merge them
and FFM pack will make an executive
decision of just creating the final
video so this is how it work works right
it tries to kind sometimes kind of guess
what the [ __ ] you want from it um so not
like it doesn't really work every time
and because of that there is a syntax
that allows you to pick certain channels
within certain inputs to merge them
properly right so maybe you want to
input from one video uh right so the
video from the um the video from the
first input but the audio from the
second output so you you can specify
what exactly you're taking from where um
right but here I think we can just
provide it like that and it will just
like merge them uh so let's go ahead and
just literally run it
right uh and see if it's going to do the
thing
um okay so it just included the
video right included the
audio but what's going to happen after
10
seconds so okay that's very interesting
uh basically it took the longest one
right and there is no more video anymore
so it just like
stopped uh that's very interesting so
essentially what we have to do the video
generator needs to generate the video
while analyzing the audio right while
analyzing the audio so it needs to
actually generate as much of the frame
as the length of the audio and as the
generates the frame it should iterate
through the wave form and do the fft
analysis and stuff like that and then we
give all of that to FFM Pac and
hopefully it will just like synchronize
and merge everything together so that's
basically the
idea uh you'll need frames from the
audio uh says c64 cost me yeah so and
because of that uh we'll have to load
that specific audio so we're going to be
giving that audio to FFM back and we're
also going to load that audio
ourselves we're going to also load it
ourselves and we're going to be
analyzing it as we generate the frames
so and that's why I suppose we're going
to how we're going to do that right so
and depending on the FPS we're going to
be handling the samples from the audio
with a different speed right so to to
make sure that we synchronize the audio
with the uh with the video so and this
is very interesting thing right
because in the actual visualization in a
mutualized visualization we don't have a
control over how we synchronize audio
and video right because the audio system
is just running in SE in a separate
threat and periodically calls our call
back giving us the new frames but as far
as I know the audio system of rip and
the video system of rip they do not
communicate with each other and they
don't try to synchronize with each other
so it's just like it works as it
works right it just works as it works
but when we're doing the offline
rendering when we're doing the like
final offline rendering we'll have
control over that and we'll have to take
this conscious decision of how how many
samples we're handling per frame but I
mean we usually know the amount of
samples per second right and it's
usually 48,000 samples per second so we
can just take 48,000 and divide it by
FPS which is 60 48 by 60 and this is how
many samples per frame we will have to
handle right so we can even see if it
div divisible right if I divide it like
that it actually divide perfectly right
so we have 48,000 samples in the audio
we divided by 60 FPS and this is
basically how many samples we handle per
per frame uh and we just put that into
fft you know analyzer and it's just like
okay so everything just like you know
works out more or less properly
um uh what happens if the application is
slow it cannot do 60 FPS it doesn't
matter because we're rendering video in
offline right so essentially we just
have to say to FFM we are rendering
video that is 60 FPS and what we have to
do we have to supply enough frames for
the duration to be at 60 FPS and the F
imp will just synchronize all of them to
60 FPS so just because we're rendering
60 FPS video doesn't mean that we have
to render the frames at 60 FPS we just
have to render enough V enough frames
for 60 FPS video right you see what I'm
talking about right we are not
displaying at 60 FPS right we're not
displaying in real time at 60 FPS we're
displaying them in offline so that means
we can take as much time as we want to
render a single frame we can render
single frame per minute but at the end
of the day we we're going to have 60 FPS
video uh
right so that's why the the actual speed
doesn't really matter that much for the
end result it would be better if it was
faster of course but I mean doesn't
matter
uh all right okay guys so that seems to
be good that seems to be good so I think
I want to make a small break I want to
make a small break and after the break
we're going to try to integrate this
entire thing into the visual into the
visualizer right so essentially let me
show you visualizer one more time so
that's the visualizer and the point of
visualizer is that you can basically
take any song and it will start
visualizing it like that uh I hope the
encoding doesn't really ruin the The
View too much uh right but essentially
our goal is to be able to then press
some button and it should call it back
and start start rendering it at very
high
resolution so uh yeah that's basically
what we're doing in
here so that's pretty cool all right
let's make break and um okay so one
thing I want to do in here right the one
thing I want to do in here is I want to
take this entire stuff that starts up
the FFM FFM pack process and put it in a
separate function right uh so maybe we
can do something like um FFM
pack start rendering uh right and it's
supposed to return the uh the pipe right
it's supposed to return the pipe uh into
which you're going to be you know um
send in the frames and stuff like that
so let me put this stuff in here um
right and
yeah I think that's fine so we probably
may want to accept two things in here
yeah we'll definitely have to accept the
um the resolution and stuff like that uh
because it might be variable so here
right now the resolution is just like
fine
mhm uh you know what I think I'm going
to start like literally uh copy pasting
this thing into the final
Mizer uh somewhere in the the in a
plugin uh somewhere here I think it
would be fine so we close the read end
but we supposed to return the Right End
right so this is the right end and you
know what I think naturally we may have
FFM pack and rendering right FFM pack
and rendering which will accept the pipe
that this thing returns right so this is
the pipe and it will close that pipe
close that pipe and wait until the
process finishes right so it needs to do
wait null uh right and after we close
the pipe we wait until the status of the
child process changes and uh that way we
sort of synchronized with with the child
process so that's actually kind of kind
of cool right so you start the rendering
and then you finish the rendering so and
in here we may accept things like uh the
width and height with which we're
rendering then
FPS uh and then maybe things like sound
file path right so sound file path that
we're going to actually put in
here uh sound file path so that's
basically uh what we can do in here so
interestingly we do a return in here if
something wrong happens and we return
one as an indication uh as as as an
error but if we're going to be returning
the pipe right if we're going to be
returning the right pipe one is a valid
pipe so what we have to do we have to go
through all of these returns and return
like a negative thing right so so now uh
the error is indicated by a negative
value and I think it makes sense and if
you return return something that is not
negative uh that's the final pipe that
you can work with um right so that's
basically what we can have that is
basically what we can have that's pretty
cool uh so and let me maybe try to
compile visualizer just to see if this
entire thing will compile in here
because we'll have to copy paste a lot
yeah okay so we'll have to include a lot
of things right because pipe is
available in a very specific place and
here's another unpleasant thing in here
chat here's another unpleasant thing it
uses a lot of Linux specific things but
here we're using only crossplatform
things right so far we been using all
cross crossplatform stuff leap C all
that stuff is available on Windows array
liap is available on Windows but now
we're using Linux specific uh
interprocess communication and that is
not available on Windows so it feels
right to maybe separate this entire
thing into like a module and whatnot
right what if we have like a separate
translation unit so we're going to have
FFM pack header right so let's have
inclusion guard right if not uh defined
f f pack H let's define that and let's
close the include guard so we can't
include header twice and we're going to
have uh
two signatures in here so start
rendering and end rendering and maybe
we're going to have a C file which is
called FFM Linux C and this is where
we're going to have Linux implementation
of this
interface what if we say okay so here is
the crossplatform interface of starting
uh FFM pack process and I'm going to
implement it only for Linux and then uh
when it's time to Port this entire thing
for Windows and Mac OS we're going to
have separate FFM pcore Windows FFM pack
andore Mac OS that use the native
mechanisms of operating system to start
the child process and communicate with
that child process right so yeah right
now like it's it's not going to compile
on Windows obviously but we leave a
little bit of a room uh to to add
Windows support for this kind of stuff
later uh and maybe we can do that on the
stream as well right so I recently
discovered such thing as you know wine
GCC and also I recently learned that MW
also available on Linux so we can
theoretically do windows development on
Linux by doing cross compilation and
just like test this kind of stuff on in
wine so that would have been interesting
um that would been
interesting yeah that's pretty
cool so much work for a niche video game
system I know
right too many gamers too many gamers uh
all right so let me let me copy paste
the implementation in
here uh and we're going to put it like
that so here we're gonna just
include uh FFM pack right so here is FFM
pack so okay and when we're building
this entire stuff um we're going to be
linking with FFM pack Linux so here
we're just building visualizer uh but
here we're building the plugin for
visualizer for hot reloading so we're
going to have SRC FFM pack Linux and
when we disable hot reloading we're
going to link everything together uh
like so
never so let's try to compile and see
how misly it's going to fail right so it
doesn't have size T but this is because
it needs the standard input you know
what I'm going to go and just like copy
paste like literally all of these
headers in here because I know that
they're going to be sufficient enough
for for compilation hopefully so read
and we don't have read and defined it's
a custom macas that just Define the
index for the array of pipe file
descriptors right because when you
create a pipe it gives you two file
descriptors and so uh I don't confuse
myself I assigned like a read end to
zero and right end to one right so at
least they have like a human readable
pneumonics uh all right so let's go and
okay so here uh we supplying wids and
height but we do it like that because it
was known at compile time we can't
really know that stuff at compile time
anymore so
what if we allocate some buffer for the
resolution uh and just basically asend
printf this stuff in here width and
height um with hide and just pass it
like that I think that's a it's a good
thing to do the same can go for FPS as
well uh but the question is how big of a
buffer this should be how big of a
buffer this should be okay so size T is
assigned a 64-bit integer so that means
its maximum value is 64 uh if you take
the amount of characters it's 20
characters right so here we have width x
height so that means we'll need this
kind of stuff twice and plus additional
character for the X so we need at least
41 character plus maybe zero uh for for
n Terminator so 42 essentially it makes
sense to just allocate 64 bytes in here
on the stack and it's going to be uh
like always enough it's always going to
be enough in here so I think that's good
right and for the frame rate FPS is also
the same thing we can just do uh frame
rate and just render it like that so
this is going to be FPS and yeah so
that's fine and you know 128 bytes on a
stack is not that much to be
fair
um so I think we can afford to have that
on the
stack we can also increase the quality
right so here I'm specifying the codic
for video uh so the codic for a audio
usually the one that works on Twitter
right by the way uploading videos on
Twitter is such a huge pain in ass
because Twitter only works with videos
with a specific video and audio codec it
has to be specifically H 264 and the
audio codec has has to be specifically
AAC if it's something else it's going to
tell you I can't process what the [ __ ]
is this sh I do not understand what the
[ __ ] you just uploaded it's just like so
frustrating so maybe it makes sense to
by default have AAC so it's uploadable
uh on Twitter right so because that's
primarily where you want to show off to
to to those plbs who can't program right
and that's why they're uh spending all
day on
Twitter
so anyway um right and we can specify
the bit rate uh so the video bit rate is
VB if I'm not mistaken so what's going
to be let's say it's going to be three
let's not put too much frame rate
because I'm not sure if it's going to be
too uh resource consuming for my PC
while I'm streaming so let's not put too
much load on
that uh okay so uh let's uh try to
compile it one more time so here we have
height oh it's as print F so I have to
provide the buffer and the size of the
buffer uh before I can actually you know
render all of that stuff so this is a
frame rate and size of uh frame rate let
we
go okay so with compiles this entire
piece of code in fact compiles that is
very
cool uh okay how we're going to be doing
all of that right from the from the
visualizer user
perspective uh I
suppose you
upload um a file and you just press a
button and it starts
rendering so what's going to be the
rendering I suppose rendering is going
to be a separate mode so plug update
it's is a very big function but what it
does it renders a single frame right it
renders a single frame so one thing it
checks it checks whether music current L
playing right if the music currently
playing it does the visualization if the
music not playing it displays uh Drag
and Drop Music here right so we can
essentially even demonstrate right so
the music currently is not playing but
there is no error uh going on so error
has not happened so it just like prints
Drag and Drop Music here uh if you drag
and drop something that it cannot read
uh it will tell you could not load the
uh the file but if you drag and drop
something that it can it starts
visualizing so this is basically three
different states in
here uh right and uh essentially all of
these three states are encoded in this
function so we need some sort of like a
another
state uh that essentially says we're
currently
rendering right we need some sort of
special state that says we're currently
rendering we can indicate that State uh
with a Boolean
rendering right if it's true we
currently rendering if it's false we're
not currently rendering I think I think
that makes sense um okay so here this
entire stuff that checks that music is
currently playing we do visualization
which is 143 lines of code and also the
else branch that uh prints the drag and
drop music and couldn't load file all of
that is behind a huge condition we are
not rendering so if we're not rendering
this is basically what we're doing here
right but if we are rendering right this
is where we're going to be basically
advancing the simulation of the uh of
this fft thingy right as as we do in the
actual rendering stuff like that and
sending the frames into FFM pack process
so if we reach this condition FFM pack
process should be already running
somehow it should be already running
somehow so uh okay but
how is it going to be running so here
we're not rendering and within the
rendering we handling different Keys
like when you press space it pauses the
music right and then pauses it so you
can see and also when you press Q it
restarts the music from the beginning we
can maybe add another key in here that
starts the rendering right another key
that starts the rendering key is pressed
which key is it going to be um maybe R
right so for for rendering and what
we're going to do in here for now at
least for now we're going to say
rendering true so now this entire
condition will be redirected to here so
effectively uh now if I press R we're
going to soft lock ourselves oh R is
already taken because it reloads the the
thing let's let's use maybe F right
because I'm pretty sure we're going to
die a lot while trying to do that so why
not call this feature F uh right so
let's call it f
uh right so uh let me restart the the
entire
thing so it's plain and I'm going to
press
F and it stopped playing effectively it
stopped playing and doesn't display
anything okay so we have an ability from
the actual plane uh switch to rendering
state but we don't do anything there so
it will be kind of nice to know that we
are in a rendering State maybe it makes
sense to just print something I'm going
to L copy paste this entire code this
the same code that displays drag and
drop and stuff like that I'm literally
copy pasting this entire code but uh
what I'm going to do I'm going to just
set the color to I think white right so
we're going to render with white and the
label is going to
be uh rendering rendering video
something like that I think that that's
good and that's it right so let's try to
recompile this entire thing and I'm
going to
quickly uh so let's put something
something from
here rendering
video Isn't that cool I think that's
pretty cool but but you can't really go
outside of that state as soon as you
went into the rendering video State you
kind of soft locked yourself right
because there's no way to get out of it
as of right now we can try to put
something like a temporary measure right
if is key uh pressed and that key is f
if you press F again we're going to
reset Plus block rendering to false
right so we's try to recompile reload
and I press again it continues playing
and now we can flip flop between these
two states right so you see how it works
so basically it's like a huge State
machine uh right so we have like a
single Boolean that switches whether
we're in a rendering state or not
rendering state so it's sort of like a
big Contraption right so programming is
basically building like a Contraption
out of the
instructions
right so and
okay uh let's go back to this state
where we start rendering right where we
start rendering when we start rendering
what we want to do I suppose we need to
reset the state of fft analyzer right we
need to reset the state of fft analyzer
because we're going to start the
simulation from scratch we need to start
the simulation from scratch so we can
render it for the video uh the whole
whole state of the fft analyzer is these
six buffers so essentially what we need
to do we need to reset them to zero so
the the first two buffers contains the
input that we receive from the audio
system then um output after fft analysis
and then smoothing some scaling and
stuff like that uh right for example the
state of the smooth depends on the
previous state of the log and the smear
depends on the smooth so they kind of
like do ass simulation and stuff like
that so uh maybe we can do fft reset
maybe
clean uh right so and in here
essentially what we have to do we have
to just like mem set to zero all of
these three things I think I can use a
little bit of emx magic to do that uh so
if I do it like that so essentially it's
going to be mem Set uh yep so I'm going
to remove that set it to zero with the
size of that uh and then boom uh look at
that we just reset all of
that so that's pretty cool uh keyf right
the first thing we have to do we have to
clean the fft state before switching to
the rendering State that's what we do um
then we have to start the FFM pack
process we have to start the FFM pack
process um so we's go ahead and do that
so we have FFM pack start render
and the question is the question is what
size do we have to provide what size do
we have to
provide we can maybe
provide um the size of the current
window but the problem is the size of
the window can
change so and funny enough funny enough
the visualization very much depends on
the size of the window
it very much depends on the size of the
window so we need to render with a fixed
size all the time so that's what we need
to do so what I'm thinking is that maybe
we have to render everything into this
separate texture so the same trick that
we've been using to sort of make the
animation Smooth by the way I still
don't [ __ ] know why R leap was not
making smooth thing but we want to
render everything in separate texture
anyway so maybe that's fine so um yeah
let's go ahead and create a separate
texture for for the screen where we're
rendering all of that so this is going
to be render texture Tod and this is the
screen uh right and when we are
initializing the plugin uh we need to
create that texture so this is going to
plug um
screen uh load render texture and we
have to provide like a very fixed size
very fixed size so what's going to be
that size let's say that it's going to
be 16 multiply by Factor some sort of a
factor 9 multip by some sort of a factor
uh right and factor for now is going to
be uh maybe
60 maybe a 60 when we reloading we don't
really need to uh reload anything so we
just allocate the structure once and
we're just like run with it so that's
basically what we do uh that is
basically what we do and when we're
starting FFM Peg uh we're going to be
supplying the width of that specific
texture so this is going to be plug
screen width so I wonder if render
texture actually has the width inside of
it right I know it's a structure uh but
I don't remember if it has a a width so
render texture okay so it has a texture
inside of it so we have to go a little
bit Yeah so this is where you can have
width and height all right so we can do
that
texture uhuh and similarly plug screw
green texture height so what's going to
be the FPS
um I don't know maybe it's going to be
some sort of a global parameter right so
it can be fixed it doesn't really have
to be whatever FPS of R is fixed right
it can be our uh sort of like a render
FPS right so let's call it render FPS
and here comes the very interesting
thing we need to supply sound
file but we don't really keep track of
the sound file to be fair when we load
uh load music right when we load the
music we just load the music and we
forget about it and I don't think when
we loaded the music the file path is
associated with the music somehow I
don't think so I I don't remember seeing
that so R where is the r so here is the
music yeah there there's no there's no
file in here so as soon as you loaded
the music the file path is lost so we
don't have that but we need to provide
that for to start the FFM back process
unfortunately
all right so one of the things we can do
we can maybe store the file path in the
plugin like
so initially it's going to be null right
and look um essentially instead of uh Lo
like getting the file path into this
variable we're going to save it into the
plugin but we have to be careful because
this thing is going to be lost as soon
as we unload the dropped files right so
we we probably have to do Str strr dup
right but that means we're going to leak
a little bit of memory every time we
draw a new file in there so that means
we need to First free whatever we have
before that whatever we got from the
previous EST dup and only then do the
next so that's basically what we're do
right initially file path is going to be
null and passing null to fre is actually
safe so just these two operations is
totally safe and it's not going to leak
in memory so it's not going to like free
corrupt memory or anything like that so
that should be fine uh and that also
means that uh we have the file path that
we can pass to here is that simple right
so we can just do plug file path and
there we go so we just started the uh
the rendering
process that's pretty cool but starting
the rendering process actually returns
you the pipe so we need to save that
pipe somewhere we can also save it in
the plugin
so yeah so here everything is reled to
related to rendering so maybe I'm going
to put like a group all of the things
related to rendering like separately so
it's clear uh here we have stuff that is
just like General stuff and here is the
rendering so in here we need to have a
FFM pack pipe right so this is FFM pack
pipe maybe we can just call it FFM pack
uh the god object I mean you can think
about this thing as a global scope like
I just put all of that stuff into into a
global scope and the the reason why it
is in in a structure so it survives
between the plugin reloads right because
we allocate this entire thing in um in a
dynamic memory right and we just keep a
pointer to it and we keep passing that
pointer between different versions of
the plugin so all of that stuff survives
between reloads right it's a persistent
State yeah so uh NRI Dev persistent yeah
so essentially that's what it is uh so
all of the global variables outside of
that structure do not survive hot
reloading everything inside survives it
so I wouldn't say it's a gut object it's
just like two Global Scopes
essentially um right two Global
Scopes
um so here maybe this stuff should also
survive hot reloading to be fair but
that means I have to do six allocations
in here for all of these buffers it's
just like I don't know
no I don't know I don't know I don't
know we'll see we'll see okay
so FF andex start so and we can just
assign plug FFM pack all right so we
cleaned fft we started FFM pack and we
switch to the rendering State
essentially and that's it so we can just
start simulating things we can just
start simulating things and sending
frames to a corresponding place I think
right but we need to factor out the
process of simulation right so because
here uh we need to trigger fft analysis
and fft Analysis requires to apply the
window to the to the input then do fft
then squash the logarithmic scale uh
normalize frequencies smooth out and
spear values you know all of that Nur
theed that we looked into in the
previous sessions right uh we need to do
that during the rendering as well so
what I'm thinking is that we need to
factor out this this sort of stuff into
a separate
function uh so how we're going to call
that so fft clear maybe we can call it
fft analyze analyze something like this
right and essentially it's just going to
do all of that stuff in there right so
it will apply the window does the actual
fft squash logarithmic scale smooth out
and stuff like that so
here it needs the Delta time right so it
needs to be aware of Delta time so that
means we'll need to pass it in there in
fact we'll probably have to pass more
stuff in here so I'll let the compil
tell me what we have to pass in there so
it needs fft so that means F of the
actual F of has to be defined a little
bit earlier so let's go ahead and Define
it somewhere in
here need to be defined a little bit
earlier so amplitude function has to be
defined even earlier than that it's a
very small function can we just do
static in line so it just like in lines
all these functions all the time right
so I didn't see any reason to not inline
that stuff so free it doesn't know what
is free this is because we never really
it doesn't know what is free oh this is
because I the file path is Con okay so
it doesn't really have to be con to be
fair because Str strr dup St dup returns
non cost thing anyway so that's fine to
not keep track of the con so I think
that's fine so fft analyze we actually
extracted it
successfully so uh let me find so that
means I can grab this entire stuff this
fft analysis thing and just replace it
with fft analyze and since it accepts
Delta time we can just get the frame
time since this is the real time
analysis we get the frame time from
rayap right we get the frame time from
Ray leap but uh in the rendering
somewhere here in the rendering when
we're going to be doing the fft analy
we're going to be doing one divided by
render
FPS right so it's going to be fixed like
that so we're not going to call it right
now but this is something that just to
to keep uh to keep to keep track of to
keep at the back of our heads um so and
this one is rather interesting so we
have to also do
fft uh
rendering oh [ __ ] by the way
this one is rather interesting fft
analyze after squashing the logarithmic
scale it computes the amount of samples
in a smaller logarithmic scale and we
kind of lose that information so that
means we have to return that thing out
of this function like so so then
later where is it I'm surprised this
[ __ ] compiled how the [ __ ] did it
compile it's yeah yeah it's not supposed
to compile how how the [ __ ] do you comp
yeah exactly that's exactly what I'm
talking about I think I forgot to
compile it let's go through the
compilation errors and yeah so we don't
have M but we can get the M out of the
fft analyze there we go yeah boy so we
don't use that stuff anymore all right
everything is coming together
nicely um okay so now I need to take
this thing um and factor out to like a
separate function something like f of
Renda right that's what we need
so uh fft anal eyes and let's do fft
Renda so fft NLS let's just grab all of
that stuff in here all of this rendering
and do fft render and by the way I think
we'll have to pass m in here right so
we'll have to pass m in here
um it's actually like uh like comes
together really nice thing like all of
the Pieces Just Like fall in places
that's kind of cool
um I think I over copy pasted [ __ ] [ __ ]
damn uh [ __ ]
bra yeah yeah I over copy pasted God Dam
so let's try this one more time but this
time do not over copy
paste uh yeah so end shade remote so
here display in circles yeah okay so
that's the that's the last place where
we're going to do fft random
pass M
fft render so this is very dangerous
oparation we're doing right now where
shuffling uh around huge pieces of code
this is called refactoring this is very
important step in our entire process
right so we're just treating these
pieces of code as black boxes and we're
shuffling them around and uh of course
when you put them in a different context
they may not compile because of that so
you have to sort of like Stitch together
to to fit them properly into the new
context but I think we're doing quite
fine uh [ __ ] okay so well I mean the
only thing we have to
provide is just the the render width and
render height so it's not that big of a
deal so we can just put it in here uh
and everything's fine everyone come down
everything's fine so moving a huge chunk
of code to a different place was not
that scary was not that scary okay so
everything is working
nicely um so so we just done that okay
okay okay cool so when we switch to a
separate like to a rendering State the
first thing we do in this rendering
State we are just telling the user that
we're rendering [ __ ] right we're
rendering shed and while we are
rendering shed we need to do fft analyze
right we do fft analyze uh and does fft
analy accept yeah so it accepts the
Delta time oh yeah we already have that
and stuff in here so it also returns
like how many Squatch samples we have
and we do fft render uh and here comes
the punch line we need to do that fft
render the same fft render we do up
there but inside of the frame buffer
right so we need to do begin texture
mode plug screen right so there we go
you have a screen and then you render
this entire thing in
there this one is rather interesting by
the way because when we we have to
render it as I already mentioned we must
render it with a fixed size and here I
made a huge mistake I'm rendering with a
size that we get from ra no
no this is not how it works my NE this
is we have to do this kind of [ __ ] right
so we have to pass it like that right
and depending on the context where you
preview or actually render in the final
video you have to render for different
resolutions we have to R it for
different this is very [ __ ] important
okay so let's go to the compilation
errors okay so this is the context what
is this context this is context of
preview in a preview we're doing get
render width so we take the uh width of
the window uh and uh height of the
window so that's totally fine okay but
in the context of final rendering we
have to take the width and height of the
texture into which we're rendering all
that this is very important different
context different parameters the same uh
the same thing with fft analyze when
we're prev
we're doing the uh the Delta time of the
radi of the actual window refresh window
and stuff like that when we are doing
like a final rendering it's fixed it's
fixed to whatever FPS we want to render
it into very important so the same with
the with the resolution
so in different context these parameters
are different okay so in here we end the
texture mode so and here we have a
texture that contains the pixels of the
current frame it contains the pixels of
the current frame so the only thing that
is left is to grab those mother flipping
pixels uh with load image from texture
like so we grab those mother flipping
pixels and we have to send them down the
pipe just look shove it down the F FFM P
pipe so it can render in code that frame
and save it into the file
system okay so here is an interesting
thing we're using write which is a Linux
specific
cull and I'm trying to not use anything
Linux specific in this place right I'm
trying to put everything Linux specific
to FFM pack uh Linux so that means that
maybe we need
to uh create another function right
something like FFM pack send
frame you know what I mean you know what
I mean so and essentially here uh you
you have to provide the pipe right so
the PIP pipe that you created with this
thing when you started the the the
process and you have to provide the data
the pixels themselves and width and
height right so this is what we're going
to do so and then I'm going to go to
Linux in here I'm just going to do it
like that and we can take this very
Linux specific piece of
shis and copy it there and it's pretty
safe to do this kind of stuff in here uh
right so and what we'll have to do in
the future in the future we'll have to
go through all of these three functions
and Implement them for Windows because
on Windows these kind of things are
going to be different uh right so that's
very important mind for that's very
important so uh okay so this is going to
be data this is just a width this is
just uh height and I think that's that's
it right so essentially what we can say
instead of doing it like that we can say
FFM pack send frame plug FFM pack and we
do image data um image width and image
height boom that's it and we have to not
forget to unload the image right so
otherwise we're going to leak some
memory so that's pretty cool isn't it I
think it is maybe we can even align this
stuff like this so it makes a little bit
more sense right so as you can see we do
fft analysis then we render our fft
visualization into this texture and then
we get the pixels of the texture and we
sending those pixels to the FFM pack
and then we're going to repeat that on
the next
iteration in fact we can keep rendering
these textures yeah we can keep
rendering these textures on the screen
so we can see what is the current frame
that we are visualizing that is not a
bad idea honestly but since the window
can be of different siid it's kind of
difficult to you'll have to feed that
thing right you have to feed that thing
so let's try to compile this entire
thing and see if it compiles all right
so uh you don't have to provide this
thing in here so you have to provide one
here and seems to be compound okay
that's pretty
cool that is pretty cool but there is
one problem in here there is one problem
in
here we don't handle the sound right we
do fft
analysis but fft analysis of
what fft analysis of what exactly excuse
me like what exactly are we analyzing so
to analyze something we should have
something in this buffer but while we're
rendering nothing puts anything into
that buffer when we do preview we have a
call back that is called but by the
sound subsystem every time something
needs to be played and it gives us a
bunch of
frames it gives us a bunch of frames and
we just put those frames into that input
row and then we can analyze that but in
case of the rendering the final render
of the video nothing is plain and in
fact nothing should be plain because we
we synchronizing the sound perfectly to
the
frames
right so how can we solve
that we need to
have this frames somewhere like the the
sound frames loaded
somewhere uh as far as I know like we
have a music music doesn't keep the
entire file in memory so that's the
problem it kind of loads it lazily as
it's playing so it's not particularly
useful for analyzing in
offline we need to load the wave of that
sound separately as far as I know R
introduces a notion of a wave right that
basically literally contains the samples
of the sound file that you just loaded
so and there is a thing like load wave
and you can take any file and it just
like loads it uh and you can analyze it
it's like loading pixels from the from
the image so you can just load this kind
of stuff and this is what we can do we
can just load it and uh as we generate
[100:01] the frames we can iterate through these
[100:03] samples of this wave and just like put
[100:05] it into fft and analyze them ourselves
[100:08] so here's the interesting thing in a
[100:10] preview it's the sound subsystem that
[100:14] gives us the
[100:15] frame but when we're rendering the line
[100:18] it's us who constantly pulling the uh
[100:21] the the the sound it's kind of
[100:24] interesting right it's different
[100:26] API uh so in the first case we were just
[100:29] like using callback uh we we didn't have
[100:32] to worry about anything so all of that
[100:34] stuff was handed to us but now it's us
[100:37] who has to constantly pull that stuff
[100:39] from uh from a sound W it's kind of
[100:42] interesting isn't it I think it
[100:46] is uh So speaking of different sizes
[100:49] will it support perspective mapping
[100:50] depends on what exactly do you
[100:53] mean what exactly do you mean
[100:55] perspective mapping so it doesn't matter
[100:59] what you're
[101:01] visualizing really FFM pack doesn't give
[101:05] a [ __ ] what exactly is on the screen as
[101:08] far as it concerned it's just
[101:11] pixels you can put whatever you want in
[101:14] there you can have 3D 4D 5D some crazy
[101:18] non ukian [ __ ] it doesn't give a [ __ ]
[101:22] really so write whatever code you want
[101:25] that puts 2D pixels and we can just give
[101:28] it to FFM pack and it will give you a
[101:30] final video it doesn't care it doesn't
[101:32] matter so that's what's cool about
[101:34] it
[101:37] um so yes yes
[101:41] yes what do you send to FM pack array a
[101:44] pix of pixels or images what's the
[101:46] difference
[101:47] exactly what's the difference between
[101:49] array of pixels and images
[101:53] well uh
[101:54] so from I would I would say that I'm
[101:57] sending array of pixels for me AR images
[102:00] are array of pixels um images can be
[102:03] compressed oh yeah we're setting array
[102:06] of pixels right so we're not the the
[102:08] format is row pixels if that's what you
[102:11] mean so what's the format the format is
[102:13] row pixels and we can even take a look
[102:15] at that stuff so yeah we specifically
[102:18] say that the format of the of the pixels
[102:20] is R GBA and the format of the whole
[102:22] video is R video that means that it
[102:24] literally expects uh a sequence of 32bit
[102:28] rgba right so these two flags mean that
[102:32] you just send 32 bits constantly and it
[102:35] will just interpret that as as
[102:37] images
[102:40] um so that's what's cool about the FM
[102:43] pack
[102:47] right it is WID height rgba yeah it's
[102:50] it's that simple surprisingly so it's
[102:53] pretty cool that you can do this kind of
[102:55] stuff as a FM pack the the bad part is
[102:58] that it's it's not really documented
[103:00] well
[103:01] anywhere right you can only find this
[103:04] kind of information in some like you
[103:06] know
[103:08] some weird like lost stack Overflow
[103:12] answer by somebody and it's just like
[103:14] yeah it's like there's no like a proper
[103:16] tutorial that explains you anything like
[103:17] that it's just like you can stumble upon
[103:19] that accidentally
[103:21] uh right but but FFM can do that FM can
[103:24] do that and actually pretty
[103:27] cool all right so we need to load the
[103:31] wave right
[103:33] so we need to load the wave just a
[103:36] second I need to
[103:40] sneeze thank you very much all right so
[103:44] we need to have a
[103:45] wave um
[103:49] and uh r rendering uh
[103:52] true so I suppose what we can do we
[103:56] already have a file path uh thank you
[103:58] thank you thank you thank you um we
[104:01] already have a file path so that means
[104:03] we can just do wave load
[104:07] wave uh plug file path look at that and
[104:11] we have a wave and that means here uh
[104:15] what we have to do we have to just
[104:16] analyze it so but to analyze it we have
[104:20] to
[104:21] like do this weird stuff where where is
[104:24] the weird stuff where's the call
[104:27] back so yeah we need to be able to do
[104:31] this kind of thing maybe I can factor
[104:34] out this process by saying something
[104:37] like
[104:38] fft uh push yeah fft push where I'm
[104:42] going to accept like a single frame that
[104:45] your push in there uh and it's going to
[104:47] do this kind of stuff automatically for
[104:49] you so I can call this thing
[104:51] during the final rendering right so and
[104:55] frame in here is just frame and here is
[104:59] this fft push and frameware pushing is
[105:03] this
[105:04] one yeah there we go so that means now
[105:09] here what I can do is just do this kind
[105:12] of thing um as we analyze in the the
[105:15] frames of the so here's the problem um
[105:20] both video and sound use the word frame
[105:25] and they mean different
[105:28] things God damn it in the context of
[105:31] sound frame is a pair of uh samples for
[105:35] left and right channels right so
[105:38] essentially um sound consists of samples
[105:43] ah what the [ __ ] uh it consists of
[105:45] samples and they're usually
[105:47] interlift uh so it's usually left
[105:51] right left right left right and so on
[105:55] and so forth so each individual sort of
[105:57] value is called sample and a pair of
[106:00] values like left and right are called
[106:04] frames
[106:06] so uh essentially wave contains the
[106:09] sound frames so like the terminology is
[106:12] super confusing I really apologize for
[106:15] that but yeah so and here comes another
[106:19] problem
[106:21] when the sound sound subsystem calls our
[106:26] callback it gives us normalized uh sound
[106:29] frames they're normalized from minus one
[106:32] to one regardless of the format of the
[106:35] music uh but if I'm not mistaken the
[106:39] wave is not
[106:41] normalized so depending on a sample size
[106:45] the data actually points to samples of
[106:48] different sizes of different types
[106:53] so that means we have to normalize it
[106:54] ourselves but I don't think so I think
[106:56] there was something that kind of decodes
[106:59] that uh we can try to find it so where
[107:01] is my coule yeah here's the coule uh so
[107:05] let's say that we
[107:08] accept um wave and return like a float
[107:12] is there something like that oh yeah
[107:14] okay so load samples data from wave as
[107:17] 32 bit okay so you can actually take the
[107:20] uh wave samples and normalize them so
[107:21] let me take a look at how it works does
[107:24] it actually normalize this thing so
[107:25] that's actually super useful um include
[107:29] no no no so let's go to software
[107:31] rip uh and it's going to be
[107:35] [Music]
[107:36] SRC so
[107:39] RN where is the
[107:42] Implement okay so it allocates some
[107:44] additional memory okay cool and all
[107:47] right look at that that's actually kind
[107:49] of funny so it through all the samples
[107:52] like it's through all the frames by the
[107:53] way right it takes the frame count
[107:55] multiplies it by channel right because
[107:56] depending on the amount of channel the
[107:58] size of the frame is different and
[108:00] depending on the sample size it just
[108:02] reinterprets the data differently and
[108:05] divides it by different values to
[108:06] normalize it from minus one to one okay
[108:09] so we already have this kind of thing so
[108:11] we don't really have to do that
[108:13] um okay so it and it also locates the
[108:16] memory so it has to be freed with unload
[108:18] wave sample this is very important so we
[108:21] should not forget to do that at some
[108:22] point um all right so that's pretty cool
[108:26] that's exactly what we need so we can
[108:27] even have something like I don't know
[108:30] float wave samples and after we load the
[108:33] wave we can do load wave samples we
[108:38] provide the plug wave uh we're going to
[108:40] keep track of this wave because we'll
[108:42] need to deallocate it later and we're
[108:43] going to reassign it to wave samples so
[108:45] we have the samples as floats so yeah
[108:50] another thing we need to keep track of
[108:52] we need to keep track of the of the
[108:54] current sample that we're currently
[108:55] processing right so because we iterating
[108:57] through the sound samples right so we
[108:59] need to somehow keep track of that oh
[109:02] boy um so let me see
[109:07] so wave
[109:10] cursor but as we start rendering as we
[109:13] start rendering the wave
[109:16] cursor uh wave cursor has to be reset to
[109:19] zero because with tile like over I think
[109:21] it's quite important uh right so this is
[109:24] the wave cursor then uh loading the wave
[109:27] loading the sample and yeah everything
[109:29] seems to be nice everything seems to be
[109:31] nice
[109:33] okay so now we need to decide how many
[109:38] samples we want to push into fft for a
[109:41] single frame for a single frame this is
[109:44] very important so how many frames we
[109:47] have at all we have plug
[109:50] wave frame count I believe this is how
[109:53] many sound frames we
[109:56] have um
[110:00] essentially we probably need to have a
[110:03] sample rate right so sample rate yes
[110:06] here is the sample rate so this is how
[110:08] many samples there are per one
[110:11] second per one second so if we divide it
[110:16] by render
[110:18] FPS uh r FPS this is how many samples we
[110:22] have to process per
[110:24] frame so uh this is sort of like a
[110:27] single chunk chunk size this is
[110:30] literally how many samples we have to
[110:33] push into fft with fft push before we
[110:37] analyze
[110:38] it so that's what we have to do so we
[110:40] can basically
[110:42] iterate from 0 to Chan size Plus+ I and
[110:47] I suppose uh we can essentially well we
[110:49] need to to take the samples right so um
[110:53] wave
[110:54] samples and depending on the amount of
[110:56] channels we have to re interpret it
[110:58] differently this is something that we
[110:59] did in a
[111:00] callback uh this is something that we
[111:02] did in a callback so I can basically
[111:04] grab this thing but if I understand
[111:06] correctly it's going to be different
[111:07] depending on the amount of channels so
[111:09] we have to be super careful so this is
[111:11] basically channels yeah like this there
[111:15] we go so this is basically channels and
[111:18] uh we have to do
[111:20] plug wave samples okay that's
[111:24] cool so this is the samples and
[111:28] essentially we start with
[111:31] plug wave cursor so this is the sample
[111:35] and we need to push that uh thing into
[111:39] the
[111:40] fft and
[111:42] afterwards uh we have to increment it by
[111:45] one so we have to be a little bit
[111:48] careful because at this point
[111:51] uh wave cursor can go
[111:53] outside of the frame frame count right
[111:57] it can go outside of the frame count so
[111:59] that
[112:00] means uh we have to do the following
[112:02] thing if this thing is less than the
[112:05] frame count so it's going to plug wave
[112:08] frame count right so this is the frame
[112:10] count we just push this thing
[112:14] otherwise I guess we can just
[112:17] push zero so it's very important because
[112:20] the whole amount of frames may not be
[112:24] divisible by the chunk size so there
[112:27] will be some sort of a trail that we
[112:28] need to like pad with zeros and that's
[112:31] why I'm doing it like that so we'll need
[112:33] to pad it with
[112:34] zeros uh so we also need to check we
[112:40] also need to
[112:41] check did we finish rendering at
[112:44] all do we finish rendering at all so we
[112:47] finished rendering I suppose when the
[112:50] plug wave cursor is greater or equal to
[112:54] plug uh wave frame count right as soon
[112:58] as we reach that we finished rendering
[113:01] and we need to do something in here so
[113:04] essentially what we need to do we just
[113:05] need to deallocate all of the memory
[113:07] that we allocated so we need to unload
[113:08] the wave uh right we need to unload the
[113:11] wave uh unload wave
[113:15] samples uh so let me see what else did
[113:18] we allocate by the way we allocated a
[113:20] lot of things uh let me see so wave
[113:24] samples wave we need to close F FM pack
[113:26] right so if we take a look at FFM pack.
[113:30] H we need to say end
[113:33] rendering all right we ending the
[113:35] rendering plug FF p and I guess that's
[113:39] it so we don't really need to deallocate
[113:42] anything else
[113:44] anymore uh yeah that's fine so and the
[113:47] last thing we have to do we have to go
[113:48] out of the rendering mode by setting it
[113:51] to
[113:51] false uh there we go so as soon as we
[113:55] reach this
[113:56] thing uh we finish the rendering uh but
[114:00] if we not we keep rendering so
[114:02] essentially we just print like a render
[114:05] that we're currently rendering the video
[114:07] uh then we push a little bit of data
[114:10] into the fft we do fft analysis then we
[114:14] render everything and then we save the
[114:15] frames and we keep repeating that over
[114:17] and over again okay that should be it so
[114:22] let's try to compile this entire thing
[114:23] and see if it works or not I don't know
[114:26] maybe maybe it does uh okay so wave
[114:30] sample C I suppose uh what else do we
[114:32] have in here FS incompatible Point blah
[114:36] blah blah I suppose I can just like like
[114:39] cast it to void star and void star is
[114:41] assignable to any poter so that should
[114:43] be fine uh so this one is
[114:47] FS float parameter oh okay so because we
[114:52] have to pick one of the channels left
[114:53] and right okay so that seems to be
[114:55] comp uh all right that's a really weird
[115:00] contration that we just came up with
[115:03] that's a really weird
[115:04] contration um so let me see when we are
[115:08] rendering what's the actual size of
[115:11] rendering we have when I do screen just
[115:14] a second plug in it I don't want to have
[115:18] a very big
[115:20] rendering size okay you know what I
[115:24] think we need to introduce something
[115:25] like render width render width and
[115:28] render
[115:29] height render width and render height
[115:32] and here we're going to have it along
[115:33] with render FPS
[115:36] render uh render
[115:38] width is going to be
[115:42] Factor uh maybe render factor
[115:46] 16 uh
[115:48] 9
[115:50] height Define render factor is going to
[115:54] be 60 nothing particular special let's
[115:57] try to compile this entire thing and
[115:58] that should be it I'm a little bit
[116:00] scared actually all right so it should
[116:02] be already working it should be already
[116:05] working uh right so let me do
[116:09] visualizer okay
[116:14] so um let's press uh let's press
[116:18] f
[116:20] broken pipe
[116:23] okay uh X leap holy [ __ ] all right uh
[116:28] file loaded
[116:30] successfully could not run FFM pack as a
[116:34] child bad
[116:36] address unknown sequence number while
[116:39] processing Q most likely this is a
[116:41] multi-threaded client uh has not been
[116:44] called sorry about
[116:48] that that is actually
[116:51] yeah that's exactly what I was talking
[116:54] about that's exactly what I was talking
[116:55] about that's weird like what the [ __ ] uh
[116:59] so let me let me
[117:02] see uh resolution
[117:05] so we ended
[117:08] everything okay so let's try to do that
[117:12] second time right so to see uh
[117:16] and yeah it's it's the same thing
[117:21] uh it specifically cannot run FFM pack
[117:25] at all but if we go back to our small
[117:30] example that we had in there right so
[117:32] FFM pack rendering with FFM pack so
[117:35] main. C uh so I
[117:37] have yeah I have a couple of things in
[117:41] here oh it's it's a wrong one actually
[117:44] so it's
[117:45] sing f p apparently I already had this
[117:49] clone but it was in a different folder
[117:51] uh all right so let me do build Dosh so
[117:55] uh yeah essentially I just copy pasted
[117:59] some [ __ ] so let me bring that [ __ ]
[118:02] back let me bring that [ __ ]
[118:05] back uhuh so let me close it in
[118:10] here so it must be something with the
[118:13] way I call FFM
[118:16] pack yeah it might be something with the
[118:18] way I call FFM pack
[118:20] because here uh it's it was working
[118:23] right we we're calling F FM pack fine
[118:25] and as you can see it is working
[118:28] right
[118:30] um all right all right all right so and
[118:33] if we take a look at this thing all
[118:36] right so that's
[118:38] fine so let's compare how we call to FFM
[118:42] pack here and in that place so what's
[118:45] the differences between those
[118:46] things uhuh so it's going so in
[118:50] visualizer SRC FFM pack Linux and what
[118:54] do we have in there
[118:58] so verbos y Row
[119:01] video we the only difference here we do
[119:04] is a resolution we rendering this as a
[119:07] resolution uh and a frame rate in this
[119:10] SN print T but that shouldn't be a
[119:14] problem uh maybe this is because of this
[119:17] thing I'm also not sure
[119:21] I'm also not sure
[119:25] so
[119:30] uhhuh back end rendering maybe we end
[119:33] rendering too
[119:36] early
[119:37] um so if we go to the plugin I don't see
[119:41] any problem honestly like what is what
[119:43] is wrong in here
[119:46] um so maybe we end the rendering too
[119:54] early what if we just like don't call to
[119:57] this
[119:58] thing frame rate instead of FPS what
[120:03] what are you talking about did they did
[120:05] they not notice something
[120:08] what what what wait what the [ __ ] is
[120:13] FPS why did
[120:17] it why the compiler just
[120:20] this is because it's a very ad holy [ __ ]
[120:23] godamn C I hate this
[120:27] language oh my God this is so freaking
[120:31] bad like it literally allows you to put
[120:33] any [ __ ] in here because it's a very
[120:35] adct and it's not even checked at
[120:37] runtime it's so
[120:41] bad very adic andc suck holy [ __ ] thank
[120:47] you so much whoever said that in the
[120:48] chat thank you so so
[120:50] much all right so yeah it's insane how
[120:56] bad it is all right so let's go ahead
[121:00] and try to do
[121:02] that okay we are
[121:05] rendering we're [ __ ] rendering
[121:07] [ __ ] we're [ __ ]
[121:11] rendering all right all right all right
[121:13] all right so as far as I know what
[121:15] what's the what's the length of that
[121:17] specific thing uh so music music um just
[121:21] another n VIP where's where's VIP just
[121:26] another there we go uh FF
[121:29] prob right FF prob what's that what's
[121:31] the length it's 30 seconds okay we're
[121:34] looking for 30
[121:35] seconds yeah so okay so it's almost done
[121:39] uh down rendering so it needs to be 30
[121:41] seconds in
[121:42] here yo what's up what's
[121:48] up
[121:51] is it done oh [ __ ] it didn't oh [ __ ]
[121:56] yeah we didn't we en
[121:59] disable we freaking disabled it but I
[122:01] mean it should actually finish rendering
[122:03] fine I think okay so here is the output
[122:10] MP4 yeah it it didn't properly finish it
[122:13] so we need to recompile it one more time
[122:14] and try to do that one more time because
[122:16] we didn't finalize everything right
[122:19] should have actually finalized it
[122:22] um
[122:24] yeah are we sending everything okay okay
[122:26] that's fine uh uhuh so let's do it like
[122:30] that and let's try to do that one more
[122:33] [Music]
[122:35] time okay let's render
[122:41] [Music]
[122:46] boom uh why run FFM back command instead
[122:50] of using their API because it [ __ ]
[122:52] sucks my ass that's
[122:55] why
[123:01] um that's literally the reason that's
[123:04] literally the reason everyone uses FFM
[123:07] pack instead of their API not only me
[123:10] it's nothing unheard of their API
[123:13] [ __ ] sucks ass to the point that it's
[123:16] literally easy to just run the F pack
[123:17] separately seriously
[123:20] that's how bad it is
[123:23] so it's so bad I try to do that even
[123:28] successfully look at that [ __ ] even
[123:31] successfully um and it's just just
[123:35] bad um okay so uh let me
[123:47] see
[123:50] okay not bad that's already not bad uh
[123:53] we just need to clean this screen every
[123:56] time uh right every time we're like
[123:59] rendering the next frame this is nothing
[124:01] special uh all right it is really easy
[124:05] to fix so essentially when we're
[124:07] rendering this entire thing before
[124:10] rendering just clean the screen with a
[124:12] background C very easy to
[124:14] fix uh all
[124:17] right one more
[124:30] time all right so let's take a look at
[124:33] uh at the video
[124:38] itself this is
[124:47] video
[124:51] it's a little bit jerky the thing it's
[124:53] kind of
[125:04] weird Okay so it also finishes too earli
[125:09] before the fft sort of settled down um
[125:13] so maybe this is something that we want
[125:15] to do in
[125:16] here uh essentially wait until it's
[125:19] settled
[125:20] down uh so how can we do that
[125:23] essentially the definition of settled
[125:26] down both smoothing and smear became
[125:31] zero or small
[125:34] enough right so this is one of the
[125:36] things we can do so essentially fft
[125:40] settled uh right and we can just
[125:44] do uh to be fair we need to maybe
[125:47] accept damn
[125:50] how can we easily do that because it's
[125:52] kind of you know we need to know M so
[125:55] that's kind of a
[125:56] problem we need to know M so we can
[126:00] decide whether it is settled or
[126:04] not we can do n right so we can just
[126:07] iterate the whole range it's not that
[126:08] big of a deal though
[126:13] [Music]
[126:15] so so then essentially if out smooth
[126:21] I is greater than certain Epsilon we say
[126:25] that it's false we're not settled yet
[126:28] and the same with smear and if we went
[126:31] through all of this stuff uh without
[126:33] trigging any of this condition that
[126:35] means we settled and let's say that
[126:37] Epsilon is going to
[126:39] be maybe
[126:42] 1,000 something like that right so in
[126:46] essentially here uh we're keep doing all
[126:49] that while while wave cursor is less
[126:53] than frame count um and fft not
[126:58] settled right so if it's greater and fft
[127:02] settled only then we finish the
[127:04] rendering and then we finish the
[127:06] rendering so interesting enough if we go
[127:09] over uh we just keep adding zeros on
[127:12] here so at the end like it going to pad
[127:14] everything with zero so it's it's fine
[127:17] it should be fine at least hope
[127:20] hopefully
[127:22] so yeah so this is the wave
[127:25] cursor it looks a little bit weird the
[127:28] final thing looks a little bit weird so
[127:31] I'm not really sure what's up with
[127:33] that uh all right so one more time one
[127:38] more time I I really like the sample
[127:41] because it's small enough right it's
[127:42] only 30 seconds so it doesn't really
[127:44] take that much time to render
[127:46] it how's it going everyone
[127:49] try different products to improve the
[127:51] video but video quality is not the
[127:52] problem that we're having is it so why
[127:56] do I troubleshoot the problem that
[127:58] doesn't exist right now what's what's
[127:59] the point of doing that excuse
[128:01] me
[128:03] [Music]
[128:07] um so what's the problem we're here
[128:09] right now oh yeah so basically uh you
[128:12] know settling why are you guys
[128:14] suggesting me to troubleshoot problems
[128:15] that I don't have like what what's wrong
[128:17] with that
[128:19] the quality of the video is not a
[128:21] problem that I'm having right now like
[128:23] it's literally not a problem the problem
[128:24] is that the video cuts too too earli
[128:27] like what the [ __ ] are you
[128:28] troubleshooting I don't
[128:31] understand so
[128:47] weird it looks weird
[129:00] honestly there we go so it actually
[129:03] settled but the the final thing looks
[129:06] really [ __ ]
[129:08] bizarre yeah so look look pay attention
[129:11] to this thing in here like this thing
[129:15] this
[129:16] end if you can see it of course because
[129:19] maybe because of the uh like rendering
[129:21] and stuff it's it's not
[129:23] visible but here that end keeps jumping
[129:28] there is some sort of a bug in
[129:30] here there is some sort of a bug like
[129:33] it's really clear because in the actual
[129:35] preview there's no such thing in here
[129:38] yeah so fft is not doing the same
[129:40] frequency so um there's some there are
[129:43] some discrepancies I we don't really
[129:45] know what exactly is going on why are
[129:46] they like that
[129:49] uh maybe we're just like not converting
[129:51] everything properly or something else so
[129:53] we need to like analyze
[129:56] this we need to analyze this this a
[129:59] little bit all right
[130:02] after you know um sometime trying to
[130:05] figure all that out we found a problem
[130:08] the problem was that we were not
[130:10] stopping the
[130:12] music uh like before starting the
[130:14] rendering and apparently for whatever
[130:17] reason uh Ray leap keeps calling uh the
[130:20] call back which keeps messing with with
[130:23] the data that is used for offline
[130:25] rendering so even though we don't really
[130:28] call update Music stream and I thought
[130:31] that if you don't call update Music
[130:33] stream the call back is not going to be
[130:34] cold it was not enough apparently like
[130:37] you literally have to stop the music
[130:39] stream uh to to make it work properly
[130:42] and that uh fixes this problem and it
[130:45] renders everything uh perfectly like
[130:49] that so
[130:51] yeah that's I suppose it for today so we
[130:55] you the thing uh so the rest of the
[130:57] stuff that we'll need to do is probably
[130:59] maybe wire up uh probably wire up all of
[131:04] this parameters to the UI right it would
[131:06] be kind of nice to let the user like
[131:07] decide what's going to be the render
[131:09] resolution what's going to be the frame
[131:11] rate uh also maybe some bit rate
[131:13] parameters like like how how high of a
[131:16] quality you want to have uh and stuff
[131:19] like that so that would be nice but I
[131:21] think I'm going to do all of that
[131:22] already off screen right because I I
[131:24] like to sort of uh stream uh the
[131:27] Milestones of the development right so
[131:29] if there's the major feature I'm just
[131:31] implementing the major feature and I
[131:33] clean it up like offc screen right
[131:34] because no nobody cares about this kind
[131:36] of stuff so I think this is a very
[131:39] educational session right so because
[131:42] yeah apparently like you can just like
[131:44] take frames of Ray leap and just shove
[131:46] them into FFM pack and get a decent
[131:49] video the frame rate of that video is
[131:51] actually quite nice so um yeah I really
[131:53] like that so I guess that's it for today
[131:57] thanks everyone who's watching me right
[131:59] now I really appreciate that have a good
[132:01] one and I see you all on the next
[132:03] Recreation programing session with aen I
[132:07] love
[132:08] you