Inside FIFA's Secret World Cup Broadcast Network

[00:03] but this is where the World Cup is made. Welcome to the International Broadcast Center. Every single World Cup game, every pass, goal, celebration, tear, [music] 16 stadiums across three countries comes through this building

[00:18] connections. >> Guys, can I touch it? I don't know guys, >> But what if I did disconnect them? Would the World Cup just stop? here is that failure is not an option. So

[00:31] >> And yet they let me in during a match. I was in the control room at kickoff. I even got this, a packet capture of the actual game. I wonder if we can decode it. The scale of this is insane. I've never seen a network like this. And it's

[00:43] temporary. So by the end of this video, you'll know how a kick in Boston ends up on a TV in Singapore. Now this was a big deal for me because I'm a massive football, soccer, fan. I played when I was younger, DREAMS OF GOING PRO.

[00:57] networking nerd instead. But never in my wildest dreams did I think that my love and IT would combine. That I would makes the World Cup happen. Are you kidding me? Also, my daughter Addy had a

[01:10] >> Hey Daddy, how do they capture the sound of the ball when the players kick it? same way again. Get your coffee ready. Let's Let's >> [music]

[01:30] in the HBS engineering department. >> Christophe is a fan and when he found place that the IBC is at for this World Cup, he thought, "Man, that would be >> [laughter] >> So I came with a mission because today

[01:45] France is playing Morocco in Boston, a stadium 1,500 miles away from the IBC. How is it going from the stadium to the IBC and then to the world? other goals. Uh number four might be kind of hard.

[01:58] >> You cannot walk on the grass. Okay, we have two snipers and they will >> I brought my ball. We'll see what happens. When we finally emerge from the back rooms, I saw it, the data center, the place that has all the answers, the

[02:10] place where all 16 stadiums connect. Will they let us in? get distracted. Don't even look at this row of replay servers that cost about

[02:22] here. Look at this. This is where all the stadiums connect, right here, this say we take a look at the other side? Field trip. Field trip. >> [music]

[02:42] And it's on Tech Street. I like that. Let's go inside. Right here is the other um connected, but why? Because the World Cup isn't made at the stadium. It's made at the IBC. No, what does that mean? Cuz the game

[02:56] happens at the stadium, right? What does it mean to make the World Cup? have to go back to the IBC. Let's go. of a sick name for a team. They make the cameras look pretty. They literally

[03:11] control the color and look and brightness and contrast of the cameras Then we have the audio mixing. All the audio mixing for a match is done at the IBC. Also, the audio listening. This guy, Mick, he listens for any anomalies

[03:25] and then writes it down. I love that. All the replays, they're done here. Remember all those servers? The team here does it. Graphics, all here. footage looked like this? >> You would get fired instantly.

[03:39] quick, back in 1994, the last time the IBC was here in Dallas, this wasn't the case. The match was actually mostly made at the stadium. All these people we just made the change. >> Purpose of centralizing all those

[03:54] functions is to to have to have better consistency. >> we've got less teams. They are here and they are controlled. they can be briefed, they can be uh trained if needed.

[04:07] >> Yeah. >> in the world because otherwise 16 16 >> the inconsistency. Yeah, that'd be insane. So, >> They wanted to have the best talent in the world making every match in one

[04:21] place consistently. Now, they had the same idea back in 1994. Some of the functions were put back at the IBC, but they couldn't do them all until now thanks to these three fiber connections. Three? Yes.

[04:33] uh over three diverse paths. >> Every stadium has three 200 gig fiber connections. Why three? >> Uh they are fiber cuts. So, basically out of the three, we always maintain two

[04:48] >> And it's sent along diverse paths just in case a fiber cut might happen, which going to see that here in a moment. But these three paths aren't just for would make the engineers in the '90s drool.

[05:02] >> And for each feed, we are transmitting transmitting the data twice. >> So, each senders are attached to the network with two uh two interfaces. packets and the red packets. >> Okay, the scale of this is crazy. Watch

[05:17] connection, that's just channel one. That's a backup. But the other two connections? So, okay. Each match at least has 45 They got helicopter cameras, ref cams, everything. Each of these is sent

[05:31] individually over a connection. But it's not just cameras, we got audio, commentaries, ball sounds. We'll get to that later. We're sending all of that what? We're like boop. Let's do it again. Clone it. Every

[05:45] single camera is sending two feeds. Every audio is sending two feeds. Why? >> And what's even crazier is they're not backups. They're both used. How? We'll chance to see one go down right before a match happened. Get your coffee ready.

[06:00] We got to go to the MCR. Through the back rooms we go. By the way, I asked bunch of other people a bunch of questions that will not end up in this video. Head over to Network Chuck Academy in the description for a special

[06:12] hours of info. It's awesome. >> Why is it down? >> Really? >> People drilling. wasn't happening in Kansas, and that stadium wasn't going to be used for

[06:24] another game. That's why everyone was so calm. But hey, we made it just in time for kickoff. Remember I wanted to be in this control room? Here's my goal. and I realized we didn't really answer the first question yet. I mean, we kind

[06:36] on the NASA monitor there, I thought, the feeds get to the IBC, but from there, where do they go?" Like how is it take one camera for example. That camera is on that screen, but this camera is

[06:50] also on this screen and this screen. This camera is being sent to broadcasters in Japan. This camera is also being sent to Fox and NHK, BBC. How does one camera feed go to all these people times, you know, 45 from the

[07:04] entire match? How do they get it all? That's where Martin comes in. Martin's sitting right there. I love Martin. Let's call him multicast Martin. >> No, multicast. Martin, how many flows do we have in the network right now?

[07:16] >> We've got about 150,000 multicast flows that we're managing at >> This is why the World Cup is run by network engineers. Watch this. Cuz that number was big, and this is going to get to a level of nerdiness that you weren't

[07:28] prepared for. It's okay. Welcome. The meter's going to go up as I keep time. You ready? Let's take a little sip of coffee. >> Oh, right, multicast. Takes one camera feed and it multicasts it or multiplies

[07:45] what does that mean? So, let's take one camera at one stadium. It sends its one When the network receives it, it looks up and goes, "Hey, um I've got camera A.

[07:58] Who wants it? Oh, you?" "You?" It duplicates it to whoever wants to see it. Fox, Telemundo, ESPN, George. And the magic is the camera's only sending one stream down the network connection.

[08:14] 100 people might want to see that camera, and without multicast, it might have to send 100 streams, killing our links, and that would kill the World Cup. But no, in this situation, that one camera sends one stream, and the network

[08:26] fireworks. Actually, it's kind of like a firework, isn't it? The coffee break. Level two. I just made that level up. I

[08:39] two, but we're at two now. Remember, our cameras are sending two feeds. So, when those two feeds arrive at the network, it's going to send those two feeds to each receiver that wants them. So, Fox, you get two feeds. Telemundo, you get

[08:52] two feeds. George. And then multiply that times, you know, 45. Or in the case of the final match, there's going to be 50 cameras. And then you add the audio in there. And then add on top of that that the IBC is equipped to run six

[09:04] matches at one time. All these games being broadcasted, every camera coming in. Isn't that crazy? Martin thinks so, too. And just so you know, a traditional regular network could not do this. It's too crazy. And that is where Martin

[09:18] comes in. >> So, we've got a >> So, we've got a uh ourselves, uh called TFC, um which has a view of the entire

[09:32] >> And how does it get Does it get a view? Does it like plug in through API to like >> Correct. So, plugging in through GRPC to all the Nexus switches as well as the few Arista switches that we have

[09:46] within the network. And then it uses the same means to throughout the network. So, for every stream, there's a backup stream.

[09:58] Not technically a backup stream because either could be active at any time. some pretty sick Cisco Nexus switches. Easy for me to say. That was a hard handle multicast networks, but not a multicast network like this. Because not

[10:13] only are we casting out to a billion devices. Don't quote me on that. It's a lot. But we're also sending a duplicate stream to each receiver. And those streams actually need to take a unique path through the network for redundancy.

[10:26] packet, which is how they refer to them, going through the same switches because that defeats the point. You want them taking a unique path through the network just in case. That's why Martin and his team, and the company he works for, has

[10:39] a software called TFC. It actually sits on top of the network. And it helps tell network, "Hey, all these multicast streams are supposed to go." Level three,

[10:52] >> coffee break. You ready? Actually, this one's not that bad. It might be. Every receiver, every single thing that receives a camera feed, it receives two streams. Let's say camera A, and it's Messi scoring a goal

[11:04] or something. As the kick comes in to the receiver, it comes in the form of network packets. That's what they call them. It will get red packets and blue use both. And here's how it does it. It's

[11:17] whichever comes first. Packet one comes in. If red packet one is first, it'll use red. Packet two comes in. If blue is first, it'll use blue. Isn't that crazy? you You to admit that sounds pretty neat. And they do this because, again,

[11:32] >> And then to fully close out our question one, those receivers, again, are like Telemundo and ESPN and Fox. It's then their jobs to broadcast it out to us. almost time for kickoff. France versus Morocco, here we go.

[11:53] want to show you something. It's 1:20 a.m. I have to post this video tomorrow, things I just built. I wanted to have you guys experience the IBC, so I made this. By the way, this video is sponsored by Hostinger. And I

[12:06] >> [laughter] >> The backrooms of the IBC. I built it with Hermes agent running on that Hostinger VPS with my ChatGPT subscription and it did okay. Need some coffee.

[12:19] one. Go to the URL. It's there. Let's go to the central equipment room. >> [laughter] next one. {forward slash} comic on this one.

[12:31] I'm running a comic on this VPS. The idea here was I want my kids to learn how all this works and maybe your kids too. The game is in Boston. Let me know going to add more stuff to this. Now, these two things were random ideas that

[12:46] I had today. If you have a random idea, go build it. There's nothing stopping you from going to hostinger.com {forward slash} chuckfifa. Get yourself a VPS, install Hermes on it, or any kind of AI, and just start

[13:01] with. And I think my coupon code still works. Let's see. Oh, yeah. Totally works. So, what are you waiting for? Go make that dumb thing wrong. Now, back to me about how we almost missed the kickoff.

[13:14] >> Is the game about to start? >> Yeah. flashing there. >> Everyone was so chill. I was expecting >> Everyone was so chill. I was expecting like NASA counting down. Here we go. No.

[13:27] had time to turn around and talk to us as we were interviewing each person. I I I asked them why. It's because they spend years preparing for this. >> This event takes three years of preparations.

[13:41] Cup last year. >> That network you see, that was fully planned, tested. They do fake matches. They do audio sync hours before the match happens. It honestly is a very well-oiled machine. And I kept asking

[13:54] wrong? What What What happens?" And like they're all like, "No, it's been a >> We are knocking on wood every day [laughter] to >> So as cool as the NASA control room was, uh kind of calm. Then we walked around

[14:08] and got to talk to people. This is Alex. He's the supervisor of the MCR. >> All those people who are sitting in the front row are taking care to receive all the feeds from the venue. We speak about hundreds of feeds both

[14:20] venue. >> Now I had to ask him the question, "Alex, do you like football?" >> I don't. Sometimes I when the match is the score, but I know that the picture was fine, the audio was fine.

[14:34] checks and the commentators are like getting ready. I love seeing that. And to those commentators and pretty much everyone involved with the World Cup. >> So I'm Wolfgang Herrmann and I'm here

[14:47] the communication engineer for the IBC. I know some people don't can't believe how many people are communicating uh from the venue to the IBC and and >> And his team that manages the commentators has an interesting problem.

[15:01] >> Like 99% of the time the commentary audio they reach me much faster than the >> Oh. >> Meaning if we don't put the right delay, goal. This would be the the worst nightmare if somebody calls a goal

[15:16] >> And then we have the most chill guys in the room that almost made us miss the >> You could call us the layer one guys. So, the cabling, the the guys who made the design hand us the designs and we make sure that all the cabling gets

[15:29] we call the sub stars. >> And I didn't know this. They actually only 21 and he's already helping make the World Cup happen. Do what that guy interfaces they were using to control everything? Steve does that.

[15:41] >> So, you're you're building dashboards and and controls and interfaces for >> These used to be physical controls and now it's all done in software from a graphs. >> We can zoom into the switches and see

[15:55] >> That's just for one that is displayed. >> Also, Martin logged into a switch and fun. >> How to? Okay. Okay.

[16:07] that talk about packets makes me want to see one. This is goal number two. We talked a lot about how the World Cup goes through the network. They broadcast the entire World Cup through the network. Because the World

[16:20] Cup is run by network engineers, it means that we can actually capture part of it as network traffic. It's called a pcap or a packet capture and I've got me. I asked Kristoff and he's like, "Sure." And then I got an email.

[16:33] You want to see it? This is an actual World Cup game but in network traffic at the moment it happens. That for me is like the coolest I'm going to open up the pcap. And this is your first time ever seeing one,

[16:47] stick with me. It's okay cuz it doesn't look very exciting at first. nothing, right? Each one of these is a packet. And look how many there are. If I scroll down to the bottom, you can see that Oh, bam.

[17:00] Let me scoot this over. 10,000 packets. You know how much of a game this is? 27 milliseconds. Now, when a receiver receives this information, these packets, it can decode it and turn it into the game. And you know what? I I we

[17:15] can do the same thing. So, I threw this in a chat GPT. in a chat GPT. >> [music] that's essentially well, let's see. Look [laughter] at that.

[17:27] That's awesome. Now, you might notice this is not the game that happened in Boston. This is not France and Morocco. Who is that? It's a Portugal and you the capture. That's when I got it. But, how cool is this? This is actual

[17:42] image from the World Cup. This was what was sent through the cameras live. >> Now, I saw this and I'm like, "Christoph, can you send me one longer? >> [snorts] >> So, he sent me this. And it's this one

[17:56] going to try and have chat GPT decode It's going to be a video, not just an image from this Wireshark capture. But, first, this is the secret sauce. We're actually seeing what we've been talking

[18:09] about this entire video. Get your coffee stinking ready. Now, this one is bigger. first. Take it a minute. Ah, here we go. 530,000 packets in this one. A whopping 1 second of footage. I'll take it. 5,196 megabits per second.

[18:25] speed? It comes in uncompressed. No codec, not MP4 like you're watching me now on YouTube, raw pixels. This is better than watching it on TV. Now, Here we go. Now, don't worry, we are going to decode the packet here in a

[18:39] But, you got to learn some stuff along the way. Now, we'll go to statistics and on a network are just talking all the time. Let's see the biggest talkers. this? Remember, this packet capture is going to have not one camera feed,

[18:55] beginnings of this right here. Notice these two conversations right here have similar amount of packets. And then right here is the multicast address that they're sending to, or addresses they're sending to. Notice, in the second octet,

[19:08] two different networks. First one's on 182, the second one's on 181. Two identical copies on two physically separate networks, otherwise known as our red and blue paths. Also worth noting is that .20 is video and .30

[19:24] is audio. Video audio. Ah, love this stuff. Okay, time to up our level of nerdiness. We're already there though, aren't we? already there though, aren't we? Up here in my filter,

[19:42] what the junk is 2110? It's a standard for how to transmit networks, which is what we're doing right now. So, we're only looking at right now. So, we're only looking at 2110 packets. So, all this

[19:56] you're watching 1080p video, each frame, which is like this, and this capture will be 1,080 rows. What's a row? SRD row number. Look, row zero.

[20:10] Four rows is one packet. Row one. Row two. Or, I'm sorry, that was one, two, three. So, one complete frame requires 4,320 packets. Let me prove it to you. If I add to the filter

[20:24] to you. If I add to the filter rtp.marker I can see the boundaries of every time we have a complete frame. So, here's a frame. Here's a frame. Here's a frame. Here's a frame. Notice

[20:38] we're only displaying 118 packets right now in total. Now, the video coming across is 1080p at 59.94 frames per second. Yet, we're seeing 118 packets. Because it's sending it twice. Look, you can see it right here. Let me remove our

[20:54] RTP marker. I added this insane filter to show you just two packets, the exact same packets. Sequence number, timestamp, exactly the same. But, notice networks. But then, I got a question for you.

[21:09] Which of these packets do you think was received as the actual video? Which one the packet that arrives first is the one that's chosen. arrived a little bit faster. Now, let's decode it and see what we got. Past

[21:24] Chuck, take over. I had ChatGPT decode this, and here's the result. And this >> [laughter] >> this one was special. Did you see it? That was it. [laughter] You want to see it again?

[21:40] part was that guy dancing in the bottom right. >> Cuz that's so good. If we go back to the IBC, and we get lost a bit into the back room. Now, here they're mixing 5.1 surround sound. A match was actually

[21:53] was pretty sick. But I cannot stop looking at the probably the >> probably the lamest looking thing on the screen, but I thought it was amazing. It's this 2D what looks like a game from the '80s. But this is what's happening

[22:06] in the game live. Notice the dot they're kicking around. That's the ball. Notice how when they kick it, based on how strong they kick it, I guess the velocity in which they kick it, the line goes out to show you that. Now, these

[22:19] metrics are coming from the ball. The ball has smarts in it, chips that where it's at. Actually has a wireless charger it sits on. You may have seen the sound. There are microphones on the field, all around the pitch, ground

[22:33] sounds. Okay, cool. That's what does it. But it's more complex than that. What you're seeing on that screen there, and why it's in the audio room, is that the ball, as it gets near to certain microphones, it's auto adjusting the

[22:47] audio, auto editing the audio, because it knows the ball is right there. So, it picks up the sound more. Isn't that amazing? Now, Kristoff told me that back this was, but they would have a guy sitting there with like a touch tablet

[23:00] pen, and be watching the game, and kind of be following the ball around and the said he didn't know which one was cheaper. I'm telling you my journey, my tour through this whole World Cup thing, I can't give you everything cuz that

[23:12] full of moments like that. Just like, "Oh, wow. They That's so cool." Okay, I have a confession. The stadium we've been going noticed that cuz we can't drive to Boston that fast. It's actually at the

[23:26] Dallas stadium right here in Dallas. Now, every stadium is the same as far as what FIFA adds to it and that's the interesting part actually. So, we arrive there and by the way, remember my goal. I want to get on that pitch. I want to

[23:39] snipers, so I don't know. But, we arrive there and the first thing we see is the technical stuff. They actually do have a street called Tech Street and FIFA Street. And what's interesting is that while they don't actually do a lot of

[23:51] make the match, they do still have everything there to make the match. So, remember the things we we looked at at the IBC, the shaders, the replay, the audio engineers who do the audio editing, all of that, they have all that

[24:06] there just in case. Can you funny to me is I was like, "Okay, so you have a guy sitting here and he's mixing stuff and no one ever hears what he does." He's like, "Yeah, yeah, it's just

[24:19] in case. Just in case." What? Then I'm like, "Okay, what situation would you need to have them do stuff?" He's like, "Well, if the three connections from our stadium to the IBC go down,

[24:33] we saw a fiber cut happen in real time. Well, it wasn't it wasn't real time, but I like to pretend it was. Just for us it happened. But, if that does go down, there, but how do they get the match to everyone else? This thing right here,

[24:46] this truck. It has a satellite on it. Just in case. They have a procedure for when this happens. They've got people on site to start mixing that match, to doing the replays. Look at the data center they have here. They have a small

[25:00] the IBC, including those really expensive replay servers. And it gets sent over a really complex satellite network. And by the way, they can satellite that back to the IBC. Or if the entire IBC is just down, no

[25:14] connection at all, they can actually satellite to the other broadcasters. The matter what. What's funny though is the things that do still occur at the The directors are still there. Every match has a director who's deciding

[25:28] which cameras to switch to and and and how the matches they're not saying, "Okay, and Messi score now." But, you know, they're just directing all the things, all the shots, and all the stuff. But, they do talk

[25:40] really crazy is that when HBS comes into the stadium, they don't use any of the of the existing fiber or existing runs or anything. They run all their own stuff. And actually, we ran into a lady who was in charge of the fiber for the

[25:54] award for like the best fiber. And they didn't even use any of it. I I don't for the moment of truth. Can I get on the pitch and and play? Like I brought my ball with me. I was walking around with my ball. I was so excited. And we

[26:08] actually got to enter where the players enter. Like where their buses come in. big and intimidating. And when you walk through the tunnel, you could start to smell the smell of fresh grass, which that's a new thing they added. Here at

[26:21] stadium, still is, but it had to be converted into a soccer stadium. The World Cup requires that we have real grass. They did a half grass job. Sorry, that was a dad joke for you. But, literally, they have turf sewed in with

[26:34] real grass. It's half and half. Not only that, they had to raise the pitch 1.2 m. You can see the difference on these stairs. Also notice that the field looks a little weird right now. They had to grow real grass. The stadium is

[26:48] indoors, felt amazing. But, grass needs sun to grow, and they needed a solution, so it's these lights. These lights take 6 hours to lower, 6 more hours to fully rise. They really have to plan that out. I

[27:01] them for the game. Now, I wasn't going to let that get in the way. I could run around the things, but what they said was true. I could not get on the pitch. it, and I'm like, "What about him?" Gavin, who was walking us around, who

[27:15] was the stadium manager, he had this patch on his chest, "Bad decisions make Ultimately, I obeyed the rules because HBS and FIFA and everyone was so kind to let us come, so I wasn't going to break the rules. But, I did juggle the soccer

[27:28] the rules. But, I did juggle the soccer ball around the stadium. I dribbled even scored a goal. Check it out. >> Even they they're putting >> World Cup goal? >> Yeah.

[27:40] make-up cams. >> I didn't >> It wasn't upright, but I got it in. In the net. Upper 90. Not sure which side.

[27:54] there. I'll take it. I kicked the ball into that goal. >> I was desperate. Now, what was fun is I actually ran into a fan. He was the all the games for free. We got to nerd out for a bit. It's great to meet you,

[28:08] by the way. What was your name? George? Alan? Bob? Forget. You'll know. I also to go into their locker room. Nine World Cup matches were played in the stadium, the most of any stadium. I went to one of those games. It was surreal to be

[28:21] players were. And I did leave a few stickers for any player who might have watching this, you're welcome. Now, the craziest thing out of the entire tour that I heard is that all of this is temporary. All of

[28:35] it. The IBC and the data center. Everything they stood up to have this intense network, the most mission critical network I've seen. They just tear it down. They built it in like 4 months. Here's a time lapse.

[28:48] And Kristoff said, "As soon as the trophy is is raised, they start packing up. They'll be home in 4 days." It takes a few I think they said like 4 or 6 here. To fully take everything down, but it's temporary. One of the most intense

[29:02] impressive networks I've ever seen is temporary. But, it runs so well good. I got to meet so many of them. It was such a joy to get to know them. I brought them coffee, so I think that my coffee is also

[29:15] contributing to the success of the World Cup. Cuz everything in IT needs coffee, and the World Cup needs IT. So, the World Cup needs my coffee. You're welcome, Messi. That's all I got. Thank you, Kristoff, Bart, Steve, and everyone

[29:27] involved for making our trip possible. I'll catch you guys next time. Hey, videos, I like to pray for you, my audience. Now, this prayer is going to right now. Making this video was actually very

[29:40] hard. We had a very short amount of time with a ton of footage. So, I'm feeling right now just a ton of anxiety and stress. So, I'm going to pray just for relief for myself and for you if you're feeling anxious or stressed. Just saying

[29:52] >> [laughter] >> So, um let's pray. God, I thank you for lives the lives we get to live. Um if uh if if the person watching this is an IT

[30:07] a blessing to be able to do the things we do, to impact the world. Networking mission-critical. I just thank you, God, that you give us And I ask right now that if there's any stress or anxiety being felt by the

[30:22] person watching this and for me right now, relieve that stress. Uh your yoke is easy. Your yoke is and we can give you our burdens and you'll take them from us, God. So, we

[30:34] ask this in your name that you just help us right now. And I I pray just now. Over you watching right now. I pray peace over you. And ultimately, I pray that you will

[30:47] you will know God. You'll know Jesus. I feel stress all the time. But it means I have someone to turn to when I do have So God, I just ask you help me right now.

[30:59] And you help the person watching. I ask this in your name, Jesus. Amen. Thank you for letting me do that. I'll catch you next time.