How YouTube Uploads Huge Files Without Breaking
30sExplains a clever chunking trick that prevents upload failures, offering a practical insight viewers can appreciate.
▶ Play ClipAI Summary
This video explains the architecture behind YouTube's video upload pipeline, detailing how a 10 GB file is processed from client upload to global CDN distribution. It covers chunked uploads via pre-signed URLs, a DAG-based processing pipeline with parallel transcoding, and the final push to edge nodes for low-latency viewing.
[00:01]
Chunked Upload with Pre-Signed URLs
The client splits the file into chunks and uploads them to blob storage using pre-signed URLs. This allows retries on individual chunks rather than the entire file.
[00:16]
DAG Fan-Out and Transcoding
After upload, the pipeline fans out into a DAG. The heaviest branch is transcoding, where the video is segmented and a job is queued for each segment at every target resolution (360p to 4K).
[00:28]
Parallel Processing Workers
Workers pull jobs from the queue and process transcoding in parallel. Other branches handle integrity checks, metadata extraction, and copyright matching.
[00:44]
Central Jobs Table and CDN Distribution
A central jobs table tracks completion across all branches. Once processing finishes, the video is pushed to CDN edge nodes globally, so viewers in Tokyo don't pull bytes from Virginia.
YouTube's upload pipeline is a masterclass in distributed systems, using chunked uploads, parallel processing, and global CDN caching to handle massive files efficiently.
Mentioned in this Video
Study Flashcards (5)
How does YouTube handle uploads of large files?
easy
Click to reveal answer
How does YouTube handle uploads of large files?
easy
Click to reveal answer
The client splits the file into chunks and uploads them via pre-signed URLs, allowing retries on individual chunks.
00:01
What is the heaviest branch in YouTube's upload DAG?
medium
Click to reveal answer
What is the heaviest branch in YouTube's upload DAG?
medium
Click to reveal answer
Transcoding, where the video is segmented and a job is queued for each segment at every target resolution.
00:16
What resolutions are targeted during transcoding?
easy
Click to reveal answer
What resolutions are targeted during transcoding?
easy
Click to reveal answer
From 360p up to 4K.
00:28
What does the central jobs table do?
medium
Click to reveal answer
What does the central jobs table do?
medium
Click to reveal answer
It tracks completion across every branch of the DAG.
00:44
Why does YouTube push videos to CDN edge nodes?
medium
Click to reveal answer
Why does YouTube push videos to CDN edge nodes?
medium
Click to reveal answer
So viewers in different regions don't have to pull bytes from a distant data center, reducing latency.
00:44
💡 Key Takeaways
🔧
Chunked Upload Strategy
Explains a key technique for handling large file uploads reliably.
00:01
💡
DAG-Based Processing
Illustrates how complex pipelines are structured for parallelism.
00:16
⚖️
Global CDN Caching
Highlights the importance of edge caching for performance.
00:44Full Transcript
[00:01] back-end after user clicks upload? Well, first, the client splits the file into to blob storage via what's called a pre-signed URL. This way, any dropped connection will retry just that one chunk, not the whole 10 GB file. Once
[00:16] uploaded, the pipeline then fans out into a DAG. The heaviest branch in this DAG is transcoding, where the video is put into segments and a job is queued for each segment at every target resolution, all the way from 360p up to
[00:28] 4K. Workers then pull off the queue and process the transcoding in parallel. branches are doing things like integrity checks, extracting metadata, generating content for copyright matches. There's also a central jobs table that tracks
[00:44] the completion across every branch. When all that processing finishes, the video pushed out to CDN edge nodes all over the world. This is so that a viewer in Tokyo isn't having to pull bytes all the way from a data center in Virginia. Read
[00:58] our full breakdown on how to design YouTube at hellointerview.com.
