#496 – FFmpeg: The Incredible Technology Behind Video on the Internet

#496 – FFmpeg: The Incredible Technology Behind Video on the Internet

The entire global video internet — YouTube, Netflix, Chrome, Discord — runs on FFmpeg, maintained by roughly 10–15 volunteers who have never been paid a meaningful salary for it.

May 6, 2026 4:23:41 Difficulty: Intermediate Played

TL;DR

FFmpeg and VLC — the open-source software powering nearly all video on the internet — are built and maintained by a tiny band of volunteers. Jean-Baptiste Kempf and Kieran Kunhya explain how video compression works, why hand-written assembly outperforms C by up to 62x, how JB repeatedly turned down tens of millions of dollars to keep VLC ad-free, the Google security-researcher drama that boosted FFmpeg donations, and why the dav1d AV1 decoder's 240,000 lines of handwritten assembly represent a modern engineering masterpiece. The single most useful takeaway: the entire digital video world runs on volunteer passion projects — support them.

#FFmpeg internals #video codec engineering #open source community #assembly optimization #H.264 / AV1 codecs #video compression algorithms #open source licensing #software reverse engineering #maintainer burnout #low-latency video streaming #video archiving #x264 encoder #dav1d AV1 decoder #Google security drama #VLC media player #FFmpeg #VLC #video codec #open source #H.264 #AV1 #assembly programming #video compression #VideoLAN #streaming #x264 #reverse engineering #multimedia #low latency #SIMD #Jean-Baptiste Kempf #Kieran Kunhya #dav1d

Jean-Baptiste Kempf (president of VideoLAN, lead VLC developer) and Kieran Kunhya (codec engineer, FFmpeg contributor) join Lex Fridman to discuss the invisible technology powering all video on the internet: FFmpeg and VLC. Topics span video codecs, containers, open source philosophy, assembly programming, reverse engineering, the Google security drama, turning down millions to keep VLC ad-free, and the future of multimedia.

Chapter list
  • Lex introduces FFmpeg and VLC, praises the open-source volunteer engineers behind them, and gives a rapid sponsor roll call before the full ad reads begin.

  • Extended sponsor reads for Larridin, Blitzy, BetterHelp, Fin, and LMNT, with Lex weaving in personal reflections on AI, mental health, and electrolytes.

  • VLC's legendary resilience: VHS capture cards, DVD Audio, Star Wars game codecs, competition files with per-frame resolution changes, and the infamous cone logo origin.

  • Step-by-step breakdown of decoding: demuxing, entropy coding, intra-prediction, DCT, GPU vs. software paths, and why 45% of files need software fallback.

  • Containers (MP4, MKV, MOV) vs. codecs (H.264, AAC); why VLC ignores file extensions; the 100–1000x compression target; YUV color space and human perception.

  • FFmpeg as a language-like toolbox; its democratizing role in podcasting and YouTube; open-source licensing (MIT, GPL, LGPL); and why changing a license requires contacting every contributor.

  • The legend of Linus Torvalds, Git built in two weeks, the meritocratic harshness of code review culture, and why a core team of 10–15 must maintain code for thousands of contributors.

  • The full story behind JB's Reddit-famous refusal of tens of millions in ad deals, the VideoLAN origin at École Centrale Paris, and why accepting would have betrayed hundreds of volunteers.

  • Google's AI-generated bug reports on obscure codecs, the 'denial of service by CVE' critique, Microsoft Teams demanding free urgent support, and how public X posts forced accountability.

  • Passion for multimedia, the school of programming that beats any university, pride of shipping code used by billions, and teenagers writing production assembly. Andrew Kelly (Zig) started as an FFmpeg developer.

  • The interdependence of VLC, FFmpeg, and x264 explained as a binary star system — VLC is to FFmpeg as Android is to Linux; x264 powers 80%+ of FFmpeg encoding pipelines.

  • Fabrice Bellard creates the concept; Michael Niedermayer's 2000s era of DivX/Xvid support; the H.264 maturity era; native decoders replacing bloated proprietary players like RealPlayer.

  • Kostya's GoToMeeting reverse engineering, stepping through millions of CPU instructions, dumping memory from VMs, needing bit-exact output — and why a 1MB blob takes a month.

  • The FATE automated testing system covers hundreds of compiler/OS/architecture combinations on volunteer hardware, catching silent miscompilations that could cascade into video glitches.

  • Why SIMD assembly beats C by 62x; dav1d's 240,000 lines of handwritten assembly; Henrik Gramner and Martin Storsjö as assembly wizards; the Moore's Law plateau making optimization essential.

  • JB and Kieran debate Rust: valuable for new projects and memory safety, but ill-suited for rewriting existing assembly-heavy codebases, and hand-written assembly breaks any Rust security model anyway.

  • The 2011 governance split, how libav's work was absorbed back into FFmpeg, how forks are a healthy part of open-source evolution, and why FFmpeg emerged stronger.

  • AI slop bug reports, the XZ backdoor caused by deliberate burnout of a solo maintainer, death threats JB received over dropping PowerPC support, and the mental health toll on critical infrastructure maintainers.

  • How anime fans rejected PSNR metrics and invented psychovisual rate distortion; adaptive quantization; the Parkjoy sample as the benchmark that sorted encoders; Blu-ray adoption of x264.

  • I-frames, P-frames, B-frames, intra-refresh, group-of-pictures, AV1 vs. H.264 codec tool collections, ProRes intra-only for editing, and why decoding order differs from display order.

  • WikiLeaks revealed the CIA distributed a fake VLC build with a DLL that exfiltrated documents during movie playback; Chinese hackers also used a signed VLC DLL; VLC was temporarily banned in India.

  • JB's Kyber startup targets 4ms glass-to-glass latency for robot teleoperation using QUIC, forward error correction, multi-stream clock synchronization, and 3D-printed demonstration rovers.

  • AV2 as 30% better than AV1; the HEVC patent disaster that cost hundreds of millions per year and drove creation of the Alliance for Open Media; dav2d as the planned AV2 decoder.

  • Two intelligence agencies requested VLC backdoors and were refused; air-gapped build machines; double-signature verification; detected state-actor fake binary injection; the sandboxing architecture challenge.

  • The archiving community led by Dave Rice uses FFmpeg and FFV1 lossless codec to preserve world multimedia heritage; the Domesday Book failure; C as the Latin of programming languages; the moral hazard of choosing what to archive.

  • VLC and FFmpeg expanding to volumetric video, XR streaming, haptic tracks, spatial audio, brainwave codecs; JB's philosophy on regret; closing tribute to open-source engineers from Linus Torvalds quote.

Codec
Short for coder-decoder; a program that compresses (encodes) and decompresses (decodes) digital audio or video data.
Container / Mux
A file format (e.g., MP4, MKV) that bundles multiple tracks — video, audio, subtitles — into one file; multiplexer = mux, demultiplexer = demux.
SIMD
Single Instruction Multiple Data; a class of CPU instructions that performs the same operation on multiple data elements simultaneously, crucial for fast video processing.
I-frame / Keyframe
A complete, independently decodable video frame, analogous to a JPEG image; all other frame types reference it.
B-frame
A bi-directionally predicted video frame that can reference both past and future frames, enabling higher compression at the cost of decoding complexity.
YUV
A color space separating luminance (Y) from chrominance (U, V); exploits human visual sensitivity to brightness over color to enable efficient compression.
DCT (Discrete Cosine Transform)
A mathematical transform that converts spatial pixel data into frequency-domain coefficients, allowing video codecs to discard perceptually insignificant detail.
Entropy coding
Lossless compression applied to the mathematical symbols in a codec bitstream; common methods include Huffman coding and arithmetic coding.
FATE
FFmpeg Automated Testing Environment; a volunteer-run continuous integration system testing FFmpeg across hundreds of compiler, OS, and architecture combinations.
LGPL
GNU Lesser General Public License; a 'weak copyleft' license allowing proprietary software to link against a library without open-sourcing the whole application.
Copyleft
A licensing principle requiring that derivative works be distributed under the same open-source license, ensuring modifications remain publicly available.
Intra-prediction
A technique where a video codec predicts pixel values from neighboring pixels within the same frame, exploiting spatial redundancy.
AV1
A royalty-free video codec developed by the Alliance for Open Media, offering roughly 40–60% better compression than H.264 at equivalent quality.
Psychovisual rate distortion
A codec optimization strategy that tunes compression decisions to match human visual perception rather than minimizing a purely mathematical error metric like PSNR.
Forward error correction (FEC)
A technique that deliberately over-transmits redundant data so the receiver can reconstruct lost packets without requesting a retransmission, reducing latency.
QUIC
A modern transport protocol built on UDP, designed for low latency and built-in encryption; used by Kyber for real-time video control streams.
PSNR
Peak Signal-to-Noise Ratio; a mathematical metric for image quality widely used in video coding research, often criticized for not correlating well with perceived visual quality.
Demux / Demultiplexer
The software component that reads a container file and separates it into its constituent tracks (video, audio, subtitles) for decoding.
Meritocratic
A system where advancement is based on demonstrated ability and output rather than seniority, status, or personal relationships; how FFmpeg and VLC evaluate code contributions.
Bit-exact
A guarantee that every compliant decoder produces identical output bits for a given input; standard from H.264 onwards but absent in MPEG-2, which caused interoperability problems.

Chapter 3 · 10:48

Weirdest things VLC opens

VLC's legendary resilience: VHS capture cards, DVD Audio, Star Wars game codecs, competition files with per-frame resolution changes, and the infamous cone logo origin.

Claims made here

Over 90% of video processing workflows online and offline involve FFmpeg in some form.

Lex Fridman no source cited

VLC has been downloaded at least 6.5 billion times.

Lex Fridman no source cited

Chapter 4 · 15:12

How video playback works

Step-by-step breakdown of decoding: demuxing, entropy coding, intra-prediction, DCT, GPU vs. software paths, and why 45% of files need software fallback.

Claims made here

Approximately 45% of video files are not GPU-decodable and require a software fallback.

Kieran Kunhya no source cited

Video codecs compress raw video by 100x to 1000x compared to uncompressed formats, compared to about 10x for MP3 audio.

Jean-Baptiste Kempf no source cited

Each new generation of video codec delivers approximately 30% better compression at the same visual quality level.

Jean-Baptiste Kempf no source cited

Technology
How Video Compression Actually Works

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

Video compression isn't just removing data — it's degrading reality in ways humans don't notice. Codecs exploit the fact that your eyes care more about brightness than color, and that the next frame is usually almost identical to the last. The target is 1000x compression, achieved through mathematical transforms, spatial prediction, and temporal redundancy removal.

Chapter 6 · 35:20

FFmpeg explained

FFmpeg as a language-like toolbox; its democratizing role in podcasting and YouTube; open-source licensing (MIT, GPL, LGPL); and why changing a license requires contacting every contributor.

Claims made here

Jean-Baptiste Kempf contacted more than 350 individual contributors to relicense VLC's core from GPL to LGPL.

Jean-Baptiste Kempf no source cited

Technology
What FFmpeg Actually Is

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

FFmpeg is not just a command-line tool — it's a complete multimedia processing universe. It handles codecs, containers, filters, and streaming, and it's embedded inside YouTube, Netflix, Chrome, VLC, OBS, and virtually every smart TV. Your grandmother's home videos and Netflix's trillion-dollar infrastructure use the exact same technology stack.

Chapter 7 · 56:20

Linus Torvalds

The legend of Linus Torvalds, Git built in two weeks, the meritocratic harshness of code review culture, and why a core team of 10–15 must maintain code for thousands of contributors.

Chapter 8 · 1:00:59

Turning down millions to keep VLC ad-free

The full story behind JB's Reddit-famous refusal of tens of millions in ad deals, the VideoLAN origin at École Centrale Paris, and why accepting would have betrayed hundreds of volunteers.

Society & Culture
Turning Down Tens of Millions to Keep VLC Free

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Society & Culture

JB didn't turn down advertising money because he hates money. He turned it down because every offer came from shady ad companies, and accepting would have betrayed hundreds of people's volunteer labor. He could be a multimillionaire on a beach — he chose to keep VLC ad-free for billions of users instead.

Chapter 9 · 1:15:17

FFmpeg & Google drama

Google's AI-generated bug reports on obscure codecs, the 'denial of service by CVE' critique, Microsoft Teams demanding free urgent support, and how public X posts forced accountability.

Technology
The Google Security Drama: AI Bug Reports as a Denial of Service

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

Google used AI to mass-generate security vulnerability reports on FFmpeg — including a 'critical' bug in an obscure 1993 game codec — and went to the media before patches could be deployed. Volunteers faced a flood of high-severity reports they never had resources to fix. The result: FFmpeg pushed back publicly, donations rose, and Google eventually started sending patches.

Business
Microsoft Teams Told Volunteers Their Bug Was 'High Priority'

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Business

A Microsoft Teams manager filed a bug on FFmpeg's public tracker, name-dropped Microsoft's brand to imply urgency, and when asked for a support contract, offered a one-time payment of a few thousand dollars. FFmpeg's response on X went viral, and the incident forced a reckoning about trillion-dollar companies freeloading on volunteer infrastructure.

Chapter 13 · 1:48:59

Reverse engineering codecs

Kostya's GoToMeeting reverse engineering, stepping through millions of CPU instructions, dumping memory from VMs, needing bit-exact output — and why a 1MB blob takes a month.

Claims made here

Hand-written SIMD assembly for pixel format conversion at Open Broadcast Systems runs 62 times faster than equivalent C code.

Kieran Kunhya no source cited

AV1 delivers 40–60% less bandwidth consumption than H.264 at equivalent visual quality.

Jean-Baptiste Kempf no source cited

AV1 encoding requires approximately two orders of magnitude more CPU cycles than H.264 encoding.

Jean-Baptiste Kempf no source cited

VLC's latest version still supports Windows XP through Windows 11 and iOS 9 through iOS 26.

Jean-Baptiste Kempf no source cited

Technology
Why Hand-Written Assembly Beats C by 62x

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

Compilers cannot auto-vectorize anywhere close to what an expert human can do by hand. The dav1d AV1 decoder proves it: 80% assembly, 62x faster than C on pixel conversion functions, running on 3 billion devices. Moore's Law is over — squeezing hardware through hand-crafted assembly is how you get more without buying more.

Technology
Reverse Engineering a Codec From a Binary Blob

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

Reverse engineering a codec means opening a binary with no documentation, stepping through millions of CPU instructions in a debugger, dumping memory from a virtual machine, and slowly inferring — frame by frame — what the compression algorithm is doing. A 1MB blob takes roughly a month. Kostya did 20–30MB blobs regularly, for fun.

Chapter 14 · 2:02:14

FFmpeg testing

The FATE automated testing system covers hundreds of compiler/OS/architecture combinations on volunteer hardware, catching silent miscompilations that could cascade into video glitches.

Chapter 15 · 2:06:21

Assembly code (handwritten)

Why SIMD assembly beats C by 62x; dav1d's 240,000 lines of handwritten assembly; Henrik Gramner and Martin Storsjö as assembly wizards; the Moore's Law plateau making optimization essential.

Claims made here

The dav1d AV1 decoder contains approximately 240,000 lines of handwritten assembly code — more than double FFmpeg's total assembly across all codecs combined.

Jean-Baptiste Kempf no source cited

Approximately 30% of Netflix video content is now delivered in AV1, and approximately 50% of YouTube content is in AV1.

Jean-Baptiste Kempf no source cited

FFmpeg is likely running on 100 million to potentially 1 billion CPUs simultaneously at any given moment.

Kieran Kunhya no source cited

Technology
dav1d: 240,000 Lines of Handwritten Assembly for AV1

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

When the Alliance for Open Media said AV1 was too complex for software decoding, VideoLAN proved them wrong with dav1d: 240,000 lines of handwritten assembly, every cycle optimized, running on 3 billion devices. It enables 720p AV1 decoding on just one or two CPU cores — on hardware that was never supposed to handle it.

True Crime
The CIA Used a Modified VLC to Spy on People

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 True Crime

WikiLeaks' Vault 7 revealed that the CIA distributed a custom VLC build with a hidden DLL that read your documents folder and encrypted and exfiltrated the data — while you watched a movie for two hours and blamed your fans on the CPU load. VideoLAN's only defense was telling people to only download from videolan.org.

Chapter 16 · 2:40:40

Rust programming language

JB and Kieran debate Rust: valuable for new projects and memory safety, but ill-suited for rewriting existing assembly-heavy codebases, and hand-written assembly breaks any Rust security model anyway.

Chapter 18 · 2:48:17

Open source burnout

AI slop bug reports, the XZ backdoor caused by deliberate burnout of a solo maintainer, death threats JB received over dropping PowerPC support, and the mental health toll on critical infrastructure maintainers.

Society & Culture
Open Source Maintainer Burnout Is Civilization's Hidden Crisis

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Society & Culture

The XZ backdoor happened because a single overwhelmed maintainer was systematically harassed at night by two attackers posing as contributors until he gave them commit access. AI-generated bug reports and corporate entitlement are now accelerating this crisis across all of open source. The entire internet runs on people like this.

Chapter 19 · 2:56:04

x264 and internet video

How anime fans rejected PSNR metrics and invented psychovisual rate distortion; adaptive quantization; the Parkjoy sample as the benchmark that sorted encoders; Blu-ray adoption of x264.

Technology
The x264 Psychovisual Revolution

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

For decades, video codec engineers optimized for PSNR — a purely mathematical metric that made video look blurry. Hobbyist anime fans ignored the sacred metrics and optimized by eye instead, inventing psychovisual rate-distortion tuning. x264 implemented their ideas and became the reference every codec since has been compared against.

Chapter 21 · 3:16:17

CIA and fake VLC

WikiLeaks revealed the CIA distributed a fake VLC build with a DLL that exfiltrated documents during movie playback; Chinese hackers also used a signed VLC DLL; VLC was temporarily banned in India.

Claims made here

FFmpeg is used by NASA's Mars 2020 rover to compress pictures sent back to Earth.

Kieran Kunhya NASA Mars 2020 rover published paper on commercial off-the-shelf technology use

Technology
Kyber: Making Distance Disappear for Robots and Drones

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

Real-time robot control demands video latency measured in milliseconds, not seconds. Kyber encodes a single video frame in 4 milliseconds, uses QUIC-based multi-stream transport with forward error correction, and handles clock drift across multi-camera robot setups to guarantee coherent training data for AI models. Current achievement: 7ms glass-to-glass. Target: 4ms (240 Hz equivalent).

Chapter 23 · 3:44:10

AV2 codec and video patents

AV2 as 30% better than AV1; the HEVC patent disaster that cost hundreds of millions per year and drove creation of the Alliance for Open Media; dav2d as the planned AV2 decoder.

Claims made here

AV2 delivers approximately 30% better compression than AV1 at equivalent visual quality.

Jean-Baptiste Kempf no source cited

Kyber has achieved 7 milliseconds glass-to-glass video latency from Windows to Windows, with approximately 3.5ms consumed by the NVIDIA hardware encoder and 2ms by the Intel decoder.

Jean-Baptiste Kempf no source cited

Chapter 24 · 3:54:27

VLC backdoors

Two intelligence agencies requested VLC backdoors and were refused; air-gapped build machines; double-signature verification; detected state-actor fake binary injection; the sandboxing architecture challenge.

Technology
Two Intelligence Agencies Asked for VLC Backdoors

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

At least two intelligence agencies approached VideoLAN requesting backdoors in VLC. The answer was an unambiguous no. VLC compiles on air-gapped machines, starting from scratch with the compiler itself, uses double-signature verification, and has caught what appears to be a state actor attempting to inject fake binaries into their servers.

Chapter 26 · 4:11:04

Future of FFmpeg and VLC

VLC and FFmpeg expanding to volumetric video, XR streaming, haptic tracks, spatial audio, brainwave codecs; JB's philosophy on regret; closing tribute to open-source engineers from Linus Torvalds quote.

No indexed bits in this chapter.

Show stoppers

Technology
The x264 Psychovisual Revolution

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Technology

For decades, video codec engineers optimized for PSNR — a purely mathematical metric that made video look blurry. Hobbyist anime fans ignored the sacred metrics and optimized by eye instead, inventing psychovisual rate-distortion tuning. x264 implemented their ideas and became the reference every codec since has been compared against.

Society & Culture
Turning Down Tens of Millions to Keep VLC Free

#496 – FFmpeg: The Incredible Technology Behind Video on th… · May 6, 2026 Society & Culture

JB didn't turn down advertising money because he hates money. He turned it down because every offer came from shady ad companies, and accepting would have betrayed hundreds of people's volunteer labor. He could be a multimillionaire on a beach — he chose to keep VLC ad-free for billions of users instead.

Snapshots ()

Key Quotes ()

This episode

Cast

  • Track
  • Track
  • Track
  • Track

Stats

Episode stats

Insight Overview

insights
chapters

Insight distribution

Sub-Categories

Speaker breakdown

Talk Time

This episode

Claims & Sources

1 / 16 cited (6%)

Factual claims made this episode, and whether a source was named.

Over 90% of video processing workflows online and offline involve FFmpeg in some form.

Lex Fridman no source cited

VLC has been downloaded at least 6.5 billion times.

Lex Fridman no source cited

Video codecs compress raw video by 100x to 1000x compared to uncompressed formats, compared to about 10x for MP3 audio.

Jean-Baptiste Kempf no source cited

Each new generation of video codec delivers approximately 30% better compression at the same visual quality level.

Jean-Baptiste Kempf no source cited

Approximately 45% of video files are not GPU-decodable and require a software fallback.

Kieran Kunhya no source cited

Hand-written SIMD assembly for pixel format conversion at Open Broadcast Systems runs 62 times faster than equivalent C code.

Kieran Kunhya no source cited

The dav1d AV1 decoder contains approximately 240,000 lines of handwritten assembly code — more than double FFmpeg's total assembly across all codecs combined.

Jean-Baptiste Kempf no source cited

Approximately 30% of Netflix video content is now delivered in AV1, and approximately 50% of YouTube content is in AV1.

Jean-Baptiste Kempf no source cited

AV1 delivers 40–60% less bandwidth consumption than H.264 at equivalent visual quality.

Jean-Baptiste Kempf no source cited

FFmpeg is likely running on 100 million to potentially 1 billion CPUs simultaneously at any given moment.

Kieran Kunhya no source cited

AV2 delivers approximately 30% better compression than AV1 at equivalent visual quality.

Jean-Baptiste Kempf no source cited

AV1 encoding requires approximately two orders of magnitude more CPU cycles than H.264 encoding.

Jean-Baptiste Kempf no source cited

FFmpeg is used by NASA's Mars 2020 rover to compress pictures sent back to Earth.

Kieran Kunhya NASA Mars 2020 rover published paper on commercial off-the-shelf technology use

VLC's latest version still supports Windows XP through Windows 11 and iOS 9 through iOS 26.

Jean-Baptiste Kempf no source cited

Kyber has achieved 7 milliseconds glass-to-glass video latency from Windows to Windows, with approximately 3.5ms consumed by the NVIDIA hardware encoder and 2ms by the Intel decoder.

Jean-Baptiste Kempf no source cited

Jean-Baptiste Kempf contacted more than 350 individual contributors to relicense VLC's core from GPL to LGPL.

Jean-Baptiste Kempf no source cited