Almost all chip area is wasted on moving data around — actual computation (the multiply-accumulate) is a tiny fraction of the silicon, and every GPU/TPU innovation is really just trying to fix that ratio.
Dwarkesh Podcast
Almost all chip area is wasted on moving data around — actual computation (the multiply-accumulate) is a tiny fraction of the silicon, and every GPU/TPU innovation is really just trying to fix that ratio.
TL;DR
Reiner Pope, CEO of MatX, delivers a blackboard lecture building chip design from first principles — starting with logic gates and AND/OR primitives, through multiply-accumulate circuits, DADA multipliers, systolic arrays, clock cycles, and pipeline registers [1] — Reiner Pope "A systolic array stores the weight matrix locally in registers right next to the compute units and reuses it across many input vectors. Thi…" 26:10 , then comparing FPGAs vs ASICs [2] — Reiner Pope "A CPU cache lets hardware silently decide whether to fetch from fast on-chip memory or slow DDR — introducing non-deterministic latency. A …" 1:03:40 , CPU caches vs scratchpads, and finally showing that a GPU is structurally a grid of tiny TPUs [3] — Reiner Pope "A TPU has a few large systolic arrays (MXUs) with a shared vector unit. A GPU is the same architecture tiled many times in miniature: each …" 1:18:20 . The single most useful takeaway: almost all chip area is spent on data movement, not computation — and every major architectural innovation (Tensor Cores, systolic arrays, scratchpads) is fundamentally an attempt to fix that ratio.
Reiner Pope, CEO of MatX, delivers a blackboard lecture building chip design from first principles — from AND/OR logic gates through multiply-accumulate circuits, DADA multipliers, systolic arrays, clock cycles, pipeline registers, FPGAs vs ASICs, cache vs scratchpad memory, and finally the architectural relationship between GPUs and TPUs.
Reiner explains why multiply-accumulate is the fundamental AI chip operation. Demonstrates partial product generation with AND gates and introduces the DADA multiplier using full adders (3-to-2 compressors). Area scales as p×q. [1] — Reiner Pope "The fundamental AI chip operation — multiply-accumulate — requires exactly p×q AND gates to generate partial products, then p×q full adders…" 03:28
Traces a CUDA core / CPU data path: register file → MUX → ALU → write back. Shows that 7/8 of circuit area is consumed by MUX data movement, not computation. [1] — Reiner Pope "In a classic CPU or CUDA core, 7/8 of the circuit area is consumed by MUX circuits just to read and write the register file — not by actual…" 20:20
Motivates systolic arrays as the solution to CUDA core inefficiency. Stores weight matrix locally; achieves x×y compute for only x bandwidth. Older TPUs used 128×128 arrays. [1] — Reiner Pope "A systolic array stores the weight matrix locally in registers right next to the compute units and reuses it across many input vectors. Thi…" 26:10
Explains why a global clock synchronizes all chip circuitry every ~nanosecond. Pipeline registers can halve logic depth and double clock speed. Feedback loops (running sums) create the hardest timing constraints. [1] — Reiner Pope "A chip's global clock forces every transistor to synchronize in lockstep every nanosecond. Without it, two paths through logic could produc…" 39:20
Contrasts FPGAs (reprogrammable, $10K first cost) with ASICs (~$30M tape-out, 10× more efficient). Explains LUT internals as truth-table-based MUXes, showing why FPGAs have ~10× overhead. [1] — Reiner Pope "An FPGA emulates any circuit using lookup tables (LUTs) and programmable MUXes, but each 4-input LUT costs ~32 gate-equivalents to implemen…" 52:00
CPU cache is 100× faster than DDR but introduces non-deterministic latency. TPU scratchpad uses explicit software instructions for each memory tier, enabling deterministic latency. Groq and TPUs advertise this. [1] — Reiner Pope "A CPU cache lets hardware silently decide whether to fetch from fast on-chip memory or slow DDR — introducing non-deterministic latency. A …" 1:03:40
CPU cache is 100× faster than DDR but introduces non-deterministic latency. TPU scratchpad uses explicit software instructions for each memory tier, enabling deterministic latency. Groq and TPUs advertise this. [1] — Reiner Pope "A CPU cache lets hardware silently decide whether to fetch from fast on-chip memory or slow DDR — introducing non-deterministic latency. A …" 1:03:40
Compares brain and GPU architectures. Brain's slow clock is explained by batch size of 1 vs GPU's 1,000+. Slowing GPU to MHz reduces energy 1,000× but not energy efficiency per operation. [1] — Reiner Pope "A GPU runs at GHz clock speeds because it processes batch sizes of 1,000 simultaneously. The brain runs at a much slower clock because it o…" 1:15:00
TPU has few large systolic arrays with shared vector units. GPU tiles many miniaturized versions across the chip — each SM is a tiny TPU. GPUs trade large-array amortization for more parallel data paths. [1] — Reiner Pope "A TPU has a few large systolic arrays (MXUs) with a shared vector unit. A GPU is the same architecture tiled many times in miniature: each …" 1:18:20
Chapter 1 · 00:00
Reiner explains why multiply-accumulate is the fundamental AI chip operation. Demonstrates partial product generation with AND gates and introduces the DADA multiplier using full adders (3-to-2 compressors). Area scales as p×q. [1] — Reiner Pope "The fundamental AI chip operation — multiply-accumulate — requires exactly p×q AND gates to generate partial products, then p×q full adders…" 03:28
The fundamental AI chip operation — multiply-accumulate — requires exactly p×q AND gates to generate partial products, then p×q full adders (3-to-2 compressors) to sum them in a DADA tree. Every atomic step in long multiplication maps directly to a physical logic gate. This is why area scales quadratically with bit width.
A p-bit by q-bit integer multiplier requires exactly p×q AND gates to generate partial products, scaling quadratically with bit width.
A DADA multiplier uses p×q full adders because each full adder eliminates one bit from the partial product tree, reducing 24 input bits to 8 output bits.
Multiplier circuit area scales quadratically with bit width, so halving precision from FP8 to FP4 should yield 4× more throughput. NVIDIA historically reported only 2×. The B3-100 finally moved to 3×, but the theoretical max is 4×. This quadratic scaling is the single biggest reason low-precision AI arithmetic works so well.
Halving the bit precision of a multiplier reduces circuit area quadratically, meaning FP4 should theoretically be 4× faster than FP8, not 2×.
Starting with the B3-100, NVIDIA began acknowledging the quadratic scaling and now specs FP4 at 3× the throughput of FP8, though the theoretical maximum is 4×.
Chapter 2 · 16:31
Traces a CUDA core / CPU data path: register file → MUX → ALU → write back. Shows that 7/8 of circuit area is consumed by MUX data movement, not computation. [1] — Reiner Pope "In a classic CPU or CUDA core, 7/8 of the circuit area is consumed by MUX circuits just to read and write the register file — not by actual…" 20:20
In a classic CPU or CUDA core, 7/8 of the circuit area is consumed by MUX circuits just to read and write the register file — not by actual computation. The multiply-accumulate unit you care about is a tiny fraction of the silicon. This is the fundamental problem that Tensor Cores and systolic arrays were invented to solve.
In a classic CUDA core / CPU data path, about 7/8 of the circuit area is consumed by MUX circuits reading the register file, not by the actual multiply-accumulate logic.
Chapter 3 · 26:10
Motivates systolic arrays as the solution to CUDA core inefficiency. Stores weight matrix locally; achieves x×y compute for only x bandwidth. Older TPUs used 128×128 arrays. [1] — Reiner Pope "A systolic array stores the weight matrix locally in registers right next to the compute units and reuses it across many input vectors. Thi…" 26:10
A systolic array stores the weight matrix locally in registers right next to the compute units and reuses it across many input vectors. This means you get x×y compute operations for only x units of register file bandwidth — a linear-vs-quadratic advantage that is the entire reason Tensor Cores exist.
A systolic array achieves quadratically more compute than a CUDA core while requiring only linearly more communication, by storing the weight matrix locally and reusing it across many input vectors.
Older Google TPU chips used a 128×128 systolic array, representing the most area-efficient known hardware circuit for matrix multiplication.
Chapter 4 · 39:11
Explains why a global clock synchronizes all chip circuitry every ~nanosecond. Pipeline registers can halve logic depth and double clock speed. Feedback loops (running sums) create the hardest timing constraints. [1] — Reiner Pope "A chip's global clock forces every transistor to synchronize in lockstep every nanosecond. Without it, two paths through logic could produc…" 39:20
A chip's global clock forces every transistor to synchronize in lockstep every nanosecond. Without it, two paths through logic could produce outputs that arrive at different times, corrupting results. Pipeline registers let you split logic to raise clock speed, but a feedback loop in the adder shows the hard limit: you can't pipeline your way out of a recurrence.
A chip's global clock synchronizes all circuitry in lockstep approximately every nanosecond, enabling coordination across 100 billion transistors.
Splitting a logic cloud with a pipeline register in the middle allows the chip to run at twice the clock frequency, at the cost of additional register area.
An FPGA emulates any circuit using lookup tables (LUTs) and programmable MUXes, but each 4-input LUT costs ~32 gate-equivalents to implement what an ASIC does with ~3 gates. That's the source of the ~10× area and energy penalty. The trade-off: first ASIC costs $30M in tape-out; first FPGA costs $10K.
The first manufactured ASIC requires a full tape-out costing around $30 million, while the first FPGA costs roughly $10,000, making FPGAs ideal for frequently changing designs.
Chapter 5 · 53:25
Contrasts FPGAs (reprogrammable, $10K first cost) with ASICs (~$30M tape-out, 10× more efficient). Explains LUT internals as truth-table-based MUXes, showing why FPGAs have ~10× overhead. [1] — Reiner Pope "An FPGA emulates any circuit using lookup tables (LUTs) and programmable MUXes, but each 4-input LUT costs ~32 gate-equivalents to implemen…" 52:00
An equivalent FPGA implementation costs approximately 10× more in silicon area and energy than a dedicated ASIC, because each programmable LUT uses ~32 gates to implement what an ASIC does with ~3.
A CPU cache lets hardware silently decide whether to fetch from fast on-chip memory or slow DDR — introducing non-deterministic latency. A TPU scratchpad gives software explicit instructions for on-chip versus HBM memory. The cache gives 100× faster access on a hit, but the non-determinism is the price.
Chapter 6 · 1:04:07
CPU cache is 100× faster than DDR but introduces non-deterministic latency. TPU scratchpad uses explicit software instructions for each memory tier, enabling deterministic latency. Groq and TPUs advertise this. [1] — Reiner Pope "A CPU cache lets hardware silently decide whether to fetch from fast on-chip memory or slow DDR — introducing non-deterministic latency. A …" 1:03:40
A CPU's on-chip cache is approximately two orders of magnitude (100×) faster than off-chip DDR memory, making it essential for reasonable CPU performance.
A modern CPU achieves roughly 1,000-way parallelism from ~100 cores each with ~16-way vector units, far less than a GPU.
Chapter 7 · 1:10:07
CPU cache is 100× faster than DDR but introduces non-deterministic latency. TPU scratchpad uses explicit software instructions for each memory tier, enabling deterministic latency. Groq and TPUs advertise this. [1] — Reiner Pope "A CPU cache lets hardware silently decide whether to fetch from fast on-chip memory or slow DDR — introducing non-deterministic latency. A …" 1:03:40
Most CPU die area goes to cache and branch predictor hardware — not ALUs. The branch predictor must guess a branch outcome 5 clock cycles before it's evaluated so the pipeline doesn't stall. GPUs strip out branch prediction and simplify register files, which is why you can fit thousands of GPU cores where a CPU has a hundred.
Chapter 8 · 1:12:00
Compares brain and GPU architectures. Brain's slow clock is explained by batch size of 1 vs GPU's 1,000+. Slowing GPU to MHz reduces energy 1,000× but not energy efficiency per operation. [1] — Reiner Pope "A GPU runs at GHz clock speeds because it processes batch sizes of 1,000 simultaneously. The brain runs at a much slower clock because it o…" 1:15:00
A GPU runs at GHz clock speeds because it processes batch sizes of 1,000 simultaneously. The brain runs at a much slower clock because it only ever processes one instance of itself. Running a GPU at MHz instead of GHz would give roughly 1,000× less energy consumption — but not a 1,000× improvement in energy efficiency per operation.
A GPU is architecturally equivalent to many small TPUs tiled across the chip — each streaming multiprocessor (SM) contains its own Tensor Core (like a mini MXU) plus a vector unit and register file.
Most of a chip's energy is consumed by charging and discharging capacitors as bits toggle between 0 and 1, known as dynamic or switching power.
Chapter 9 · 1:18:15
TPU has few large systolic arrays with shared vector units. GPU tiles many miniaturized versions across the chip — each SM is a tiny TPU. GPUs trade large-array amortization for more parallel data paths. [1] — Reiner Pope "A TPU has a few large systolic arrays (MXUs) with a shared vector unit. A GPU is the same architecture tiled many times in miniature: each …" 1:18:20
A TPU has a few large systolic arrays (MXUs) with a shared vector unit. A GPU is the same architecture tiled many times in miniature: each SM has a Tensor Core (mini-MXU) plus its own vector unit. The GPU trades larger systolic array amortization for more parallel data paths between vector and matrix units.
No indexed bits in this chapter.
In a classic CPU or CUDA core, 7/8 of the circuit area is consumed by MUX circuits just to read and write the register file — not by actual computation. The multiply-accumulate unit you care about is a tiny fraction of the silicon. This is the fundamental problem that Tensor Cores and systolic arrays were invented to solve.
A TPU has a few large systolic arrays (MXUs) with a shared vector unit. A GPU is the same architecture tiled many times in miniature: each SM has a Tensor Core (mini-MXU) plus its own vector unit. The GPU trades larger systolic array amortization for more parallel data paths between vector and matrix units.
Multiplier circuit area scales quadratically with bit width, so halving precision from FP8 to FP4 should yield 4× more throughput. NVIDIA historically reported only 2×. The B3-100 finally moved to 3×, but the theoretical max is 4×. This quadratic scaling is the single biggest reason low-precision AI arithmetic works so well.
This episode
Factual claims made this episode, and whether a source was named.
A p-bit by q-bit integer multiplier requires exactly p×q AND gates to generate all partial products.
A DADA multiplier for a p-bit by q-bit multiply-accumulate requires exactly p×q full adders.
In a classic 8-entry register file CUDA core, approximately 7/8 of circuit area is spent on MUX circuits for data movement rather than the multiply-accumulate logic unit.
Multiplier circuit area scales quadratically with bit width, so FP4 should theoretically deliver 4× the throughput of FP8.
NVIDIA's B3-100 chip specs FP4 at 3× the throughput of FP8, compared to the 2× ratio used in earlier generations like B100/B200.
The first manufactured ASIC requires a tape-out costing approximately $30 million.
An ASIC is approximately 10× more area- and energy-efficient than an equivalent FPGA implementation.
A 4-input LUT in an FPGA costs approximately 32 gate-equivalents to implement logic that an ASIC can implement in 3 gates.
Older Google TPU chips used a 128×128 systolic array.
A CPU's on-chip cache is approximately 100× faster than off-chip DDR memory.
A modern CPU achieves roughly 1,000-way parallelism from ~100 cores each with ~16-way vector units.
Crusoe was one of only five GPU clouds to achieve gold tier in SemiAnalysis' most recent ClusterMAX report.
Gold-tier GPU cloud providers like Crusoe had total cost of ownership 5-15% lower than silver-tier providers, even with identical GPU pricing.
Running a chip at 1/1,000th the clock speed reduces energy consumption approximately 1,000× but does not provide a substantial advantage in energy efficiency per operation.
A GPU is structurally equivalent to many small TPUs tiled across the chip, with each streaming multiprocessor containing a Tensor Core analogous to a TPU matrix multiply unit.
This episode
Discussed as the dominant GPU maker whose FP4/FP8 spec ratios reflect chip area quadratic scaling; Tensor Cores and the Volta/B3-100 generations are analyzed in detail.
Featured as a sponsor; Dwarkesh interviewed senior Jane Streeters Ron Minsky and Dan Pontecorvo about AI use in trading and open technical positions.
Referenced as the leading chip foundry that provides PDKs with primitive logic cells; chip designers send tape-out files to TSMC for manufacturing.
Episode sponsor; described as one of five gold-tier GPU clouds in SemiAnalysis' ClusterMAX report with 5-15% lower TCO than silver-tier providers.
AI chip startup founded by guest Reiner Pope; host Dwarkesh Patel discloses he is an angel investor. Publicly developing a 'splittable systolic array' architecture.
Mentioned as an example of a chip company that has publicly advertised deterministic latency in its AI processor, like TPUs.
Semiconductor research firm whose ClusterMAX report ranked ~100 GPU clouds; used by host to contextualize Crusoe's gold-tier status.
Used as the primary example of a systolic-array-based AI accelerator with scratchpad memory and deterministic latency, contrasted throughout with GPU architecture.
NVIDIA's general-purpose GPU compute cores; used as the example of the pre-Tensor Core architecture where most area is consumed by register file MUXes.
NVIDIA's matrix multiply accelerator units introduced in the Volta GPU generation; described as the hardware implementation of systolic arrays solving the data movement problem.
Episode sponsor and tool used by host Dwarkesh Patel for research, coding internal tools, and building essay contest review interfaces.
Off-chip memory used in AI accelerators like TPUs, contrasted with the on-chip scratchpad; analogous to DDR in the CPU memory hierarchy.
Stats
We use essential and analytics cookies to run Vuci. To understand how the site is used: Privacy Policy.
Install Vuci on your phone
Add it to your home screen for a faster, app-like experience.
Install Vuci on your phone
Tap the Share button, then “Add to Home Screen”.
A new version is available
Reload to get the latest Vuci.