Speaker
Reiner Pope
Appearances over time
1 episodes
Episodes
1
Podcasts
Quotes & moments
A p-bit by q-bit integer multiplier requires exactly p×q AND gates to generate partial products, scaling quadratically with bit width.
A DADA multiplier uses p×q full adders because each full adder eliminates one bit from the partial product tree, reducing 24 input bits to 8 output bits.
In a classic CUDA core / CPU data path, about 7/8 of the circuit area is consumed by MUX circuits reading the register file, not by the actual multiply-accumulate logic.
Halving the bit precision of a multiplier reduces circuit area quadratically, meaning FP4 should theoretically be 4× faster than FP8, not 2×.
Starting with the B3-100, NVIDIA began acknowledging the quadratic scaling and now specs FP4 at 3× the throughput of FP8, though the theoretical maximum is 4×.
The first manufactured ASIC requires a full tape-out costing around $30 million, while the first FPGA costs roughly $10,000, making FPGAs ideal for frequently changing designs.
An equivalent FPGA implementation costs approximately 10× more in silicon area and energy than a dedicated ASIC, because each programmable LUT uses ~32 gates to implement what an ASIC does with ~3.
Older Google TPU chips used a 128×128 systolic array, representing the most area-efficient known hardware circuit for matrix multiplication.
A CPU's on-chip cache is approximately two orders of magnitude (100×) faster than off-chip DDR memory, making it essential for reasonable CPU performance.
A modern CPU achieves roughly 1,000-way parallelism from ~100 cores each with ~16-way vector units, far less than a GPU.
Most of a chip's energy is consumed by charging and discharging capacitors as bits toggle between 0 and 1, known as dynamic or switching power.
A GPU is architecturally equivalent to many small TPUs tiled across the chip — each streaming multiprocessor (SM) contains its own Tensor Core (like a mini MXU) plus a vector unit and register file.
Splitting a logic cloud with a pipeline register in the middle allows the chip to run at twice the clock frequency, at the cost of additional register area.
A chip's global clock synchronizes all circuitry in lockstep approximately every nanosecond, enabling coordination across 100 billion transistors.
A systolic array achieves quadratically more compute than a CUDA core while requiring only linearly more communication, by storing the weight matrix locally and reusing it across many input vectors.
In a classic CPU or CUDA core, 7/8 of the circuit area is consumed by MUX circuits just to read and write the register file — not by actual computation. The multiply-accumulate unit you care about is a tiny fraction of the silicon. This is the fundamental problem that Tensor Cores and systolic arrays were invented to solve.
The fundamental AI chip operation — multiply-accumulate — requires exactly p×q AND gates to generate partial products, then p×q full adders (3-to-2 compressors) to sum them in a DADA tree. Every atomic step in long multiplication maps directly to a physical logic gate. This is why area scales quadratically with bit width.
Multiplier circuit area scales quadratically with bit width, so halving precision from FP8 to FP4 should yield 4× more throughput. NVIDIA historically reported only 2×. The B3-100 finally moved to 3×, but the theoretical max is 4×. This quadratic scaling is the single biggest reason low-precision AI arithmetic works so well.
A systolic array stores the weight matrix locally in registers right next to the compute units and reuses it across many input vectors. This means you get x×y compute operations for only x units of register file bandwidth — a linear-vs-quadratic advantage that is the entire reason Tensor Cores exist.
A chip's global clock forces every transistor to synchronize in lockstep every nanosecond. Without it, two paths through logic could produce outputs that arrive at different times, corrupting results. Pipeline registers let you split logic to raise clock speed, but a feedback loop in the adder shows the hard limit: you can't pipeline your way out of a recurrence.
An FPGA emulates any circuit using lookup tables (LUTs) and programmable MUXes, but each 4-input LUT costs ~32 gate-equivalents to implement what an ASIC does with ~3 gates. That's the source of the ~10× area and energy penalty. The trade-off: first ASIC costs $30M in tape-out; first FPGA costs $10K.
A CPU cache lets hardware silently decide whether to fetch from fast on-chip memory or slow DDR — introducing non-deterministic latency. A TPU scratchpad gives software explicit instructions for on-chip versus HBM memory. The cache gives 100× faster access on a hit, but the non-determinism is the price.
Most CPU die area goes to cache and branch predictor hardware — not ALUs. The branch predictor must guess a branch outcome 5 clock cycles before it's evaluated so the pipeline doesn't stall. GPUs strip out branch prediction and simplify register files, which is why you can fit thousands of GPU cores where a CPU has a hundred.
A GPU runs at GHz clock speeds because it processes batch sizes of 1,000 simultaneously. The brain runs at a much slower clock because it only ever processes one instance of itself. Running a GPU at MHz instead of GHz would give roughly 1,000× less energy consumption — but not a 1,000× improvement in energy efficiency per operation.
A TPU has a few large systolic arrays (MXUs) with a shared vector unit. A GPU is the same architecture tiled many times in miniature: each SM has a Tensor Core (mini-MXU) plus its own vector unit. The GPU trades larger systolic array amortization for more parallel data paths between vector and matrix units.
Analysis
What they talk about
- Technology 100%
