Release: 2026/11/17 12:09 Reading: 0
Original author:Ohmhytte
Original source:https://www.youtube.com/embed/HePsdhmLf_A
A modern AI chip can do about a thousand trillion calculations a second. Ask it one question, and the answer still arrives a fragment of a word at a time, while the part of the chip that does the arithmetic spends most of its time waiting. Waiting for what? We follow one question, "Why is the sky blue?", through the chip. The text becomes tokens and then whole numbers; each number becomes a list of thousands of numbers; and every layer of the model is mostly matrix multiplication, cut into tiles and shared across the chip's tensor cores. The question is read all at once. The answer can only be written one token at a time, and every new token needs nearly every one of the model's weights read from memory again: for a model with eight billion 16-bit weights, about 15 GB per token. At the 3.35 terabytes a second of NVIDIA's H100, that takes about 4.5 thousandths of a second, while the arithmetic itself would take about 15 millionths. Add the KV cache, the keys and values kept for every earlier token, and you see why AI chips are built around memory, and why data centres batch many conversations into one read, store weights in fewer bits and split big models across several chips. The token ids and the next-token probabilities shown are real: they come from a small open-weight model asked the same question for this film. The H100 figures are NVIDIA's published peak specifications. The chip on screen is a generic, stylised accelerator drawn for this film, not any real chip's layout; the timings are an idealised count (peak rates, no overheads), so real systems are slower; and the 8-billion-weight model has the published shape of Meta's Llama 3 8B. Chapters 0:00 One question, as numbers 1:15 Lists of numbers 2:15 Arithmetic in tiles 3:34 The memory wall 5:06 One token, one layer, one choice 6:31 More people, fewer bits, more chips 8:01 The answer, one piece at a time Sources - NVIDIA Technical Blog, "NVIDIA Hopper Architecture In-Depth" (2022): https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ - NVIDIA, H100 Tensor Core GPU specifications: https://www.nvidia.com/en-us/data-center/h100/ - NVIDIA Technical Blog, "NVIDIA Ampere Architecture In-Depth" (2020): https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/ - NVIDIA Technical Blog, "Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server and FasterTransformer": https://developer.nvidia.com/blog/accelerated-inference-for-large-transformer-models-using-nvidia-fastertransformer-and-nvidia-triton-inference-server/ - NVIDIA Technical Blog, "Mastering LLM Techniques: Inference Optimization": https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/ - NVIDIA Deep Learning Performance Guide, "Matrix Multiplication Background": https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html - A. Vaswani et al., "Attention Is All You Need", NeurIPS 2017: https://arxiv.org/abs/1706.03762 - R. Sennrich, B. Haddow, A. Birch, "Neural Machine Translation of Rare Words with Subword Units", ACL 2016: https://arxiv.org/abs/1508.07909 - J. Kaplan et al., "Scaling Laws for Neural Language Models" (2020): https://arxiv.org/abs/2001.08361 - R. Pope et al., "Efficiently Scaling Transformer Inference", MLSys 2023: https://arxiv.org/abs/2211.05102 - W. Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023: https://arxiv.org/abs/2309.06180 - J. Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", EMNLP 2023: https://arxiv.org/abs/2305.13245 - Llama Team, AI @ Meta, "The Llama 3 Herd of Models" (2024): https://arxiv.org/abs/2407.21783 - T. Dettmers et al., "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", NeurIPS 2022: https://arxiv.org/abs/2208.07339 - E. Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023: https://arxiv.org/abs/2210.17323 - J. Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", MLSys 2024: https://arxiv.org/abs/2306.00978 - SK hynix Newsroom, "SK hynix Develops Industry's First 12-Layer HBM3": https://news.skhynix.com/sk-hynix-develops-industrys-first-12-layer-hbm3/ Music and sound effects: original, made for this film. Fonts: Source Sans 3 (Adobe) and Figtree (Erik Kennedy), SIL Open Font License 1.1. #AI #Technology #Science #Engineering #Computing
子敬.轮回珠宝
2026-11-22 23:55
kumis patiler
2026-11-22 23:55
Ha Ms
2026-11-22 23:55
CRYPTO_BEN
2026-11-22 23:55
CryptoMetric
2026-11-22 23:55
劇翻天 Drama Go
2026-11-22 16:36
CryptoMason
2026-11-22 16:22
Lý Tử Ly
2026-11-22 14:42
Pidoku
2026-11-22 14:15
Select Currency
US Dollar
USD
Chinese Yuan
CNY
Japanese Yen
JPY
South Korean Won
KRW
New Taiwan Dollar
TWD
Canadian Dollar
CAD
Euro
EUR
Pound Sterling
GBP
Danish Krone
DKK
Hong Kong Dollar
HKD
Australian Dollar
AUD
Brazilian Real
BRL
Swiss Franc
CHF
Chilean Peso
CLP
Czech Koruna KČ
CZK
Singapore Dollar
SGD
Indian Rupee
INR
Saudi Riyal
SAR
Vietnamese Dong
VND
Thai Baht
THB
Select Currency
US Dollar
USD-$
Chinese Yuan
CNY-¥
Japanese Yen
JPY-¥
South Korean Won
KRW -₩
New Taiwan Dollar
TWD-NT$
Canadian Dollar
CAD-$
Euro
EUR - €
Pound Sterling
GBP-£
Danish Krone
DKK-KR
Hong Kong Dollar
HKD- $
Australian Dollar
AUD-$
Brazilian Real
BRL -R$
Swiss Franc
CHF -FR
Chilean Peso
CLP-$
Czech Koruna KČ
CZK -KČ
Singapore Dollar
SGD-S$
Indian Rupee
INR -₹
Saudi Riyal
SAR -SAR
Vietnamese Dong
VND-₫
Thai Baht
THB -฿