Price: $0.05526 0.295%
Market Cap: $8.67B 0.417%
Volume (24h): 407.79M 0%
Dominance: 0.417%
Price: $0.05526 0.295%
Market Cap: $8.67B 0.417%
Volume (24h): 407.79M 0%
Dominance: 0.417% 0.417%
  • Price: $0.05526 0.295%
  • Market Cap: 8.67B 0.417%
  • Volume (24h): 407.79M 0%
  • Dominance: 0.417% 0.417%
  • Price: $0.05526 0.295%
Home > 视频 > What Happens Inside an AI Chip When You Ask One Question

What Happens Inside an AI Chip When You Ask One Question

Release: 2026/11/17 12:09 Reading: 0

Original author:Ohmhytte

Original source:https://www.youtube.com/embed/HePsdhmLf_A

A modern AI chip can do about a thousand trillion calculations a second. Ask it one question, and the answer still arrives a fragment of a word at a time, while the part of the chip that does the arithmetic spends most of its time waiting. Waiting for what? We follow one question, "Why is the sky blue?", through the chip. The text becomes tokens and then whole numbers; each number becomes a list of thousands of numbers; and every layer of the model is mostly matrix multiplication, cut into tiles and shared across the chip's tensor cores. The question is read all at once. The answer can only be written one token at a time, and every new token needs nearly every one of the model's weights read from memory again: for a model with eight billion 16-bit weights, about 15 GB per token. At the 3.35 terabytes a second of NVIDIA's H100, that takes about 4.5 thousandths of a second, while the arithmetic itself would take about 15 millionths. Add the KV cache, the keys and values kept for every earlier token, and you see why AI chips are built around memory, and why data centres batch many conversations into one read, store weights in fewer bits and split big models across several chips. The token ids and the next-token probabilities shown are real: they come from a small open-weight model asked the same question for this film. The H100 figures are NVIDIA's published peak specifications. The chip on screen is a generic, stylised accelerator drawn for this film, not any real chip's layout; the timings are an idealised count (peak rates, no overheads), so real systems are slower; and the 8-billion-weight model has the published shape of Meta's Llama 3 8B. Chapters 0:00 One question, as numbers 1:15 Lists of numbers 2:15 Arithmetic in tiles 3:34 The memory wall 5:06 One token, one layer, one choice 6:31 More people, fewer bits, more chips 8:01 The answer, one piece at a time Sources - NVIDIA Technical Blog, "NVIDIA Hopper Architecture In-Depth" (2022): https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ - NVIDIA, H100 Tensor Core GPU specifications: https://www.nvidia.com/en-us/data-center/h100/ - NVIDIA Technical Blog, "NVIDIA Ampere Architecture In-Depth" (2020): https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/ - NVIDIA Technical Blog, "Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server and FasterTransformer": https://developer.nvidia.com/blog/accelerated-inference-for-large-transformer-models-using-nvidia-fastertransformer-and-nvidia-triton-inference-server/ - NVIDIA Technical Blog, "Mastering LLM Techniques: Inference Optimization": https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/ - NVIDIA Deep Learning Performance Guide, "Matrix Multiplication Background": https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html - A. Vaswani et al., "Attention Is All You Need", NeurIPS 2017: https://arxiv.org/abs/1706.03762 - R. Sennrich, B. Haddow, A. Birch, "Neural Machine Translation of Rare Words with Subword Units", ACL 2016: https://arxiv.org/abs/1508.07909 - J. Kaplan et al., "Scaling Laws for Neural Language Models" (2020): https://arxiv.org/abs/2001.08361 - R. Pope et al., "Efficiently Scaling Transformer Inference", MLSys 2023: https://arxiv.org/abs/2211.05102 - W. Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023: https://arxiv.org/abs/2309.06180 - J. Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", EMNLP 2023: https://arxiv.org/abs/2305.13245 - Llama Team, AI @ Meta, "The Llama 3 Herd of Models" (2024): https://arxiv.org/abs/2407.21783 - T. Dettmers et al., "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", NeurIPS 2022: https://arxiv.org/abs/2208.07339 - E. Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023: https://arxiv.org/abs/2210.17323 - J. Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", MLSys 2024: https://arxiv.org/abs/2306.00978 - SK hynix Newsroom, "SK hynix Develops Industry's First 12-Layer HBM3": https://news.skhynix.com/sk-hynix-develops-industrys-first-12-layer-hbm3/ Music and sound effects: original, made for this film. Fonts: Source Sans 3 (Adobe) and Figtree (Erik Kennedy), SIL Open Font License 1.1. #AI #Technology #Science #Engineering #Computing

Selected Topics

  • Dogecoin whale activity
    Dogecoin whale activity
    Get the latest insights into Dogecoin whale activities with our comprehensive analysis. Discover trends, patterns, and the impact of these whales on the Dogecoin market. Stay informed with our expert analysis and stay ahead in your cryptocurrency journey.
  • Dogecoin Mining
    Dogecoin Mining
    Dogecoin mining is the process of adding new blocks of transactions to the Dogecoin blockchain. Miners are rewarded with new Dogecoin for their work. This topic provides articles related to Dogecoin mining, including how to mine Dogecoin, the best mining hardware and software, and the profitability of Dogecoin mining.
  • Spacex Starship Launch
    Spacex Starship Launch
    This topic provides articles related to SpaceX Starship launches, including launch dates, mission details, and launch status. Stay up to date on the latest SpaceX Starship launches with this informative and comprehensive resource.
  • King of Memes: Dogecoin
    King of Memes: Dogecoin
    This topic provides articles related to the most popular memes, including "The King of Memes: Dogecoin." Memecoin has become a dominant player in the crypto space. These digital assets are popular for a variety of reasons. They drive the most innovative aspects of blockchain.