Implementing Kimi K3 from scratch in PyTorch

Original author：Prasoon Mahawar

Original source：https://www.youtube.com/embed/U6sobPCsdaY

Kimi K3 is a 2.8 trillion parameter open-weight model with a 1 million token context window by and in this video we implement every core architectural idea behind it from scratch in @PyTorch . Paper Link: https://arxiv.org/abs/2607.24653 Blog: https://www.kimi.ai/blog/kimi-k3 Code Repo: https://github.com/prasoonmhwr/fromScratch/blob/main/KimiK3/kimiK3.py For more content like this, subscribe to my channel: (https://www.youtube.com/@prasoonmahawarYT) X (Twitter): https://www.x.com/prasoonmahawar Instagram: (https://www.instagram.com/prasoonmahawar) 👇 Comment your thoughts ╔═╦╗╔╦╗╔═╦═╦╦╦╦╗╔═╗ ║╚╣║║║╚╣╚╣╔╣╔╣║╚╣═╣ ╠╗║╚╝║║╠╗║╚╣║║║║║═╣ ╚═╩══╩═╩═╩═╩╝╚╩═╩═╝ #ai #airesearch #python #pytorch #moonshotai #kimik3 #llm #llms 00:00 — Intro & Hook 00:05 — Overview of Architecture 05:12 — The 11 Novel Contributions: Our Roadmap 07:03 — Kimi Delta Attention 13:03 — Gated MLA + NoPE 16:21 — Attention Residuals (Full + Block) 20:10 — Stable LatentMoE overview 22:27 — SiTU-GLU 24:44 — Quantile Balancing 26:58 — Per-Head Muon 29:14 — Model Configuration 36:31 — Building Blocks: RMSNorm, ShortConv, SiTU-GLU 49:48 — Quantile Balancing implementation 1:01:32 — Stable LatentMoE implementation 1:16:36 — KDA: recurrent forward + chunkwise forward 1:39:04 — Gated MLA implementation 1:40:49 — Block Attention Residuals implementation 1:42:20 — Per-Head Muon optimizer 1:43:59 — Complete KimiK3Block + KimiK3 model 1:45:31 — All 11 contributions recap 1:47:14 — Conclusion