There is an extra icon in the Mac menu bar, and 90% of people don’t understand this move |oMLX

Original author：AI 雷達站

Original source：https://www.youtube.com/embed/31fBird82lk

A small icon hidden in the macOS menu bar received 370 stars on GitHub in one day. It's called oMLX (jundot/omlx, Apache 2.0), which is a local large model inference server running on Apple Silicon. The focus of everyone's forwarding is "Mac can finally run local models well", but what this video wants to dismantle is the move behind it: it divides the KV cache into two layers, and keeps the hot ones in the memory. When it is full, it writes it to the SSD in the safetensors format. Next time the same context is encountered, it can be directly retrieved without recalculating the prefill. Coupled with continuous batch processing using MLX BatchGenerator (default 8 requests at the same time), there is also an OpenAI / Anthropic compatible interface. The video also organizes what each represents to developers, content creators, and general users, the radar station’s judgment for the next three months, and three signals that you can track yourself. Project link: https://github.com/jundot/omlx Threshold: macOS 15.0+, Apple Silicon (M1–M4), Python 3.11–3.13 #oMLX #localAI #AppleSilicon The narration and some scenes of this video are generated by AI