On July 16, Moonshot AI released Kimi K3. On the Artificial Analysis Intelligence Index it scores 57, against Claude Fable 5's 60 and GPT-5.6 Sol's 59. The closest other open-weights model, GLM-5.2, sits at 51.
Read that gap again. Three points. And on a number of benchmarks K3 does not trail at all — it wins:
The frontier is no longer behind an API key.
Except it is behind something else: hardware. Kimi K3 has 2.8 trillion parameters. The released checkpoint is 1.56 terabytes. Loading it the normal way needs something in the neighborhood of forty H200s. That is a data center, not a hobby.
So the interesting question is not whether K3 is good. It is whether you can run it.
We did. On one GPU. Using 3.72GB of VRAM.
Not a distillation. Not a smaller variant. The full 2.8T checkpoint, generating real tokens.
Here is how.
01 — Why the usual trick stops working
Two years ago I wrote about running a 70B model on a 4GB card using layer-wise inference. The idea is simple divide and conquer: a transformer runs its layers in sequence, one at a time, each consuming the previous one's output. So you never need all the layers in memory at once. Load a layer from disk, compute, free it, move on. For a 70B model that turns a 130GB problem into a 1.6GB one.
That technique is the foundation of AirLLM, and for two years it kept working — 405B Llama, 671B DeepSeek-V3.
On Kimi K3, it breaks.
The reason is what K3 is made of. It is a Mixture-of-Experts model, and an extreme one. Each of its 93 layers holds 896 separate expert networks. Stream a whole K3 layer and you are loading all 896.
Do the arithmetic. One K3 layer is about 17GB as stored, and roughly 56GB once expanded to compute with. Our card has 48GB. The oldest trick in the book — one layer at a time — no longer fits in one layer.
02 — The new technique: per-expert streaming
Here is the thing about a Mixture-of-Experts model. It has 896 experts per layer, but a token does not use them.
K3 routes each token to 16 of the 896. That is the entire point of the architecture — 2.8 trillion parameters exist, but only about 104 billion activate for any given token. The other 98% of that layer is dead weight for this particular token.
So stop loading layers. Load experts.
Instead of one hook per layer, AirLLM now places a hook on every individual expert — 82,432 of them across the model. When the router picks an expert, that expert's weights load. It computes. It is immediately evicted. The 880 experts the token did not ask for are never read from disk at all.
The difference is not incremental:
- A K3 layer's experts, expanded: ~55GB
- What one token actually needs: ~1GB
That is the whole trick. Divide and conquer, one level deeper than before. The previous generation of AirLLM asked "which layer am I in?" This one asks "which expert did the router actually want?"
03 — Move it packed, expand it late
K3 ships its expert weights in MXFP4, a 4-bit format. The library that handles this format decompresses weights to full precision before running the model.
For K3 that is fatal. Decompressing every expert up front is how you get from 17GB to 56GB per layer, and straight past the edge of the card.
So AirLLM keeps the weights packed for as long as possible. The 4-bit bytes travel from disk across the PCIe bus still compressed, and only expand once they are already on the GPU, one expert at a time.
Two wins from one change. Peak memory never sees an expanded layer, and there is 4x less data crossing the bus, which matters enormously when disk throughput is the bottleneck in the entire system.
04 — A 1.56TB model on a 3TB disk
A problem we did not anticipate.
AirLLM pre-processes a checkpoint by re-sharding it: one file per layer, so that loading a layer reads exactly that layer and nothing else. Disk reads are the slowest thing in this whole pipeline, so this matters a great deal.
But re-sharding writes a second copy. Original 1.56TB, plus split copy 1.56TB, equals 3.12TB — on a 3TB disk. Fifty gigabytes short of impossible.
The escape hatch came from looking closely at how Moonshot packaged the weights. K3's shards turn out to be pure: each file contains exactly one module, nothing mixed in. Which means the re-sharding step has nothing to actually rearrange.
So AirLLM detects that case and hardlinks instead of copying. The split layer files point at the original bytes. Disk usage stays at 1.56TB, and the setup step that would have taken hours of copying takes seconds.
05 — The numbers
Measured end to end on a single RTX 6000 Ada (48GB), against the full 1.56TB checkpoint, generating real tokens:
Let me be straight about that last row, because it is the honest cost of this approach.
This is not fast. Roughly five minutes per token. The bottleneck is not compute — it is disk. Every token walks through 1.56TB of weights, and no amount of clever memory management changes how fast an SSD reads.
So this is not your chatbot. What it is good for is exactly what it was good for at 70B: offline work where latency does not matter and quality does. Batch analysis. Document processing. Evaluating whether K3 is worth renting real hardware for. Research on a frontier model you could not otherwise touch. Fine-tuning experiments on architecture you can now actually inspect.
And there is a broader point. 3.72GB is not "a big GPU." It is a laptop. The gap between "frontier open-source model" and "hardware a student owns" just became a question of patience rather than capital.
06 — Run it yourself
pip install airllm compressed-tensors flash-attn
from airllm import AutoModel
model = AutoModel.from_pretrained("moonshotai/Kimi-K3")
input_text = ['What is the capital of United States?']
input_tokens = model.tokenizer(input_text,
return_tensors="pt",
return_attention_mask=False,
truncation=True,
max_length=128,
padding=False)
generation_output = model.generate(
input_tokens['input_ids'].cuda(),
max_new_tokens=20,
use_cache=True,
return_dict_in_generate=True)
print(model.tokenizer.decode(generation_output.sequences[0]))That is the whole thing. AirLLM detects the architecture, re-shards the checkpoint on first run, and sets up expert streaming without being asked.
Three requirements specific to K3, none of them optional:
- flash-attn is mandatory. K3's model code overwrites whatever attention implementation you request, so there is no way to run it without.
- A CUDA 12 build of torch, because no prebuilt flash-attn wheel exists for CUDA 13 yet.
- transformers 4.56.x — K3's remote code does not load on 5.x.
You will also want fast storage and roughly 1.6TB free. Disk speed is the single biggest lever on how long a token takes.
07 — What this means
For two years the story of local LLMs has been the same shape: the open models get better, and the hardware to run them stays out of reach, so most people rent an API and hope the terms do not change.
Kimi K3 is the moment that story gets interesting, because the model is genuinely at the frontier — three points off the best proprietary system, ahead of it on agentic coding — and the weights are public.
Whoever holds the weights sets the rules. Runs it offline. Fine-tunes it. Inspects it. Studies why it works.
That used to require a data center. Today it requires a graphics card and some patience.
AirLLM is open source and MIT licensed: https://github.com/lyogavin/airllm
If it is useful to you, a star helps more people find it. And if you want support for a model we have not covered, open an issue — that is exactly how K3 support got built.
We will keep open sourcing the newest and most effective techniques as they arrive. Follow along.