Kimi
Kimi is a large language model based on the technique of Mixture-of-Experts (MoE), developed by Moonshoot AI.
Source:
Kimi K2 has 32 billion activate parameters and 1 trilion total parameters. Uses MuonClip optimizer.
During post-training, K2 undergoes a multi-stage post-training process, highlighted by a large-scale agentic data synthesis pipeline and a joint reinforcement learning (RL) stage, where the model improves its capabilities through interactions with real and synthetic environments.
Performance on Workbenchs: Kimi K2 obtains 66.1 on Tau2-Bench, 76.5 on ACEBench (En), 65.8 on SWE-Bench Verified, and 47.3 on SWE-Bench Multilingual — surpassing most open and closed-sourced baselines in non-thinking settings. . It also exhibits strong capabilities in coding, mathematics, and reasoning tasks, with a score of 53.7 on LiveCodeBench v6, 49.5 on AIME 2025, 75.1 on GPQA-Diamond, and 27.1 on OJBench, all without extended thinking.
MuonClip: Stable Training with Weight Clipping: Kimi K2 uses a token-efficient Muon optimizer that uses both weight decay and consistent update RMS scaling. Authors claims that Muon outperforms AdamW.
One of the problems of Muon is that when scaling there is instability due to exploding attention logits. Existing mitigation strategies are insufficient.
Authors propose a novel weight-clipping mechanism called QK-Clip to explicitly constrain attention logits. QK-Clip works by rescaling the query and key projection weights post-update to bound the growth of attention logits.
Let the input representation of a transformer layer be
where
Let the max logit (a per-head scalar) be defined as the maximum input to softmax in this batch
The core idea of QK-Clip is to rescale
On improvement from previous is the introduction of synthetic data generation strategy to increase token utility. The synthetic rephrasing framework applied to Kimi is composed of the following key components:
- Style- and perspective-diverse prompting: inspired by WRAP, a range of carefully engineered prompts are applied to enhance linguistic diversity
- Chunk-wise autoregressive generation: this is done in order to preserve global coherence and avoid information loss. Texts are divided into segments, rephrased individually, and then stitched back together to form complete passages. This method mitigates implicit output length limitations that typically exist with LLMs.
- Fidelity verification: to ensure consistency between original and rewritten content, they perform fidelity checks that compare the semantic alignment of each rephrased passage with its source.
Mathematics Data Rephrasing: To enhance mathematical reasoning capabilities, authors rewrite high-quality mathematical documents into a “learning-note” style, following the methodology introduced in SwallowMath.
Pre-training Data Overall: the Kimi K2 pre-training corpus comprises 15.5 trillion tokens of curated, high-quality data spanning four primary domains: Web Text, Code, Mathematics, and Knowledge.
The number of attention head is 64. The reason is: increasing the number of attention head increases inference overhead that scales with context length. This becomes a major limitation in agentic applications, where efficient long context processing is essential.
Supervised Fine-Tuning: authors used the Muon optimizer and suggests using it for finetuning with K2.
Large-Scale Agentic Data Synthesis for Tool Use Learning:
- Tool spec generation: we first construct a large repository of tool specs from both real-world tools and LLM synthetic tools;
- • Agent and task generation: for each tool-set sampled from the tool repository, we generate an agent to use the toolset and some corresponding tasks;
- Trajectory generation: for each agent and task, we generate trajectories where the agent finishes the task by invoking tools.