Apex-3

A 21B-parameter mixture-of-experts chat model, trained from scratch: 70B pretraining tokens with a SuperBPE tokenizer, then SFT, DPO and RLVR. It answers directly, without a reasoning block.

Thank you for your interest in this model.

Apex-3 is a mixture-of-experts language model with 21B total and 3.36B active parameters, trained from scratch with no external pretrained weights. The pretraining ran on NVIDIA GH200 superchips, up to three at a time, combined through DiLoCo. Post-training used a single GH200. This is a model trained on a small budget: expect a capable small model for math word problems, short code and everyday questions, not a replacement for a large frontier model.

Here is what Apex-3 offers:

• A fine-grained MoE. 64 routed experts per layer, 4 active, plus a shared expert that every token passes through. Attention alternates five sliding-window layers (window 1,024) with one global layer, and uses grouped-query attention (20 query heads, 4 key/value heads) with QK-norm. A sigmoid router balances the experts mainly with a selection bias updated outside the gradient, helped by a small sequence-level balance loss.
• SuperBPE. Tokens may span several words, so the same text needs fewer tokens than with an ordinary BPE vocabulary of the same size (about 21% fewer, as we measured on the pretraining data mix).
• Pretraining. 70.0B tokens in 12,268 optimizer steps: 59B tokens at a constant learning rate, then 11B tokens of linear decay toward code and math. Optimization used Muon and AdamW with FP32 master weights. From 4B tokens on, the attention, shared-expert and dense-layer linears ran in FP8.
• Post-training for direct answers. Supervised fine-tuning on reasoning-style data (about 1B tokens), then DPO and GRPO with verifiable code rewards, both run without a reasoning block. See No-think mode for what this means for you.
• Short context, on purpose. Everything was trained at 8,192 tokens. config.json says 8,192 and the model was not extended to longer contexts.

Evaluation

All numbers were measured by us on the final checkpoint, in BF16. The generation benchmarks use the no-think prompt (system turn /no_think) and greedy decoding; the log-likelihood tasks are scored without a chat template. Each number is a single run. We looked at several of these benchmarks while choosing the post-training settings (DPO and RLVR learning rates, data), so they are development scores, not untouched test results. MMLU was not evaluated.

Log-likelihood tasks

Zero-shot, full evaluation sets, no chat template (lm-evaluation-harness). acc_norm divides each answer's log-likelihood by its length in characters (lm-evaluation-harness 0.4.13).

WinoGrande uses partial scoring: each option fills the blank, and the shared continuation is scored. LAMBADA counts an example as correct when every token of the last word is the greedy prediction. SuperBPE tokens can cross word boundaries, so treat LAMBADA as a check between checkpoints of this model rather than a number to compare with other tokenizers.

Generation tasks

The prompt is the benchmark question in a chat turn, after the system message /no_think. Greedy decoding, up to 8,192 new tokens.

• Math answers are graded by a math checker on the final answer (the prompt asks for it in \boxed{}); code is run against the benchmark tests in a sandbox without network access.
• IFEval: our implementation of the instruction checks, prompt-level (all instructions of a prompt must pass). 520 of the 541 prompts are used; the other 21 exceed the prompt-length limit of our runner. Standard IFEval numbers from lm-evaluation-harness are in the next table.
• LiveCodeBench: a fixed 300-problem subset of the 878 v5 problems with tests (stdin and call-based), taken from the evaluation split of DeepCoder-Preview-Dataset. Not the full benchmark and not the official harness.

Standard harnesses

Chat prompts with the system message /no_think, greedy, zero-shot, using EvalPlus 0.3.1 and lm-evaluation-harness 0.4.13. These runs use each harness's own answer extraction and generation limit, and some of the limits are short, so read the "at limit" column. No response in these runs opened a reasoning block.

Documents overlapping these evaluation sets were removed from the pretraining data by 13-gram matching, and the post-training data was filtered the same way. This removal is a lower bound, and the FLAN data in the decay stage may contain the training splits of ARC and HellaSwag.

These scores are not directly comparable with numbers other models report under other prompts, shot counts, answer extraction and generation limits, so this card does not rank Apex-3 against other models.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOON1v/Apex-3"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Write a Python function that checks whether a number is prime."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

• trust_remote_code=True is required because this is a custom architecture.
• Requires transformers 4.53 or newer: the model keeps its routers in FP32 through _keep_in_fp32_modules_strict, which older versions do not have (they would round the routers to BF16 on load). Verified on CPU with transformers 4.53.0, 4.57.6, 5.20.2 and 5.21.0 (torch 2.14): with torch_dtype="auto" the 58 router tensors stay FP32, the chat template gives the same token ids as the training code, and logits and greedy decoding (with and without the KV cache) match the training code exactly on short test prompts. transformers 5.x may warn about a "Mistral regex" when loading the tokenizer; ignore it, the token ids were checked against the training tokenizer.
• The BF16 weights take 39.1 GiB, so use a 48 GB or larger GPU. With device_map="auto", layers that do not fit are offloaded to CPU memory, which is slow.
• The Transformers code in this repository is a simple reference implementation, not an optimized one: it builds a dense attention mask and runs the experts in a loop. The routers are kept in FP32, as in training.
• vLLM and SGLang cannot load this architecture yet.
• Generation stops at <|end|> (also at <|endoftext|> and the role tokens; all are listed in generation_config.json). generation_config.json samples with temperature 0.7 and top-p 0.95; the evaluation above used greedy decoding (do_sample=False), which is the better choice for math and code.
• Keep prompt plus answer within 8,192 tokens.

No-think mode

Apex-3 does not use a reasoning block. The tokenizer has <|think|> and <|/think|> tokens, and the first supervised fine-tuning data contained long reasoning traces, but this release is trained and evaluated only without them:

• The trained and evaluated no-think format is a system turn that holds only /no_think: <|system|>/no_think<|end|>. The chat template writes it by default when you give no system message. The no-think SFT conversations, DPO, RLVR and every benchmark above used exactly this, so it is the supported way to use the model.
• If you pass your own system message, the template merges it as /no_think, a blank line, and then your text (the same merge scripts/gen_eval.py --no-think does). That combination was not trained or evaluated: SFT v4 did not mark conversations with a system prompt as no-think, DPO and RLVR left out prompts with system messages, and none of the benchmarks above carries one. Expect weaker adherence to such system prompts.
• Why: in our tests the reasoning mode of the SFT model often failed to finish its reasoning and looped until the 8K limit (44% of GSM8K, 76% of MATH-500, 61% of HumanEval+ and 89% of IFEval answers), while the no-think mode scored higher on code, MATH-500 and IFEval. DPO and RLVR were therefore run in no-think mode only.
• tok.apply_chat_template(..., no_think=False) renders your system message as given. That mode was not trained further, is not evaluated, and is not recommended.
• The model was not trained on tool calls. The template can still render tool definitions and tool results in the apex3-chat-v1 format, and apex3_tool_parser.py can parse tool-call blocks, but no tool-use behavior was trained or measured in this release.

The template also breaks special-token strings inside message text, so user text cannot forge control tokens:

<|system|>/no_think<|end|><|user|>...<|end|><|assistant|>answer<|end|>

Training details

Pretraining

• Batch and length: 8,192-token sequences; the global batch grew from 2M to 8M tokens over the first 33B tokens.
• DiLoCo: the workers exchanged weight differences every 1B tokens, using an outer Nesterov SGD with learning rate 0.7 and momentum 0.9. In the last synchronization of stage 3 only two of the three workers contributed, because the third had failed shortly before the end: the final average is over those two, and the last 1B-token interval covers about two thirds of the planned tokens.
• Optimizers: Muon (momentum 0.95, Nesterov) for weight matrices; AdamW (β 0.9/0.95, ε 1e-8) for embeddings, output layer, norms and routers; weight decay 0.1; gradient clipping 1.0; peak learning rate 4e-4.
• FP8 precision: tensor-wise FP8 for the attention, shared-expert and dense-layer linears, about half of the FLOPs. Routed experts, routers and the output layer stayed in higher precision.
• Memory layout: FP32 master weights, gradient accumulation and optimizer state were kept in the GH200's CPU memory, with BF16 working weights on the GPU.
• Throughput: about 50% model FLOPs utilization per GPU as logged.

Data, stages 1–2 (0 → 59B)

Data, stage 3 (59B → 70B, decay)

• Synthetic data: SwallowCode-v2 and SwallowMath-v2 (rewritten by Qwen3-235B-A22B, Apache-2.0), Cosmopedia v2 (written by Mixtral-8x7B-Instruct), and the reasoning sets (generated by DeepSeek-R1 and QwQ-32B).
• Filtering: code was filtered for permissive licenses; secrets and personal data were removed. Documents overlapping 24 evaluation sets were dropped by 13-gram matching.

Post-training

• SFT sources (token share): SYNTHETIC-1 (33%), OpenMathReasoning (22%), OpenR1-Math (16%), OpenScienceReasoning-2 (12%), AceReason-1.1-SFT code and math (8%), OpenCodeReasoning (5%), OpenThoughts3 (3%), LIMO. The conversations are math, code, science and general reasoning; there is little everyday chat or instruction-following data, which is visible in the IFEval scores.
• DPO sources: the pairs above, plus the prompts of OpenR1-Math, DeepScaleR, DeepCoder, TACO-verified, Nemotron-RL instruction following, RLVR-GSM-MATH-IF and Dolci-Think-RL for the on-policy pairs.
• What each step did: in no-think mode, 10.5% of the SFT model's answers on our generation benchmarks ran into the 8,192-token limit (repetition loops). DPO v3 cut that to 3.6% and halved the average answer length. The code RLVR added about one point on MBPP+ and LiveCodeBench, but two numbers went down: lm-eval MATH-500 with 2,048 new tokens (28.2 → 25.6) and EvalPlus HumanEval+ (23.8 → 22.6). The other benchmarks changed within noise.
• Generators: according to their dataset cards, the reasoning sets were mostly generated by DeepSeek-R1 (or R1-0528), QwQ-32B and Qwen models, and the Dolci DPO answers by Qwen3. A provenance filter dropped rows whose source field named a GPT, Claude, Gemini, Llama, Orca or OpenMath-2 style generator. We did not audit the generator of every source individually (for example LIMO), so this card makes no license claim for them; check the dataset cards.
• Decontamination: SFT conversations overlapping the evaluation sets were removed by 13-gram matching (1,599 of 503,204 conversations, 0.3%).
• Safety: there was no dedicated safety tuning or RLHF. Do not rely on the model to refuse harmful requests.

Intermediate checkpoint

The raw training state at the end of the constant-learning-rate phase (59B tokens, before the decay and post-training) is public as YOON1v/Apex-3-59B-stable. It is in the training code's own checkpoint format and meant for continued pretraining and research, not for inference.

The training code is at https://github.com/DW-dev-UE/LLM-from-scratch.

Before you download

What are its limits?

• It is a small-budget model: 3.36B active parameters and 70B pretraining tokens. Knowledge is limited and it hallucinates, especially on facts, names, numbers and citations. Check anything that matters.
• Instruction following is weak. Constraints such as "exactly three paragraphs" or "no commas" are often missed (IFEval prompt-level 13–17%).
• Code ability is modest: it solves simple functions (HumanEval+ 22.6–26.8, MBPP+ 38.1–43.4) but fails most real-world and competition problems (LiveCodeBench 7.0).
• Math is limited to school level: about a third of GSM8K and a quarter of MATH-500 problems are solved.
• It is English-centric, with code and math. Other languages were not a training goal.
• Context is 8,192 tokens.
• Long answers sometimes fall into repetition loops; IFEval answers hit the 8,192-token limit 17.7% of the time. Use a token limit and, if you can, a repetition penalty or a stop condition on your side.
• Because SuperBPE tokens can cross word boundaries, results can degrade when a prompt ends in the middle of a word (code completion, fill-in-the-middle). Token healing at serving time helps.
• No safety tuning and no tool-use training (see above).

Is it free to use? The weights are free to download. License: Apache 2.0.

License

The model weights and code are released under the Apache License 2.0. Some training sources carry terms of their own, for example CC BY-SA (StackExchange, Wikipedia), ODC-By (FineWeb-Edu, Cosmopedia, peS2o, Dolci) and CC BY 4.0 (the NVIDIA reasoning sets), and the licenses of the models that generated the synthetic data. Review them before choosing a license for derivatives.

---

한국어 안내

저의 모델에 관심을 가져주셔서 감사합니다.

Apex-3는 총 21B·활성 3.36B 파라미터의 MoE 채팅 모델이며, 외부 사전학습 가중치 없이 처음부터 학습했습니다. SuperBPE 토크나이저로 70B 토큰을 사전학습한 뒤 SFT, DPO, RLVR(코드 GRPO)을 거쳤습니다.

사용할 때 지킬 것:

• 생각 모드가 없는 모델입니다. 채팅 템플릿이 시스템 메시지 /no_think를 자동으로 넣으므로 apply_chat_template을 그대로 쓰세요.
• 문맥 길이는 8,192토큰입니다.
• 작은 예산으로 만든 모델이라 지식이 한정적이고, 사실과 다른 내용을 말할 수 있습니다. 지시 따르기(IFEval)와 코드 실력은 낮은 편입니다.
• trust_remote_code=True가 필요하며 vLLM·SGLang은 아직 지원하지 않습니다.