RWKV7-G1j-10B-A1B

A RWKV7 dense-to-MoE initialization with 80 routed experts, one shared expert, and top-4 routing in every layer.

This checkpoint starts from RWKV7-G1j-2.9B-20260831, preserves its complete 32-layer recurrent backbone, and expands its channel-mixing FFNs into a sparse expert architecture. It is intended as a starting point for continued pretraining, MoE recovery training, and architecture experiments.

This is an initialized checkpoint. It has not undergone language-model recovery training after conversion. Its experts and router have not yet learned specialized roles through continued training. The source model is a Base LM; this release does not establish instruction-following or benchmark performance.

繁體中文摘要： 本模型由 RWKV7-G1j 2.9B 初始化，保留完整主體，每層採用 80 個路由專家、1 個共享專家及 top-4 路由。每個 token 使用 4 個路由專家與始終啟用的共享專家。總參數為 9.770B；啟用參數為 1.465B（不含 embedding／輸出頭）或 1.801B（包含）。已完成權重轉換、輸出尺度校準與運算驗證，尚未進行語言模型恢復訓練。

Architecture

The shared expert is additional to top-4: each token activates four routed branches plus one shared branch. Routing decisions can differ between tokens and layers. The FFN token shift and x_k remain shared across branches, so this expansion does not create a separate recurrent state for every expert.

For a channel-mixing input x, the layer computes:

output = shared(x) + sum(weight[e] * expert[e](x) for e in selected_top4)

The selected sigmoid scores are normalized to sum to one. The shared branch is outside this normalization and is not selected by the router.

Exact parameter accounting

The repository name is a short size label. The exact counts for this shared-1 / top-4 release are:

Both active counts include every layer's router and shared expert. Buffers are excluded. The inclusive convention counts the full embedding and output-head matrices; this is parameter accounting, not a claim that every embedding row is read for each token or a FLOPs estimate.

The BF16/FP32 checkpoint contains approximately 19.55 GB of tensor data. Sparse activation reduces the expert computation used by a token; all expert weights still need to be stored. Training memory also depends on gradients, optimizer states, activations, and sharding.

Initialization method

Complete FFN coverage

Each original 10,240-channel FFN is divided into 16 contiguous groups of 640 channels. Every group is copied five times, producing 80 independently stored routed experts. Key rows and the corresponding value columns are moved together. Every original FFN channel therefore occurs exactly five times in the routed bank, with no remainder or dropped channels.

The embedding, output head, normalization, time-mixing weights, and FFN token-shift parameters are copied from the source. Router rows are initialized independently in FP32 with Gaussian standard deviation 0.02 and seed 20260912 + layer_index; expert-bias buffers start at zero. Copied experts initially share source functions, and independent router initialization does not by itself establish useful specialization.

Shared branch and joint scale calibration

For each layer, one of the 16 source groups is also used to initialize the 640-channel shared expert. The group and two strictly positive output scales are selected jointly to minimize reconstruction error on calibration fit inputs from the original dense teacher:

teacher_ffn_output ≈ routed_scale * routed_top4_output
                   + shared_scale * shared_group_output

The shared source group is still present in the routed bank. This overlap is explicit; joint fitting adjusts the two contributions. The construction does not claim disjoint source features or already-learned common knowledge.

Scales are folded into the BF16 value projections. Calibration uses 673 tokens from eight fit documents, followed by 234 tokens from three separate check documents. These are short English, Chinese, and Python texts created for initialization diagnostics. They are not a benchmark or a continued-pretraining corpus.

• Routed scale range: 0.1877–4.9224.
• Shared scale range: 0.7767–2.6273.
• Median per-layer relative squared reconstruction error on check documents, after actual BF16 scale folding: 0.6086.

This error is measured using teacher hidden states. It does not measure accumulated error in the complete MoE model, perplexity, generation quality, or recovery-training progress. The calibration documents should be excluded from independent quality evaluation.

Exact source-channel mappings, chosen shared groups, scales, documents, and provenance are recorded in initialization_map.json, initialization_calibration.json, and conversion_report.json.

initialization_sources.zip contains the conversion, calibration, verification, and smoke-check scripts, their local model-code dependencies, and unit checks. See REPRODUCE.md inside the archive for the command sequence. release_manifest.json records the packaged source hashes and validation environment.

Loading and raw text continuation

The model uses custom Transformers modules included in this repository. Load it with trust_remote_code=True. The example below uses one CUDA GPU and raw Base-LM text continuation.

The local validation environment used Python 3.12, PyTorch 2.10.0+cu130, Transformers 5.8.0, Accelerate 1.11.0, and Safetensors 0.8.0. After installing a suitable CUDA PyTorch build:

pip install "transformers==5.8.0" "accelerate==1.11.0" "safetensors==0.8.0"

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "win10/RWKV7-G1j-10B-A1B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map={"": "cuda:0"},
).eval()

prompt = "A computer program can organize incoming tasks in a queue. When a new task arrives, "
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=1.0,
        top_p=0.5,
        top_k=0,
    )
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

top_k=0 in this generation example is a vocabulary-sampling setting. It does not change the model's MoE top-4, which is configured by num_experts_per_tok=4.

The inherited tokenizer and chat template are included for compatibility with the source checkpoint. Raw continuation is the appropriate basic evaluation format for this initialization; a chat template does not establish post-conversion instruction tuning.

FLA acceleration

The included model implementation retains Flash Linear Attention's RWKV7 kernel. The local FLA forward check used flash-linear-attention==0.5.2 with Triton 3.6.0.

pip install "flash-linear-attention==0.5.2"

After loading the model on CUDA:

model.config.wkv_implementation = "fla"

FLA requires FP16/BF16 activations and FP32 WKV state; this checkpoint uses BF16 weights and wkv_state_dtype="float32". It accelerates the RWKV WKV computation. Expert execution is configured separately through moe_backend, whose default here is torch.

FLA is Flash Linear Attention. It is not enabled through Transformers' attn_implementation="flash_attention_2" setting.

Validation performed

These checks establish conversion and execution integrity in the recorded environment. They do not establish language-model recovery, downstream benchmark performance, long-context accuracy, FLA numerical equivalence to other backends, or training throughput. No speedup ratio is claimed for this checkpoint.

Continuing training

This checkpoint is intended for recovery training before assessing its language capability. Useful measurements include held-out next-token loss, teacher divergence, raw continuations, expert utilization, and replica specialization. Evaluate language behavior and routing correctness separately.

Changing top-k, shared width, routing normalization, or branch scaling changes the initialized computation. Revisit the joint scale calibration when making those changes. Record any future training corpus, token budget, optimizer setup, and evaluation results in the model card for the resulting trained release.

License and attribution

Distributed under Apache-2.0; see the included LICENSE and NOTICE files. The checkpoint derives from the official RWKV7-G1j-2.9B release. The MoE implementation combines RWKV7 ReLU² channel mixing with Ling/Bailing-style sigmoid routing and a directly added shared expert.

Related architecture and initialization references:

• RWKV-7
• DeepSeekMoE: shared-expert isolation and fine-grained experts
• Upcycling Large Language Models into Mixture of Experts

The cited papers provide background; their benchmark results are not results for this checkpoint.