⚡ Each donation funds the next large quant.

I host free GGUF or MoE quants as independent research.
Local hardware: Mechrevo Kuangshi GM7AG0M — RTX 3060 Laptop 6GB GDDR6, 64GB DDR5, i7-12700H (14C/20T, 4.7GHz), Windows 11, Samsung 990 Pro.
Good for imatrix and 0.6–35B-class work in RAM. 9B+ and searches need rented H200/Blackwell, typically $100 per quant.

🎉 Boosty🦄  |  ☕ Buy Me a Coffee🦄  |  ⭐ DonationAlerts🦄

💚 Thanks to Hugging Face for extra storage.🦄

---

NOESIS / AMAImedia

Released as part of the NOESIS Professional Multilingual Dubbing Automation Platform (framework: DHCF-FNO — Deterministic Hybrid Control Framework for Frozen Neural Operators).

• Founder: Ilia Bolotnikov
• Organization: AMAImedia.com
• X (Twitter): @AMAImediacom
• LinkedIn: Ilia Bolotnikov
• Telegram: @djbionicl
• NOESIS version: v16.1
• Release date: 2026-09-18

Kimi-K3-0.40B Kazakh CPT — Step 200

An experimental Kazakh continued-pretraining checkpoint based on inference-optimization/Kimi-K3-0.40B.

This is an early smoke-test checkpoint created to validate a complete Kazakh language-model pretraining pipeline on Kaggle.

Important limitation

The base checkpoint is a tiny architecture-development version of Kimi K3. Its weights were initialized from scratch and trained only on a toy copypasta dataset. It is not a distilled or compressed version of the trained 2.8T Kimi K3 model.

This checkpoint is therefore an educational research artifact, not a production language model.

Training

• Dataset: kz-transformers/multidomain-kazakh-dataset
• Objective: causal language modeling
• Optimizer steps: 200
• Tokens processed: 1,638,400
• Sequence length: 512
• Effective batch: 8,192 tokens
• Initial learning rate: 3e-4
• Hardware: 1× NVIDIA T4
• Runtime: approximately 12.7 minutes
• Tokenization: native Kimi tokenizer accelerated with Gigatoken

Results

Example generations

The model has started learning Kazakh Cyrillic, morphology, punctuation, and sentence structure, but its generations remain semantically incoherent and contain invented words. This is expected after only 1.64M training tokens.

Usage

Install the required libraries:

pip install -U torch transformers

Run a minimal Kazakh text-generation example:

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "AMAImedia/Kimi-K3-0.40B-Kazakh-CPT-step200"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="auto",
)

prompt = "Қазақстан —"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.language_model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

For reproducible comparisons, use a fixed random seed and report the prompt, decoding parameters, Transformers version, and hardware. Because this is an intermediate checkpoint, outputs should be treated as experimental and inspected manually.

Intended use

This checkpoint is intended for studying Kazakh language adaptation, inspecting Kimi K3's small architecture-development configuration, testing dataset and pretraining pipelines, and comparing intermediate language-acquisition checkpoints.

It should not be used for factual, safety-critical, medical, legal, financial, or production tasks. The checkpoint has not been instruction-tuned or safety-aligned for general deployment.

License

This repository is released under the MIT License. Review the base model and dataset terms before redistribution or commercial use.