Sparse-AST-BWM — latest checkpoint, step 93,213

This repository contains one current checkpoint of the serial old/new Sparse-AST family for byte-level Blender Python completion. It was recovered from results (4).zip, the newest relevant training-results archive found in the local Downloads search, and its saved training state confirms step 93,213. It continues from step 90,213 through 3,000 successful updates, with no recorded non-finite skips.

Updated 16 October 2026. The current release is under latest/ and uses SafeTensors for model weights and training-state tensors. That release contains no .pt checkpoints. Existing remote variant folders are retained as historical artifacts; they are not the latest checkpoint. The pre-update legacy code remains available at revision 7c1001debae58e03d2a3e7073819de4c50ca0cfe.

Files in latest/

training_state.safetensors is resumable state for the same model version, not a second model checkpoint. Inference needs only the model weights, config and Python architecture files.

Architecture and context

• 18 serial stages, each retaining its own legacy/new branch pair.
• Width 704; feed-forward width 1,408; eight attention heads.
• 512-entry output vocabulary; inputs and generated text use UTF-8 bytes.
• Stored positional embedding capacity: 512 positions.
• This continuation trained with 128-byte contexts, not a verified 512-token or 4K context window.
• 201,380,162 model parameters (about 201.38M), stored in FP32.

Branches are mixed locally at each serial stage. This is not a top-k router over interchangeable stages. Legacy branches remain frozen during training. The actual saved input embedding and output projection differ and are preserved as separate matrices; the loader does not retie or change them.

Load and generate

Install PyTorch 2.4 or newer, safetensors and huggingface_hub. This is a custom PyTorch model; use the bundled loader.

import sys
from pathlib import Path
from huggingface_hub import snapshot_download

revision = "main"  # Pin a commit hash for reproducible deployments.
folder = snapshot_download(
    "Chaman1234/Sparse-AST-BWM",
    revision=revision,
    allow_patterns=["latest/*.py", "latest/config.json", "latest/model.safetensors"],
)
folder = str(Path(folder) / "latest")
sys.path.insert(0, folder)
from model import load_model, generate

model = load_model(folder, device="cpu")
print(generate(model, "import bpy\nbpy.ops.mesh.", max_new_bytes=80))

Generation defaults to the 128-byte training context and greedy decoding over byte values 0–255. The longer positional table does not establish reliable performance at longer contexts. Generated code should be inspected and executed in an appropriate Blender environment before use.

Resume state without pickle

from model import load_training_state

state = load_training_state("Chaman1234/Sparse-AST-BWM", revision=revision)
assert state["step"] == 93213
optimizer.load_state_dict(state["optimizer"])
scaler.load_state_dict(state["scaler"])

Construct the model's optimizer using model.trainable_parameter_groups() before restoring its state. Restore python_rng, torch_rng, and, on compatible CUDA hardware, cuda_rng; restore the recorded order, cursor and progress counters in the training loop. The loader reconstructs original dictionary key types and tuples from JSON and SafeTensors. No pickle is used to load the published release.

Evaluation and limits

The archived run reports the following teacher-forced byte-completion results on 256 examples per split:

These are archived training-run metrics, not a newly reproduced benchmark, full-script success rate, Blender execution result or visual-quality assessment. The data audit reports family-disjoint train/validation/test splits within this sampled dataset; historical overlap with every earlier training corpus is not established. Source quality checks cover JSON/AST syntax and bpy imports, not runtime or visual correctness.

The release conversion independently verifies all model tensors exactly, all non-model training state exactly, finite parameters, and forward logits within FP32 tolerance (absolute 0.0001, relative 0.00001) with identical predicted tokens on two Blender-related prompts. See validation.json for measured differences and hashes. This establishes conversion fidelity, not general coding reliability.

Provenance

• Archive: results (4).zip.
• Member: sparse_ast_bigtrain_3k_output/checkpoint_latest.pt.
• Recorded final step: 93,213; parent step: 90,213.
• Recorded successful continuation updates: 3,000; attempts: 3,000; non-finite skips: zero.
• Training source: compact Blender task/code pairs described in data_audit.json.
• Conversion used PyTorch's restricted weights_only=True loader, followed by lossless SafeTensors serialization and parity checks.

Format references: SafeTensors and shared tensors, Hugging Face upload documentation.