BAGEL-7B-MoT actor-only GRPO captioner (step 250)
A trainable-parameter delta for ByteDance-Seed/BAGEL-7B-MoT, trained with closed-loop GRPO (captions rewarded by how well BAGEL reconstructs the image from them).
Format
model.safetensors holds 233 tensors (39,004,736 parameters):
LoRA (rank 16, alpha 32) on the Qwen2 understanding-expert attention q/k/v/o projections, plus the fully
trained llm2vae, vae2llm, latent_pos_embed and time_embedder. It is not a standalone model and not a
plain PEFT adapter (PeftModel.from_pretrained() won't load it). It needs the base model and the
BAGEL-VLM code (revision d8923f8fe4fd8fce83bbc614763def410c83c443), which builds BAGEL and injects LoRA
with the same module names.
Loading
git clone https://github.com/arj-shriv/BAGEL-VLM.git && cd BAGEL-VLM && git checkout d8923f8fe4fd8fce83bbc614763def410c83c443
pip install -r requirements.txt
hf download ByteDance-Seed/BAGEL-7B-MoT --local-dir pretrained_models/BAGEL-7B-MoT
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from train_closed_loop.launch import apply_lora_and_freeze, build_model
model, vae_model, tokenizer, new_token_ids, vae_transform, vit_transform = build_model(
"pretrained_models/BAGEL-7B-MoT", vae_max_size=768, vae_min_size=192,
quantize_nf4=True, device=0) # NF4, as in training
model = apply_lora_and_freeze(model, lora_rank=16, lora_alpha=32)
state = load_file(hf_hub_download("<user>/<repo>", "model.safetensors"))
params = dict(model.named_parameters())
assert set(state) == {n for n, p in params.items() if p.requires_grad}
with torch.no_grad():
for n, v in state.items():
params[n].copy_(v.to(params[n].device, params[n].dtype))
model.eval()
Repo tools that take --checkpoint (e.g. tools/comprecap_caption.py) expect the original .pt format,
{"step": N, "state_dict": {...}}. To use them, rebuild one with
torch.save({"step": 250, "state_dict": load_file("model.safetensors")}, "step_000250.pt").
Training setup
| Setting | Value |
|---|---|
| Optimizer step of this checkpoint | 250 |
| Images per optimizer step | 42 |
| Caption rollout group size | 4 |
| LoRA rank / alpha | 16 / 32 |
| Learning rate | 2e-05 |
| KL coefficient (beta_kl) | 0.02 |
| Reconstruction-loss coefficient (alpha_rec) | 0.0 |
| GRPO enabled | True |
| Max caption tokens | 128 |
| Sampling temperature | 0.95 |
| Noise time range | [0.5, 1.0] |
| VAE max size | 768 |
| Base precision during training | NF4 |
| Seed | 42 |
Full arguments: hyperparams.json (absolute paths there are the original run's, not portable).
Run history. Actor-only run configs/grpo4_lora16_t05_shared_alpharec0_15k_7gpu.yaml (alpha_rec 0: only the GRPO caption loss + KL trains; the reconstruction pass is used for the reward only). Steps 0-100 ran first; the run stopped at step 107 and was resumed from step 100 (configs/grpo4_lora16_t05_shared_alpharec0_15k_7gpu_resume101.yaml, AdamW state restarted) for steps 101-250+. hyperparams.json is the resumed segment's. This checkpoint is after 251 optimizer steps x 42 images (~10.5k images of a 14,994-image 4KLSDB subset), 7 GPUs.
Result. CompreCap (560 images, prompt "Please describe this image.", 128-token cap, GPT-4o-mini judge instead of the paper's Llama-3, so not comparable to published numbers): unified 44.25 vs 41.00 for base BAGEL (+3.25, 95% CI [+2.18, +4.39]). At a 256-token cap there is no gain (53.19 vs 53.32): the model writes shorter captions that front-load content.
Integrity
The original .pt checkpoint was loaded with torch.load(..., weights_only=True), written to safetensors, reloaded,
and compared tensor by tensor: key set, dtypes, shapes and values identical. Hashes are in manifest.json.
License
Apache 2.0, as for the base model. Users must also comply with the base model's terms.