elephant-1 — ROCmFPX GGUF

baselabs/elephant-1, Base Labs' 9B decision model (post-trained from Qwen3.5-9B), quantized to the ROCmFPX formats for AMD GPUs. Give it a state and a set of questions; it returns a probability for every option of every question from one shared prefill.

These files run with llama-decide, a small tool in HarshaVardhan3002/ROCmFPX that reads the same request format as Base Labs' own runtime. A normal chat server is not the way to use this model.

Start with Q4_0_ROCMFP4_STRIX. It is the closest 4-bit file to the original.

These files only load in a ROCmFPX build of llama.cpp. They have not been tested on AMD hardware yet.

Not an official release of Base Labs or Baseten. All credit for the model goes to Base Labs.

Files and measured quality

Measured against the BF16 model (perplexity 7.5591) on WikiText-2 test, 40 chunks of 512 tokens. KLD is the mean KL divergence of the next-token distribution from BF16 (lower is better). Same top is how often the quant picks the same next token as BF16. These numbers use ordinary left-to-right attention on plain text; they show how much each file changes the weights, not decision accuracy, which was not measured.

The 6-bit file is named elephant-1-Q6_K_ROCMFPX.gguf so that Hugging Face lists it as a 6-bit quant. Its weights are the ROCmFPX Q6_0_ROCMFPX type, not a stock llama.cpp Q6_K, and it needs the same fork as the other files.

Not published: Q3_0_ROCMFPX (4.9 GB, PPL 7.9236, KLD 0.0750, same top 86.74%) is larger than Q4_0_ROCMFP4_FAST and further from BF16.

Decisions

The GGUF files carry the decision settings from the original checkpoint (qwen35.decision.*: block-bidirectional attention on the full-attention layers and the 256 answer labels), so llama-decide needs no extra files.

git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-decide -j
build/bin/llama-decide -m elephant-1-Q4_0_ROCMFP4_STRIX.gguf -ngl 999 -f request.json

request.json: a state (text or JSON) and questions, as in Base Labs' card:

{
  "state": {
    "task": "Pick up the red mug from the table and place it on the shelf.",
    "observations": ["The gripper closed around the handle of the red mug.",
                     "The arm lifted the mug 12 cm above the table.",
                     "The mug slipped and fell back onto the table, upright and intact."],
    "gripper_force_newton": 4.1
  },
  "questions": {
    "success": {"type": "noul", "instructions": "Was the task completed successfully?"},
    "failure_mode": {"type": "choice", "instructions": "What is the main reason for the outcome?",
      "criteria": {"grasp_slip": "The object slipped out of the gripper.", "collision": "The arm or the object hit something.",
                   "wrong_object": "The robot handled the wrong object.", "no_failure": "Nothing went wrong."}},
    "progress": {"type": "score", "instructions": "How far did the robot get with the task?",
      "criteria": ["did not start", "grasped the object", "lifted the object", "placed the object on the shelf"]}
  }
}

Answer from elephant-1-Q4_0_ROCMFP4_STRIX.gguf (CPU backend): success 0.020; failure_mode grasp_slip 0.996; progress most likely "grasped the object" (0.719), then "lifted the object" (0.223).

Options: --debias N (average over N cyclic shifts of the answer labels), --causal (plain causal attention instead of the trained block-bidirectional mask). llama-decide was checked against Base Labs' PyTorch runtime on this model (probabilities within 0.004).

How these were made

1. convert_hf_to_gguf.py --outtype bf16 from the fork, which also writes the decision settings.
2. Importance matrix from the BF16 model on the GPU over a general calibration text (calibration_datav3), 512-token chunks.
3. llama-quantize --imatrix, one run per type (ROCmFPX fb08d7c merged onto upstream llama.cpp).
4. Quality: the BF16 reference ran on an NVIDIA GPU with stock BF16 kernels; each quant was scored with the fork's CPU backend, which runs the ROCmFPX types directly. AMD GPU kernels compute the same math in a different order, so results on an AMD GPU can differ slightly.

The 8-bit file is not as close to BF16 as a stock Q8_0 usually is: ROCmFPX block scales have a fixed smallest step, so 8-bit blocks of small weights keep fewer than 8 bits (ROCmFPX#118).

License

MIT, same as the base model.