CLEF Calibrated INT8 ConvRot

CLEF turns text, JSON, images, and video into probabilities over the allowed answers to typed questions, in a single forward pass.

Benefits

• Lower VRAM: 43.3% lower measured process peak than BF16.
• Self-contained weights and processor: loads directly into INT8/BF16 storage, without downloading or loading the original BF16 checkpoint first.
• Calibrated INT8 text projections reduce drift from BF16 compared with naive INT8. Embeddings, vision tower, small projections, and decision head stay BF16.
• H100 CuTeDSL kernels fuse calibrated scales, biases, and MLP SwiGLU.

Measured in fresh processes on H100 80 GB, batch size 1, using the same 23 text records (298-2,270 tokens) and one 224x224 image loading/parity probe. NVML process memory is sampled every 5 ms. The BF16 baseline uses the unmodified source runtime, without CuTe patches.

Warm median latency over 30 repeats, including the decision head. Both INT8 paths use the same calibrated weights. The Torch path uses ATen integer GEMMs and native pointwise operations, without CuTe kernels. Both use Comfy Kitchen's CUDA ConvRot activation quantizer. Measurements. Usage.

Batchable throughput on H100: 1,502 tokens per record, eight questions per record, four options per question. Each answered question counts as one decision. Warm medians over 10 repeats include the backbone and decision head. Both INT8 curves use the calibrated weights. Batch measurements. These are equal-length synthetic batches; tokenization, request queuing, and batch assembly are excluded. Batch latency, peak VRAM, and decision agreement with batch size 1 are included in the measurements.

Limitations

• Inference-only custom loader; not a drop-in AutoModel.from_pretrained checkpoint.
• CuTeDSL backend requires H100/SM90 and the dependencies listed in Usage.
• BF16 can be faster, particularly on short inputs; memory reduction is the primary benefit, not a universal latency improvement.
• Calibration and evaluation use small synthetic text sets. Agreement below is with BF16, not ground-truth accuracy; image/video quality is not evaluated.
• One BF16 decision disagreement remains. Validate representative inputs before deployment.

Evaluation

16 held-out records and seven regression records, with 80 decisions in total. BF16 is the reference. Calibration uses separate training and selection records.

Calibration reduces mean held-out logit drift by 15.2% and hidden-state drift by 6.2%. The direct loader preserves calibrated hidden states and logits exactly on all 23 text records and the image parity probe. Evaluation details.