Byrne-DFINE-N

4M D-FINE object detector with the SpikeWhale HRM family trait. Pretrained on Objects365, fine-tuned on COCO (80 classes). Exports a TensorRT-compatible ONNX graph (no GridSample, explicit deformable sampling). Runs from a laptop CPU to a Blackwell GPU.

Object detection, live results - TensorRT on an RTX 5060 Ti (Blackwell, sm_120), ~26 ms end-to-end.

What it is

• Architecture: D-FINE-N (nano) - a DETR-style detector whose FDR decoder already performs iterative box refinement. On top of it sits the HRM trait: a zero-init, gated iterative-refinement block added to each decoder layer (init == vanilla D-FINE, so the trait only ever helps). The gate learned to 0.38, and rose monotonically every training run - the model genuinely uses the trait.
• Training recipe: ImageNet HGNetV2-B0 backbone → 50k steps Objects365 pretrain → warm-start → 30k steps COCO fine-tune. This recipe cut clean-view COCO det-loss -18% vs a plain-COCO-from-random baseline (26.4 ← 32.2; bbox -28%, giou -30%, vfl -31%).
• Size / speed: 4.0 M params · fp32 TensorRT engine ≈ 23 MiB · ~26 ms end-to-end on an RTX 5060 Ti.
• Note: this is an early checkpoint (~5% of D-FINE's 72-epoch budget), so expect solid detections on common classes and some noise on rare ones - raise the confidence threshold to trade recall for precision.

Files

Input / output spec (all backends)

From meta.json:

• Input images: float32[N, 3, 640, 640], RGB, stretch-resized to 640×640, divided by 255 (mean 0 / std 1), channel layout NCHW.
• Output logits: float32[N, 300, 80] - apply sigmoid for per-class scores (the model is NMS-free; just threshold).
• Output boxes: float32[N, 300, 4] - cxcywh, normalized to [0, 1]. Convert to pixels with the original image size.

---

Usage

The repo is gated. Authenticate first: huggingface-cli login, or pass token= / set HF_TOKEN.

1. PyTorch (model.pt)

Loading the checkpoint needs the vendored model code (the HRM trait is not in upstream D-FINE-seg), which lives in src/ in this repo.

import os, sys, json
import numpy as np, torch
from PIL import Image
from huggingface_hub import snapshot_download

repo = snapshot_download("Quazim0t0/Byrne-DFINE-N")   # pulls model.pt, meta.json, src/
sys.path.insert(0, repo)
os.environ["DFINE_USE_HRM"] = "1"                       # the checkpoint was trained with it
from src.d_fine.dfine import build_model

meta  = json.load(open(f"{repo}/meta.json"))
names = meta["class_names"]; S = meta["input_h"]        # 640

model = build_model("n", len(names), False, "cpu", img_size=[S, S]).eval()
model.load_state_dict(torch.load(f"{repo}/model.pt", map_location="cpu", weights_only=False)["model"])
model.cuda()                                            # or leave on CPU

img = Image.open("your.jpg").convert("RGB"); W, H = img.size
x = np.asarray(img.resize((S, S)), np.float32) / 255.0
t = torch.from_numpy(x).permute(2, 0, 1).unsqueeze(0).cuda()

with torch.no_grad():
    out = model(t)
logits, boxes = out["pred_logits"][0], out["pred_boxes"][0]
scores = logits.sigmoid(); conf, cls = scores.max(-1)
for s, c, (cx, cy, bw, bh) in zip(conf.tolist(), cls.tolist(), boxes.tolist()):
    if s < 0.5: continue
    x1, y1 = (cx - bw/2) * W, (cy - bh/2) * H
    x2, y2 = (cx + bw/2) * W, (cy + bh/2) * H
    print(f"{names[c]} {s:.2f}  [{x1:.0f},{y1:.0f},{x2:.0f},{y2:.0f}]")

2. ONNX Runtime - CPU (model.onnx, self-contained, no model code)

import json, numpy as np, onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download

onnx = hf_hub_download("Quazim0t0/Byrne-DFINE-N", "model.onnx")
meta = json.load(open(hf_hub_download("Quazim0t0/Byrne-DFINE-N", "meta.json")))
names = meta["class_names"]; S = meta["input_h"]

sess = ort.InferenceSession(onnx, providers=["CPUExecutionProvider"])  # or CUDAExecutionProvider
img = Image.open("your.jpg").convert("RGB"); W, H = img.size
x = np.asarray(img.resize((S, S)), np.float32) / 255.0
x = np.transpose(x, (2, 0, 1))[None]                     # NCHW
logits, boxes = sess.run(["logits", "boxes"], {"images": x})
scores = 1 / (1 + np.exp(-logits[0]))                    # sigmoid
cls = scores.argmax(1); conf = scores.max(1)
for i in np.where(conf > 0.5)[0]:
    cx, cy, bw, bh = boxes[0, i]
    print(names[cls[i]], round(float(conf[i]), 2),
          [round((cx-bw/2)*W), round((cy-bh/2)*H), round((cx+bw/2)*W), round((cy+bh/2)*H)])

3. TensorRT (model.onnx → engine)

The ONNX is already TRT-ready (explicit deformable sampling, no GridSample). Build fp32 - the D-FINE decoder is numerically sensitive and must stay FP32.

Option A - trtexec:

trtexec --onnx=model.onnx --saveEngine=byrne_dfine_n.engine \
        --minShapes=images:1x3x640x640 --optShapes=images:1x3x640x640 --maxShapes=images:8x3x640x640

Option B - the dfine-cpp C++/TensorRT runtime (build engine + run detect/eval/bench). On Blackwell (RTX 50xx, sm_120) build the Docker image from nvcr.io/nvidia/tensorrt:25.08-py3 with --build-arg CUDA_ARCH=120:

# inside the dfine-cpp container
./dfine_build model.onnx byrne_dfine_n.engine     # ONNX -> fp32 engine (~85 s, ~23 MiB)
./dfine_detect byrne_dfine_n.engine your.jpg      # ~26 ms/frame

Apply the same preprocessing as above (RGB, stretch-resize 640, /255) and post-processing (sigmoid on logits, cxcywh-normalized boxes, no NMS).

---

Fine-tuning further

The checkpoint is a warm-start for more detection training. Keep the HRM trait enabled (DFINE_USE_HRM=1) so the trained gate is preserved, and load model.pt with --init.

Streaming recipe (no dataset download) - the reference trainer streams detection-datasets/coco and toilaluan/object365 straight from the Hub and builds D-FINE targets on the fly (boxes normalized, category ids remapped to 0-79):

DFINE_USE_HRM=1 python train_family_dfine.py \
    --init  model.pt \
    --dataset coco \
    --model n --steps 30000 --batch-size 8 --lr 1e-4 \
    --num-workers 2 --save-every 5000 \
    --out runs/byrne_dfine_n_finetune

Guidelines:

• Warm-starting a different class set: --init is shape-filtered - matching tensors load, the 80-class heads reinitialize if your class count differs (Objects365→COCO reinitialized 9/686 tensors this way).
• Backbone: add --pretrained-backbone to start the HGNetV2 backbone from ImageNet (much faster convergence) when training a fresh stage; omit it to keep the current weights.
• Fair evaluation: compare det-loss on a fixed clean-view batch (eval_det_loss.py), not the augmented training loss.
• Export after fine-tuning: produce the runtime ONNX with dfine-cpp/trt-files/scripts/export_dfine_onnx.py --deform explicit --dfine-src <this src/>. The custom RMSNorm in the HRM trait is decomposed from primitives specifically so the graph exports to ONNX opset 16 (fused aten::rms_norm has no symbolic).

For a conventional (downloaded-COCO) fine-tune, the vendored src/ is a drop-in modified D-FINE-seg - use its standard training loop with DFINE_USE_HRM=1.

---

Provenance & credits

• Base detector: D-FINE (Peng et al.) - https://github.com/Peterande/D-FINE
• Export-compatible arch: D-FINE-seg (ArgoHA) - https://github.com/ArgoHA/D-FINE-seg
• HRM trait: SpikeWhale family-trait line (zero-init gated iterative refinement).
• Datasets: Objects365 (pretrain), COCO (fine-tune), both streamed from the Hub.

License: Apache-2.0.