OmniShotCut v1.5 — Core ML, fp32

uva-cv-lab/OmniShotCut_v1.5, the weights of OmniShotCut (arXiv 2604.24762), converted to Core ML for the GPU by scripts/convert.py, for the Swift package OmniShotCutCoreML:

hf download starkdmi/OmniShotCut-v1.5-coreml-fp32 --local-dir OmniShotCut-v1.5-coreml-fp32

import OmniShotCutCoreML

let model = URL(fileURLWithPath: "OmniShotCut-v1.5-coreml-fp32/OmniShotCut-v1.5-fp32.mlpackage")
let detector = try await OmniShotCut(model: model, computeUnits: .cpuAndGPU)
let segments = try await detector.segments(in: videoURL)

It is also the verifier of exact mode, next to the fp16 conversion on the Neural Engine, which it runs again only on near-tie windows:

let exact = try await OmniShotCut(model: fp16Model, verifier: model)

• The official graph, sequence-major, with the batch norms folded into the convolutions in float64 and every tensor rounded once to fp32; computed in fp32. 365 ops, all placed on the GPU; 207 MB.
• Input frames, Float16 [100, 3, 96, 128]: one window of 100 frames, RGB 0–255 as ffmpeg -s 128x96 -pix_fmt rgb24 decodes them (exact in fp16), black past the end of the video. Outputs, Float32 [24, C]: the logits intra (10 classes), inter (7) and end (102) of 24 queries; each query's prediction is the argmax over every class but the last. The package reads the video, forms the windows and stitches them as the official code does.
• Runs on .cpuAndGPU. It loads in 70–80 ms with Core ML's cache warm and 210–590 ms without; the first prediction takes about 105 ms.
• Deployment target macOS 13 / iOS 16; the Swift package needs macOS 14 / iOS 17. The window size, query and class counts and both layouts are in the model's user-defined metadata.

Against the official PyTorch code's 219 cuts on three Blender open movies (© Blender Foundation, CC BY 3.0), the package with this model finds all 219 on the same frame with the same labels. Its logits are within 3.4e-4 of PyTorch's in fp64.

On a Mac mini M4 (16 GB) a 100-frame window takes 55.3–55.4 ms, against 72.1–72.7 ms for OmniShotCutMLX with fp32 weights on the GPU, and the 12-minute Tears of Steel 12.1–13.2 s end to end, against 16.9–17.0 s. The process running the windows peaks at 481–498 MB, against 1246 MB. The package's README has all configurations. Memory is in MB of 2^20 bytes.

License

MIT (LICENSE). The conversion is Copyright (c) 2026 Dmitry Starkov; OmniShotCut's code and weights are Copyright (c) 2026 UVA Computer Vision Lab, under the same licence.

@article{wang2026omnishotcut,
  title={OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer},
  author={Wang, Boyang and Xu, Guangyi and Zhang, Jiahui and Tang, Zhipeng and Cheng, Zezhou},
  journal={arXiv preprint arXiv:2604.24762},
  year={2026}
}