OmniShotCut v1.5 — Core ML, fp16

uva-cv-lab/OmniShotCut_v1.5, the weights of OmniShotCut (arXiv 2604.24762), converted to Core ML for the Neural Engine by scripts/convert.py, for the Swift package OmniShotCutCoreML:

hf download starkdmi/OmniShotCut-v1.5-coreml-fp16 --local-dir OmniShotCut-v1.5-coreml-fp16

import OmniShotCutCoreML

let model = URL(fileURLWithPath: "OmniShotCut-v1.5-coreml-fp16/OmniShotCut-v1.5-fp16.mlpackage")
let detector = try await OmniShotCut(model: model)   // on the Neural Engine
let segments = try await detector.segments(in: videoURL)

With the fp32 conversion as its verifier, exact mode runs this model on every window and the fp32 one again on near-tie windows:

let exact = try await OmniShotCut(model: model, verifier: fp32Model)

• Laid out for the Neural Engine: activations [1, C, 1, S], every linear a 1×1 convolution, attention as two einsums per head. The batch norms, the ImageNet input normalization and the attention scale are folded into the weights in float64, then rounded once to fp16. 664 ops, all placed on the Neural Engine; 108 MB.
• Input frames, Float16 [100, 3, 96, 128]: one window of 100 frames, RGB 0–255 as ffmpeg -s 128x96 -pix_fmt rgb24 decodes them, black past the end of the video. Outputs, Float16 [1, C, 1, 24]: the logits intra (10 classes), inter (7) and end (102) of 24 queries; each query's prediction is the argmax over every class but the last. The package reads the video, forms the windows and stitches them as the official code does.
• Runs on .cpuAndNeuralEngine with MLOptimizationHints.specializationStrategy = .fastPrediction, the package's default on macOS 15 and iOS 18. A first load compiles the model for the Neural Engine, 7.7 s on an M4 (3.9 s without .fastPrediction); Core ML caches the result per process and compiled model location, and later loads take 65 ms. The package compiles the .mlpackage once into its caches.
• Deployment target macOS 13 / iOS 16; the Swift package needs macOS 14 / iOS 17. The window size, query and class counts and both layouts are in the model's user-defined metadata.

Against the official PyTorch code's 219 cuts on three Blender open movies (© Blender Foundation, CC BY 3.0), the package with this model finds 218 on the same frame with the same labels. The two differences are near-ties: a boundary label PyTorch in fp64 prefers by 0.0023, and an extra cut where it prefers to continue the segment by 0.046. fp16 arithmetic is not the same on every chip, so another Mac can decide a near-tie or two the other way. Exact mode finds all 219 cuts: it checks a window again with the fp32 model only where a decision is closer than 1.0 logit, one window in six on these videos, for 31% more time per window than this model alone on Tears of Steel.

On a Mac mini M4 (16 GB) a 100-frame window takes 20.3 ms, against 72.4 ms for OmniShotCutMLX's default on the GPU, and the 12-minute Tears of Steel 4.5 s end to end, against 16.9–17.1 s. The process running the windows peaks at 167 MB: 39 MB of footprint and the Neural Engine program's 128 MB, which the kernel charges to the process outside its footprint; with OmniShotCutMLX's default it peaks at 1338 MB. The package's README has all configurations. Memory is in MB of 2^20 bytes.

License

MIT (LICENSE). The conversion is Copyright (c) 2026 Dmitry Starkov; OmniShotCut's code and weights are Copyright (c) 2026 UVA Computer Vision Lab, under the same licence.

@article{wang2026omnishotcut,
  title={OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer},
  author={Wang, Boyang and Xu, Guangyi and Zhang, Jiahui and Tang, Zhipeng and Cheng, Zezhou},
  journal={arXiv preprint arXiv:2604.24762},
  year={2026}
}