ViveKi-0.1-20B

Real-time decisions on text, images and video. No generated tokens.

ViveKi is an open-source video language model created by Ekalabya Ghosh (Hugging Face: not-ekalabya). It scores plain-language questions against live video, images and text in one forward pass.

Ask a question in plain language, give the options, get a score for each option back in a single fast forward pass. Describe the event or state you care about, and the model watches for it.

• Fast. About 0.26 s per decision on an A100, so it can re-check a feed every couple of seconds.
• Cheap. About $0.33 per hour of video on a rented A100, with one decision every 2 s.
• Zero-shot. Your question is the only configuration: "Is a forklift carrying a tall stack of crates?", "Did the ball cross the line?"
• Any modality. Text, images, video frames, or all three in one request, with as many options and questions as you need.
• Shows where. localize returns a heatmap of where in the frame the answer comes from.

Built for event and state detection on live feeds. It is not a tool for searching or answering questions over long videos.

See it work

Live monitoring. One running question across real footage: it sits at NO through the build-up, flips to YES the instant the ball crosses the line, fires on a real fight clip and stays at NO on a real non-fight clip. Same question, same model, no per-clip tuning. (Video above.)

Localization. A plain-text description of an unsafe state on real 1920x1080 factory CCTV: "Is a forklift carrying a tall stack of three or more crates?" The heat appears only while the model answers YES; the strip underneath is its probability over time. It misses once (t = 6.0 s, the cab is hidden behind the stack), and that frame is left in.

_{Clip 3_te1 from Önal & Dandıl (2024), Safe and Unsafe Behaviours dataset, CC BY 4.0 (Mendeley Data, doi:10.17632/xjmtb22pff.1).}

Performance

Detection quality. On a 1-hour stream built from 30 events across 5 action classes, at the default 0.5 threshold the model caught 52% of events with 100% precision and zero false alarms per hour; ranking quality (AUROC) was 0.81. It failed on one class (basketball dunk). This is a small, synthetic test, so treat it as a rough guide and check it on your own footage.

_{Single measurements on one A100-80GB, bf16, plain forward pass (no caching, batching or compilation). Over the network a caller sees about 0.75 s more: the client-side median was ~1.0 s in the same test.}

Quickstart

pip install "transformers>=5.8.0" torch accelerate pillow torchvision flash-linear-attention

from transformers import pipeline

decide = pipeline("viveki-decision", model="not-ekalabya/viveki-0.1-20B", trust_remote_code=True,
                   torch_dtype="bfloat16", device_map="auto")

result = decide({
    "video": frames,                       # list of bytes / base64 str / PIL.Image, in order
    "questions": [
        {"instruction": "Is a vehicle entering?", "options": ["yes", "no"]},
        {"instruction": "What is happening?", "options": ["loading", "idle", "unloading"]},
    ],
})

[
  {"instruction": "Is a vehicle entering?",
   "scores": {"yes": {"score": 0.0421, "probability": 0.91},
              "no":  {"score": -0.0117, "probability": 0.09}}},
  ...
]

score is the model's raw readout; probability is a softmax over the scores added by the pipeline (softmax_temperature= sets how sharp it is).

What you can pass

One request per call. Combine any of these; every question in the request is scored against the same content in one pass, so five questions cost one pass, not five.

At least one of image(s), video, text is required. The only size limit is the base model's 262,144-token context.

Without the pipeline

from transformers import AutoModel, AutoProcessor

processor = AutoProcessor.from_pretrained("not-ekalabya/viveki-0.1-20B", trust_remote_code=True)
model = AutoModel.from_pretrained("not-ekalabya/viveki-0.1-20B", trust_remote_code=True,
                                  torch_dtype="bfloat16").to("cuda")

outputs = model(**processor({
    "text": "Customer wrote: 'this broke after one day, I want a refund.'",
    "questions": [{"instruction": "What is the customer asking for?",
                    "options": ["refund", "replacement", "complaint only", "praise"]}],
}))
print(outputs.scores[0][0])   # raw per-option scores for the first question

Localization

model.localize(...) returns a coarse heatmap of where in each frame the answer comes from. No boxes, no generated tokens; turning a map into boxes or alerts is up to you.

out = model.localize(processor, frames,    # PIL frames; pass full resolution for finer maps
                     "Is a forklift carrying a tall stack of three or more crates?")

out.heatmaps        # (T, gh, gw) in [0, 1]; one cell = a 32x32 px patch of the input
out.probabilities   # (T, 2) over [yes, no]

for frame, heat, p in zip(frames, out.heatmaps, out.probabilities[:, 0]):
    if p < 0.5:     # only trust the map when the model answers YES
        continue
    heat = cv2.resize(heat, frame.size, interpolation=cv2.INTER_CUBIC)

Works best for scene-level states described in plain text (a forklift, a stack of crates, a worker at a machine). Small or fine details, such as an open guard or a person outside a marked walkway, gave weak maps. Validated by eye on factory footage, with no box-accuracy benchmark. localize is a method of the model, not of the pipeline.

Several questions at once (experimental)

Pass a list of questions to answer all of them in one shared pass per frame. Outputs gain a question axis: heatmaps (T, Q, gh, gw), probabilities (T, Q, n_options). This is about 2x cheaper for three questions, but in our one test (19 frames) answers were noticeably worse than asking each question alone, and two of three questions lost most of their frame-to-frame signal. Check it on your own data first; one question per call is the validated path.

Requirements

• transformers>=5.8.0, torch>=2.12.0, accelerate, trust_remote_code=True
• flash-linear-attention is required; without it the linear-attention layers fall back to a much slower path.
• A GPU that fits a ~27B-parameter model in bf16 (an 80GB-class card, or device_map="auto" across smaller ones). The forward pass exits early, so about 20B parameters execute per request, but all of the base model is loaded.

Author

ViveKi is built and maintained by Ekalabya Ghosh.

Citation

@misc{ghosh2026viveki,
  title        = {ViveKi-0.1-20B: Real-time decisions on text, images and video},
  author       = {Ghosh, Ekalabya},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/not-ekalabya/viveki-0.1-20B}}
}

License

Apache 2.0 for the code in this repository. The model also loads the base model it wraps (Qwen/Qwen3.8-27B); see that repository for its license terms.