VoiceMem-Jev

VoiceMem-Jev selects the reasoning depth for a voice conversation turn. It combines the current transcript, recent dialogue and optional speech-state observations to return ordinary or deep, together with their probabilities.

The model supports Chinese and English and uses a 140.8M-parameter jhu-clsp/mmBERT-small encoder with a custom classification head. Speech-state inputs use shared physical quantities, so integration does not depend on a particular ASR provider.

Inputs and outputs

The output contains label, p_ordinary and p_deep. A deep probability of 0.5 or greater selects deep. Omit unavailable observations; an observed zero and a missing value have different masks.

ASR, turn detection, memory retrieval, reply generation and speech synthesis are provided by the host application.

Repository layout

Keep each weight file with its matching configuration and tokenizer. The classification head requires the supplied runtime.

CPU inference

hf download Sorbett/VoiceMem-Jev --revision main \
  --include 'onnx/*' 'python/*' 'README.md' 'LICENSE' 'NOTICE.txt' 'licenses/*' \
  --local-dir ./VoiceMem-Jev
python -m pip install './VoiceMem-Jev/python[onnx]'

from voicemem_jev import OnnxRouter

router = OnnxRouter('./VoiceMem-Jev/onnx')
decision = router.predict(
    'Can you compare these options and explain the trade-offs?',
    history=[{'role': 'user', 'content': 'I am choosing an apartment.'}],
    observations={'eot_probability': 0.9, 'silence_s': 0.4, 'span_s': 3.0},
)
print(decision.label, decision.p_deep)

CPU inference uses NumPy, Tokenizers and ONNX Runtime. The standalone CPU runtime does not require Studio, PyTorch or Transformers.

CUDA inference

hf download Sorbett/VoiceMem-Jev --revision main \
  --include 'eager/*' 'python/*' 'README.md' 'LICENSE' 'NOTICE.txt' 'licenses/*' \
  --local-dir ./VoiceMem-Jev
python -m pip install './VoiceMem-Jev/python[eager]'

from voicemem_jev import Router

router = Router(
    './VoiceMem-Jev/eager', backend='eager', device='cuda:0',
)
print(router.predict('How are you today?'))

For reproducible deployments, replace the branch name in download commands with the published commit SHA.

VoiceMem Studio integration

Install the runtime in Studio's Python environment. In Settings > Components > Router > Adapter, configure:

CPU ONNX options:

{"backend":"onnx","device":"cpu","allow_prototype":true}

CUDA options with compilation:

{"backend":"eager","device":"cuda:0","allow_prototype":true,"compiled":true}

The adapter prewarms five sequence-length buckets before serving. Compilation requires a CUDA device. The compiled runtime can recover to the same-weight eager CUDA model when its execution environment changes.

The adapter requires Studio's studio.core.utils.reply_modes.encoder_depth.EncoderDepthRouter interface. Studio owns asynchronous execution, cancellation, caching and device admission. The checkpoint retains its evaluation-stage deployment flag, so the adapter requires allow_prototype=true.

Model and training

The current transcript and recent dialogue are encoded separately. Their representations and elementwise interaction are combined with speech-state features and missing-value masks for classification.

Training uses cross-entropy supervision followed by RLCD optimization, with public and authored dialogue, synthetic ASR variants and Qwen3-8B supervision. Temperature calibration is fitted on a separate calibration split. Training settings are recorded in TRAINING.json.

The runtime retains the last four user/assistant messages, with 80 characters per message, and encodes up to 384 tokens per text. The calibrated temperature is 0.6. Keep these preprocessing and calibration settings with the weights.

Evaluation

The development regression contains 664 machine-labelled examples: 303 English and 361 Chinese, with 150 deep and 514 ordinary labels.

The regression was used during development and contains related translations and variants. It does not establish accuracy on independent natural speech. An authored context diagnostic with 12 scenario families and 360 variants reports 80.83% accuracy and 8.33% context-deep recall; context-dependent follow-up routing remains a limitation.

Warm single-decision latency includes tokenization and disables result caching:

Initial CUDA compilation and prewarming take approximately 61 seconds, excluding model loading. Measurements use Linux and exclude queueing and the rest of the voice pipeline. Native macOS, Windows and concurrent ASR/TTS performance have not been benchmarked. See EVALUATION.json for details.

License

VoiceMem-Jev is distributed under Apache-2.0. The mmBERT base model is MIT-licensed. Attribution and source revisions are provided in LICENSE, NOTICE.txt, SOURCE.json and licenses/.