Sensor-Language-Action Models

🔥 News

• [2026-10-06] Our paper is available on arXiv.
• [2026-10-06] Code released on GitHub, and model released on HuggingFace.
• [2026-10-06] Project website is live!

📖 Introduction

OpenSLA is a new family of models - Sensor-Langauge-Action (SLA) models. It takes a multi-channel sensor history together with language context and produces a structured action prediction and a sensor-state caption. We release two variants:

• OpenSLA-B: sensor tokens are projected and prepended to the language prompt as a flat token prefix.
• OpenSLA-H: builds on B with a hierarchical sensor encoder that compresses the signal into local, per-channel, and global memory tokens.

📖 Table of Contents

1. Installation
2. Quick Start
3. Pretrained Weights
4. Usage
5. Datasets
6. Citation

💿 Installation

git clone https://github.com/yang-ai-lab/Sensor-Language-Action-Model.git
cd Sensor-Language-Action-Model
pip install -r requirements.txt

Dependencies

• Python >= 3.10
• PyTorch >= 2.5
• Transformers >= 5.8.1 (for the Qwen3.5 backbone)
• PEFT >= 0.12

🚀 Quick Start

demo.ipynb loads a checkpoint, builds an input batch, and runs prediction. The same flow in Python:

import torch
from opensla import OpenSLA, SensorBatch

model = OpenSLA.from_checkpoint(
    "pretrained_weights/opensla_h_clinical.pt", domain="clinical",
    dino_checkpoint="pretrained_weights/waveform_encoder.ckpt",
    numeric_config="pretrained_weights/numeric_config.json",
    action_group_types="pretrained_weights/action_group_types.json",
)

batch = torch.load("data/preprocessed_batch.pt", weights_only=True)
predictions = model.predict(batch["input_text"], SensorBatch(**batch["sensors"]))

📦 Pretrained Weights

Download the files from this repository and put them under pretrained_weights/ in the code repository.

👩‍💻 Usage

Input Format

The model takes preprocessed signals as a .pt dictionary with domain, input_text (one text context per sample), and sensors. The sensors are:

• Waveform: waveform of shape [B, M, C, T], with M one-minute slots, C physical channels, and T = 7500 samples per slot (60 s at 125 Hz), plus a boolean waveform_channel_mask ([B, M, C]) marking observed channel-minutes and int64 waveform_modality_ids ([B, C]) indexing opensla.WAVEFORM_MODALITIES[domain].
• Numeric: observed events as parallel [B, E] tensors (values, rel_time_min in minutes before the decision, measure_ids, source_ids, event_mask), plus per-measure summary features (summary_features, summary_measure_ids).

Command line

Point configs/template.json at your weights and input batch, then:

opensla --config configs/template.json --dry-run   # print the resolved config
opensla --config configs/template.json             # write outputs/predictions.jsonl

On Slurm:

sbatch --account=YOUR_ACCOUNT --partition=YOUR_PARTITION scripts/run_template.sbatch --config configs/template.json

📊 Datasets

The models are trained and evaluated on six datasets from three healthcare settings, with an additional MIMIC-IV held out as an external clinical cohort. All of them are publicly available and can be accessed through the following links.

📝 Citation

If you use this code or models in your research, please cite our paper:

@misc{xu2026opensla,
  title  = {Sensor-Language-Action Models},
  author = {Xu, Yuekai and Shuai, Zitao and Yang, Yuzhe},
  journal = {arXiv preprint},
  year = {2026}
}

Acknowledgments

The waveform encoder adapts from the sensor encoder from OSF (MIT License).