SmolVLA + LIBERO + EEG — an 8-variant ablation on injecting a brain signal into a VLA

Eleven checkpoints and three EEG encoders from a study of whether an EEG motor-imagery signal can be made to steer a vision-language-action policy, and where in the architecture it has to enter for that to happen.

Every model here was trained on a single M5 MacBook (Apple MPS, no NVIDIA GPU).

Code: https://github.com/twu3202/SmolVLA_cl Companion release: Twu31/smolvla-cross-embodiment-mps · Twu31/rt1-lerobot-mps

---

The question, and how it was narrowed down

The project ran in eight stages, each one an answer to what the previous stage had ruled out.

The short version: the injection site was a real bottleneck and fixing it helped; the remaining ceiling is not. Nothing tried reaches 4/4, and the four directions trade off against each other rather than improving together.

---

Read this before you use the weights

The VLM backbone in v1–v7 is randomly initialised, not pretrained.

The EEG experiments branch off the Stage-1 run libero_spatial, which was trained with LOAD_VLM=0. Verified by tensor-wise comparison against lerobot/smolvla_base:

So the token-level injection results (v2–v7) describe EEG tokens entering a randomly initialised SmolVLM2 transformer that is then held frozen. Only stage1_libero_spatial_pretrained_vlm and v8 carry pretrained HuggingFaceTB/SmolVLM2-500M-Video-Instruct weights. Treat the controllability numbers as statements about the injection site and the encoder, not about pretrained multimodal grounding.

These are research artifacts from an architecture demo, not a working BCI. The headline result is a negative one — see Results.

---

Contents

dataset_stats.pt                                  # LIBERO spatial state/action mean+std — required for inference
encoders/
  eegnet_10subj_40.7/encoder_only.pt              # EEGNet,  10 subjects, 40.7% val acc  (used by v1–v3)
  eegnet_109subj_51.6/encoder_only.pt             # EEGNet, 109 subjects, 51.6% val acc  (used by v4, v7)
  eegnet_109subj_51.6/eeg_encoder.pt              # ...same run, with the 4-class MLP head attached
  atcnet_109subj_57.7/encoder_only.pt             # ATCNet, 109 subjects, 57.7% val acc  (used by v5, v6, v8)
stage1_libero_spatial_random_vlm/                 # SmolVLA on LIBERO, random VLM       (2000 steps)
stage1_libero_spatial_pretrained_vlm/             # SmolVLA on LIBERO, pretrained VLM   (2000 steps)
stage2_libero_spatial_eeg_additive/               # + EEG, randomly paired, 50% dropout (1000 steps)
v1_libero_spatial_eeg_synth/                      # additive injection, deterministic pairing
v2_libero_spatial_eeg_token/                      # token-level injection
v3_libero_spatial_eeg_token_balanced/             # + class-balanced sampling
v4_libero_spatial_eeg_token_balanced_v2/          # + better encoder  ← best result
v5_libero_spatial_eeg_token_final/                # ATCNet + auxiliary classification loss
v6_libero_spatial_eeg_token_atcnet_only/          # ATCNet, no aux loss
v7_libero_spatial_eeg_token_eegnet_5000/          # EEGNet, 5000 steps
v8_libero_spatial_eeg_token_atcnet_scratch/       # ATCNet, expert trained from scratch
eval_output/                                      # every figure and .npz behind the tables below

Each variant directory holds one step_*.pt plus the train_log.jsonl for that run. optimizer_state has been removed from every checkpoint — inference and evaluation work normally, resuming someone else's optimizer trajectory does not.

---

Results

Controllability: does switching the EEG class move the action in the expected direction?

The test fixes camera, robot state and language input, swaps only the EEG epoch, and checks whether the output shifts the way the training pairing prescribes (left_fist → Δx < 0, right_fist → Δx > 0, both_fists → gripper closes, both_feet → Δy > 0). A direction counts as passed when the shift has the right sign and the bootstrap confidence interval for that condition excludes the no-EEG baseline mean. Each condition is measured over 5 demos × 10 tasks × 30 steps = 1,500 predictions.

No configuration passes all four directions. v4 is the best at 3/4; v7 is the only run in which left_fist ever passes, and it loses two other directions to get there.

Four findings, three of them counter-intuitive

Why the stronger encoder loses

5-fold logistic-regression probes on the two encoders' 64-dim embeddings:

ATCNet buys its higher MLP-head accuracy with non-linear fusion that a downstream module reading it linearly cannot exploit, and it concentrates capacity on two classes. Across all ATCNet runs (v5, v6, v8), left_fist fails — exactly the class where its embedding is weakest. Per-class, linearly-readable discriminability matters more than headline accuracy when the encoder feeds another network.

Representation analysis (Stage 4)

The EEG representation is genuinely discriminative and nearly orthogonal to the action space; it carries a discrete intent class, not continuous kinematics. That orthogonality is why a 14-dim additive nudge cannot deliver it and why the injection site mattered so much.

Stage-1 open-loop baselines (no EEG)

A pretrained VLM buys +7.7 pp of gripper accuracy and almost nothing on translation — it helps the semantic when to grasp decision, not fine kinematics.

---

Architecture

SmolVLATokenLevelEEG (v2–v8):

EEG epoch (64ch × 320 samples @160Hz, 2 s)
  → frozen EEGNet / ATCNet                        → 64-dim embedding
  → eeg_proj: Linear(64,256) · GELU · Linear(256,960) · LayerNorm(960)
  → forward hook on vlm.get_input_embeddings() replaces the position-0 embedding
  → SmolVLM2 transformer (frozen; gradients pass through it)
  → flow-matching action expert                   → 50 × 7 action chunk

The additive variants (stage2, v1) instead use eeg_proj: Linear(64, 14) added onto the normalised robot state — the shallow baseline the study set out to beat.

train_expert_only=True, freeze_vision_encoder=True — the whole SmolVLM2 backbone stays frozen, which the weight diff above confirms independently.

Input / output

INPUT
  observation.images.image     (3, 256, 256)  ← LIBERO agentview camera
  observation.images.image2    (3, 256, 256)  ← wrist camera
                                                 both padded-resized to 512×512 internally
  observation.state            (14,)          ← eef_pos(3) + eef_quat(4) + joint_pos(7), z-scored
  language tokens              (48,)          ← LIBERO task string
  EEG epoch                    (64, 320)      ← 2 s @ 160 Hz, 4–40 Hz bandpass  [EEG models]

OUTPUT
  action chunk                 (50, 7)        ← Δxyz(3) + Δrpy(3) + gripper(1), z-scored

Training: 1000–5000 steps, batch 4 × grad-accum 4 (effective 16), LR 5e-5 cosine, grad-norm clip 10, ~0.93 step/s on M5 MPS — the token-level hook costs nothing measurable over additive injection. Runs took 18 min (1000 steps) to 92 min (5000 steps).

---

Usage

These are plain torch.save dicts, not lerobot-loadable model folders. You need the config and wrapper classes from the GitHub repo.

import torch
from huggingface_hub import hf_hub_download

REPO = "Twu31/smolvla-libero-eeg"
path = hf_hub_download(REPO, "v4_libero_spatial_eeg_token_balanced_v2/step_003000.pt")
ck   = torch.load(path, map_location="cpu", weights_only=False)

ck["step"]             # 3000
ck["vlm_hidden_dim"]   # 960
ck["policy_state"]     # SmolVLAPolicy state_dict, 500 tensors, bfloat16
ck["eeg_proj_state"]   # Linear(64,256) / Linear(256,960) / LayerNorm(960)

Full reconstruction:

import torch
from huggingface_hub import hf_hub_download
from libero_smolvla_config import make_libero_smolvla_config   # from the GitHub repo
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
from eeg_encoder import EEGNet                                  # or eeg_encoder_atcnet.ATCNet

REPO = "Twu31/smolvla-libero-eeg"
device = "mps"

cfg = make_libero_smolvla_config(device)
cfg.load_vlm_weights = False          # the checkpoint carries the full VLM state
policy = SmolVLAPolicy(cfg).to(device)
ck = torch.load(hf_hub_download(REPO, "v4_libero_spatial_eeg_token_balanced_v2/step_003000.pt"),
                map_location=device, weights_only=False)
policy.load_state_dict(ck["policy_state"])

enc = EEGNet(n_channels=64, n_timepoints=320, n_classes=4, embed_dim=64).to(device)
enc.load_state_dict(torch.load(hf_hub_download(REPO, "encoders/eegnet_109subj_51.6/encoder_only.pt"),
                               map_location=device, weights_only=False)["backbone_state"],
                    strict=False)

stats = torch.load(hf_hub_download(REPO, "dataset_stats.pt"),
                   map_location="cpu", weights_only=False)   # normalisation mean/std

Encoder ↔ checkpoint pairing is not optional — loading v4 or v7 with the 40.7% encoder, or v5/v6/v8 with an EEGNet, silently produces meaningless embeddings:

---

Training data

EEG preprocessing: 4–40 Hz bandpass, 60 Hz notch, 2 s epochs at +0.5 s from cue onset, 9,377 epochs over 4 classes. The preprocessed tensor is not redistributed here — it is one command to rebuild (download_eeg_data.py) via mne.datasets.eegbci.

No human subjects were recruited for this work. All EEG comes from the public PhysioNet release, whose own consent and IRB coverage governs it.

The EEG↔action pairing in v1–v8 is synthetic: an epoch of class k is attached to a training frame whose action matches a hand-written rule. It is a controllability probe, not evidence that a person's motor imagery drives this robot.

---

Limitations

• Random VLM backbone for v1–v7 (see the warning above).
• Open-loop evaluation only. No LIBERO closed-loop success rates are reported here.
• Effect sizes are small: |Δ| ≈ 0.01–0.10 in normalised action units. The pass criterion establishes that a shift is reliable, not that it is large enough to control anything.
• Single seed per variant. The v4/v7 trade-off (which directions pass) is exactly the kind of effect that could move under reseeding, and was not tested.
• One suite (libero_spatial), one embodiment, ~500 PhysioNet epochs per run.

---

Related releases

Where this repo asks "can a new modality be added?", smolvla-cross-embodiment-mps asks "how far does one architecture carry across robots?"

---

Citation

@software{smolvla_libero_eeg_2026,
  author = {Twu31},
  title  = {SmolVLA + LIBERO + EEG: an 8-variant ablation on injecting motor-imagery
            EEG into a vision-language-action policy},
  year   = {2026},
  url    = {https://github.com/twu3202/SmolVLA_cl}
}

Built on SmolVLA (Shukor et al., 2025), LIBERO (Liu et al., 2023), EEGNet (Lawhern et al., 2018) and ATCNet (Altaheri et al., 2023).