LightNav-0

LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation

LightNav-0 is a compact generalist embodied navigation model that elicits the spatial intelligence of a pretrained vision-language model (Qwen3-VL) and aligns it with navigation, without task-specific prediction heads. Diverse tasks share one token interface: dual-channel pointing expresses task-, scene- and embodiment-agnostic spatial intent, and a residual vector-quantized (RVQ) action tokenizer maps that intent to precise, embodiment-specific trajectories — so instruction following, open-vocabulary object navigation and visual tracking live in a single model.

Inference, serving and evaluation code: LightNav-0 repository.

Files

Path	Content
`config.json`, `model-*.safetensors`, `model.safetensors.index.json`	Qwen3-VL weights (bf16) with the navigation token embeddings
`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`, `processor_config.json`	tokenizer / processor
`eval_config.json`	processing parameters that must match training (video size, pooling, SlowFast history tiers, task settings); read automatically by the inference code
`action_tokenizer/`	RVQ action-tokenizer bundle shared by navigation and tracking (one 3 × 256 residual codebook, horizon 10)

The bundle holds manifest.json, the per-level codebooks (codebook_l{0,1,2}.npy, 3 × 256 codes), jacobian_weights.npy and alpha_per_source.json. The model emits one <act_l{level}_{code}> token per level; the bundle decodes them into a (10, 3) chunk of robot-local waypoints [forward_m, lateral_m (+left), yaw_rad (+ccw)]. eval_config.json references the bundle by a path relative to this directory, so the inference code finds them without extra flags.

Usage

pip install "lightnav[vllm,video] @ git+https://github.com/lightrobo/LightNav-0"
hf download LightOriginsHQ/LightNav-0 --local-dir ./LightNav-0-ckpt

# offline prediction on a clip
lightnav-predict --model_path ./LightNav-0-ckpt --backend vllm_local \
    --video clip.mp4 --fps 4 --instruction "follow the person in the red shirt"

# serve (tracking prompt) and drive with the reference client
lightnav-serve --task tracking --model_path ./LightNav-0-ckpt --backend vllm_local --port 8050
lightnav-ws-client --server ws://localhost:8050 --video clip.mp4 --fps 4 \
    --instruction "follow the person in the red shirt"

# navigation prompt (instruction following / object navigation)
lightnav-serve --task vln --model_path ./LightNav-0-ckpt --backend vllm_local --port 8051

Habitat VLN-CE / ObjectNav and EVT-Bench evaluation recipes, the WebSocket protocol and the real-robot deployment guide are in the code repository's docs/.

Model details

Backbone: Qwen3-VL (4B), bf16; the trajectory / pointing tokens are ordinary rows of the embedding table, so the checkpoint loads with stock transformers.
Input: a SlowFast-compressed history of first-person RGB frames (model input 256×448, 4 fps) plus a natural-language instruction.
Output: dual-channel pointing tokens (<apos_*> / <opos_*>) followed by three RVQ action tokens (<act_l0_*><act_l1_*><act_l2_*>) decoded to a 10-step waypoint chunk.

Community

Questions, deployment notes and release news — join us on Discord, or scan to join the WeChat group:

License

Apache License 2.0.

Loading…

LightNav-0

LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation

Inference, serving and evaluation code: LightNav-0 repository.

Files

Path	Content
`config.json`, `model-*.safetensors`, `model.safetensors.index.json`	Qwen3-VL weights (bf16) with the navigation token embeddings
`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`, `processor_config.json`	tokenizer / processor
`eval_config.json`	processing parameters that must match training (video size, pooling, SlowFast history tiers, task settings); read automatically by the inference code
`action_tokenizer/`	RVQ action-tokenizer bundle shared by navigation and tracking (one 3 × 256 residual codebook, horizon 10)

Usage

pip install "lightnav[vllm,video] @ git+https://github.com/lightrobo/LightNav-0" hf download LightOriginsHQ/LightNav-0 --local-dir ./LightNav-0-ckpt # offline prediction on a clip lightnav-predict --model_path ./LightNav-0-ckpt --backend vllm_local \ --video clip.mp4 --fps 4 --instruction "follow the person in the red shirt" # serve (tracking prompt) and drive with the reference client lightnav-serve --task tracking --model_path ./LightNav-0-ckpt --backend vllm_local --port 8050 lightnav-ws-client --server ws://localhost:8050 --video clip.mp4 --fps 4 \ --instruction "follow the person in the red shirt" # navigation prompt (instruction following / object navigation) lightnav-serve --task vln --model_path ./LightNav-0-ckpt --backend vllm_local --port 8051

Habitat VLN-CE / ObjectNav and EVT-Bench evaluation recipes, the WebSocket protocol and the real-robot deployment guide are in the code repository's docs/.

Model details

Backbone: Qwen3-VL (4B), bf16; the trajectory / pointing tokens are ordinary rows of the embedding table, so the checkpoint loads with stock transformers.

Input: a SlowFast-compressed history of first-person RGB frames (model input 256×448, 4 fps) plus a natural-language instruction.

Output: dual-channel pointing tokens (<apos_*> / <opos_*>) followed by three RVQ action tokens (<act_l0_*><act_l1_*><act_l2_*>) decoded to a 10-step waypoint chunk.