Maverick XR Agent for Unity

Control a Unity scene with typed or spoken English. This package runs Maverick-4B-Unity-XR-Agent on the player's own machine through llama.cpp: no account, no internet connection and no API key.

You choose which objects the agent may use. For each command the model reads those objects and calls one of 11 tools, which Unity then carries out: "put the red mug on the table", "turn on the lamp", "where is the stapler?". When a command could mean more than one object, it asks which one. When it cannot do something, it says so.

The article Offline Voice Control for Unity XR Scenes explains how the model was trained and tested.

Requirements

• Unity 2022.3 LTS or later
• Windows x64 (Linux and macOS are untested)
• About 3 GB of free GPU memory. Without a GPU the model runs on the CPU, but slowly.
• About 3 GB of disk space for the model and llama.cpp

Installation

1. In Unity, open Window > Package Manager, click +, choose Add package from git URL and enter:

   https://github.com/mcbu-xrlab/Maverick-Unity.git#v1.0.0
   

   This package is also mirrored here on Hugging Face: https://huggingface.co/ErenAta00/Maverick-Unity.git. The GitHub repository has the full integration guides.

2. Choose Tools > Scene Agent > Open Model Folder. This creates Assets/StreamingAssets/SceneAgent/.

3. Copy two things into that folder:

   • llama.cpp for Windows from the llama.cpp releases: the CUDA 12 build for NVIDIA GPUs, the Vulkan build for other GPUs, or the CPU build. Copy every file in the archive, not only llama-server.exe.
   • The model Maverick-4B-Unity-XR-Agent-Q4_K_M.gguf from the model page.

Unity copies StreamingAssets into your build, so players do not need to install anything.

Quick start

1. Choose Tools > Scene Agent > Set Up Scene. This adds a Scene Agent object with all components connected, including a chat box for testing.
2. Give your objects plain names, such as Red Mug, Table or Floor Lamp.
3. Select the objects the agent may use and choose Tools > Scene Agent > Make Selected Interactive. Each one gets a SceneItem with an id (mug_1), a label (mug), a colour (red), a room and the object it stands on.
4. Choose Tools > Scene Agent > Validate Scene and fix anything it reports.
5. Enter Play mode, type put the red mug on the table and press Enter. If the agent asks which mug you mean, click one. F1 hides the chat box.

Using it from code

using System.Collections.Generic;
using SceneAgent;
using UnityEngine;

public class VoiceCommands : MonoBehaviour
{
    public SceneAgentSession session;   // on the Scene Agent object

    void Start()
    {
        session.Line += text => Debug.Log(text);
        session.Finished += result =>
        {
            if (result.clarificationQuestion != null) ShowChoices(session.Candidates);
        };
        session.world.ToolExecuted += (tool, args, result) =>
        {
            if (tool == "teleport") MoveRigTo(session.world.playerRoom);
        };
    }

    // Typed text or the output of a speech recogniser.
    public void OnCommand(string text) => StartCoroutine(session.Command(text));

    // The user's answer to "Which one do you mean?".
    public void OnPick(string itemId) => StartCoroutine(session.Answer(itemId));

    void ShowChoices(List<string> itemIds) { /* your UI */ }
    void MoveRigTo(string room) { /* your camera rig */ }
}

• Rooms. Give each SceneItem a room. The model sees only the player's room. It searches other rooms with find_objects and moves the player with teleport, which sets SceneWorld.playerRoom; move your camera rig in ToolExecuted.
• "This" and "that". SceneAgentGaze points them at the object under the mouse, or at the centre of the view in XR. The objects need colliders.
• Follow-up answers. When the agent asks which object you mean, the user can pick one (session.Answer) or reply in words, such as "the red one".
• Speech. Pass the recognised text to session.Command. For a fully offline app, use a local recogniser such as whisper.cpp.

Tools

Unity refuses a call that does not fit the object, such as picking up a table, and tells the model why. What each object can do comes from its label and a catalogue of about 200 everyday objects. For a label outside the catalogue, turn on SceneItem.overrideAffordances and set the flags yourself; Validate Scene lists these labels.

Components

fallbackDecoding checks every answer before it reaches your scene: a call to a tool or value that does not exist is turned into a refusal. guardSubstitutes stops the model from acting on a similar object when the one the user named is in another room. Leave both on.

Performance

• Cap the frame rate. An uncapped renderer on the same GPU can make the model several times slower. LocalModelServer caps rendering at 30 frames per second while the model runs on the GPU. In XR, set capFrameRate to 0 and let the headset set the frame rate.
• Keep rooms small. Under about 60 interactive objects per room, multi-step commands fit the default 4,096-token context. For larger rooms, raise LocalModelServer.contextSize, which needs more GPU memory.

Limitations

• English only, and only the 11 tools. The agent cannot create or delete objects or change materials, lighting or physics.
• It finds and fetches objects in other rooms ("bring me the book"), but when the lamp is in another room, "turn on the lamp" gets "there is no lamp here". The user should name the room or go there first.
• Placed objects go to the centre of the target, so two objects placed on the same shelf overlap. Arrange them in ToolExecuted if this matters in your scene.
• The model reads the scene as data, not through the camera. Keep object names and colours accurate.
• Not intended for robots or safety-critical control.

Troubleshooting

Licence

The package is released under the MIT License. The model is released under Apache-2.0, like its base model, Qwen3-4B. llama.cpp is released under the MIT License.