HobbyLM-1B Computer Use

HobbyLM-1B Computer Use is a LoRA adapter for HobbyLM-1B Instruct. It turns a plain-English request into actions on a Mac, one step at a time, and runs entirely on the device. This repository contains the adapter, the app that runs it (Hobby), the evaluation records and the training code.

Contents

• Skills
• How it works
• Fine-tuning
• Evaluation
• On-device speed
• Quickstart
• Safety design
• Scope
• Files
• Licence and credits

Skills

Every example below is a real request from the sealed round-4 test, or from the recorded demo, with the action the model wrote.

A recorded session looks like this:

you › split 600 among 4 people
  0.3s  Hobby → OPEN Calculator
  ↳ Calculator opened.
  0.3s  Hobby → TYPE 600/4=
  ↳ display changed from 0 to 150.
  0.2s  Hobby → DONE
  ✓ done

How it works

1. The request and the live state go to the model. Hobby reads the Mac's current state (Calculator display, front app, brightness, volume, appearance) and sends it with the request.
2. The model writes one action from a fixed vocabulary of ten.
3. The harness validates the action against an allow-list and executes it through macOS. Calculator input goes through mcp-server-macos-use; everything else uses direct macOS calls.
4. The result goes back to the model as Last action: … Result: …, and the loop continues until the model writes DONE (at most 6 steps).

Prompt format

The model uses HobbyLM's native SYSTEM: / USER: / ASSISTANT: format with greedy decoding. The system prompt lists the allowed actions (see hobby/hobbymac_data.py, SYSTEM_V2). A user turn from the evaluation log:

Request: what's 82 plus 47

State:
Calculator: open, display shows 0
Front app: Code
Brightness: 56%  Volume: 100%
Dark mode: on

Last action: OPEN Calculator. Result: Calculator opened.

Next action:

The model answers TYPE 82+47=, and Calculator shows 129.

Action vocabulary

Fine-tuning

The base model is unchanged. All the new behaviour comes from a LoRA adapter trained on synthetic one-step decisions: given a request and a Mac state, write the single correct next action.

Four rounds

Each round started from the base model, added coverage informed by the previous round's results, and was evaluated on a new sealed test written for that round.

Each row uses a different test, so the scores in this table aren't directly comparable. Like for like, on the round-4 test:

Keeping the test honest

• Independent tests. A separate model (Grok) wrote each round's test requests.
• Sealed before training. Each test file was fingerprinted with sha256 before any training example was finalised. The hashes are recorded in validation/.
• Lookalikes removed. Any training example with word-set Jaccard similarity ≥ 0.5 to a test request was dropped (788, 920, 532 and 791 examples in rounds 1 to 4).
• Live, not simulated. Every trial ran on a real MacBook Air. Independent checkers read macOS state: the Calculator display through the Accessibility API, display brightness, volume and mute, the appearance setting, the front app, Finder's window path, timers, screenshot files and a calendar read-back. The harness restored the Mac after each trial.

Evaluation

Pass bars. Neither pre-registered pass bar was met in round 4:

• The frozen bar: every skill had to clear it (Calculator ≥ 90%, refusals ≥ 80%, every other skill ≥ 85%). Only refusals did.
• The owner's 75% bar: apps, folders and refusals cleared it.

The score file records both results (all_pass: false, all_pass_owner_75: false).

Calendar checks. In the app, plain code cross-checks each EVENT against the user's sentence before asking for confirmation:

• emails are copied exactly
• an explicit am/pm time wins
• the day comes from the sentence
• only people who were named are invited

Re-scored on the same logged outputs, calendar requests complete 27 / 30, with no correct answer changed. These checks were written after the round-4 run, so 27 / 30 is a post-hoc figure. The headline result is the model alone: 21 / 30.

Per-round scores, training records and parity runs are in validation/. The retired test sets are in data/frozen/, with emails replaced by example.com.

On-device speed

How these were measured: on a MacBook Air at FP32, as model time only.

• The round-4 figures come from replaying all 262 decisions of the round-4 live run through the release runtime; all 262 came out identical. In the live run itself, with apps launching alongside, the logged mean was 0.81 s per decision.
• The PyTorch figures and the MLX-vs-PyTorch parity check (203 / 203 identical decisions) were measured with the round-1 adapter.
• The records are in validation/mlx-speed-r4.json and validation/mlx-parity-r2.json.

The speed-up comes from a KV cache and Apple's MLX runtime, with the LoRA applied in memory from compact A/B matrices.

Quickstart

Requirements: an Apple-silicon Mac, macOS 14 or later, Python 3.10 to 3.13 and the Xcode command-line tools.

pip3 install -U huggingface_hub
hf download Hobbylm/hobbylm-1B-computer-use --local-dir hobby-mac && cd hobby-mac
chmod +x scripts/setup.sh && ./scripts/setup.sh
.venv/bin/python hobby/hobby_demo.py r4

setup.sh creates a virtual environment with pinned packages and downloads HobbyLM-1B Instruct at the pinned revision. It also fetches the MLX runtime, builds mcp-server-macos-use at a pinned commit, and compiles the small Swift helpers that read brightness and the Calculator display. After setup, Hobby runs without a network connection.

Try these:

split 600 among 4 people
make my screen bright
I need to write something down
grab a screenshot
turn on dark mode
what's the weather tomorrow

Hobby bar (⇧⌘H)

Start data/HobbyBar after setup; press ⇧⌘H anywhere to open the panel. Type a request and press Enter to see each decision and result. Calendar events still need explicit confirmation: choose Add event or Cancel. Press Esc to hide the panel.

• Traffic lights: red hides, yellow collapses to the input line, green makes the transcript taller.
• Clear removes the chat history. Drag the panel by its header or an empty area; it remembers its position.
• Hotkey: HOBBY_BAR_HOTKEY=cmd-shift-h is the default. Alternatives are ctrl-option-space, option-space and cmd-shift-space. Optional cmd-h uses ⌘H and replaces the system Hide shortcut while Hobby runs.

Permissions. macOS asks you to grant these yourself; Hobby never grants anything.

• Accessibility, for the app that runs Hobby (System Settings → Privacy & Security → Accessibility). Hobby checks this at startup.
• Calendar and Contacts, for calendar events only. Turn the skill on with HOBBY_ENABLE_CALENDAR=1, and choose a calendar with HOBBY_CALENDAR="<calendar name>".

Re-run the evaluation:

.venv/bin/python hobby/hobbymac_run.py --model trained --round r4 --adapter r4 --runtime mlx
.venv/bin/python hobby/hobbymac_score.py <run folder>
.venv/bin/python hobby/test_harness.py        # 10-action parser and allow-list, no model needed
.venv/bin/python hobby/test_offline.py        # System Settings agent parser, no model needed
.venv/bin/python hobby/test_server.py         # Hobby bar worker protocol, no model needed

Safety design

• Allow-list execution. Only the ten actions above can run, and invalid actions are rejected. The harness spells x, ×, ÷ and − as Calculator keys.
• Calendar cross-checks. Calendar fields are checked against your sentence, and any change is shown before you confirm.
• Confirmation before calendar changes. Every event is shown in full (title, date, time, time zone, invitee) and needs an explicit y.
• Real invites. If the event has an email address, confirming it sends a real invitation from the calendar you chose.
• Exact invitees. Emails are copied from your sentence or resolved from your own Contacts. Hobby invites only people you named.
• Local by default. The model, the adapter and every action run on your Mac. Calendar invites are delivered by your own calendar account, if you choose one.
• Independent verification. In evaluation, results are read back from macOS rather than taken from the model's output.

Scope

• Actions: the ten actions listed above, for the apps and folders named there.
• Platform: Apple Silicon running macOS 14 or later, with Accessibility granted.
• Language: English, including Indian-English phrasing.
• Requests: one task per request; the model completes it in up to six steps.
• Calendar: times resolve in the Mac's own time zone unless the request names one.
• Measurement: each skill was measured on a set of 4 to 30 sealed requests.
• Use: not intended for security, financial or medical decisions.

Files

Licence and credits

• Licence: Apache-2.0 for this adapter and its code. The base model keeps its own licence.
• Credits: Built by Hariharan and Prabhurajhan at Fuel Labs. Architecture based on HobbyLM by Harish.
• Calculator UI actions: these use mediar-ai/mcp-server-macos-use, which setup.sh downloads; it is not bundled here.
  • Its licence: Business Source License 1.1, with the Additional Use Grant "Production use is permitted for non-commercial, educational purposes only". It converts to MIT on 2028-04-09.
  • Commercial use needs a licence from Mediar.
• Related work: UC Berkeley's TinyAgent, on small on-device function calling for the Mac.