archai-jev-games-1.5b-v2

A small specialist that chooses the next move of two toy games, Snake and a pellet maze, from sensors that the game computes. It is a fine-tune of nickprock/archai-jev-qwen-1.5b (Qwen2.5-1.5B-Instruct) trained on synthetic data, and it is meant to be used through archai-jev, which returns a typed value with calibrated probabilities instead of free text.

What it is, and what it is not

• It does not read a board. For every legal move the game works out the numbers the model needs (what is in the next cell, how far the nearest ghost and pellet are, how many cells are free, ...) and shows them as a JSON state. The model answers one Choice question over the legal moves (up, down, left, right).
• It imitates a fixed policy (the "oracle") that is a function of exactly those numbers. It plays like that policy, not better. The policy is not a champion: in the maze the oracle itself loses more than half of its games.
• It is a demo of fast typed decisions. It says nothing about spatial reasoning.
• It is not a general-purpose model. It only makes sense for the format of games.py (game-sensors-v1). Do not use it for other questions.

Use it

from archai_jev import Choice, Jev

jev = Jev.from_pretrained("nickprock/archai-jev-games-1.5b-v2")   # archai-jev 0.2.0 or later

state = {"game": "maze", "power_left": 0, "pellets_left": 63, "moves": {
    "up":   {"next_cell": "pellet", "ghost_distance": 7, "scared_distance": None,
             "pellet_distance": 0, "exits": 3},
    "left": {"next_cell": "empty",  "ghost_distance": 3, "scared_distance": None,
             "pellet_distance": 2, "exits": 2}}}
question = Choice("Pellet maze: which move should the player make? Avoid the ghosts "
                  "(ghost_distance), eat pellets, and chase scared ghosts while power_left lasts.",
                  ["up", "left"])
answer = jev.ask(state, {"move": question}).choices["move"]
answer.value, answer.probabilities     # the most probable move and the probability of each one

The state above only shows the format: the numbers are made up, not taken from a real game.

A complete game with a window is in the repository of the library: python demos/010_pellet_maze.py (add --device cuda:0 on a GPU build). The game code of the demo (demos/_games.py) is the games.py of this repository, the file the model was trained with, but for one word of a docstring.

Results

Two sets of numbers, because the model was measured twice, on two different artifacts. The file you download and run is the GGUF, so the second set is the one that describes it.

1. On Kaggle, right after training (4-bit base + LoRA, PyTorch)

Training: one epoch, 5,333 Snake decisions and 16,000 maze decisions, best checkpoint chosen on a validation split (validation accuracy 99.4%, loss 0.032). Test states equal to a training state were removed (177 Snake and 48 maze). Prompts: mean 182 tokens, longest 246.

Whole games with fixed seeds never seen in training, 10 games per row, at most 200 moves (the numbers are noisy because 10 games is few):

The untrained base model, on a test set of the same kind from an earlier run of the same script (not the set above), picked a best move in 0.61 (Snake) and 0.57 (maze) of the cases.

2. On the GGUF (Q8_0) through archai-jev, same states and seeds

The merged weights are converted to a Q8_0 GGUF, and the adapter learned on a 4-bit base was merged into a 16-bit one, so the numbers move a little. Measured with archai-jev, on the same test states and seeds as above:

• Against the reference: the probabilities of the GGUF differ from those of the 32-bit weights by at most 0.044 on the 24 reference decisions, and the most probable move never flips.
• Calibration: the raw probabilities are already well calibrated on the states with a single best move (ECE 0.5% Snake, 1.2% maze). A temperature fitted on the validation split (1.16) made it slightly worse, so the library uses 1.0.
• Whole maze games, fixed seeds, 200 moves at most:

The model and the oracle are not distinguishable at these sample sizes. The Snake was not played as whole games on the GGUF.

The maze is the weaker game: 3 states in 100 get a move the policy would not choose, and in a game the mistakes add up. A longer training would probably help (the run on Kaggle was cut to one epoch to fit the time limit of the platform).

Limits

• One policy for two games, imitated well but not improved on. A different policy needs new labels and a new training.
• Calibrated only for these two games with the parameters of game-sensors-v1: 2 ghosts and a chase of 0.5 were the setting of every number above. The model saw 1 to 3 ghosts and chases of 0.35, 0.5 and 0.65 in training, but those were not measured.
• Questions it is not meant for are not detected as such: the library refuses more than 4 options and Score questions, but cannot tell that a question is outside what the model learned.
• The library requires a self-check vector for yes/no questions for this family. The one of this model was recorded from the inference engine itself, so it only guards against regressions of the engine; it says nothing about yes/no quality, and the model is not for yes/no questions.
• English only; prompts up to 512 tokens (the longest of the games has 247).

Training data

Synthetic. Every label is computed by a program (games.py, policies snake-v1 and maze-v2) as a function of the sensors shown to the model, and every tie is kept: the target is uniform over all the best moves. States come from games played by the oracle with 20% random moves, so they include the situations a mistaken player reaches. The order of the moves is shuffled in every example, to remove any bias for a position. No third-party dataset was used.

Files

The tokenizer that archai-jev uses is the one of the base model, Qwen/Qwen2.5-1.5B-Instruct (the library fixes it by hash).

License

Apache-2.0 (the base model is Apache-2.0 and the training data is ours and synthetic).