Hibs Jeep
Hibs Jeep is a compact (~300M parameter) encoder-decoder model for code generation and tool-call traces, built on a JEPA-style (Joint-Embedding Predictive Architecture) text encoder with a causal decoder and a normalized multi-step latent planner.
Architecture
prompt tokens --> [ JEPA Text Encoder (12L, bidirectional, 768) ] | +--> [ Latent Planner: K=4 normalized steps ] | - cross-attends to prompt representation | - residual MLP update per step | - L2-normalized every step (anti-drift) v cross-attn memory = [4 plan tokens ; up to 512 prompt tokens] | v [ Causal Decoder (12L, 768, causal + cross-attn) ] | v token logits (GPT-2 BPE, 50257)
| Component | Config |
|---|---|
| Encoder | 12 layers, 768 d, 12 heads, bidirectional, sincos pos-emb (512) |
| Prompt window | 512 tokens (fits a full JSON tool schema) |
| Target window | 256 tokens |
| Latent planner | 4 steps, 768 d, 8 heads, L2-normalized states |
| Decoder | 12 layers, 768 d, 12 heads, causal + cross-attn, dropout 0.1 |
| Vocab | 50,257 (GPT-2 BPE), tied output embedding |
| Trainable during SFT | 156.97M (decoder + planner + encoder LoRA r=32) |
The planner is the normalized long-horizon reasoning component: instead of letting the decoder attend only to token-level encoder features, it iteratively refines a global plan and injects the K normalized plan states as extra cross-attention memory.
Training stages
| Stage | Data | Steps |
|---|---|---|
| SFT (code-dominant) | 23,893 instruction to code pairs | 29,865 |
| 512-window SFT | 32,893 pairs (code + tool schemas) | 65,784 |
| Planner SFT | same, with latent planner | 65,784 |
| Long SFT (+6 epochs) | same | 98,676 |
| RL (RLOO/GRPO) | 11 verifiable tasks, unit-test reward | 800 |
Data: iamtarun/python_code_instructions_18k_alpaca, NousResearch/hermes-function-calling-v1, glaiveai/glaive-function-calling-v2, teknium/OpenHermes-2.5 (code-filtered subset).
RL reward is fully verifiable (no reward model): 1.0 if generated code passes the task unit tests, +0.1 if it parses. Group-relative advantage (mean baseline), no critic.
Evaluation (measured by executing generated code)
| Benchmark | Score |
|---|---|
| HumanEval pass@1 (164) | 0 / 164 |
| MBPP pass@1 (100) | 0 / 100 |
| In-domain 20 simple tasks (strict) | 8 / 20 (40%) |
| In-domain 20 tasks (function rename allowed) | 11 / 20 (55%) |
| Tool-call JSON format rate | 0.50 |
| Tool selection accuracy (given schema) | 0 / 4 |
Generalization curve (train11 = tasks used by RL, holdout9 = never trained):
| checkpoint | train11 | holdout9 | HumanEval |
|---|---|---|---|
| planner SFT | 0.182 | 0.000 | 0/60 |
| long SFT (10 ep) | 0.500 | 0.000 | 0/60 |
| + RL 800 steps | 0.909 | 0.100 | 0/60 |
Limitations (please read)
- Does NOT solve standard algorithm benchmarks. HumanEval and MBPP are 0.
- Writes correct code only for short in-distribution function tasks (reverse_string, is_prime, sum_list, ...).
- Highly sensitive to prompt phrasing; symbolic or long code-prefix prompts fall back to memorized templates.
- Emits function-call JSON but does not reliably select the right function from a schema.
- RL raised in-distribution accuracy to ~91% but produced only a weak, unstable generalization signal (1/9 held-out tasks, sampling-sensitive).
- Research artifact, not a production model.
Usage
pip install -r requirements.txt
python example.py
Minimal loading code:
import torch
from transformers import AutoTokenizer
from textjepa.models.seq2seq import TextSeq2Seq
ckpt = torch.load("hibs-jeep.pth", map_location="cpu")
cfg = ckpt["config"]["model"]
model = TextSeq2Seq(**cfg)
model.encoder.load_state_dict(ckpt["encoder"], strict=False)
model.decoder.load_state_dict(ckpt["decoder"], strict=False)
if model.planner is not None and "planner" in ckpt:
model.planner.load_state_dict(ckpt["planner"], strict=False)
model.eval()
tok = AutoTokenizer.from_pretrained("gpt2")
tok.pad_token = tok.eos_token
prompt = ("<|system|>\nYou are a helpful coding agent.\n"
"<|user|>\nWrite a python function that reverses a string.\n<|assistant|>\n")
ids = tok.encode(prompt)
L = cfg["max_src_len"]
ids = ids[-L:] if len(ids) > L else ids + [tok.eos_token_id] * (L - len(ids))
src = torch.tensor([ids], dtype=torch.long)
out = model.generate(src, max_len=96, temperature=0.8, top_k=50, eos_id=tok.eos_token_id)
print(tok.decode(out[0].tolist(), skip_special_tokens=False))
Files
| File | Description |
|---|---|
| hibs-jeep.pth | weights (encoder + decoder + planner) + full config |
| config.json / config.yaml | model configuration |
| textjepa/ | model code |
| example.py | minimal end-to-end example |
| requirements.txt | dependencies |
Provenance
Research artifact exploring whether a JEPA-style representational encoder can be turned into an autoregressive agent model. All numbers above were produced by executing generated code against real unit tests.