LFM2–MaleCNS–Titans VL 3B V2 — SFT

Repository: win10/Lfm2-MaleCNS-Titans-VL-3B-V2-SFT. Checkpoint ID: 95c663dda67d4dbe8ec77a4681db3fc2. This repository contains a complete merged model, including the native vision/language backbone and custom dual-memory runtime.

The two training stages are separately versioned: visual/agent SFT and subsequent HashHop memory learning. A linked stage may still be in progress; only a populated checkpoint repository represents a completed upload.

Completed training stage

All 464/464 optimizer updates completed (one epoch, 3,712 examples: 2,718 VisualWebInstruct and all 994 image-only Omnimodal examples). Native LlamaFactory owns preprocessing, batching, Trainer and optimization. Outer LoRA rank 8 / alpha 16 covers the language, vision, projector, memory, VAE and controller modules; the adapters are merged. Physical batch is 8, accumulation 1, native cutoff 4,096. No custom SFT chunk scheduler is used.

This checkpoint has not undergone the subsequent HashHop objective. The inherited parent already had memory PT and an earlier partial SFT. The final run resumed from a validated step-352 checkpoint after a worker interruption, retaining optimizer, RNG and data position.

Merge verification: 424 effective tensors audited; probe logit KL 0.00071211, maximum absolute difference 0.14062500. BF16 merge is not claimed to preserve every prediction exactly.

Measured memory and generation behavior

The primary result uses the model’s own memory routing. Oracle-unit and source-visible controls are diagnostics. Forty-two held-out hash questions include unseen graphs, 10-hop extrapolation, and cross-unit chains. Complete generated outputs are retained, including errors and repetitions. These development tests informed implementation repairs; they are not an untouched final benchmark.

The suite also produced 32 multi-turn coherence outputs; 52 outputs across the full suite reached their configured limit. Exact hash scores do not certify general coherent generation. Repetition and invented details were observed in SFT diagnostics; reliable multi-hop recall and universal factual consistency are not established.

Read-interface repair and evidence scope

The release indexes the complete raw question separately from chat role/control tokens. At read time, the native query BOS precedes the recalled feature sequence; no stored feature is removed. These implementation repairs did not alter SFT weights or the archived baseline units. An earlier, unrepaired SFT test scored 0/42 with native routing and 0/42 with oracle units; its source-visible control scored 4/42. The reader version associated with the table is identified above; the old and repaired protocols must be interpreted separately.

The initial SFT LlamaFactory template omitted the native leading BOS. The subsequent HashHop template includes it and its token IDs were checked against the native HF chat template. The completed SFT history is preserved rather than described as having used the corrected serialization.

Post-SFT J-lens calibration and functional interventions were also run. Their limited controls provide evidence of controllable internal representations, not proof of every global-workspace property. The English architecture, experiment code and checkpoint-separated evidence are maintained at LFM2-JSpace-Memory.

Loading and ordinary generation

Use the recorded PyTorch/Transformers environment from the linked research repository. The model uses custom architecture code and requires trust_remote_code=True. FlashAttention-2 is used for training; install a compatible native FA2 or the pinned community kernel used in the experiments.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "win10/Lfm2-MaleCNS-Titans-VL-3B-V2-SFT"
model = AutoModelForImageTextToText.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda",
    attn_implementation="kernels-community/flash-attn2@f50dc99ed079b35990bc895d43fd353ea0cb376d",
).eval().requires_grad_(False)
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)

Opening an archive is explicit; ordinary model.generate() does not silently choose a disk archive.

archive = model.open_physical_archive("./my_memory")
source = processor(text="A newly observed fact belongs here.", return_tensors="pt").to("cuda")
archive.session.observe(**source, use_cache=False, logits_to_keep=1)
# Optional physical FFN learning: labels must come from the observed source.
archive.session.learn(**source, labels=source["input_ids"].clone(), learning_rate=0.001)
saved = archive.append()  # Native concept address + complete immutable physical unit.
query = processor.tokenizer.apply_chat_template(
    [{"role": "user", "content": "What was the newly observed fact?"}],
    tokenize=True, add_generation_prompt=True, return_tensors="pt", return_dict=True,
).to("cuda")
address = processor(text="What was the newly observed fact?", return_tensors="pt").to("cuda")
result = archive.generate(address, generation_inputs=dict(query), top_k=1,
                          max_new_tokens=64, do_sample=False)
answer_ids = result["tokens"][0, query["input_ids"].shape[1]:]
print(processor.tokenizer.decode(answer_ids, skip_special_tokens=True))
archive.close()

The 64-token output limit above is an example argument, not a built-in maximum. Ordered recall checks the complete selected memory/query/output budget against the native context length. Cold memory units are checkpoint- and codec-bound; old addresses are not silently relabeled after fine-tuning.

Architecture: fixed backbone, physical memory units, and native concept addressing

1. Native LFM2.5-VL inheritance

Lfm2TitansForConditionalGeneration inherits Hugging Face's Lfm2VlForConditionalGeneration. The native vision tower, multimodal projector, tokenizer, chat template, hybrid language stack, and output head remain in the model. The inherited language stack has 30 layers, combining 22 short-convolution layers and 8 attention layers, with hidden width 2,048. Language memory ports attach at zero-based layers 4, 14, and 26. The visual port reads native vision patch/tile features; image observations are not replaced by generated captions.

Each port has its own feature projections and gates. There is no shared, narrow cross-modal input/output latent. The port interface is still parameter-intensive; adding dense interface parameters is not presented as an increase in persistent memory capacity.

2. Sparse MaleCNS–Titans substrate

The inherited selected topology has 90,839 nodes, 1,983,608 directed edges, four channels, and two graph microsteps. Edge indices are stored in memory_graph.safetensors, rather than millions of JSON integers. Destination/source order is part of the checkpoint contract. Four channels provide 7,934,432 fast scalars; graph fast weights plus momentum require 63,475,456 bytes in FP32, before other unit components.

For a port feature x, independent projections drive its assigned input nodes. A dendritic graph operator propagates for two steps; the port reads its assigned read nodes and projects back into the native feature space. Compiled finite-step execution plans and CSR sparse multiplication evaluate the reachable computation. Custom sparse autograd preserves the outer derivatives needed to train an online gradient writer. Small compute tiles bound temporary tensor sizes; these are not SFT sample windows.

Within one observation block, every read uses the same snapshot W_t. After the complete causal forward has produced its observations, a single commit creates W_(t+1). Thus a later write cannot alter an earlier read in that block.

The writer uses next-feature associations for language and observed-feature reconstruction for vision. It differentiates a mean squared reconstruction objective with respect to physical synaptic weights. A separate write-key projection and learned value projection are present. In the selected configuration:

g = sum of port gradients masked by edge/channel ownership
g_normalized = g / (RMS over writable scalars + epsilon)
momentum_next = beta * momentum + g_normalized
W_next = W - learning_rate * momentum_next

Non-writable entries preserve both their previous weights and momentum. Automatic decay is disabled. RMS normalization remains enabled, and the optional surprise-magnitude gate is not active in this inherited checkpoint. Accordingly, it would be inaccurate to claim that raw gradient magnitude presently controls write strength.

3. Structural write ownership

For port p, an edge e=(u,v) is structurally eligible when:

distance(input_nodes[p], u) + 1 + distance(v, read_nodes[p]) <= microsteps

Eligible edge/channel scalars receive an owner, rotated across channels with seed 17. A shared-association budget of 0.25 is applied only where multiple ports can reach an edge. The resulting masks restrict plasticity, not reading. This separates independently writable capacity from intentionally shared associations.

Structural eligibility is an upper bound on influence. It is not a measured number of facts the model can remember. The retained visual-protection mask covers 6,210 input-subgraph edges; ownership improves write separation, but that mask alone is not proof that every visual association is protected. This model does not establish that the biological topology outperforms a degree-rewired random graph.

There is no persistent neuron-activation vector in this checkpoint. Persistent graph memory is synaptic weight plus momentum; graph activations are recomputed for each feature. The design must not be described as a complete recurrent simulation of the fly brain.

4. Direct FFN physical weights

Each memory unit also contains rank-16 physical factors for w1, w2, and w3 at the three memory depths. These change the native FFN's computation. They are separate from the rank-8 outer SFT LoRA used to train this release.

The caller owns these factors and their first/second optimizer moments. An observation-derived supervised loss can update them while the backbone remains frozen. Immutable units include their physical FFN state, graph state, VAE codes, ordered observations, and addressing metadata. Loading a memory is therefore loading numerical state used by the forward pass, rather than attaching a textual description to a prompt.

5. Dual memory: VAE observations and Titans weights

The VAE participates in ordinary memory storage, not just dreaming. Independent per-port encoders produce a posterior mean and log-variance, with per-feature normalization statistics. Their hidden width is 64 and latent width is 16. The new native_input head is initialized as an exact copy of the prior first-language-port head, then trained for the actual input-embedding distribution.

Ordered native input embeddings include the native projected visual features at image positions. Their VAE means are decoded in order. Unit-local residual factors and sparse numerical correction values preserve information the shared decoder cannot reconstruct. Each decoded segment is checked against the original feature checksum. Encoding tiles bound SVD scratch space; all observed feature rows are stored.

This is not a fixed-size lossless compressor. Incompressible observations can require more bytes than raw BF16 features; the previous short-source pilot measured roughly 2.1 times raw feature size for its ordered representation. A checksum proves feature reconstruction, not semantic recall or multi-hop reasoning.

At recall, the selected ordered features are decoded and passed through all native language layers, while the selected graph and FFN physical weights contribute to the forward computation. Native attention/convolution caches are rebuilt and remain ephemeral. Stored token IDs, source strings, pixel arrays, or KV caches are not the serialized representation of the ordered memory.

The native query's leading BOS is placed before the recalled feature sequence. Every recalled feature and query token remains present. This avoids a second conversation-start marker after the memory prefix. The same framing helper is used by the production reader and the memory-dependent training objective.

The current ordered reader accepts a fresh, unpadded, batch-one text query; stored observations may include images. Selected features plus query and generation budget must fit the backbone's native 32,768-position context. Exceeding this raises an explicit error. Unbounded disk growth does not imply unlimited simultaneous attention to every archived observation.

6. J-lens-derived addresses

For each calibrated layer, a complete 2,048 × 2,048 averaged Jacobian maps that layer's native residual features toward the final language layer. The estimator sums over valid causal target positions, averages over valid source positions, then averages over calibration prompts. It is not merely a same-position diagonal derivative.

The vocabulary output vectors, final normalization weights, and Jacobian define an overcomplete dictionary of directions. Positive matching pursuit followed by projected nonnegative least-squares refitting selects 16 coordinates per observed position. The stored representation includes vocabulary-coordinate indices, nonnegative coefficients, and residual energy. Vocabulary IDs identify numerical dictionary directions; decoding them to words is only a display operation. No hand-written concept list or generated JSON controls storage.

All observed positions at each memory language depth are retained. Query and memory codes are compared using sparse late interaction on CPU or GPU. The present index performs an exact scan; there is no deployed million-unit ANN throughput claim or alternative pooled-key reader. Physical weight tensors load only after the query chooses candidate units.

For text queries, the address encoder receives the complete question text, separately from the native chat serialization used for generation. Role and conversation control tokens do not contribute to the address. This separation does not require source IDs, manually supplied concept labels, or omission of any question text.

Addresses include a checkpoint identity, lens hash, recorded time, optional event time, logical time, and causal parent hashes. A content-derived SHA-256 reference supports exact programmatic lookup. Agent ownership, permissions, and application goals remain caller concerns; the memory layer does not impose a hard-coded multi-agent policy.

7. Disk units, hot mounting, and composition

The archive is append-only and has no software unit-count ceiling. Disk space grows with retained information. An active reader pins a complete immutable unit; a background loader stages another unit before an atomic swap. Unsaved changes cannot be silently discarded by mounting a different unit. Readers already holding a snapshot finish with that snapshot.

Multiple selected units keep separate graph/FFN evaluations. Their contributions are combined at the read interface; weights are not blindly averaged. Active composition has an explicit resident-unit bound. The default archive generation call selects one unit, and callers may request more within the configured bound. Selection quality and interference still require measurement.

The published runtime accepts the unified format-6 physical unit with ordered observations and version-2 native concept addresses. Old label-driven, pooled, graph-only, and associative-softmax recall branches are absent. Old artifact conversion, if needed, is an offline operation, not an inference fallback.

8. Dream consolidation

A shared weight VAE encodes normalized chunks of 1,024 values through hidden width 128 and latent width 32. It reconstructs graph and FFN deltas. Per-port feature VAEs provide source-conditioned replay. A small MaleCNS-derived sleep controller scores reconstruction-based scheduling signals.

Dreaming constructs a candidate state without mutating the current unit. Ordered decoded observations are replayed through the native model; accepted visual latent replay contributes native visual features. Actual graph commits update the candidate. A caller-provided functional retention validator must accept it before atomic publication. Latent-cycle similarity alone is not treated as a guarantee that an imagined observation is historically true. Existing physical units remain retained.

The sleep-controller topology is biological inspiration. This implementation and its current experiments do not demonstrate biological dreaming or a measured capacity gain from consolidation.

9. Joint SFT and release compatibility

Native LlamaFactory owns preprocessing, batching, Trainer, backward, and optimizer updates. The model returns native supervised cross-entropy plus memory auxiliary loss. Real Titans writes/readbacks train the memory, while VAE reconstruction/KL and scheduling objectives train the Dream components. FFN write gradients are captured from the native supervised backward without modifying those gradients; a detached batch of actual updates trains the weight VAE on the following batch.

The backbone, vision tower, projector, graph interfaces, physical FFN priors, VAE heads, and controller receive outer LoRA updates. The final checkpoint merges those adapters. J-lens calibration is then repeated because the native weights changed. Checkpoint-bound old units and addresses are not silently relabeled as compatible with new weights.

See TRAINING.md for the exact data, auxiliary sampling, timing, and prior-update provenance; see EVIDENCE.md for measured behavior.

Joint visual and memory SFT

Data and native workflow

This run uses exactly two local dataset views:

The 685 video trajectories are not part of this run. “Complete” means all 994 rows of the selected image-only view, not all records in the upstream multimodal dataset. This local view contains 1,044 image references. The combined normalized data contain 2,505 image references before native truncation. Data are not deduplicated.

LlamaFactory's native ShareGPT loader, SupervisedDatasetProcessor, multimodal collator, Trainer, and backward/optimizer loop process the data. Role and tool-call normalization converts the selected local formats to the native LFM chat/tool template while retaining thought and tool text. There is no per-dataset training scheduler, trajectory-specific loop, or custom SFT chunking. Long samples are truncated by the native 4,096-token cutoff, as selected for this run; their truncated tails are not represented as having been trained.

The LFM vision plugin uses the original HF processor's image placeholder expansion, including image tiles and start/end markers. Eleven samples whose native cutoff intersected an image span required removal of the incomplete span and matching trailing image metadata. These remain single examples, and their retained assistant labels are unchanged. No fake placeholder image is added to text-only rows.

The resulting native HF DatasetDict contains 3,712 rows, 5,658,894 input tokens, and 1,652,087 supervised assistant targets. Its columns are the ordinary native token/label/image fields; it contains no record-window/session scheduling fields. Packing is not enabled. Native length grouping reduces padding.

Optimization settings

LoRA tensor ranks are capped by the dimensions of a target tensor: four graph channels permit rank four, while scalars/vectors use the appropriate rank-one parametrization. “Rank eight” does not imply an impossible rank-eight scalar update. There are 26,855,601 trainable outer parameters in this run. All requested module groups produced finite, nonzero gradients in actual training. Disabled forgetting scalars and some write-scale factors do not receive a gradient in this objective; the report does not claim every scalar updates.

The direct physical FFN memory rank is 16 and is independent of the outer training LoRA rank. The latter is merged into the inference checkpoint after SFT.

Loss and acceleration

The primary loss is native LFM supervised cross-entropy over the native assistant label mask. One native VL forward executes per batch. The model adds an auxiliary memory loss with coefficient 0.05:

1. Real differentiable Titans writes followed by associative reconstruction.
2. Per-port feature VAE reconstruction and KL (beta 0.001).
3. Graph and FFN weight-VAE reconstruction/KL.
4. A reconstruction-derived scheduling loss for the sleep controller.

Thirty-two feature pairs per port and example are sampled for the auxiliary objective. This sampling applies only to that auxiliary training objective; it is not a storage limit or an inference context cap. All native targets retained by the chosen cutoff remain in the primary CE.

FFN updates reuse gradients from the native supervised backward. Hooks return the original gradients unchanged and separately rescale captured values to per-example target means. Actual physical optimizer updates provide a detached replay batch for the following batch's weight-VAE loss. The replay buffer is checkpointed and restored. This avoids an additional full language-model source backward.

Expanded custom LoRA weights are cached within a forward and restored consistently during non-reentrant checkpoint recomputation. Shared fresh graph priors can be read together across batch rows. A graph compute tile of 256 bounds internal sparse work; it does not split a training example.

Actual 8 × 4,096-token batches with the earlier extra-source-backward implementation took 53–57 seconds of compute and peaked at 87.24 GiB. Subsequent batches using native-backward capture took about 32–36 seconds and peaked around 63–64 GiB. These are different batches of the same shape, not a controlled identical-input speed benchmark. Liger/fused CE is not enabled. A compute/rest ratio of 1:1 is retained for the user's low-pressure WSL constraint, so wall time per update also includes rest and checkpoint I/O.

Provenance and continuation

The starting checkpoint already contained memory PT (536 updates within the prior two-hour budget) and an earlier partial SFT checkpoint at native step eight. It is not presented as a fresh untouched backbone. A dedicated native_input VAE head was added by exact copying of the old first-language-port head; all existing parameters were checked unchanged before the new SFT.

The first two genuine optimizer updates of this run used the extra source-gradient auxiliary implementation. Updates three onward reuse the native backward and next-batch FFN replay. Valid first updates were retained; the run resumed from step four with optimizer, RNG, and data position state. This is an auxiliary-training schedule change, not a claim of identical loss to the earlier implementation. Main native CE remained unchanged.

All 464 updates subsequently completed with exit code zero. After a later worker interruption, training resumed from a validated step-352 checkpoint with optimizer, scheduler, RNG and data position retained. The merged checkpoint is 95c663dda67d4dbe8ec77a4681db3fc2: 424 effective adapter tensors were audited, with no unmerged parameters. The merge probe measured logit KL 0.0007121 and maximum absolute logit difference 0.140625; identical predictions on every input are not claimed.

Fresh J-lens calibration and cold-process checks were performed. A changed backbone invalidates the previous calibration; reusing its UUID or merely copying its old lens is not accepted. Training logs and optimizer artifacts remain outside the model directory.

The original LlamaFactory lfm2_vl template lacked the leading BOS emitted by the HF native chat template. This was corrected for the subsequent HashHop continuation, not retroactively for these 464 completed updates. The published ordered reader also places the query BOS before recalled features, and indexes complete query text separately from chat control tokens. Neither repair changes the completed SFT weights. Matched before/after evaluation artifacts identify the reader version.

The subsequent memory-dependent HashHop stage has a separate repository and checkpoint identity. Its extra training must not be attributed to this visual/agent SFT stage.

Attribution and license

The original LFM Open License v1.0 is included verbatim in LICENSE. The model is derived from LiquidAI/LFM2.5-VL-3B. The MaleCNS topology derives from MaleCNS v1.0 under its dataset attribution terms. J-lens methods follow Anthropic research; this is an independent implementation and evaluation.