This card is the README of our recipe, bertholomus/deepseek-v4.1-tensorfold-tp4-4xgb10 (v0.3.7). The files it names (ENGINE_COMMIT, RUNNING.md, scripts/, tools/, chat/, results/) are in that repository. This repository holds its kit/, with the five files the recipe's kit/verify_kit.sh makes (the four RoPE tables and kit/vision/gate_bias_vl.safetensors). The v0.1.1 kit, for the 2.9 bpw weights, stays at this repository's revision f8ee0e77.

DeepSeek-V4.1-Flash-1017 on four DGX Spark (GB10) nodes: a TensorFold TP4 engine

Weights: bertholomus/DeepSeek-V4.1-Flash-1017-EXL3-4.0bpw (MIT), our EXL3 quant of deepseek-ai/DeepSeek-V4.1-Flash-1017 (MIT). This repository is a recipe: the served launch, our measurements, the gates the build passed, the chat template, the step that readies the downloaded weights and, in kit/, the kernel kit the engine loads. It holds no weights and no engine source.

The engine is branch deepseek-v41-zig-tp4 of our TensorFold fork, bertholomus/TensorFold. The file ENGINE_COMMIT pins the exact commit. It adds a DeepSeek-V4.1-Flash family to TensorFold 1.0.12's native (Zig) engine and runs it tensor-parallel over four NVIDIA GB10 nodes, one rank per node, linked by RoCE. It ports our two-node family (deepseek-v4.1-tensorfold-tp2-2xgb10) and extends it to an exact two-dimensional split. We wrote it independently and re-implemented the model math from DeepSeek's MIT inference code and tech report (see License and attribution).

What it does:

• Four-node exact split: on the 2.9 bpw weights of v0.1.1 its replies equal the two-node family's token for token in our gates; on the 1017 weights, burst and staggered requests equal the solo replies 16 of 16.
• Image input in user turns and tool results (the vision tower runs on rank 0), as DeepSeek's encoding places it.
• Up to 1,048,576 tokens a request, 16 streams sharing a 20,971,520-position window, with kept-prompt reuse: a request that repeats a kept prompt's prefix, images included, resumes from it instead of prefilling again.
• Structured output: response_format (a JSON schema or a JSON object), guided_choice and guided_regex hold a reply to its grammar.
• Tool calls in DeepSeek's DSML; with tool_choice "required" or a named function, the reply is one calls block, each call an offered tool with its parameters within their schema.
• Exact DSpark speculative decoding: drafted output equals serial output token for token.
• OpenAI-compatible HTTP API (TensorFold's server): chat and text completions, streaming, reasoning output, tool calls.

v0.3.7: what changed

Measured by one night4 gate, gate-v037-20261104T0904Z (47 checks: every reply, structured output, tool call, kept prompt and needle equal between the arms — 974/974 seeded shas — except code C8 -1.1 %, which the A B C A B C rerun did not reproduce, -0.1 %, replies equal; see below), passes A B A B on the same four nodes: A = the v0.3.6 settings on pin 7a2bc4c2, B = the v0.3.7 build with the settings below, both at the 20,971,520 pool. Every cell is computed from the slot's files (results/served-v037/) by rows_v037.py; none is typed.

v0.3.7 is the served line's S5 streams work: the engine is v0.3.6's 7a2bc4c2 plus 5 commits, exported on the branch's public line as ba03026c; the work that stays private is not in it.

The gate's one failed check was code C8, -1.1 % (242.9 -> 240.3): the same-day A B C A B C rerun did not reproduce it — code C8 235.6 -> 235.3 (-0.1 %), all 152 replies equal in every arm — and the release row is level (8 streams 224.2 -> 224.2). The headline: concurrent decode at the new C24/C32 (+14.3 % / +8.7 % code, +11.7 % / +10.1 % prose), sustained 24/32 streams 256.0 -> 286.0 (+11.7 %) and 255.7 -> 290.1 (+13.5 %) with TTFT p50 8.01 s -> 0.33 s and 15.88 s -> 0.42 s; prefill 93K -0.2 %, TTFT behind -0.5 %, needle 300K +0.2 %; replies 974/974 equal as a multiset.

No outside contributor: the five commits are all ours (the export re-authors each of them bertholomus). Our own commits: rounds of up to 64 rows (round.max_rows, draft.max_absorb 48 -> 64), a decode window of 64 rows, up to 32 streams a pool (draft.max_streams 16 -> 32), a Markov step of more than NP streams running _bias in NP-row pieces, and the block draft in passes of at most 128 / block streams.

v0.3.6: what changed

Measured by one night4 gate, gate-v036-20261102T2005Z (45 checks, all PASS: every reply, structured output, tool call, kept prompt and needle equal between the arms), passes A B A B on the same four nodes: A = the v0.3.5 settings on pin ebf6ae332, B = the v0.3.6 build with the settings below, both at the 20,971,520 pool. Every cell is computed from the slot's files (results/served-v036/) by rows_v036.py; none is typed.

v0.3.6 is the served line's absorb fixes plus the four-node ring, the outside patches and the tokenizer/server work: the engine is v0.3.5's ebf6ae33 plus 16 commits, exported on the branch's public line as 953bd6fd; the work that stays private is not in it.

New from outside contributors, each kept under its author's name: pull request #17 by LeetJoe (the image-placeholder escape), upstream TensorFold's template fix by Nathan McBride (@nmcbride), and tonyd2wild's two MIT patches (TF_RDMA_GID, default 5 unchanged — the approach ecohash-co described on ashhart/TensorFold #559; and TF_SEED_RANDOM / TF_TOP_K_MAX). Our own commits: TF_DS_TWO_HOP (a four-node ring read as two pairs, off unless set), the rank-stop watch, the DSML block reads, the salted compaction's cache_salt, the reply-ids marker text, and the tokenizer's encode scratch and memo.

v0.3.5: what changed

v0.3.4 (a TF_DS_2D_YP build) did not pass its gate and was not released; v0.3.5 follows v0.3.3.

Measured in: pnc4 ran the two builds in turn, v0.3.3, v0.3.5, v0.3.3, v0.3.5, on our four nodes in one session, with fresh prompts each time (cached 0 on every row); the 12 decode replies have the same sha256 (610cba14f553). pnc4g is the gate run (43 checks, v0.3.3's build and v0.3.5's on the same port in turn). The raw outputs are in results/served-v035/. The gain is on prompts up to about 32K tokens, where the first pass also no longer runs slower than the next ones; from about 93K tokens on, prefill is level (-0.9 % and -1.0 %, inside the run-to-run spread). Decode is unchanged.

Engine change since v0.3.3 (one commit, 0a0b69a; the served build is the same change on 39155041, built as ebf6ae33):

• TF_DS_WAVES=1 on TF_DS_2D_YX: a prompt pass of more than 2,048 rows runs as two waves of 2,048 rows, layer by layer, and each wave's partial gathers run on a second stream over a second NCCL communicator while the other wave computes. The gathered values and the sum order are the same, so replies do not change.
• NCCL_MAX_NCHANNELS=4, in place of NCCL_NCHANNELS_PER_NET_PEER=8: with two communicators, eight channels a peer let them starve each other, and prefills above 8K tokens hung. Four channels a communicator fixes the hang.

The gates (pnc4g; every reply compared equals v0.3.3's build's in the same session): the API gates 6 of 6; 16 replies solo, as a burst and staggered, 16/16 each and equal to v0.3.3's; exact chat 71 of 71; vision 6/6, 9/9 and four at a time 18/18; the prompt shield 14 of 14; structured output 11 of 11 with burst and staggered 12 of 12; forced and "auto" tool calls 16 of 16 rows; reply ids 16 of 16; the sampler, the accept stats, the kept states 8 of 8, the resume probe 4 of 4, the keepalive 2 of 2 and the fill check 2 of 2. After the promotion, on the served port: the API gates 6 of 6 and the 16 replies solo, burst and staggered 16/16 each (promote.log).

v0.3.5 adds no pull request; the engine branch's ATTRIBUTION.md and this recipe's ATTRIBUTION.md are unchanged.

v0.3.3: what changed

Measured in: each row's numbers come from the gate slot its "Measured in" column names, on our four nodes; their raw outputs are in results/served-v033/. zyx is the gate run of the build v0.3.3 serves (engine 39155041, the served settings; the slot's phase c-v033). In its comparison rows the v0.3.2 column holds the build served before it's, measured in the same session (zyx's phase a). Against that build in the same session, v0.3.3's build moved its 16 speed rows by -0.4 % (set a prose) to +7.5 % (prefill 128K); the decode rows moved -0.2 / +0.3 / 0.0 %, inside the run-to-run spread — decode is unchanged.

The 93,335-token prefill row: v0.3.3's 2,949 tok/s is the median of three runs of cmp1021_bench.py prefill 3 on the 93,335-token prompt on the served build, client-only on the served port after the promotion (its time to first token 31.65 s; row34). v0.3.1's README row, 2,552 tok/s, is the same bench on the same prompt (cmpp25): ++15.6 % since v0.3.1. The same session's build served before it read 2,917 tok/s, derived from its three long-prompt time-to-first-token rows (the gate ran no 93,335-token prefill bench in the slot; +6.4 % against v0.3.3's bench number, and the 8,192 / 32,768 / 131,072-token rows above are the like-for-like session comparison). v0.3.2's own README row was 2,757 tok/s, derived the same way in its slot.

Against v0.3.1's README rows (its gate run z3334, engine 37df889): the 93,335-token prefill 2,552 → 2,949 tok/s (++15.6 %, the same bench both times); prefill 8,192 / 32,768 / 131,072 tokens 2,770 / 2,833 / 2,779 → 2,974 / 3,107 / 3,140 tok/s (+7.3 / +9.7 / +13.0 %, the same row method both slots); set b's decode 136.7 / 78.6 / 185.3 → 135.6 / 78.6 / 183.7 tok/s (-0.8 / 0.0 / -0.9 %, level — decode is unchanged since v0.3.1). The 300K needle: three runs (seeds 101, 102, 103) of a 299,865-token prompt with 3 needles each, cached 0 every run, all 3 found in every run, 98.65 s median against v0.3.2's 106.59 s on the same check (-7.4 % time; row34).

Memory: the window is v0.3.2's 20,971,520 positions and stays there; the nodes' lowest free memory in zyx's speed rows was 11 / 13 / 13 / 14 GiB (the floor is 9 GiB).

Engine changes since v0.3.2 (one commit; the fork's v0.3.3 commit is it):

• TF_DS_2D_YX=1 with NCCL_NCHANNELS_PER_NET_PEER=8 (both now on as served; TF_DS_2D_YX was off unless set): the prompt's cross-rank gathers carried the four fp32 quarters of the sublayer partials, and the posting kernels take only their pair-summed bf16 y from them. With the switch on, the pair partners swap their fp32 partials of each half of the rows, sum their own half in rank order, and the four nodes all-gather the bf16 halves: a node receives 5 n·w bytes instead of 12 n·w, and the quarters copies become one widen. The gathered values are the same bits (an unused gather slot holds -0.0, which the kernels read as before), so the GPU kernels are unchanged. The pair swap runs on eight NCCL channels per peer (the default is one or two); without the channels the gathers alone bought nothing (+0.1 % on the 93,335-token prompt, pnc), and with them that prompt read 2,921 tok/s (+6.1 %, pnc) before the gate's final rows above.
• The 300K needle run above is a measurement, not an engine change.

Off unless set (unset as served): TF_DS_SOLO_OVERHEAD=1 (pull request #33), TF_DS_NO_COPIES=1 (pull request #34), TENSORFOLD_REPLY_IDS=1 and TF_DS_PASS_ROWS / TF_DS_PASS_FROM (the prompt-pass rule at its default: 2,048-row passes below position 8,192 and 4,096 above), as in v0.3.2.

Thank you to @Blackwellboy for pull requests #33, #34 and #38 and to Niklas Lenz (niklaslenz.com) for the bring-up report behind v0.3.2's two recipe fixes, as in v0.3.2. v0.3.3 adds no pull request; the engine branch's ATTRIBUTION.md and this recipe's ATTRIBUTION.md are unchanged.

v0.3.2: what changed

Measured in: each row's numbers come from the gate slot its "Measured in" column names, on our four nodes; their raw outputs are in results/served-v032/. zsalt is the gate run of the build v0.3.2 serves (engine 1935b48a, the served settings; run 4 of the slot). In its comparison rows the v0.3.1 column holds v0.3.1's own numbers, from the gate run of the build v0.3.1 serves (z3334, engine 37df889); the two time-to-first-token rows and the 93,335-token prefill row name no v0.3.1 number (v0.3.1 carried no short-chat row), so their v0.3.1 column is the build served before it in the same session (zsalt's phase a). Against that build in the same session, v0.3.2's build moved its 16 speed rows by -1.4 % (16 streams) to +5.2 % (prefill 128K).

Memory: the smaller window frees memory on every node. At 20,971,520 positions the lane leaves about 12 GiB free on each Spark: our nodes' lowest free memory was 12 / 13 / 14 / 14 GiB in zsalt's speed rows. Nodes that need more room can use 16,777,216 positions (TF_DS_WINDOW=16777216 and the followers' --pool 16777216; config.env.example says where): in v0.3.1's sessions they left 18 / 20 / 20 / 20 GiB free at the lowest, with every reply equal (README, v0.3.1).

Engine changes since v0.3.1 (the fork's v0.3.2 merge commit lists them):

• TF_FILL_CHUNKS=1 with TF_FILL_MANY=1 (both now on as served; TF_FILL_MANY was off unless set): a request is admitted while a prompt pass fills, and a pass steps its chunks only while a queued request, another pass, a decoding stream or a cancel waits — a pass alone runs as a whole one. A short chat behind a cold 93,335-token prefill answers in 0.76 s (35.60 s for the build served before it, same session); the long prompt's first token came at 33.85 s (36.54 s).
• cache_salt (engine-side, no flag): a request may carry a salt, and its prompt states are kept and resumed only for that salt, vLLM semantics. A fresh salt caches 0 of a 6,014-token prompt, the same salt 4,096, no salt after a salted one 0 again and no salt after that 4,096 (10 of 10 checks); requests without a salt are as before.
• The prompt passes run by position: 2,048-row passes below position 8,192 and 4,096 above (TF_DS_PASS_ROWS and TF_DS_PASS_FROM set the rule; unset as served). The Engram read-ahead is sized by the next pass, so a shorter pass takes rows read ahead that start at it and cover it. The same bits at any rule: a pass ends on the 2,048-row resume grid either way.
• The pool is 25,165,824 → 20,971,520 positions, from free memory (the floor 9 GiB + margin 2 rule on the gate's MiB watch): 12 / 13 / 14 / 14 GiB lowest during the speed rows.
• TF #38 by @Blackwellboy, cherry-picked (the commit keeps its author): a refused Engram row id no longer leaks the AIO batch's buffers.

Two durable recipe fixes from an outside tester's bring-up (Niklas Lenz, his report's 5.2 and 5.3): scripts/serve.sh now runs a node that is this host locally — by name, by the name its ssh config gives it, or by its address — so no ssh key to itself is needed, and it gained preflight, which checks on rank 0's node every file rank 0 opens at start (the model folder's small files, each file the index maps a vision tower tensor to — the vision tower's shard model-00041-of-00041.safetensors —, the node's weight file, the token map, the kit and the engine), names any missing one, and starts and stops nothing; start runs it first. RUNNING.md section 2 lists the files rank 0 opens.

Off unless set (unset as served): TF_DS_SOLO_OVERHEAD=1 (pull request #33), TF_DS_NO_COPIES=1 (pull request #34) and TENSORFOLD_REPLY_IDS=1, as in v0.3.1.

Thank you to @Blackwellboy for pull requests #33, #34 and #38, and to Niklas Lenz (niklaslenz.com) for the bring-up report behind the two recipe fixes. The engine branch's ATTRIBUTION.md credits Blackwellboy, and this recipe's ATTRIBUTION.md credits both.

v0.3.1: what changed

Measured in: each row's numbers come from the sessions it names, on our four nodes; their raw outputs are in results/served-v031/<session>/. z3334 is the gate run of the build v0.3.1 serves (engine 37df889, the served settings); in its rows the v0.3.0 column holds v0.3.0's own numbers (zreqfix, at 4,194,304 positions). In the same session, against the build served before it (v0.3.0's engine 4964e38, at 25,165,824 positions), v0.3.1's build moved its 16 speed rows by -1.0 % (prefill 128K) to +1.2 % (set b code). cmpp16 and cmpp25 ran v0.3.0's build twice, one launch after the other, with the window raised in between: 4,194,304 then 16,777,216 positions (cmpp16), and 16,777,216 then 25,165,824 (cmpp25). Their rows take the v0.3.0 column from cmpp16's first launch and the v0.3.1 column from cmpp25's second; at 16,777,216 positions the same prompt read 2,542 and 2,548 tok/s.

Memory: the larger window takes memory on every node. At 25,165,824 positions the lane leaves about 10 GiB free on each Spark: our nodes' lowest free memory was 11 / 12 / 13 / 13 GiB, in cmpp25's checks and in z3334's speed rows alike. Nodes that need more room can use 16,777,216 positions (TF_DS_WINDOW=16777216 and the followers' --pool 16777216; config.env.example says where): in cmpp16 they left 18 / 20 / 20 / 20 GiB free at the lowest, and every reply equaled those at 4,194,304.

Engine changes since v0.3.0: two settings for a drafted stream, from pull requests by @Blackwellboy, both off unless set (unset as served; the fork's v0.3.1 merge commit lists them). Each was measured in z3334 against the same build with both unset. With either or both set, every reply the gates compare (the 16 replies, the image prompts, structured output, reply ids, the DSML rows) equaled v0.3.0's build's.

• TF_DS_SOLO_OVERHEAD=1 (pull request #33): a lone stream prices its head windows at the fixed draft allowance. Every speed row moved -0.8 % to +1.0 %.
• TF_DS_NO_COPIES=1 (pull request #34): a drafted stream drops its suffix-lookup copy windows, so the draft head proposes alone. Replies that copy JSON from the prompt decoded +4.9 %, copied code +2.2 % and set a's structured replies +3.2 %, while quoted text decoded -2.5 %; the other rows moved -1.3 % to +1.1 %. With both set: +5.2, +2.0, +3.6 and -3.0 %, the other rows -0.9 % to +0.9 %.

Thank you to @Blackwellboy for pull requests #33 and #34. The engine branch's ATTRIBUTION.md credits them.

v0.3.0: what changed

Measured in: each row's numbers come from the sessions it names, on our four nodes; their raw outputs are in results/served-v030/<session>/. zreqfix is the gate run of the build v0.3.0 serves (engine 4964e38, its served settings); in the rows that name only it, the v0.2.0 column holds v0.2.0's own numbers. The rows of zbigsolo and z1017x compare two launches of one session: zbigsolo ran engine 00da56a (the v0.3.0 line before its kept-state fixes) with v0.2.0's window, prompt cache and no keepalive, then with v0.3.0's; z1017x ran v0.2.0's served engine 5c97635 with TF_DS_IDX_SPLIT unset, then set. A row that names two sessions takes its v0.2.0 column from the first: for the image turns, zkeptk's engine e3c6a7d (before pull request #28); for the keepalive, zbigsolo's first launch. Decode and the streams are level with v0.2.0 (within 1 %); prefill gains with the prompt's length; large conversations now run together and resume.

Engine changes since v0.2.0, all on in the served settings (the fork's v0.3.0 merge commit lists them):

• Structured output: a request with response_format (json_schema or json_object), guided_choice or guided_regex draws within its grammar, drafted rows checked against it too. The grammar comes from libtfgrammar, built from xgrammar and loaded from the kit (TENSORFOLD_GRAMMAR_LIB).
• Forced tool calls: tool_choice "required" or a named function answers with one DSML calls block, each call an offered tool (the named one) with its parameters within their schema. The DSML marker is always its own token, so sampled calls stay valid (36 of 36; 29 of 36 before that fix).
• The prompt shield (TENSORFOLD_PLAIN_SPECIALS=1): text in a user turn, a tool result or a tool call's arguments that spells one of the tokenizer's special tokens is encoded as ordinary text; only the chat template's own markers and the operator's text are control tokens. A prompt in which no sender wrote a marker is tokenized as before.
• Kept prompt states: prompts with images keep and resume their states, matched by the images' digests, and a resumed prompt skips the vision tower for the images it holds (pull request #28 by Manrique, ported); a prefix two conversations share no longer stales the first one's states (pull request #26 by Manrique); a prompt whose newest matching state ends too near its end resumes from an earlier one instead of from the start.
• The keyed indexer layers' long rows split by columns over the four nodes (TF_DS_IDX_SPLIT=1): the same columns out, prefill faster the longer the prompt.
• The 4,194,304-position window and 2 GiB of kept prompt states (--prompt-cache-gib 2): eight 400K-token conversations run at once, and their next turns resume.
• The stream keepalive (TENSORFOLD_SSE_KEEPALIVE=15): a streamed reply that has sent nothing for 15 s sends : keepalive, an SSE comment clients skip, so a long prefill does not trip a client's idle timer.
• Draft statistics: /metrics counts the draft head's accepted chains by depth, and TENSORFOLD_STATS_EVERY=60 writes a stats line to rank 0's log at most once a minute.
• Sampling: top_k 0 or -1 (off) and top_k above 56 draw as top_p allows instead of failing the request.

Off unless set (unset as served):

• TF_FILL_CHUNKS=1 (pull request #29 by Manrique, ported): a prompt pass runs in pieces between the other streams' rounds. While a 65,010-token prompt filled, another stream's longest stall fell from 23.4 s to 0.8 s, and the long prompt's first token came +19.2 % later (23.5 s to 28.0 s; zkeptk, engine 1cd544e).
• TENSORFOLD_REPLY_IDS=1: a reply's end of reasoning and its DSML calls are read at their token ids, not in the decoded text, so a reply that quotes those markers as text keeps them as words. Every other reply was equal and the speed rows moved -1.2 % to +0.4 % (zdirdk, engine e3c6a7d).
• TF_DS_NEIGHBOR_RDMA=1 (pull request #30 by Scott Woolsey Biggart): the neighbor transport, adjacent-edge RDMA exchanges for four nodes cabled as a physical ring, in place of the RDMA rings. Not measured on our switched network; the engine's docs/recipes/deepseek-v41-neighbor.md describes it, with its author's receipts.

Pull request #31 by mrsergiopascal adds a CPU test and a reproducer of the replay cut's position alignment, with its write-up; the branch keeps its own replay cut, which the test checks.

Thank you to Manrique (@hashd1ve) for pull requests #26, #28 and #29, Scott Woolsey Biggart (@swbiggart) for #30, @mrsergiopascal for #31, @den13501, whose report led to the keepalive, and @Capicua25x, whose report led to the loop guard's novelty signal. The engine branch's ATTRIBUTION.md credits each of them, and the ideas taken from DeepSeek's DeepInfer and from xgrammar.

v0.2.0: what changed

The 4.02 bpw weights decode 15 to 17 % slower than the 2.9 bpw ones; prefill is level. The served settings did not change, and the model id stays DeepSeek-V4.1-Flash-TP4.

Engine changes since v0.1.1 (the fork's v0.2.0 merge commit lists them):

• TensorFold 1.0.12 under the family. Two of its reply changes reach clients: with tools offered, a reply with tool calls keeps its prose in message.content (it was null), and the tensorfold block's runtime.seconds is the decode's seconds.
• Each rank checks its settings against rank 0's at the start. A launch with no captured Triton variant for a prompt length runs the closest more general one. The image decoder's child process starts again if it dies. A request the pool cannot hold yet waits for a stream to end.
• From contributors: the prompt indexer passes each row block 16-byte aligned, so every prompt length answers (pull request #4, Lucas Peters); the loader reads each layer's expert run from the weight file whole, a faster load (pull request #18, lepetitprince716-prog).
• Off unless set: the stream keepalive (TENSORFOLD_SSE_KEEPALIVE), the loop guard's novelty signal (it acts only under --loop-guard; after a report by @Capicua25x) and TF_DS_IDX_SPLIT (the keyed indexer layers' long rows split by columns over the nodes).

Results

All numbers are ours, measured with our own clients (kit_bench.py and the gate tools of the engine branch) on the

four nodes, on the served launch of v0.3.3: engine 39155041, the 1017 weights, this kit and its served settings, unless a row names another session; v0.3.5's rows are in its section above. ENGINE_COMMIT (0a0b69a) is v0.3.5's engine source (ebf6ae33, 39155041 with the waves commit, which builds the same binaries but for nvcc's build ids) on the branch's public line, which adds pull requests #30 (off unless TF_DS_NEIGHBOR_RDMA is set) and #31 (a test and its write-up; see RUNNING.md). Greedy decoding, thinking off, unless a row says otherwise. results/served-v035/, results/served-v033/, results/served-v032/ and results/served-v031/ hold the raw outputs, a folder for each session (results/README.md).

Speed

The gate run of the served build (z3334):

The gate run of the served build (zyx, phase c-v033):

Long context and kept prompts

These rows are v0.3.0's, at its 4,194,304-position window. On v0.3.1's build the kept-state, resume, keepalive and fill checks gave replies equal to v0.3.0's build's (z3334). On v0.3.2's build they ran in its gate's phase b and equaled the build served before it's (zsalt). On v0.3.3's build they ran in its gate's phase b (the candidate with both settings unset): the kept-state replies 8 of 8, the resume probe's 4 of 4 and the fill replies 2 of 2 equal to the build served before it's, and the keepalive re-ran on the served port after the promotion, replies 2 of 2 equal (zyx). The 300K needle ran on the served build after the promotion (row34): 3 of 3 needles found in each of three runs, median 98.65 s. The 999,843-token needle check still has not run on the served settings, as in v0.3.1 and v0.3.2 (the README keeps v0.2.0's needle record and says what it ran on).

The 999,843-token needle check and the four 160K prompts of v0.2.0 ran with the split unset and the 2,097,152-position window; their results are in the v0.2.0 README.

Gates

On the gate launch of the served build (z3334), and where a line says so, again on the served port after the

promotion and in the slot's earlier phases. Every reply these gates compare equals that of the build served before it (v0.3.2's engine 1935b48a) in the same session:

• Exact chat: 71 of 71 chats (system and developer prompts, tools and tool calls, tool results, reasoning and its effort, multi-turn, response formats) give the same token ids through the server's /tokenize as DeepSeek's reference encoding (encoding.py, MIT) gives for them; 71 of 71 on the served port after the promotion.
• The API gates 6 of 6: the model list, a chat probe, a streamed reasoning, a tool call, drafted equals serial, and a burst of four; 6 of 6 on the served port.
• 16 replies sent as a burst and staggered: equal to their solo replies 16/16 and 16/16, and the solo replies equal the build served before it's.
• Vision: the image gate 6/6; 9 image prompts' token ids, then the same 9 four at a time, twice: 18/18 equal to the solo run.
• The prompt shield: 14 of 14 cases in the slot's phase b (in 5 no sender wrote a marker, and they tokenize as before; in 9 one did, and the marker stays text); 14 of 14 on the served port.
• Structured output: 11 of 11 requests as asked (a JSON schema, a JSON object, a choice, a regex; greedy, sampled and with thinking), burst and staggered equal to solo; 11 of 11 on the served port.
• Forced tool calls: 16 rows (8 requests, whole and streamed) of "required", named and "auto" calls, the forced ones each an offered tool within its schema and the "auto" ones equal to those of the build served before it (v0.3.2's engine 1935b48a); and sampled forced requests: 36 of 36 valid, 36 of 36 on the served port.
• cache_salt: 10 of 10 checks. A fresh salt caches 0 of the 6,012-token prompt, the same salt 4,096, no salt after a salted one 0 again and no salt after that 4,096 (the completions and the chat forms both).
• A short chat behind the cold 93,335-token prompt: 0.68 s to the first token (gate 0.89 s), against 0.76 s for the build served before it in the same session; the long prompt itself 32.00 s (34.04 s), and the six replies equal.

Run-time inputs from the public weights

The weights, their readying and each node's weight file are v0.2.0's: the served lane loads the same four weight files (results/served-1017/inputs.txt), so the first three rows are v0.2.0's checks.

Earlier releases

v0.3.3 served engine 39155041 with NCCL_NCHANNELS_PER_NET_PEER=8; its tables are above and in the v0.3.3 README (tag v0.3.3), and its raw outputs in results/served-v033/. v0.3.2 served engine 1935b48a with a 20,971,520-position window; its tables are in the v0.3.2 README (tag v0.3.2) and its raw outputs in results/served-v032/. v0.3.1 served engine 37df889 with a 25,165,824-position window; its tables are in the v0.3.1 README (tag v0.3.1) and its raw outputs in results/served-v031/. v0.3.0 served engine 4964e38 with a 4,194,304-position window; its tables are in the v0.3.0 README (tag v0.3.0) and its raw outputs in results/served-v030/. v0.2.0 served engine 5c97635 on these weights; its tables are in the v0.2.0 README (tag v0.2.0) and its raw outputs in results/served-1017/. v0.1.0 and v0.1.1 served the 2.9 bpw weights on engine ad70f4f and 5940208. Their tables are in the v0.1.1 README (tag v0.1.1), and their raw outputs stay in results/ (served-ad70f4f/, inputs-5940208/, throughput-1ca1c10/, probes-1ca1c10/).

The served settings

Rank 0 runs tensorfold-native serve with TF_TP_WORLD=4; ranks 1-3 run tf-dsv41-lanes, which follows it. The

settings as served: --context 1048576 --parallel 16 and --prompt-cache-gib 2 on rank 0, TF_DS_WINDOW=20971520 (the streams' shared positions; the followers' --pool 20971520), TF_DS_ROUND_ROWS=48, TF_DS_STREAMS=16, TF_DS_2D_GU=parity (the balanced experts split), TF_DS_SERVED_K=solo, the measured round costs in TF_DS_ROUND_MS, TF_DS_IDX_SPLIT=8192, TF_FILL_CHUNKS=1, TF_FILL_MANY=1, TF_DS_2D_YX=1, TF_DS_WAVES=1, NCCL_MAX_NCHANNELS=4, TENSORFOLD_PLAIN_SPECIALS=1, TENSORFOLD_GRAMMAR_LIB (the kit's grammar/libtfgrammar.so), TENSORFOLD_SSE_KEEPALIVE=15, TENSORFOLD_STATS_EVERY=60, CUDA graphs, the side stream, the paced L2 prefetch, Engram reads by AIO, and the decode-size exchanges over the RDMA rings. New since v0.3.1: TF_FILL_MANY with TF_FILL_CHUNKS (a request admitted while a prompt pass fills; a pass alone runs as a whole one), cache_salt (engine-side, no flag), the prompt passes by position, and the 20,971,520-position window (25,165,824 before; nodes that need more room can use 16,777,216, see v0.3.2's section). New since v0.3.2: TF_DS_2D_YX=1 with NCCL_NCHANNELS_PER_NET_PEER=8 (the prompt's cross-rank gathers carry the pair-summed bf16 y instead of the four fp32 quarters, and the ring's send/recv pairs run on eight channels a peer). New since v0.3.3: TF_DS_WAVES=1 with NCCL_MAX_NCHANNELS=4 in place of NCCL_NCHANNELS_PER_NET_PEER=8 (two waves a prompt pass, each one's gathers on a second stream and communicator; four channels a communicator). New since v0.3.6: the draft-cost table C in TF_DS_ROUND_MS and TF_DS_IDX_SPLIT=8192; TF_DS_TWO_HOP (a four-node ring read as two pairs, exchanging only between direct partners), TF_RDMA_GID (the RDMA rings' source GID index, default 5), TF_SEED_RANDOM and TF_TOP_K_MAX (a per-request sampling seed and a top-k cap) are new and off unless set. New since v0.3.7: rounds of up to 64 rows (TF_DS_ROUND_ROWS=64, with a 64-row decode window), up to 32 streams a pool (TF_DS_STREAMS=32), the draft-cost table's 64 entries (TF_DS_ROUND_MS, table C + 16 entries at slope 1.642), --parallel 32 and followers --streams 32 --round-rows 64. TENSORFOLD_REPLY_IDS, TF_DS_NEIGHBOR_RDMA, TF_DS_SOLO_OVERHEAD, TF_DS_NO_COPIES, TF_DS_TWO_HOP, TF_RDMA_GID, TF_SEED_RANDOM and TF_TOP_K_MAX are unset (the last four are v0.3.6's new opt-ins). TF_DS_PASS_ROWS and TF_DS_PASS_FROM are unset (the pass rule at its default). The round costs are the 64-entry table (gated in gate-v037-20261104T0904Z, results/served-v037/; the first 48 entries are v0.3.6's table C unchanged, and the gate measured the whole 64-entry table on this release's build): they guide how many tokens a round drafts, and drafted output equals serial output, so no reply depends on them. scripts/serve.sh spells the launch out; config.env.example holds the site settings.

Requirements

• Four NVIDIA GB10 nodes on one RoCE network, each with the NVIDIA PyTorch container nvcr.io/nvidia/pytorch:26.07-py3 (Pillow 12.3.0 inside it decodes the image formats other than PNG).
• Memory: the served window leaves about 12 GiB free on each node; nodes that need more room can use the 16,777,216-position window (see v0.3.2's section).
• The weights: bertholomus/DeepSeek-V4.1-Flash-1017-EXL3-4.0bpw (about 275 GB), readied by tools/serving_1017.py, and the original checkpoint's shards 47 and 48 for the Engram tables, on every node.
• The engine built from ENGINE_COMMIT (zig build with the CUDA toolchain; see RUNNING.md).
• The run-time inputs, made from the public weights (see RUNNING.md): each node's weight file and the compressed token map, one command a node (node_inputs.sh of the engine branch; scripts/serve.sh inputs runs it on all four), and the kernel kit in kit/, whose RoPE tables and image bias kit/verify_kit.sh makes before it checks every file against kit/MANIFEST.

License and attribution

This recipe is licensed under Apache-2.0 (LICENSE). The engine branch keeps TensorFold's licenses (Apache-2.0 from TensorFold 0.6.0, and the MIT notice of the code written before it), lists the upstream files it changes in its NOTICE and every outside source in its ATTRIBUTION.md. The weights are MIT (DeepSeek; our EXL3 quant inherits it). The chat template is ours, written to render what DeepSeek's reference encoding (MIT) renders. kit/ holds compiled code: our own Triton kernels (Apache-2.0), the EXL3 kernels of TensorFold's EXL3 module with our additions (Apache-2.0, and TensorFold's MIT notice for its code written before 0.6.0), PyTorch's attention kernel for the vision tower, unmodified (BSD-3-Clause), and libtfgrammar, xgrammar (Apache-2.0) built with TensorFold's shim and the headers it bundles (dlpack, Apache-2.0; picojson, BSD-2-Clause); kit/KIT-LICENSES.md gives each file's source and license. ATTRIBUTION.md here lists what this recipe uses. "DeepSeek" belongs to DeepSeek and "DGX Spark" and "GB10" to NVIDIA; this recipe is not affiliated with or endorsed by DeepSeek, NVIDIA, the TensorFold authors or Mia-AiLab.