Nemotron-3.5-Lightning-30B-A3B, JANG bit-allocation + GPTQ calibration + ipsupport-code coding-agent LoRA (MLX)

▶ A coding-agent model — run it in LLMTray, drive it with IPSupport Code

This model is fine-tuned for IPSupport Code, a local AI coding agent for real repositories (analyze · fix · test · report). It is not used standalone: run it in LLMTray, which serves it over an OpenAI-compatible API, then point IPSupport Code at that endpoint. Tool-calling is trained in (see "Tool calling is trained in" below).

NVIDIA's Nemotron-3.5-Lightning-30B-A3B, quantized to MLX using the JANG component-type bit allocation (reverse-engineered from dealignai/Nemotron-3.5-Lightning-30B-A3B-JANG_2L-CRACK's published config.json) applied through this project's own GPTQ Hessian calibration (sequential, block-by-block) instead of JANG's presumed naive RTN — and merged with a LoRA fine-tune for ipsupport-code, a local terminal coding agent, trained to fix real tool-calling reliability failures observed in production usage.

Bit allocation (JANG recipe)

Group size: 64. Average: 4.237 bits/weight, ~16.4GB on disk.

What's different about THIS release vs. this project's earlier JANG-recipe release

• Calibration corpus: mixed wikitext-2-raw + real ipsupport-code tool-calling transcripts (not wikitext-only), so quantization error is compensated for the actual agentic/tool-calling text distribution this model is meant to run, not just generic prose.
• Sequential calibration: each block's activations are captured against the model with all prior blocks already quantized in place (not one-shot against the pristine model).
• Built on the LoRA-merged model: the base model was first fine-tuned (QLoRA, lora_r=32, targeting attention q/k/v/o + MoE expert up/down_proj) on real + synthetic ipsupport-code tool-call examples, merged back into full precision, and THEN GPTQ-quantized -- so the quantization is calibrated against the actual post-fine-tune model, not the vanilla base.

Perplexity

Slightly worse on generic wikitext prose (+0.8%), meaningfully better on the actual target domain (-3.5%) -- the intended trade for a model meant to run as a coding-agent backend, not a general chatbot.

Inference speed (M-series Mac, single-request decode)

Component-type mixed-precision recipes (this release, the earlier wikitext-only release, and JANG_2L-CRACK) all run at essentially the same speed (~42-43 tok/s) regardless of calibration method or claimed average bpw, because runtime speed is governed by the bit-allocation STRUCTURE (which components get 8/6/4/3 bits), not by the specific quantized values. For comparison, a uniform 3-bit release of this same base model runs at ~65-68 tok/s. See this project's session findings, §7r for the full root-cause writeup.

Tool calling is trained in — intended behaviour, not a glitch

This model was fine-tuned for coding agents (ipsupport-code): the training data is tool-call-heavy, so the model has a strong habit of acting through tools. That habit shows up when you give it tools in a general chat. Measured on the -MTP release (same fine-tune as the non-MTP one), with an image-generation tool in the prompt (temperature 1.0, 20 samples each):

Occasionally (3/20) it also closes its <think> block twice and repeats the answer.

This is what the fine-tune is for. In a coding agent, "reach for a tool" is the right default: read the file, run the command, edit the code. It is the wrong default for small talk in a general chat UI.

What helps if you use it as a chat model with tools attached:

• A system rule such as "Only call a tool when the user's latest message explicitly asks you to perform that action; for greetings, small talk and questions, answer in text" plus a tool description that says the same. This cut tool calls on questions from 14/20 to 4–6/20. Greetings stay at 7–11/20. LLMTray sends this rule by default.
• Lower temperature: 0.6 cut greetings to 1/10 in one run.
• Don't attach tools the conversation doesn't need.
• For plain chat without tools, the base quant without the LoRA doesn't have this habit.

Details and methodology: FINDINGS §2.5.

The IPSupport local-AI stack

Local-first AI tools for macOS by IPSupport — nothing leaves your Mac.

• LLMTray — your local AI workstation for macOS: chat with local LLMs, generate and edit images, make music, run agents, and serve an OpenAI-compatible API. Downloads models from Hugging Face in-app, with per-model profiles.
• IPSupport Code — your AI coding agent for real repositories: analyze, fix, test, report.

Run this model in LLMTray — it serves an OpenAI-compatible API — then point IPSupport Code at that endpoint. That pairing is how it's meant to run.

License

Distributed under NVIDIA's OpenMDW License Agreement v1.1 (same license as the base model) — see LICENSE in this repo.

Usage

pip install mlx-lm
python -m mlx_lm.generate --model roman220220/Nemotron-3.5-Lightning-30B-A3B-JANG-GPTQ-ipsupport-code-lora --prompt "..."

Produced with this project's GPTQ pipeline (poc/gptq_stock_convert.py --quant-recipe-mode component --component-recipe jang --sequential --extra-calib-file <flattened tool-call data>, poc/mlx_convert_recipe.py --mode component). The LoRA adapter used to produce the fine-tuned base is at roman220220/ipsupport-code-nemotron-lora.