Run DiffusionGemma as a local coding agent on a 16 GB GPU
This is a ready-to-run DiffusionGemma stack for Windows + WSL2. It turns the 26B-A4B model into a repository-aware coding agent on a single 16 GB NVIDIA GPU, without requiring you to compile llama.cpp, convert weights, assemble CUDA libraries, or build an agent wrapper yourself.
This release contains more than a quantized GGUF:
- an IQ3 model profile that fits entirely in 16 GB VRAM;
- a matched custom llama.cpp/CUDA backend for DiffusionGemma;
- pinned CUDA 13 runtime libraries and launch settings;
- repository retrieval instead of sending the whole codebase to the model;
- focused file editing through Aider, automatic tests, session artifacts, and rollback when validation fails;
- a localhost OpenAI-compatible endpoint for supported coding clients.
Why use this instead of a standalone quantization?
| Standalone GGUF | This runtime |
|---|---|
| Supplies model weights | Supplies a tested model, backend, CUDA runtime, and agent workflow |
| Requires finding a compatible DiffusionGemma runner | Includes the exact custom backend used by the release |
| Requires manual offload, memory, and launch tuning | Ships a full-GPU preset tested on a 16 GB RTX 3080 Laptop GPU |
| Primarily provides raw generation/chat | Searches a repository, edits focused files, runs tests, and records results |
| Leaves integration and reproducibility to the user | Installs and operates through one versioned CLI |
Use another GGUF if you only need weights or want to assemble your own inference stack. Use this release when you want a reproducible local coding agent that is already wired together for 16 GB NVIDIA hardware.
Easiest Windows setup
You do not need Python, a CUDA Toolkit, a compiler, or ML setup experience.
- Download the standalone Windows installer.
- Open DiffusionGemma Agent after installation.
- Let the app check Windows, WSL2, the GPU, VRAM, disk, and download access.
- Review the model and CUDA licenses and click Download and install.
- Choose a Git repository, enter one concrete task, and review the generated diff and test output in the app.
The desktop app includes its own Python runtime. It automatically reuses a compatible runtime or IQ3 GGUF already present in common download and Hugging Face cache folders. The separate 13.2 GB download, when needed, is resumable. The current alpha installer is not code-signed, so Windows may show an unknown-publisher warning; GitHub publishes its SHA-256 checksum beside the download.
Command-line alternative
Run in Windows PowerShell:
python -m pip install --upgrade diffusiongemma-agent
dg-agent doctor
dg-agent install --accept-licenses
dg-agent status
Run a focused repository task:
dg-agent run --repo C:\work\repo --task "Fix src/x.py and run its tests" --file src/x.py
Stop the service and release GPU memory with dg-agent stop.
What a task looks like
your request
-> bounded repository search
-> focused edit session
-> detected tests and verification
-> final diff and saved session artifacts
-> rollback if validation fails
This works best for concrete file-level changes, focused bug fixes, and tests. It is not intended to replace a large-context cloud agent for broad, underspecified repository-wide work.
Hardware and storage
- Windows 10/11 with an initialized WSL2 Ubuntu distribution;
- NVIDIA GPU with at least 16 GB VRAM;
- current NVIDIA Windows driver with WSL CUDA support;
- at least 15 GiB free for download, with 30 GiB recommended for the download plus the installed WSL copy;
- network access during first installation.
The tested system is an RTX 3080 Laptop GPU with 16 GB VRAM. CPU-only, native Linux, macOS, AMD, and sub-16-GB GPU installations are not supported by this release.
Contents
- IQ3 GGUF derived from
google/diffusiongemma-26B-A4B-it; - Linux x86-64 custom llama.cpp DiffusionGemma backend;
- private CUDA 13 runtime libraries required by that backend;
- checkpointed repository supervisor, bounded retrieval, Haystack, and Aider adapters;
- service launchers for localhost ports
4100and8090; manifest.jsonwith file sizes and SHA-256 values;- complete third-party notices under
LICENSES/.
Limits
The default fast agent profile uses a 768-token effective input context and up to 256 output tokens. It compensates with bounded repository retrieval; it does not load an entire large repository into the model context. IQ3 quantization reduces quality relative to the original model. Native model-selected tool calls are disabled in the default route.
A short warmed probe measured approximately 19.6 words/s on the tested RTX 3080 Laptop 16 GB. Full agent tasks are slower because retrieval, diffusion passes, edits, and tests add latency. This is not a performance guarantee.
Status and scope
This is alpha, hardware-specific software. It is not an official Google, NVIDIA, Hugging Face, Aider, or upstream llama.cpp distribution.
Research context
This repository is an engineering runtime, not the implementation artifact of the papers below. For problem-first research guides on adjacent evaluation and control questions, see:
- How to tell whether a compressed text generator fails in the codec or the generator · Русский · 简体中文 · 한국어
- How can AI edit code without regenerating the entire program? · Русский · 简体中文 · 한국어
Safety
The agent can modify files and run repository tests. Checkpointing and rollback are not a security sandbox. Use a clean Git worktree, inspect generated diffs, and do not expose secrets to untrusted repositories. Services bind to localhost by default.
Source code:
See the public source README for lifecycle commands, troubleshooting, update, uninstall, and architecture details.
Licenses
Review LICENSES/NOTICE.md before installation or redistribution. The runtime
contains Apache-2.0 model-derived material, MIT-licensed llama.cpp code, and
NVIDIA redistributable libraries governed by the included CUDA EULA.