Spark-X2.6-4B-SQL-Cypher-Agent

A 4B tool-calling agent that answers questions about a database by writing one read-only query: SQLite SQL against a relational schema or Neo4j Cypher against a graph schema. It thinks first, then calls whatever query tool you give it. It's a LoRA fine-tune of XHToken/Spark-X2.6-4B, merged into the base weights, trained on a single 16 GB consumer GPU (AMD RX 7800 XT).

• This repo: the merged model in bf16 (Transformers, trust_remote_code=True).
• GGUF for llama.cpp (Q8_0, Q6_K_XL, Q6_K, Q4_K_XL, Q4_K_M): trjxter/Spark-X2.6-4B-SQL-Cypher-Agent-GGUF

Results

Execution accuracy on a held-out test set of 1,000 questions, 250 from each dataset's own test split. None of the questions appear in training. The same harness ran both models: llama.cpp, the same prompts, greedy decoding, a thinking budget of 1,024 tokens.

+10.7 points overall (95% interval ±2.9). Compared question by question on the 856 questions both runs could score: the fine-tune alone got 129 right, the base alone got 37 right (sign test p ≈ 4e-13). The unmerged adapter on the bf16 base scored 43.9% on the same test, so merging and quantising to Q8_0 cost nothing measurable.

How an answer is scored. A question counts only if the query returns the same result as the dataset's reference query, not if it merely looks similar:

• gretel: both queries run on the example's real rows.
• SQaLe: both run on several randomly filled copies of the schema. This is the test-suite idea from Spider: two queries that agree on several random databases very likely mean the same thing. Some copies place values right next to the query's literals, so > against >= or LIKE against = give different rows.
• CypherBench: both run on the dataset's own sampled graphs in Neo4j.
• text2cypher: both run on random graphs built from the prompt's schema.

Results are compared as multisets of rows, and row order counts when the reference sorts. A question where no database tells the two queries apart is left out, which is why the scored counts are just under 1,000. Exact-match scores would be far lower and mean little here.

How to use it

The model expects the setup it was trained on:

• a system message with the schema, optional example queries and a few rules;
• the question as the user message;
• one query tool in tools.

The tool's name and argument don't matter (training used run_query(query), execute_sql(sql), cypher_query(cypher), ...), and unrelated tools in the list are ignored. If the schema can't answer the question, the model is trained to say so in one sentence instead of calling the tool.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "trjxter/Spark-X2.6-4B-SQL-Cypher-Agent"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, torch_dtype="bfloat16",
                                             device_map="auto")

schema = """CREATE TABLE teams (id INTEGER PRIMARY KEY, name TEXT, city TEXT);
CREATE TABLE games (id INTEGER PRIMARY KEY, season INTEGER, home_id INTEGER, away_id INTEGER,
                    home_score INTEGER, away_score INTEGER);"""
system = f"""You answer questions about the database below by calling the `run_query` tool with ONE read-only SQLite SQL query.

{schema}

Examples (tested queries on this database):
(none)

Rules:
- One statement, read only (SELECT or WITH ... SELECT). No comments. The query runs as written, so use literal values, not parameters.
- Return named columns: the names and the numbers the answer needs. Never return whole rows of every column unless the question asks for them.
- Match names and values as the data spells them.
- Return exactly the rows the question asks for, in its shape: one row for a single answer ("which is the most", "the highest": ORDER BY the measure, LIMIT 1); one row per entity for "which ...", "every", "list"; the top N for "top N".
- If the database can't answer the question, don't call the tool: say why in one short sentence."""
tools = [{"type": "function", "function": {
    "name": "run_query",
    "description": "Run one read-only SQLite SQL query against the database and return the result rows.",
    "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "The query text."}},
                   "required": ["query"]}}}]
messages = [{"role": "system", "content": system},
            {"role": "user", "content": "Which team scored the most points at home in 2024?"}]

ids = tok.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=2048, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:]))
# ... thought ...</think><tool_call>run_query<arg_key>query</arg_key><arg_value>SELECT ...</arg_value></tool_call>

For a graph, use the same shape with "graph database", "Neo4j Cypher", MATCH ... RETURN and "nodes", and give the schema as node labels, relationship types and their properties.

Thinking is on by default (enable_thinking=True, reasoning_effort="high"; "low" also works). Pass enable_thinking=False to apply_chat_template for a direct call. The scores above use thinking with a 1,024-token budget; the median thought is about 650 characters.

Decoding. Every score above uses greedy decoding (do_sample=False, or temperature=0 on a server). generation_config.json is the base model's sampling default (temperature 1.0, top-p 0.95), which suits chat better than query writing.

Serving. For llama.cpp, use the GGUF repo. It needs llama.cpp b10828 or newer for the Spark architecture. llama-server's OpenAI-compatible API returns the thought in message.reasoning_content and the call parsed in message.tool_calls. Pass thinking options as "chat_template_kwargs": {"enable_thinking": false} or {"reasoning_effort": "low"}. Spark's template accepts only high and low, so don't set the server-wide --reasoning-effort medium.

Training

• Method: LoRA, rank 16 / alpha 32 / dropout 0.05, on every linear layer of every block (q_k_v_proj, out_proj, gate_proj, up_proj, down_proj). AdamW, lr 1e-4, 3% warmup, cosine decay, weight decay 0.01, batch 32, max length 4,096, seed 42. The loss is on the assistant turn only.
• Data: 94,745 examples, about 66M tokens, one epoch.
  • 90,549 gold calls: each dataset's reference query, written as a thinking-off tool call. SQaLe 27,887, gretel 28,695, text2cypher 25,871, CypherBench 8,096.
  • 4,196 self-generated traces: the base model's own thought and tool call on training questions, kept only when the query passed the execution checks above (thinking high 2,829 / low 1,129 / off 238). These teach it to reason before calling without drifting from its own style.
• Checkpoint chosen: optimizer step 1,500 of 2,961. A snapshot was scored every 500 steps on a separate 200-question evalset. The final adapter and the best mid-run one (step 1,500) were both run on the test set. Step 1,500 scored 43.9% against 42.2% for the final one, and both held-out sets pointed the same way, so step 1,500 shipped.
• Hardware: one AMD Radeon RX 7800 XT (16 GB, ROCm) under WSL, about 24 hours for the run. The pipeline (trace generation, checking, training, evaluation) is a personal project.

Limitations

• About 44% of answers return the reference result. It's a 4B model. Run its queries read-only, and check them before trusting a number, especially on unfamiliar schemas.
• One query per answer, one turn. It wasn't trained to read a tool result and try again (multi-step agent loops are the next stage of the project).
• Dialects: trained on SQLite SQL and Neo4j Cypher. Other SQL dialects work as far as their syntax overlaps with SQLite.
• English only, and questions in the style of the four datasets: single-database analytics, joins, groups, top-N, filters.
• It sometimes adds SELECT DISTINCT where the reference doesn't, which changes row counts on duplicate-heavy tables.
• text2cypher is the weakest set (27.7%); its databases aren't public, so its answers are checked on random graphs, a harder and noisier test.

Licence and attribution

Apache-2.0, like the base model. Base model: XHToken/Spark-X2.6-4B (Apache-2.0).

Training data, used under their licences: