Thaki-BrandPrior-4B

Thaki-BrandPrior-4B scores how likely an AI assistant is to name a given brand on its own when a user asks a product question without naming any brand ("what's a good CRM for a 10-person sales team?"). Give it the question and a list of candidate brands; it ranks the brands by predicted mention likelihood. The intended use is to estimate, before running any LLM, which brands an assistant would surface for a category of questions, and where a brand sits in that ranking.

This is a query-brand reranker, not a text-generation model. It outputs one score per (question, brand) pair and cannot produce text.

What the score means: the relative propensity of the observed no-search LLMs (three chat models answering without web search) to mention the brand for that question. It is not market share, consumer preference, purchase probability, or a measure of how all ChatGPT, Gemini, or Claude users experience these assistants, and it does not predict what a search-enabled assistant would cite.

Input format

Qwen3-Reranker prompt with this instruction; the document is the candidate brand and the question's category. The score is logit("yes") - logit("no"); higher means more likely to be mentioned. It is a ranking score, not a probability. See example_inference.py.

Instruct: Given a user's question to an AI assistant that names no brand, judge whether the assistant would mention this brand in its answer without searching the web.
Query: <user question>
Document: Brand: <brand name>\nCategory: <category, e.g. technology software>

Training data

• Questions: 3,261 product-seeking user turns from WildChat-4.8M (allenai, ODC-BY), selected by an in-house LLM as product discovery, comparison, or purchase questions that name no brand. Questions with any person name, email, phone number, or URL were excluded.
• Responses: each question was sent once (temperature 0, no web search) to openai/gpt-4.1-mini, google/gemini-2.5-flash, and mistralai/mistral-medium-3.1. Brands in the responses were extracted by an in-house LLM, keeping only strings that appear verbatim in the response.
• Label: the share of the three models that mentioned the brand (0, 1/3, 2/3, 1). This is an LLM-observed silver label, not a human gold label.
• Candidates: all observed brands for the question, plus brands observed for other questions in the same category (label 0), up to 30 per question.
• Size: 3,246 questions, 98,188 question-brand pairs. Training used LoRA (r=16) for 2 epochs on one NVIDIA B200 (36 minutes).

This repository contains no WildChat text, no LLM responses, and no training pairs.

Evaluation

Held-out questions; "unseen categories" are five product categories that never appear in training. Only questions with at least one mentioned brand are scored.

Paired bootstrap 95% intervals for the gain over the base reranker (test): Recall@5 +0.026 [0.017, 0.036], nDCG@10 +0.083 [0.069, 0.097], Spearman +0.078 [0.062, 0.095].

Negative candidates are brands that appeared for other questions in the same category. Because of that, the base reranker already reaches a Precision@3 of 0.921: it usually puts a mentioned brand on top. The main improvement from training is in ordering candidates by mention frequency (rank correlation 0.481 to 0.560), not in discovering whether a brand is relevant at all.

Limitations

• Mention behaviour of three specific models, sampled once at temperature 0 in October 2026. Other models, later versions, sampling, or search-enabled assistants can differ.
• Questions are mostly English chatbot traffic from 2023-2024.
• Brand names are matched as written; variants ("HubSpot" vs "HubSpot CRM") can split the label.
• Not a measure of market share, quality, consumer preference, or purchase probability.

Release status

Research preview.