Retail-TW-Qwen3.5-4B-Base

Retail-TW-Qwen3.5-4B-Base is a continually pre-trained (CPT) checkpoint of Qwen/Qwen3.5-4B-Base, adapted to Taiwan (Traditional Chinese) retail and e-commerce knowledge.

Note: This is a base checkpoint. It is neither instruction-tuned nor safety-aligned, and it is intended as a foundation for downstream supervised fine-tuning (SFT).

Model Details

Key Features

• 🎯 Retail-Domain Adaptation: Continual pre-training on 2.56B tokens, including 7.30M proprietary Taiwan product listings across 15 retail categories.
• 🇹🇼 Taiwan Retail Knowledge: Synthetic brand and contributor wikis, together with name variants (aliases, foreign-language names, Zhuyin and Pinyin spellings, and typographical errors), generated with NVIDIA NeMo Data Designer.
• 🌐 General-Language Retention: A replay mix of general-domain text (Wikipedia in zh-TW, zh-CN, and en, plus en↔zh translation pairs) preserves general and multilingual language modeling ability.

Training Data

Data Mixture

Token counts are computed with the base model tokenizer. A 0.1% split of each dataset is held out for validation.

Token Mix at a Glance

Share of CPT Tokens Product Listings53.7% · 1.38B Taiwan Retail Wiki26.3% · 674.4M Replay Data20.0% · 512.9M Replay Composition (% of Replay Tokens) zh-TW Wikipedia40.0% en↔zh Translation Pairs30.0% en Wikipedia20.0% zh-CN Wikipedia10.0% Product Listings1.38Btokens7,300,108 samples · 53.7% Taiwan Retail Wiki674.4Mtokens2,762,847 samples · 26.3% Replay Data512.9Mtokens2,774,330 samples · 20.0% Total2.56Btokens12,837,285 samples · 100% Taiwan retail wiki (synthetic, NVIDIA NeMo Data Designer): brand and contributor wikis, plus name variants (aliases, foreign-language names, Simplified/Traditional script, Zhuyin and Pinyin spellings, typos). Replay: general-domain text mixed into CPT to mitigate catastrophic forgetting. Sources: wikimedia/wikipedia (zh-TW, zh-CN, en) and en↔zh translation pairs from wmt/wmt19. zh-TW = Chinese converted to Traditional Chinese with OpenCC (translation pairs: Chinese side converted to zh-TW).

Product Listings

The product listings are proprietary Taiwan retail data comprising title, brand, category, price, and description. The data was cleaned with NVIDIA NeMo Curator, which removed SEO filler text, emojis, contact details, hyperlinks, and policy terms, followed by exact and fuzzy deduplication.

Category distribution (15 categories, share of listings)

GeneralMerchandise14.67%Apparel, Shoes &Accessories12.72%Food & Beverage11.98%Beauty & PersonalCare10.6%Toys, VideoGames &Collectibles9.88%Sports,Outdoors &Fitness7.14%3C,Computers &Electronics6.36%Home, Kitchen& DailySupplies5.6%Books,Media &Stationery4.2%HomeAppliances4.19%Health &Medical4.16%PetSupplies4.09%Maternity &Baby3.93%Automotive&Motorcycle0.45%Tickets · 0.03%

Evaluation

Evaluation Setup

The CPT model is compared against its base model, Qwen/Qwen3.5-4B-Base. Statistical significance is assessed with a 95% paired bootstrap over items (1,000 resamples):

🟢 significant improvement · 🔴 significant degradation · ⚪ no significant difference

Language Modeling

Metric: Bits per byte (BPB); lower is better.

Held-Out Domain Data

BaselineCPT modelBaselineCPTΔ BPB Product Listings1.3920.359−1.032 Retail Wiki (Brand & Contributor)1.0110.574−0.437 Name Variants (Brand & Contributor)1.1430.348−0.795

General-Domain Replay Data

BaselineCPT modelBaselineCPTΔ BPB Chinese (Traditional, OpenCC-Converted)0.9290.854−0.074 Chinese (Simplified)0.9310.900−0.031 English0.6840.647−0.038 English–Chinese Translation0.8100.541−0.270

Knowledge Probes

Metric: PMI-normalized accuracy, which adjusts for the prior likelihood of each option; higher is better.

Setup: Zero-shot, five-option multiple choice (chance level: 20%). Each option is scored by its log-likelihood as a continuation of the prompt.

Data: Internal probe sets derived from the retail training data. Alias and wiki-fact labels are LLM-derived. Scores are not comparable with results reported for other models.

Summary by Probe Set

BaselineCPT modelBaselineCPTΔ PP Product & Work Knowledge50.5%64.9%+14.4% Retail Wiki Knowledge54.6%65.6%+10.9% Name Variants (Brand & Contributor)78.6%83.6%+5.0%

Product and Work Knowledge

These probes test knowledge about products and about the works (books, music, and other media) that products represent.

BaselineCPT modelBaselineCPTΔ PP Product Title → Brand61.4%69.9%+8.5% Product Title → Brand (Brand Masked)31.6%34.5%+2.9% Product → Category64.7%84.3%+19.7% Brand → Main Category41.4%65.0%+23.6% Work → Contributor53.7%70.8%+17.2%

Retail Wiki Knowledge

A contributor is a credited person or organization on a product or work, such as an author, singer, artist, illustrator, translator, editor, or publisher. Brand origin is the country or region where the brand was founded.

BaselineCPT modelBaselineCPTΔ PP Brand/Contributor: Wiki Passage → Name48.2%51.9%+3.7% Brand: Founding Year57.4%66.5%+9.1% Brand: Origin64.6%74.0%+9.4% Brand: Parent Company60.6%71.5%+10.9% Contributor: Nationality76.8%88.5%+11.8% Contributor: Occupation53.0%64.5%+11.5% Contributor: Credit Role21.8%42.0%+20.3%

Name Variants

A name variant is an alternative form of a brand or contributor name: a foreign-language alias, a different script, a phonetic spelling, or a typographical error. The canonical name is the standard form of the name.

Query Normalization (Variant → Canonical Name). Given a noisy variant of a brand or contributor name, the model selects the canonical name. A typographical error is either a Latin-letter typo or a same-sound Chinese character substitution.

BaselineCPT modelBaselineCPTΔ PP Foreign-Language Alias66.7%70.9%+4.2% Typographical Error90.2%86.3%−3.9% Simplified Chinese Script98.7%91.0%−7.7% Zhuyin (Bopomofo)66.1%69.8%+3.7% Zhuyin Keystroke Sequence20.6%19.6%−1.0% Pinyin (Toneless)92.4%84.3%−8.1% Pinyin (Tone Numbers)89.5%80.4%−9.2%

Variant Generation (Canonical Name → Variant). Given a canonical brand or contributor name, the model selects the correct variant.

BaselineCPT modelBaselineCPTΔ PP Foreign-Language Alias79.6%90.0%+10.4% Simplified Chinese Script100.0%100.0%+0.0% Zhuyin (Bopomofo)98.9%100.0%+1.1% Zhuyin Keystroke Sequence20.4%95.0%+74.6% Pinyin (Toneless)99.6%100.0%+0.4% Pinyin (Tone Numbers)99.6%99.9%+0.3%

Citation

@misc{retailtwqwen354bbase,
  title     = {Retail-TW-Qwen3.5-4B-Base},
  author    = {President Information Corp. (PIC) and Uni-President Information Philippines Corp. (UNI-PIPC)},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/UNI-PIC/Retail-TW-Qwen3.5-4B-Base}
}

Contact

• Taiwan: pic.net.tw/contactUs
• Philippines: pic.net.ph/about/contactUs