xVITA-Embed2-zhTW-GGUF
繁體中文(台灣)檢索用的多模態嵌入模型,文字、照片、語音輸出到同一個768維向量空間,由云碩科技(xCloudinfo)以google/embeddinggemma-2微調。
本倉庫提供GGUF格式,文字主模型七個精度階,另附圖片與語音編碼器(mmproj),可用llama.cpp在地端執行。
這是嵌入模型,輸出的是向量,不生成文字。
這顆解決什麼問題
EmbeddingGemma 2把文字、照片、語音放進同一個向量空間,一套索引就能用文字找照片、用照片找說明、用錄音找逐字稿。 我們把它接進繁中知識庫時,碰到兩個問題:中文段落檢索的命中率輸給前一代的自家模型,中文語音更是幾乎找不到對應文字。
直接拿繁中語料微調可以把文字分數拉上來,但代價在別處。早期一輪實驗的文字段落檢索從0.8125升到0.8780, 同一個檢查點用中文圖說找照片卻從0.6726掉到0.4934,照片找中文圖說也從0.6394掉到0.3864。 主幹被繁中文字拉走,圖文對齊跟著散掉。只看文字分數的話,這顆看起來是進步的。
所以這顆的發佈條件是一道門檻,而不是單一分數:每一項跨模態檢索相對底模的退步都不能超過1個百分點,過了門檻之後才比文字分數。 要讓文字分數在這道門檻下繼續往上,靠的是台灣自己的圖說資料:我們用瀏覽器逐頁收錄國家文化記憶庫中授權可商用的素材, 做成標題配照片、照片配說明的訓練對,並把短標題同時做成照片找標題的反方向配對。 少了反方向那一半時,照片找中文圖說一直卡在比底模低3.6個百分點,補上後反而比底模高了4.3個百分點。
成績
以下是原始權重(safetensors)在同一套量尺上的結果,指標都是hit@1。 所有測試集都沒有進訓練:DRCD用官方test分割;照片與語音的測試樣本在建立訓練料前就先保留出來。
| 測試 | 底模embeddinggemma-2 | 本模型 | 差 |
|---|---|---|---|
| DRCD段落檢索(3,493題) | 0.8125 | 0.8683 | +5.58 |
| DRCD整篇檢索(3,493題) | 0.8128 | 0.8299 | +1.71 |
| XM3600:中文圖說找照片 | 0.6726 | 0.6745 | +0.19 |
| XM3600:照片找中文圖說 | 0.6394 | 0.6822 | +4.28 |
| XM3600:英文圖說找照片 | 0.5946 | 0.5853 | −0.93 |
| XM3600:照片找英文圖說 | 0.6183 | 0.6236 | +0.53 |
| 台灣老照片:標題找照片(999張) | 0.3353 | 0.5145 | +17.92 |
| 台灣老照片:照片找標題(999張) | 0.3624 | 0.5185 | +15.61 |
| 行政院公報:主旨找頁面圖 | 0.9100 | 0.9740 | +6.40 |
| 行政院公報:頁面圖找全文 | 0.9950 | 0.9970 | +0.20 |
| Common Voice中文:語音找文字 | 0.2452 | 0.7638 | +51.86 |
| Common Voice中文:文字找語音 | 0.4209 | 0.7732 | +35.23 |
差的單位是百分點。XM3600是3,600張照片、7,174句中文圖說。台灣老照片是國家文化記憶庫的保留集,素材從沒進過訓練。
語音那兩列要保守看待。訓練用的語音同樣來自Common Voice,測試只排除了句子、沒有排除講者, 所以這兩列有一部分是「同一批人的聲音」。我們另外用一組不同來源、從沒進訓練的中文錄音驗證過,語音檢索的進步同樣存在, 但那組錄音不對外公開,數字不列在這裡。
英文圖說找照片是唯一退步的一項,退0.93個百分點,在門檻之內。
和其他嵌入模型比
我們用同一套程式量了六顆模型,分成兩把尺:DRCD代表繁體中文的主場,C-MTEB代表簡體中文的通用網路檢索。
| 模型 | DRCD段落 | DRCD整篇 | C-MTEB原版 | C-MTEB轉繁體 |
|---|---|---|---|---|
| Llama-xVITA-Embed-1B-zhTW(云碩,2048維) | 0.8809 | 0.8586 | 56.85 | 55.78 |
| 本模型(768維) | 0.8677 | 0.8305 | 61.79 | 60.73 |
| xVITA-Embed-300M-zhTW(云碩前一代,768維) | 0.8600 | 0.7927 | 58.90 | 57.22 |
| intfloat/multilingual-e5-large(1024維) | 0.8285 | 0.7060 | 63.66 | 61.57 |
| BAAI/bge-m3(1024維) | 0.8219 | 0.7741 | 65.32 | 63.71 |
| google/embeddinggemma-2(底模,768維) | 0.8125 | 0.8128 | 61.26 | 59.87 |
DRCD是hit@1;C-MTEB是八項中文檢索任務(T2、MMarco、Du、Covid、Cmedqa、Ecom、Medical、Video)的nDCG@10平均, 「轉繁體」是把同一份題目與文件用OpenCC轉成繁體字形後再量一次。C-MTEB一律截在512個token, multilingual-e5-large在這把尺上量到的平均是63.66,與MTEB官方公布的數字相同,我們用它確認這把尺可以和公開排行榜對照。 bge-m3官方是用8192個token量的,這裡截在512,數字會比官方低一些。DRCD整篇檢索讀到8192個token,multilingual-e5-large只吃得下512,所以它那一格偏低。
兩把尺的結論不一樣。在繁體中文上,本模型的段落與整篇檢索都贏multilingual-e5-large與bge-m3; 在簡體的通用網路檢索上,本模型是表中排第三,落後multilingual-e5-large約1.9分,差最多的是百度搜尋題的DuRetrieval(75.19對85.31)。 八項裡有三項贏multilingual-e5-large:T2Retrieval(76.79對76.04)、EcomRetrieval(60.67對54.86)與VideoRetrieval(60.49對58.26)。
如果只做純文字、而且常要檢索整篇長文件,我們的Llama-xVITA-Embed-1B-zhTW在DRCD上還是比本模型高,整篇檢索高2.8個百分點; 本模型的強項是同一個向量空間同時放文字、照片與語音,以及只有768維、Q8_0只有310 MB的體積。
檔案
文字一致度是同一批20句中文分別送GGUF與原始權重,比較兩邊向量的cosine。
| 檔案 | 大小 | 與原始權重的cosine(最低/平均) | 建議 |
|---|---|---|---|
xVITA-Embed2-zhTW-BF16.gguf |
558 MB | 0.99991 / 0.99998 | 要最接近原始權重時用 |
xVITA-Embed2-zhTW-Q8_0.gguf |
310 MB | 0.99934 / 0.99971 | 一般部署首選 |
xVITA-Embed2-zhTW-Q6_K.gguf |
246 MB | 0.99406 / 0.99852 | |
xVITA-Embed2-zhTW-Q5_K_M.gguf |
213 MB | 0.99452 / 0.99653 | |
xVITA-Embed2-zhTW-Q4_K_M.gguf |
182 MB | 0.97281 / 0.98884 | 記憶體吃緊時用 |
xVITA-Embed2-zhTW-IQ4_XS.gguf |
169 MB | 0.97969 / 0.98711 | |
xVITA-Embed2-zhTW-IQ2_M.gguf |
131 MB | 0.87301 / 0.89867 | 不建議用於檢索 |
xVITA-Embed2-zhTW-mmproj-BF16.gguf |
982 MB | 見下節 | 要處理照片或語音時才需要 |
Q6_K以下都以imatrix量化,校準語料是700行繁中段落、問題、新聞句與照片標題,並且照服務時的格式加上檢索前綴。 IQ2_M的一致度只有0.87,對檢索來說名次會明顯亂掉,照慣例一併提供,但請不要拿它建索引。
照片與語音在llama.cpp上的表現
圖片向量和原始權重逐張比,cosine在0.92到0.96之間,BF16與Q8_0一樣,代表差距不是量化造成,而是llama.cpp與原始實作的圖片前處理不同。 光看cosine無法判斷檢索會不會變差,所以我們把整套照片測試都在llama.cpp上重跑一次,查詢端與文件端都走GGUF:
| 測試 | 原始權重 | GGUF BF16 | GGUF Q4_K_M |
|---|---|---|---|
| XM3600:中文圖說找照片 | 0.6745 | 0.6641 | 0.6620 |
| XM3600:照片找中文圖說 | 0.6822 | 0.6881 | 0.6831 |
| 台灣老照片:標題找照片 | 0.5145 | 0.5395 | 0.5385 |
| 台灣老照片:照片找標題 | 0.5185 | 0.5355 | 0.5325 |
四項中三項持平或略高,中文圖說找照片低約1個百分點。Q4_K_M和BF16的差距都在0.5個百分點以內。 語音向量和原始權重的cosine最低0.995(BF16),Q4_K_M最低0.972。
這組數字是在每張圖固定280個圖片token的設定下量的,部署時請用同樣的設定(見下方指令)。
用法
前綴(必讀)
本模型沿用EmbeddingGemma 2的前綴,建索引與查詢時都要加,而且要和下面一字不差,用錯不會出現錯誤訊息,只會讓檢索無聲變差:
- 查詢:
task: search result | query: - 文件:
title: none | text:
照片與語音本身不加前綴。用照片找文字時,照片是查詢、文字端照樣加文件前綴。
llama-server(文字)
llama-server -m xVITA-Embed2-zhTW-Q8_0.gguf --embeddings --port 8080
curl -s localhost:8080/v1/embeddings -H 'Content-Type: application/json' \
-d '{"input": "task: search result | query: 玉山有多高?"}'
池化與正規化已寫進GGUF檔頭,不需要另外指定--pooling。
llama-server(照片與語音)
llama-server -m xVITA-Embed2-zhTW-Q8_0.gguf --mmproj xVITA-Embed2-zhTW-mmproj-BF16.gguf \
--embeddings --image-min-tokens 280 --image-max-tokens 280 --port 8080
送照片或語音要走原生的/embeddings端點。媒體標記由伺服器每次啟動時產生,要先從/props讀media_marker:
import base64, json, urllib.request
URL = "http://localhost:8080"
marker = json.load(urllib.request.urlopen(URL + "/props"))["media_marker"]
def embed_media(path):
body = {"content": {"prompt_string": marker,
"multimodal_data": [base64.b64encode(open(path, "rb").read()).decode()]}}
req = urllib.request.Request(URL + "/embeddings", data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
r = json.load(urllib.request.urlopen(req))
e = r[0]["embedding"] if isinstance(r, list) else r["embedding"]
return e[0] if isinstance(e[0], list) else e # 依版本可能多包一層
vec = embed_media("photo.jpg") # 語音檔(wav)用同一個函式
Matryoshka截維
本模型保留底模的截維能力,向量取前N維再重新正規化即可。訓練時的開發集(500題)上,768維hit@1為0.778,512維0.762,256維0.740,128維0.640。 最高就是768維,沒有更高維度的版本。
訓練方式
只訓練文字主幹的Transformer層(約137M參數),圖片編碼器、語音編碼器與詞嵌入(合計約608M參數)全部凍結。 損失函數是對比式學習,困難負例由底模自己挖。為了保住底模原本的能力,另外放了兩組錨點: 3,000段語音與5,000句台灣新聞句子,老師向量取自訓練前的底模,作用是拉住這些輸入的向量不要漂走,本身不是配對訓練料。
訓練完成後,在底模與微調後的權重之間做線性內插,從1.0到0.5每0.05取一點,共11點,每一點都量完整套跨模態測試。 選取規則是先過門檻(每項跨模態退步不超過1個百分點),再取文字分數最高的一點,最後選定的是微調權重0.6、底模0.4。
訓練資料
| 資料 | 用途 | 筆數 | 授權 |
|---|---|---|---|
| DRCD訓練集(台達研究院) | 中文段落檢索 | 26,936題 | CC BY-SA 3.0 |
| 國家文化記憶庫(文化部) | 標題配照片、照片配說明 | 21,472對 | 公眾領域、CC0、CC BY、政府資料開放授權 |
| 行政院公報 | 主旨配頁面圖、頁面圖配全文 | 6,000對 | 政府公開資訊 |
| Common Voice 19中文(台灣) | 語音配逐字稿 | 21,843對 | CC0 |
國家文化記憶庫的素材只收圖像授權與文字授權兩者都允許商業使用的,非商業、禁止改作、相同方式分享三類一律不收, 每筆素材頁上的授權在收錄時都再檢查一次。
限制
- 影片輸入沒有測試過。底模支援影片,本模型未針對影片調整,也沒有量過。
- 檢索分數是在新聞、公報、百科類段落與台灣歷史照片上量的,換成專業領域文件(醫療、法律、程式碼)時請先用自己的資料驗證。
- 語音測試的保留條件見「成績」一節,對語音檢索的數字請保守解讀。
授權
本模型是google/embeddinggemma-2的衍生物,依該模型的Apache License 2.0釋出。
關於云碩科技
云碩科技(xCloudinfo Corp. Limited)專注於地端私有化人工智慧解決方案。xVITA是云碩自研模型的品牌名稱。
其他模型請見 https://hf.co/xCloudinfo。
xVITA-Embed2-zhTW-GGUF — English
A multimodal embedding model for Traditional Chinese (Taiwan) retrieval, which outputs text, photos and speech into the same 768-dimensional vector space, fine-tuned by xCloudinfo (云碩科技) on google/embeddinggemma-2.
This repository provides the GGUF format, with seven precision tiers for the main text model, plus image and speech encoders (mmproj), runnable on-premises with llama.cpp.
This is an embedding model; it outputs vectors, it does not generate text.
What this one solves
EmbeddingGemma 2 puts text, photos and speech into the same vector space, so a single index can find photos from text, find captions from photos, and find transcripts from recordings. When we connected it to a Traditional Chinese knowledge base, we hit two problems: the hit rate of Chinese passage retrieval lost to our previous in-house model, and Chinese speech could hardly find the corresponding text at all.
Fine-tuning directly on Traditional Chinese corpus can raise the text score, but the cost shows up elsewhere. In an early round of experiments, text passage retrieval rose from 0.8125 to 0.8780, but the same checkpoint, finding photos from Chinese captions, dropped from 0.6726 to 0.4934, and finding Chinese captions from photos also dropped from 0.6394 to 0.3864. The backbone is pulled away by Traditional Chinese text, and the image-text alignment scatters along with it. Looking only at the text score, this one looks like an improvement.
So the release condition for this one is a threshold, not a single score: the regression of every cross-modal retrieval item relative to the base must not exceed 1 percentage point, and only after passing the threshold do we compare text scores. To keep pushing the text score up under this threshold, we rely on Taiwan's own caption data: we used a browser to collect, page by page, the commercially-licensed materials in the National Cultural Memory Bank (國家文化記憶庫), turned them into training pairs of title-with-photo and photo-with-caption, and also made the short titles into reverse-direction pairs of photo-finding-title. Without that reverse half, finding Chinese captions from photos stayed stuck 3.6 percentage points below the base; once added, it was instead 4.3 percentage points above the base.
Results
The following are the results of the original weights (safetensors) on the same measuring stick, with hit@1 as the metric throughout. None of the test sets entered training: DRCD uses the official test split; the photo and speech test samples were held out before the training data was built.
| Test | Base embeddinggemma-2 | This model | Diff |
|---|---|---|---|
| DRCD passage retrieval (3,493 questions) | 0.8125 | 0.8683 | +5.58 |
| DRCD whole-article retrieval (3,493 questions) | 0.8128 | 0.8299 | +1.71 |
| XM3600: find photos from Chinese captions | 0.6726 | 0.6745 | +0.19 |
| XM3600: find Chinese captions from photos | 0.6394 | 0.6822 | +4.28 |
| XM3600: find photos from English captions | 0.5946 | 0.5853 | −0.93 |
| XM3600: find English captions from photos | 0.6183 | 0.6236 | +0.53 |
| Taiwan old photos: find photos from titles (999 photos) | 0.3353 | 0.5145 | +17.92 |
| Taiwan old photos: find titles from photos (999 photos) | 0.3624 | 0.5185 | +15.61 |
| Executive Yuan Gazette (行政院公報): find page images from subject lines | 0.9100 | 0.9740 | +6.40 |
| Executive Yuan Gazette (行政院公報): find full text from page images | 0.9950 | 0.9970 | +0.20 |
| Common Voice Chinese: find text from speech | 0.2452 | 0.7638 | +51.86 |
| Common Voice Chinese: find speech from text | 0.4209 | 0.7732 | +35.23 |
The unit of the diff is percentage points. XM3600 is 3,600 photos and 7,174 Chinese captions. "Taiwan old photos" is a held-out set from the National Cultural Memory Bank (國家文化記憶庫); the materials never entered training.
Those two speech rows should be read conservatively. The training speech also comes from Common Voice, and the test only excludes sentences, not speakers, so part of these two rows is "the same batch of people's voices". We separately validated with a set of Chinese recordings from a different source that never entered training, and the improvement in speech retrieval holds there too, but that set of recordings is not public, and the numbers are not listed here.
Finding photos from English captions is the only item that regressed, by 0.93 percentage points, within the threshold.
Compared with other embedding models
We measured six models with the same program, split into two measuring sticks: DRCD represents the home ground of Traditional Chinese, and C-MTEB represents general web retrieval in Simplified Chinese.
| Model | DRCD passage | DRCD whole-article | C-MTEB original | C-MTEB converted to Traditional |
|---|---|---|---|---|
| Llama-xVITA-Embed-1B-zhTW (xCloudinfo, 2048-dim) | 0.8809 | 0.8586 | 56.85 | 55.78 |
| This model (768-dim) | 0.8677 | 0.8305 | 61.79 | 60.73 |
| xVITA-Embed-300M-zhTW (xCloudinfo previous generation, 768-dim) | 0.8600 | 0.7927 | 58.90 | 57.22 |
| intfloat/multilingual-e5-large (1024-dim) | 0.8285 | 0.7060 | 63.66 | 61.57 |
| BAAI/bge-m3 (1024-dim) | 0.8219 | 0.7741 | 65.32 | 63.71 |
| google/embeddinggemma-2 (base, 768-dim) | 0.8125 | 0.8128 | 61.26 | 59.87 |
DRCD is hit@1; C-MTEB is the nDCG@10 average of eight Chinese retrieval tasks (T2, MMarco, Du, Covid, Cmedqa, Ecom, Medical, Video), and "converted to Traditional" is measuring the same questions and documents again after converting them to Traditional-Chinese glyphs with OpenCC. C-MTEB is always truncated at 512 tokens, and the average measured for multilingual-e5-large on this stick is 63.66, the same as the number officially published by MTEB, which we use to confirm that this stick can be compared against public leaderboards. bge-m3 is officially measured with 8192 tokens; truncated at 512 here, its numbers are a bit lower than official. DRCD whole-article retrieval reads up to 8192 tokens, and multilingual-e5-large can only take 512, so its cell there is on the low side.
The two sticks reach different conclusions. On Traditional Chinese, this model beats both multilingual-e5-large and bge-m3 on passage and whole-article retrieval; on Simplified general web retrieval, this model ranks third in the table, behind multilingual-e5-large by about 1.9 points, with the biggest gap on the Baidu-search task DuRetrieval (75.19 vs. 85.31). It wins against multilingual-e5-large on three of the eight: T2Retrieval (76.79 vs. 76.04), EcomRetrieval (60.67 vs. 54.86) and VideoRetrieval (60.49 vs. 58.26).
If you do pure text only and often need to retrieve whole long documents, our Llama-xVITA-Embed-1B-zhTW is still higher than this model on DRCD, 2.8 percentage points higher on whole-article retrieval; this model's strengths are putting text, photos and speech in the same vector space at once, and being only 768-dimensional with a Q8_0 size of only 310 MB.
Files
Text consistency is the same batch of 20 Chinese sentences sent to the GGUF and the original weights separately, comparing the cosine of the vectors from both sides.
| File | Size | Cosine with original weights (min / mean) | Recommendation |
|---|---|---|---|
xVITA-Embed2-zhTW-BF16.gguf |
558 MB | 0.99991 / 0.99998 | use when you want closest to the original weights |
xVITA-Embed2-zhTW-Q8_0.gguf |
310 MB | 0.99934 / 0.99971 | general deployment first choice |
xVITA-Embed2-zhTW-Q6_K.gguf |
246 MB | 0.99406 / 0.99852 | |
xVITA-Embed2-zhTW-Q5_K_M.gguf |
213 MB | 0.99452 / 0.99653 | |
xVITA-Embed2-zhTW-Q4_K_M.gguf |
182 MB | 0.97281 / 0.98884 | use when memory is tight |
xVITA-Embed2-zhTW-IQ4_XS.gguf |
169 MB | 0.97969 / 0.98711 | |
xVITA-Embed2-zhTW-IQ2_M.gguf |
131 MB | 0.87301 / 0.89867 | not recommended for retrieval |
xVITA-Embed2-zhTW-mmproj-BF16.gguf |
982 MB | see next section | only needed when processing photos or speech |
Q6_K and below are all quantized with imatrix; the calibration corpus is 700 lines of Traditional Chinese passages, questions, news sentences and photo captions, with the retrieval prefix added in the service-time format. IQ2_M's consistency is only 0.87, which clearly scrambles the ranking for retrieval; it is provided as usual by convention, but please do not use it to build an index.
Photos and speech on llama.cpp
Comparing image vectors against the original weights photo by photo, the cosine is between 0.92 and 0.96, the same for BF16 and Q8_0, which means the gap is not caused by quantization but by llama.cpp's image preprocessing differing from the original implementation. Cosine alone cannot tell whether retrieval gets worse, so we re-ran the whole photo test once on llama.cpp, with both the query side and the document side going through GGUF:
| Test | Original weights | GGUF BF16 | GGUF Q4_K_M |
|---|---|---|---|
| XM3600: find photos from Chinese captions | 0.6745 | 0.6641 | 0.6620 |
| XM3600: find Chinese captions from photos | 0.6822 | 0.6881 | 0.6831 |
| Taiwan old photos: find photos from titles | 0.5145 | 0.5395 | 0.5385 |
| Taiwan old photos: find titles from photos | 0.5185 | 0.5355 | 0.5325 |
Three of the four are level or slightly higher; finding photos from Chinese captions is about 1 percentage point lower. The gap between Q4_K_M and BF16 is within 0.5 percentage points in all cases. The cosine of speech vectors with the original weights is at least 0.995 (BF16), and at least 0.972 for Q4_K_M.
This set of numbers was measured in a setting of a fixed 280 image tokens per image; deploy with the same setting (see the command below).
Usage
Prefix (must read)
This model keeps EmbeddingGemma 2's prefix; it must be added both when building the index and when querying, and must match the following character for character. Getting it wrong produces no error message, it only makes retrieval silently worse:
- Query:
task: search result | query: - Document:
title: none | text:
Photos and speech themselves take no prefix. When finding text from a photo, the photo is the query, and the text side still adds the document prefix.
llama-server (text)
llama-server -m xVITA-Embed2-zhTW-Q8_0.gguf --embeddings --port 8080
curl -s localhost:8080/v1/embeddings -H 'Content-Type: application/json' \
-d '{"input": "task: search result | query: 玉山有多高?"}'
Pooling and normalization are already written into the GGUF header, so --pooling does not need to be specified separately.
llama-server (photos and speech)
llama-server -m xVITA-Embed2-zhTW-Q8_0.gguf --mmproj xVITA-Embed2-zhTW-mmproj-BF16.gguf \
--embeddings --image-min-tokens 280 --image-max-tokens 280 --port 8080
Sending photos or speech goes through the native /embeddings endpoint. The media marker is generated by the server on each startup, so first read media_marker from /props:
import base64, json, urllib.request
URL = "http://localhost:8080"
marker = json.load(urllib.request.urlopen(URL + "/props"))["media_marker"]
def embed_media(path):
body = {"content": {"prompt_string": marker,
"multimodal_data": [base64.b64encode(open(path, "rb").read()).decode()]}}
req = urllib.request.Request(URL + "/embeddings", data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
r = json.load(urllib.request.urlopen(req))
e = r[0]["embedding"] if isinstance(r, list) else r["embedding"]
return e[0] if isinstance(e[0], list) else e # 依版本可能多包一層
vec = embed_media("photo.jpg") # 語音檔(wav)用同一個函式
Matryoshka truncation
This model keeps the base's truncation ability: take the first N dimensions of the vector and re-normalize. On the development set used during training (500 questions), 768-dim hit@1 is 0.778, 512-dim 0.762, 256-dim 0.740, 128-dim 0.640. 768-dim is the highest; there is no higher-dimensional version.
Training method
Only the Transformer layers of the text backbone (about 137M parameters) are trained; the image encoder, speech encoder and token embeddings (about 608M parameters in total) are all frozen. The loss function is contrastive learning, with hard negatives mined by the base itself. To preserve the base's original ability, two sets of anchors are also placed: 3,000 speech segments and 5,000 Taiwanese news sentences, with teacher vectors taken from the base before training, serving to hold the vectors of these inputs from drifting away, and are not themselves paired training data.
After training is complete, a linear interpolation is done between the base and the fine-tuned weights, taking one point every 0.05 from 1.0 to 0.5, 11 points in all, and the full set of cross-modal tests is measured at each point. The selection rule is to first pass the threshold (each cross-modal regression not exceeding 1 percentage point), then take the point with the highest text score; the final choice is fine-tuned weight 0.6, base 0.4.
Training data
| Data | Use | Count | License |
|---|---|---|---|
| DRCD training set (Delta Research Center, 台達研究院) | Chinese passage retrieval | 26,936 questions | CC BY-SA 3.0 |
| National Cultural Memory Bank (Ministry of Culture, 文化部) | title-with-photo, photo-with-caption | 21,472 pairs | public domain, CC0, CC BY, Open Government Data License |
| Executive Yuan Gazette (行政院公報) | subject-with-page-image, page-image-with-full-text | 6,000 pairs | government public information |
| Common Voice 19 Chinese (Taiwan) | speech-with-transcript | 21,843 pairs | CC0 |
The National Cultural Memory Bank materials only take those where both the image license and the text license allow commercial use; the three categories of non-commercial, no-derivatives and share-alike are not taken at all, and the license on each material's page is re-checked at the time of collection.
Limitations
- Video input has not been tested. The base supports video, but this model is not adjusted for video and has not been measured on it.
- The retrieval scores were measured on news, gazette and encyclopedia-type passages and Taiwanese historical photos; for professional-domain documents (medical, legal, code), please first validate with your own data.
- See the "Results" section for the hold-out condition of the speech test, and read the speech-retrieval numbers conservatively.
License
This model is a derivative of google/embeddinggemma-2, released under that model's Apache License 2.0.
About xCloudinfo
xCloudinfo Corp. Limited (云碩科技) builds on-premises, private AI solutions. xVITA is xCloudinfo's in-house model brand.
More models: https://hf.co/xCloudinfo