Overview of bert-japanese-12M

• The bert-japanese-12M model is a transformer-based model with BERT architecture, which is designed to be used on Japanese text. In addition, it is small enough to be run on consumer low-end GPUs, or even CPUs with acceptable latency.
• The model was trained from scratch on an cc100-ja and Japanese Wikipedia Cirrussearch 20240701 dump file.
• The pre-training process of this model heavily follows the practices of bert-japanese but it was performed from scratch with the training data was randomed differently, and tokenizer's vocab is also different.
• This model is designed based on the tradeoff between accuracy and performance.

Model description

• The model has BERT architecture with the following details:
  • Model size: 12,963,584 parameters
  • Output vector size (same as hidden size): 256
  • Number of layers: 6
  • Max position embeddings: 2048 (it means the model can consume inputs with length up to 2048 tokens)
• The model employs BertJapaneseTokenizer as its tokenizer
  • Word-level tokenization with MeCab wrapped by fugashi.
  • WordPiece is used for subword tokenization.
  • Tokenizer's vocabulary has 32,768 tokens.

Intended uses & limitations

• Example code:

from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline
MODEL_PATH = "nptdat/bert-japanese-12M"
model = AutoModelForMaskedLM.from_pretrained(MODEL_PATH, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)

pipe = pipeline("fill-mask", model=model, tokenizer=tokenizer)
text = "東京は[MASK]の首都です。"
pipe(text)

• Because this model is very small, its accuracy may be low compared with other bigger models. However, it is light and can be run on low-end GPUs, or even CPUs to cut inference cost.
• We recommend the following practices when using this model
  • Use the model on specific domain, where small models can work well
  • Use the model on use-cases where low inference cost and low latency are necessary
  • Should fine-tune the model on downstream task before using

Training procedure

• Tokenizer training
  • The tokenizer is trained from 10M Japanese Wikipedia sentences.
• Model training are performed 2 phases on different datasets
  • First phase: pre-training on cc100-ja dataset
    • We pre-trained a model from scratch with 122M sentences from cc100-ja (67GB of text), validated on 1M sentences.
  • Second phase: pre-training on Japanese Wikipedia dataset
    • The model was then pre-trained more on 38M Wikipedia sentences (5.6GB of text), validated on 1.2M sentences.
• Both phrases of training are performed on a single consumer Geforce RTX 3090 GPU. It took about 34 hours and 11 hours respectively.

Training hyperparameters

The following hyperparameters were used during training on both datasets

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.08
• num_epochs: 1
• mixed_precision_training: Native AMP

Training results

This is the results of the 2nd phase, when the model was trained and validated on Japanese Wikipedia text.

Framework versions

• Transformers 4.42.0
• Pytorch 2.3.1+cu121
• Datasets 2.18.0
• Tokenizers 0.19.1