bert-tiny

google/bert_uncased_L-4_H-256_A-4 (BERT-Mini: 4 layers, hidden size 256, 4 attention heads, ~11M parameters), converted to model.safetensors (float32) with its masked language modeling head.

Used in the Transformer Architecture course at Lviv Polytechnic, where students implement BERT inference from scratch in pure Python.

The weights are unchanged. The MLM decoder is tied to bert.embeddings.word_embeddings.weight, so it is not stored separately.

Original paper: Turc et al., Well-Read Students Learn Better: On the Importance of Pre-training Compact Models (2019).

Papers

• Vaswani et al., Attention Is All You Need (2017). Переклад українською (PDF)
• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019). Переклад українською (PDF)