gpt2 (Quantized)

Description

This model is a 4-bit quantized version of the original gpt2 model, optimized for reduced memory usage while maintaining performance.

Quantization Details

• Quantization Type: 4-bit
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: bfloat16
• bnb_4bit_quant_storage: uint8
• Original Footprint: 510.34 MB (FLOAT32)
• Quantized Footprint: 213.07 MB
• Memory Reduction: 58.2%

Usage

from transformers import AutoModel, AutoTokenizer

repo_id = "manu02/gpt2-bnb-4bit-nf4"

model = AutoModel.from_pretrained(
    repo_id,
    device_map="auto",
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)