NanoLlama (Public)

A compact Llama-based language model optimized for efficient inference and deployment. This is the public version with open access.

Model Details

Model Description

NanoLlama is a small-scale language model based on the Llama architecture, designed for lightweight applications and resource-constrained environments. This model provides a good balance between performance and computational efficiency.

• Developed by: svc-nai-cci
• Model type: Causal Language Model
• Language(s): English
• License: Apache 2.0
• Finetuned from: Llama architecture
• Access: Public (Open Access)

Model Architecture

• Architecture: LlamaForCausalLM
• Hidden Size: 4096
• Number of Layers: 4
• Number of Attention Heads: 4
• Number of Key-Value Heads: 2
• Vocabulary Size: 32000
• Max Position Embeddings: 4096
• Hidden Activation: SiLU

Uses

Direct Use

This model can be used for:

• Text generation
• Conversational AI
• Code completion
• Creative writing
• Question answering

Downstream Use

The model can be fine-tuned for specific tasks such as:

• Domain-specific text generation
• Task-specific instruction following
• Specialized conversational agents

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load the model and tokenizer
model_name = "svc-nai-cci/nanollama-public"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Generate text
input_text = "Hello, how are you?"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100, temperature=0.7)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

Technical Specifications

Model Architecture and Objective

The model uses the standard Llama architecture with:

• RMSNorm for layer normalization
• RoPE (Rotary Position Embedding) for positional encoding
• SwiGLU activation function
• Grouped Query Attention (GQA)

Performance Characteristics

• Model Size: Compact design for efficient deployment
• Memory Requirements: Optimized for low-memory environments
• Inference Speed: Fast inference suitable for real-time applications

Limitations

• Limited context length (4096 tokens)
• May not perform as well as larger models on complex reasoning tasks
• Primarily trained/fine-tuned for English text

Citation

If you use this model, please cite:

@misc{nanollama2024,
  title={NanoLlama: A Compact Llama-based Language Model},
  author={svc-nai-cci},
  year={2024},
  url={https://huggingface.co/svc-nai-cci/nanollama-public}
}

Contact

For questions or issues, please contact: svc-nai-cci