Sign up

Jimmi42/MonkeyOCR-Apple-Silicon

🤗 Hugging Face sourcemit14 MBother⚠ No SHA-256 weightsupdated today
No torrent yet →

🚀 MonkeyOCR-MLX: Apple Silicon Optimized OCR

A high-performance OCR application optimized for Apple Silicon with MLX-VLM acceleration, featuring advanced document layout analysis and intelligent text extraction.

🔥 Key Features

  • ⚡ MLX-VLM Optimization: Native Apple Silicon acceleration using MLX framework
  • 🚀 3x Faster Processing: Compared to standard PyTorch on M-series chips
  • 🧠 Advanced AI: Powered by Qwen2.5-VL model with specialized layout analysis
  • 📄 Multi-format Support: PDF, PNG, JPG, JPEG with intelligent structure detection
  • 🌐 Modern Web Interface: Beautiful Gradio interface for easy document processing
  • 🔄 Batch Processing: Efficient handling of multiple documents
  • 🎯 High Accuracy: Specialized for complex financial documents and tables
  • 🔒 100% Private: All processing happens locally on your Mac

📊 Performance Benchmarks

Test: Complex Financial Document (Tax Form)

  • MLX-VLM: ~15-18 seconds ⚡
  • Standard PyTorch: ~25-30 seconds
  • CPU Only: ~60-90 seconds

MacBook M4 Pro Performance:

  • Model loading: ~1.7s
  • Text extraction: ~15s
  • Table structure: ~18s
  • Memory usage: ~13GB peak

🛠 Installation

Prerequisites

  • macOS with Apple Silicon (M1/M2/M3/M4)
  • Python 3.11+
  • 16GB+ RAM (32GB+ recommended for large documents)

Quick Setup

  1. Clone the repository:

    git clone https://huggingface.co/Jimmi42/MonkeyOCR-Apple-Silicon
cd MonkeyOCR-Apple-Silicon

  2. Run the automated setup script:

    chmod +x setup.sh
./setup.sh

    This script will automatically:

    • Download MonkeyOCR from the official GitHub repository
    • Apply MLX-VLM optimization patches for Apple Silicon
    • Enable smart backend auto-selection (MLX/LMDeploy/transformers)
    • Install UV package manager if needed
    • Set up virtual environment with Python 3.11
    • Install all dependencies including MLX-VLM
    • Download required model weights
    • Configure optimal backend for your hardware

  3. Alternative manual installation:

    # Install UV if not already installed
curl -LsSf https://astral.sh/uv/install.sh | sh

# Download MonkeyOCR
git clone https://github.com/Yuliang-Liu/MonkeyOCR.git MonkeyOCR

# Install dependencies (includes mlx-vlm)
uv sync

# Download models
cd MonkeyOCR && python tools/download_model.py && cd ..

🏃‍♂️ Usage

Web Interface (Recommended)

# Activate virtual environment
source .venv/bin/activate  # or `uv shell`

# Start the web app
python app.py

Access the interface at http://localhost:7861

Command Line

python main.py path/to/document.pdf

⚙️ Configuration

Smart Backend Selection (Default)

The app automatically detects your hardware and selects the optimal backend:

# model_configs_mps.yaml
device: mps
chat_config:
  backend: auto  # Smart auto-selection
  batch_size: 1
  max_new_tokens: 256
  temperature: 0.0

Auto-Selection Logic:

  • 🍎 Apple Silicon (MPS) → MLX-VLM (3x faster)
  • 🖥️ CUDA GPU → LMDeploy (optimized for NVIDIA)
  • 💻 CPU/Fallback → Transformers (universal compatibility)

Performance Backends

Backend Speed Memory Best For Auto-Selected
auto ⚡ 🧠 All systems (Recommended) ✅ Default
mlx 🚀🚀🚀 🟢 Apple Silicon 🍎 Auto for MPS
lmdeploy 🚀🚀 🟡 CUDA systems 🖥️ Auto for CUDA
transformers 🚀 🟢 Universal fallback 💻 Auto for CPU

🧠 Model Architecture

Core Components

  • Layout Detection: DocLayout-YOLO for document structure analysis
  • Vision-Language Model: Qwen2.5-VL with MLX optimization
  • Layout Reading: LayoutReader for reading order optimization
  • MLX Framework: Native Apple Silicon acceleration

Apple Silicon Optimizations

  • Metal Performance Shaders: Direct GPU acceleration
  • Unified Memory: Optimized memory access patterns
  • Neural Engine: Utilizes Apple's dedicated AI hardware
  • Float16 Precision: Optimal speed/accuracy balance

🎯 Perfect For

Document Types:

  • 📊 Financial Documents: Tax forms, invoices, statements
  • 📋 Legal Documents: Contracts, forms, certificates
  • 📄 Academic Papers: Research papers, articles
  • 🏢 Business Documents: Reports, presentations, spreadsheets

Advanced Features:

  • ✅ Complex table extraction with highlighted cells
  • ✅ Multi-column layouts and mixed content
  • ✅ Mathematical formulas and equations
  • ✅ Structured data output (Markdown, JSON)
  • ✅ Batch processing for multiple files

🚨 Troubleshooting

MLX-VLM Issues

# Test MLX-VLM availability
python -c "import mlx_vlm; print('✅ MLX-VLM available')"

# Check if auto backend selection is working
python -c "
from MonkeyOCR.magic_pdf.model.custom_model import MonkeyOCR
model = MonkeyOCR('model_configs_mps.yaml')
print(f'Selected backend: {type(model.chat_model).__name__}')
"

Performance Issues

# Check MPS availability
python -c "import torch; print(f'MPS available: {torch.backends.mps.is_available()}')"

# Monitor memory usage during processing
top -pid $(pgrep -f "python app.py")

Common Solutions

  1. Patches Not Applied:

    • Re-run ./setup.sh to reapply patches
    • Check that MonkeyOCR directory exists and has our modifications
    • Verify MonkeyChat_MLX class exists in MonkeyOCR/magic_pdf/model/custom_model.py

  2. Wrong Backend Selected:

    • Check hardware detection with python -c "import torch; print(torch.backends.mps.is_available())"
    • Verify MLX-VLM is installed: pip install mlx-vlm
    • Use backend: mlx in config to force MLX backend

  3. Slow Performance:

    • Ensure auto-selection chose MLX backend on Apple Silicon
    • Check Activity Monitor for MPS GPU usage
    • Verify backend: auto in model_configs_mps.yaml

  4. Memory Issues:

    • Reduce image resolution before processing
    • Close other memory-intensive applications
    • Reduce batch_size to 1 in config

  5. Port Already in Use:

    GRADIO_SERVER_PORT=7862 python app.py

📁 Project Structure

MonkeyOCR-MLX/
├── 🌐 app.py                    # Gradio web interface
├── 🖥️ main.py                   # CLI interface  
├── ⚙️ model_configs_mps.yaml    # MLX-optimized config
├── 📦 requirements.txt          # Dependencies (includes mlx-vlm)
├── 🛠️ torch_patch.py           # Compatibility patches
├── 🧠 MonkeyOCR/               # Core AI models
│   └── 🎯 magic_pdf/           # Processing engine
├── 📄 .gitignore               # Git ignore rules
└── 📚 README.md                # This file

🔥 What's New in MLX Version

  • ✨ Smart Patching System: Automatically applies MLX-VLM optimizations to official MonkeyOCR
  • 🧠 Intelligent Backend Selection: Auto-detects hardware and selects optimal backend
  • 🚀 3x Faster Processing: MLX-VLM acceleration on Apple Silicon
  • 💾 Better Memory Efficiency: Optimized for unified memory architecture
  • 🎯 Improved Accuracy: Enhanced table and structure detection
  • 🔧 Zero Configuration: Works out-of-the-box with smart defaults
  • 📊 Performance Monitoring: Built-in timing and metrics
  • 🛠️ Latest Fix (June 2025): Resolved MLX-VLM prompt formatting for optimal OCR output
  • 🔄 Always Up-to-Date: Uses official MonkeyOCR repository with our patches applied

🔬 Technical Implementation

Smart Patching System

  • Dynamic Code Injection: Automatically adds MLX-VLM class to official MonkeyOCR
  • Backend Selection Logic: Patches smart hardware detection into initialization
  • Zero Maintenance: Always uses latest official MonkeyOCR with our optimizations
  • Seamless Integration: Patches are applied transparently during setup

MLX-VLM Backend (MonkeyChat_MLX)

  • Direct MLX framework integration
  • Optimized for Apple's Metal Performance Shaders
  • Native unified memory management
  • Specialized prompt processing for OCR tasks
  • Fixed prompt formatting for optimal output quality

Intelligent Fallback System

  • Hardware Detection: MPS → MLX, CUDA → LMDeploy, CPU → Transformers
  • Graceful Degradation: Falls back to compatible backends if preferred unavailable
  • Cross-Platform: Maintains compatibility across all systems
  • Error Recovery: Automatic fallback on initialization failures

🤝 Contributing

We welcome contributions! Please:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit changes (git commit -m 'Add amazing feature')
  4. Push to branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

  • Apple MLX Team: For the incredible MLX framework
  • MonkeyOCR Team: For the foundational OCR model
  • Qwen Team: For the excellent Qwen2.5-VL model
  • Gradio Team: For the beautiful web interface
  • MLX-VLM Contributors: For the MLX vision-language integration

📞 Support

🚀 Supercharged for Apple Silicon • Made with ❤️ for the MLX Community

Experience the future of OCR with native Apple Silicon optimization