LLaVA-via: Fine-Tuned LoRA Adapter for Scene Narration and Navigation Guidance

Overview

LLaVA-via is a PEFT LoRA adapter fine-tuned on top of LLaVA-1.5-7B to improve scene narration and navigation guidance for assistive vision-language applications.

The objective of this work is to generate contextually grounded, coherent, and navigation-aware responses that can support visually impaired assistance scenarios. Rather than answering general visual questions, the adapter is optimized to describe surrounding environments and provide guidance-oriented information from input images.

This repository contains only the LoRA adapter and associated tokenizer/processor configuration files. It does not contain the full LLaVA model weights.

---

Base Model

This adapter must be loaded on top of the following Hugging Face model:

Base Model

llava-hf/llava-1.5-7b-hf

The base model is not included in this repository.

---

Repository Contents

This repository includes:

• adapter_model.safetensors — Fine-tuned LoRA adapter weights
• adapter_config.json — PEFT configuration
• Tokenizer files
• Processor configuration
• Image preprocessor configuration
• Chat template
• Model card (README)

Training checkpoints, optimizer states, and scheduler files are intentionally excluded.

---

Intended Use

LLaVA-via is intended for research and development involving assistive vision-language systems, including:

• Scene narration
• Environmental understanding
• Navigation guidance
• Assistive AI research
• Vision-language model benchmarking

Example applications include:

• Assistive mobile applications
• Smart glasses
• Vision-language research
• Accessibility-focused AI systems
• Human-centered AI

---

Fine-Tuning Method

• Architecture: LLaVA-1.5-7B
• Fine-tuning: PEFT LoRA
• Task: Vision-Language Conditional Generation
• LoRA Rank (r): 16
• LoRA Alpha: 32
• Target Modules: q_proj, v_proj
• LoRA Dropout: 0.05

---

Evaluation

The fine-tuned adapter was evaluated using multiple automatic metrics covering lexical similarity, semantic similarity, vision-language alignment, and hallucination analysis.

The evaluation includes lexical overlap metrics (BLEU, METEOR, ROUGE-L), semantic similarity metrics (BERTScore and SBERTScore), vision-language alignment metrics (CLIPScore and PickScore), a task-specific evaluation (SPECS-VIA and PickScore-VIA), and hallucination analysis using the semantically refined CHAIR-i metric.

---

Loading the Adapter

The adapter can be loaded using the Hugging Face Transformers and PEFT libraries.

from transformers import AutoProcessor, LlavaForConditionalGeneration
from peft import PeftModel

base_model = LlavaForConditionalGeneration.from_pretrained(
    "llava-hf/llava-1.5-7b-hf"
)

processor = AutoProcessor.from_pretrained(
    "llava-hf/llava-1.5-7b-hf"
)

model = PeftModel.from_pretrained(
    base_model,
    "SobanHM/LLaVA-via"
)

---

Example Use Cases

The adapter is designed for prompts such as:

• Describe the surrounding environment.
• Identify important objects in the scene.
• Provide navigation guidance.
• Describe possible obstacles.
• Explain the scene for a visually impaired user.

---

Limitations

This repository provides a fine-tuned PEFT LoRA adapter evaluated using offline automatic metrics.

The reported evaluation does not include:

• Real-time deployment on electronic white canes
• Smart glasses deployment
• Mobile-device latency evaluation
• Embedded hardware benchmarking
• User studies involving visually impaired participants

Accordingly, inference latency, energy consumption, real-time responsiveness, and human-subject usability remain outside the scope of the reported experiments.

---

Citation

If you use this repository in your research, please cite this work and the original LLaVA and PEFT publications.

A BibTeX entry for this work may be added once the associated thesis or publication becomes available.

---

Authors

Soban Hussain Department of Computer Science Sukkur IBA University

Praih Alias Faiza Department of Computer Science Sukkur IBA University

Tasmia Department of Computer Science Sukkur IBA University

---

Acknowledgements

This work builds upon the following open-source projects:

• LLaVA
• Hugging Face Transformers
• PEFT
• PyTorch

We gratefully acknowledge the developers and research community whose contributions made this work possible.