Voice Assistant with RAG

This project implements a voice assistant that uses Retrieval-Augmented Generation (RAG) and speech recognition to provide responses to user queries. The assistant can listen to voice input, process it, and respond with synthesized speech based on the knowledge base you passed.

Features

• Speech recognition using Google Speech Recognition
• Text-to-Speech (TTS) using Mozilla TTS
• RAG-based question answering using Langchain and FAISS
• Integration with Ollama for language model processing

Prerequisites

Before running this project, make sure you have the following dependencies installed:

• Python 3.7+
• PyTorch
• Transformers
• SpeechRecognition
• soundfile
• playsound
• TTS
• Langchain
• FAISS
• Ollama

How to get started with project

1. Clone this repository.

git clone https://huggingface.co/foduucom/Voice-Assistant-using-RAG

2. Create conda environment.

conda create -n VoiceAI python==3.10
conda activate VoiceAI

3. You can install most of these dependencies using pip:

pip install -r requirements.txt

For Ollama, follow the installation instructions on their official website https://ollama.com/library/llama3.

Setup

1. Clone this repository to your local machine.
2. Install the required dependencies as mentioned above.
3. Make sure you have the KnowledgeBase.pdf file in the same directory as the script. This file will be used to create the knowledge base for the RAG system.
4. Ensure that Ollama is running on http://localhost:11434 with the llama3 model loaded.

Usage

To run the voice assistant, execute the following command in your terminal:

python Voice_Assistant.py

The assistant will start listening for your voice input. Speak clearly into your microphone to ask questions or give commands. The assistant will process your input and respond with synthesized speech.

How It Works

1. The script loads the knowledge base from KnowledgeBase.pdf and creates a FAISS vector store using sentence embeddings.
2. It sets up a Retrieval QA chain using Ollama as the language model and the FAISS vector store as the retriever.
3. The main loop continuously listens for voice input using the computer's microphone.
4. When speech is detected, it's converted to text using Google's Speech Recognition service.
5. The text query is then processed by the RAG system to generate a response.
6. The response is converted to speech using Mozilla TTS and played back to the user.

Customization

• To use a different knowledge base, replace KnowledgeBase.pdf with your own PDF file and update the filename in the script.
• You can experiment with different embedding models by changing the model_name in the HuggingFaceEmbeddings initialization.
• To use a different Ollama model, update the model parameter in the Ollama initialization.
• Try to use other TTS frameworks - Melo TTS, coqui TTS, Mars5 TTS.

Further Improvements

1. Works on achieving lower latency for responses.
2. Understand speech better, even with background noise or accents.
3. Learn to speak and understand more languages.
4. Have better conversations by remembering what we talked about before.
5. Sound more natural when I speak, maybe even express emotions.

Troubleshooting

• If you encounter issues with speech recognition, ensure your microphone is properly connected and configured.
• For TTS issues, make sure you have the necessary audio drivers installed on your system.
• If the RAG system is not working as expected, check that your knowledge base PDF is properly formatted and contains relevant information.

Model Card Contact

For inquiries and contributions, please contact us at info@foduu.com.

@ModelCard{
    author       = {Nehul Agrawal and Roshan Kshirsagar}
    title         = {Voice Assistant},
    year           = {2024}
}