This project implements a Retrieval-Augmented Generation (RAG) system without using LangChain. It uses the following core dependencies:
- OpenAI: For LLM capabilities
- Pinecone: Vector database for storing and retrieving embeddings
- Tiktoken: OpenAI's tokenizer
- Python-dotenv: For managing environment variables
- Create a virtual environment (recommended):
python -m venv venv
source venv/bin/activate # On Unix/macOS- Install dependencies:
pip install -r requirements.txt- Create a
.envfile and add your API keys:
OPENAI_API_KEY=your_openai_api_key
PINECONE_API_KEY=your_pinecone_api_key
PINECONE_INDEX_NAME=your_pinecone_index_name
- Index documents:
python main.py -l /path/to/documents- Ask a question:
python main.py "What is the best way to do great work?"- Both index and ask:
python main.py -l /path/to/documents "What is the best way to do great work?"The project consists of several key components:
main.py: Core application logic and CLI interfaceutils.py: File reading utilitiestokenization.py: Token-related functions for text processingrequirements.txt: Project dependencies.env.example: Template for environment variables
- Document processing and chunking
- Semantic search using OpenAI embeddings
- Vector storage with Pinecone
- RAG-based question answering
- Source citations in responses
- Progress tracking for long operations
This project is licensed under the BSD 3-Clause License - see the LICENSE file for details.