An AI agent for log analytics and querying.
Problem Statement 3 – Real-Time Data Experiences with Fluvio
AutoAgentDesk addresses the challenges of log analytics and querying by providing an AI-powered solution that simplifies the process. It enables users to interact with logs in natural language, analyze them in real-time, and generate actionable insights.
- Problem:
- Log analytics is time-consuming and complex, especially with large datasets from multiple sources.
- Traditional tools lack real-time capabilities and require manual intervention.
- Solution:
- AutoAgentDesk automates log analytics and querying with an AI agent that supports natural language interaction.
- It provides real-time insights, report generation, and multi-source log integration.
- Target Audience:
- Developers, DevOps engineers, and data analysts.
- Organizations needing real-time log monitoring and analytics.
- Value Proposition:
- Saves time and reduces complexity in log analysis.
- Makes log querying accessible to non-technical users.
- Enables faster issue resolution with real-time capabilities.
AutoAgentDesk (Solo hacker)
- Rohith Singh (Team Lead)
- Researched existing tools and identified gaps.
- Designed a scalable architecture for real-time log processing and natural language interaction.
- Built the core functionality using Python, FastAPI, LangChain, and Fluvio.
- Integrated Chromadb for log storage and trained the AI on diverse log queries.
- Created a web interface for user-friendly interaction.
- Iteratively improved the agent based on feedback.
- Frontend: HTML, CSS, JavaScript (ReactJS)
- Backend: Python, FastAPI
- Database: Chromadb, PostgreSQL
- APIs: Fluvio (log streaming), LangChain (NLP)
- AI Models: Groq AI, Gemini AI, OpenAI, Anthropic
- Hosting: Docker (containerization), Redis (caching) - not yet implemented
- Other Tools: GitHub (version control), Postman (API testing)
- ✅ Multi-model AI support (Groq AI, Gemini AI, OpenAI, Anthropic)
- ✅ Real-time log streaming with Fluvio
- ✅ Natural language querying for logs
- ✅ Multi-source log integration (frontend, backend, database)
- ✅ RESTful API and web interface
- ✅ Plugin system for extensibility
- ✅ Support for JSON, XML, and other log formats
- ✅ Timeline tracking and state management
- ✅ Report generation and batch processing
- Demo Video Link: YouTube
- Pitch Deck / PPT Link: Google Slides
- Python 3.8+: Ensure Python is installed on your system.
- Node.js: Required for the frontend (if applicable).
- Docker: For containerization (optional, not yet implemented).
- Fluvio: For real-time log streaming.
- API Keys: Required for AI models and services:
GROQ_API_KEYOPENAI_API_KEYANTHROPIC_API_KEYGOOGLE_API_KEY
- Environment Variables: Set up a
.envfile in the root directory. Refer to the.env_samplefile for required variables. - Chromadb: For log storage.
- PostgreSQL: For database storage (optional, not yet implemented).
- Redis: For caching (optional, not yet implemented).
git clone https://github.com/rohittcodes/autoagentdesk
cd autoagentdesk
pip install -r requirements.txt
python main.pyAccess the API at http://localhost:8000.
Click to view architecture diagrams and snapshots
- 📈 Add more integrations and advanced analytics features.
- 🛡️ Enhance security and scalability.
- 🌐 Support localization and multi-language capabilities.
- 🔔 Implement alerting and notification systems.
- 🖥️ Improve the web interface with ReactJS.
- APIs: Fluvio, LangChain, Groq AI, OpenAI, Anthropic
- Tools: Chromadb, FastAPI, Docker, Redis
- Acknowledgements: HackHazards team and community
My hackathon journey building AutoAgentDesk has been both challenging and rewarding! Here's a glimpse into my experience:
- Working with ChromaDB's query format was a significant hurdle. When implementing filters, I discovered that ChromaDB has specific requirements for field names and operators. After debugging the "Invalid where clause" errors, I learned the importance of proper field name mapping and using the correct query operators (
$invs$eq). - The context variable definition bug in the LogAnalysisAgent class taught me to be more careful about variable scoping in asynchronous Python code. Always capture return values!
- Balancing token limits with comprehensive log analysis was tricky. I developed a smart log prioritization system that ensures critical database and memory-related errors are never truncated, even when dealing with thousands of logs.
- Integrating Groq's API was seamless once I understood its token limitations. The performance improvement over other models for this specific use case was impressive - responses are noticeably faster.
- Fluvio's streaming capabilities transformed the project from a basic log analysis tool into a real-time monitoring platform. The producer-consumer architecture enables complex real-time pipelines that traditional log systems can't match.
- Building a context-aware AI agent that maintains conversation history and previous findings created a more natural and insightful user experience than I initially expected.
- Special thanks to the Groq and InfinyOn teams for their documentation and sample projects that helped make integration smoother.
I'm excited to continue developing AutoAgentDesk beyond this hackathon, with plans to implement the alerting system and ReactJS frontend improvements next. This project has deepened my understanding of AI-assisted observability tools and real-time data processing architectures.
/logs: Endpoint for analyzing logs. You can access this endpoint to send log data to the AI agent for analysis. The agent will process the logs and return insights based on the data./queries: Endpoint for querying logs. You can send a query to this endpoint, and the AI agent will return the relevant log data.curl -X POST "http://localhost:8000/queries/" -H "Content-Type: application/json" -d '{"query": "What WARN level logs occurred in the API service in the last 24 hours?", "max_logs": 789B 100}'/report: Endpoint for generating reports. You can send a request to this endpoint, and the AI agent will generate a report based on the log data. (yet to be implemented)
- Implement visualization capabilities
- Add support for more log sources
- Improve natural language understanding
- Enhance performance and scalability
- Add more advanced analytics features
- Implement user authentication and authorization
- Create a web interface for easier interaction
- Update the web interface to use ReactJS
- Add support for real-time log monitoring
- Implement alerting and notification features
- Add support for more languages
- Improve documentation and examples
- Add more test cases and improve test coverage
- Implement a plugin system for extensibility
- Add support for more log formats (e.g., JSON, XML)
We welcome contributions to AutoAgentDesk! If you have ideas for new features, improvements, or bug fixes, please open an issue or submit a pull request. Please make sure to follow the contribution guidelines when contributing to the project.
This project is licensed under the MIT License. See the LICENSE file for details.