Build apps with "any modality input (x), reasoning (R), any modality output (x)."
xRx is a framework for building AI-powered reasoning systems that interact with users across multiple modalities, where "x" represents the flexible integration of text, voice, and other interaction forms.
We believe that the future of software interactions lies in multimodal experiences, and xRx is at the forefront of this movement. It enables developers to build sophisticated AI systems that seamlessly integrate various input and output modalities, providing users with a truly immersive experience.
To get the xRx system up and running, follow our comprehensive quickstart guide.
xRx is a set of building blocks for developers looking to build next-generation AI-powered user experiences. Whether you're developing voice-based assistants, text-based chatbots, or multimodal applications, xRx provides the building blocks you need.
- Multimodal Input and Output: Integrate audio, text, and other modalities effortlessly.
- Advanced Reasoning: Utilize comprehensive reasoning systems to enhance user interactions.
- Modular Architecture: Easily extend and customize components to fit your specific needs.
System Architecture
The xRx system is composed of several key components, each playing a crucial role in delivering a seamless multimodal experience.
flowchart TD
A[Client] <-->|audio/text| B[Orchestrator]
B -->|Send audio| C[STT]
C -->|Return text| B
B <-->|text| G[Guardrail Proxy]
G <-->|text| D[Agent]
D[Agent] <-->|text / API requests| F[External Services]
B -->|Send text| E[TTS]
E -->|Return audio| B
style A fill:#FFCDD2,stroke:#B71C1C,stroke-width:2px,color:#000000
style B fill:#BBDEFB,stroke:#0D47A1,stroke-width:2px,color:#000000
style C fill:#C8E6C9,stroke:#1B5E20,stroke-width:2px,color:#000000
style D fill:#FFF9C4,stroke:#F57F17,stroke-width:2px,color:#000000
style E fill:#D1C4E9,stroke:#4A148C,stroke-width:2px,color:#000000
style F fill:#FFECB3,stroke:#FF6F00,stroke-width:2px,color:#000000
style G fill:#E1BEE7,stroke:#4A148C,stroke-width:2px,color:#000000
xRx includes xrx-core, which is a set of containers that provide the core functionality of the system reusable across different applications, and sample-apps, which are example applications that demonstrate how to use xrx-core.
This code is application-specific.
- Client: Front-end app rendering UI and handling WebSocket communication.
- Reasoning Agent: Processes inputs and generates responses.
This code is reusable across applications.
- Orchestrator: Manages data flow between components.
- STT (Speech-to-Text): Converts audio to text.
- TTS (Text-to-Speech): Converts text to audio.
- React xRx Client Library: Reusable UI components and utilities.
- xRx Agent Framework: Foundation for building reasoning agents.
- Guardrails Proxy: A safety layer for the reasoning system.
These components then communicate via the following sequence diagram
sequenceDiagram
participant Client
participant Orchestrator
participant STT
participant Agent
participant TTS
Client->> Orchestrator: Send audio on websockets port 8000
Orchestrator->>STT: Send audio on websockets port 8001
STT ->>Orchestrator: Return text
STT ->>Orchestrator: Return text
Orchestrator->>Agent: Send text on port 8003
Agent->>Orchestrator: Return text
Orchestrator->>TTS: Send text on port 8002
TTS ->>Orchestrator: Return audio
Orchestrator->>Client: Return audio, text, and application widgets
Reasoning Apps
xRx core is meant to be imported as a submodule into your project. Check out the Sample-apps repository for how to set up xRx core in your project.
To showcase the capabilities of xRx, we've created multiple reasoning applications:
-
Simple Tool Calling App: For developers looking to create their own reasoning apps, we've created Simple App. Simple App is a simple tool calling app that demonstrates basic functionality. This App has access to tools like weather and time retrievers, and stock price lookup. It shows how any Python-based reasoning App can be deployed into the xRx system.
-
Shopify App: We have built a sophisticated reasoning system that interacts with a Shopify store. The Shopify App allows users to interact with a reasoning system built on top of Shopify, handling tasks like product inquiries, order placement, and customer service.
-
Wolfram Assistant App: The Wolfram Assistant App leverages Wolfram Alpha's conversational API to provide answers to user queries, particularly useful for mathematical and scientific questions. This App enhances the dialogue with refined language processing to deliver a smooth and engaging user experience.
-
Patient Information App: The Patient Information App is designed to collect and manage patient information before a doctor's visit. It demonstrates how xRx can be applied in healthcare scenarios, gathering essential medical data in a conversational manner.
Contributing
We welcome contributions from the community. Whether you're adding new features, fixing bugs, or improving documentation, your efforts are valued.
For more information on contributing, see our Contribution Guide.
Documentation
See our documentation here