First things first! You can install the SDK with pip as follows:
pip install observersWe differentiate between observers and stores. Observers wrap generative AI APIs (like OpenAI or llama-index) and track their interactions. Stores are classes that sync these observations to different storage backends (like DuckDB or Hugging Face datasets).
To get started you can run the code below. It sends requests to a HF serverless endpoint and log the interactions into a Hub dataset, using the default store DatasetsStore. The dataset will be pushed to your personal workspace (http://hf.co/{your_username}). To learn how to configure stores, go to the next section.
import os
from observers.observers.models.openai import wrap_openai
from openai import OpenAI
api_key = os.environ["HF_TOKEN"]
openai_client = OpenAI(
base_url="https://api-inference.huggingface.co/v1/", api_key=api_key
)
client = wrap_openai(openai_client)
response = client.chat.completions.create(
model="Qwen/Qwen2.5-Coder-32B-Instruct",
messages=[{"role": "user", "content": "Tell me a joke."}],
)- OpenAI and every other LLM provider that implements the OpenAI API message formate
The wrap_openai function allows you to wrap any OpenAI compliant LLM provider. Take a look at the example doing this for Ollama for more details.
| Store | Example | Annotate | Local | Free | UI filters | SQL filters |
|---|---|---|---|---|---|---|
| Hugging Face Datasets | example | ❌ | ❌ | ✅ | ✅ | ✅ |
| DuckDB | example | ❌ | ✅ | ✅ | ❌ | ✅ |
| Argilla | example | ✅ | ❌ | ✅ | ✅ | ❌ |
To view and query Hugging Face Datasets, you can use the Hugging Face Datasets Viewer. You can find example datasets on the Hugging Face Hub. From within here, you can query the dataset using SQL or using your own UI. Take a look at the example for more details.
The default store is DuckDB and can be viewed and queried using the DuckDB CLI. Take a look at the example for more details.
> duckdb store.db
> from openai_records limit 10;
┌──────────────────────┬──────────────────────┬──────────────────────┬──────────────────────┬───┬─────────┬──────────────────────┬───────────┐
│ id │ model │ timestamp │ messages │ … │ error │ raw_response │ synced_at │
│ varchar │ varchar │ timestamp │ struct("role" varc… │ │ varchar │ json │ timestamp │
├──────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┼───┼─────────┼──────────────────────┼───────────┤
│ 89cb15f1-d902-4586… │ Qwen/Qwen2.5-Coder… │ 2024-11-19 17:12:3… │ [{'role': user, 'c… │ … │ │ {"id": "", "choice… │ │
│ 415dd081-5000-4d1a… │ Qwen/Qwen2.5-Coder… │ 2024-11-19 17:28:5… │ [{'role': user, 'c… │ … │ │ {"id": "", "choice… │ │
│ chatcmpl-926 │ llama3.1 │ 2024-11-19 17:31:5… │ [{'role': user, 'c… │ … │ │ {"id": "chatcmpl-9… │ │
├──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴───┴─────────┴──────────────────────┴───────────┤
│ 3 rows 16 columns (7 shown) │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘The Argilla Store allows you to sync your observations to Argilla. To use it, you first need to create a free Argilla deployment on Hugging Face. Take a look at the example for more details.
See CONTRIBUTING.md