Paper: https://arxiv.org/abs/2406.12045
| Strategy | Pass^1 | Pass^2 | Pass^3 | Pass^4 |
|---|---|---|---|---|
| TC (claude-3-5-sonnet-20241022) | 0.460 | 0.326 | 0.263 | 0.225 |
| TC (gpt-4o) | 0.420 | 0.273 | 0.220 | 0.200 |
| TC (claude-3-5-sonnet-20240620) | 0.360 | 0.224 | 0.169 | 0.139 |
| TC (mistral-large-2407) | ?? | ?? | ?? | ?? |
| TC (gpt-4o-mini) | 0.225 | 0.140 | 0.110 | 0.100 |
| Act (gpt-4o) | 0.365 | 0.217 | 0.160 | 0.140 |
| ReAct (gpt-4o) | 0.325 | 0.233 | 0.185 | 0.160 |
| Strategy | Pass^1 | Pass^2 | Pass^3 | Pass^4 |
|---|---|---|---|---|
| TC (claude-3-5-sonnet-20241022) | 0.692 | 0.576 | 0.509 | 0.462 |
| TC (gpt-4o) | 0.604 | 0.491 | 0.430 | 0.383 |
| TC (claude-3-5-sonnet-20240620) | 0.626 | 0.506 | 0.435 | 0.387 |
| TC (mistral-large-2407) | ?? | ?? | ?? | ?? |
| TC (gpt-4o-mini) | ?? | ?? | ?? | ?? |
| Act (gpt-4o) | ?? | ?? | ?? | ?? |
| ReAct (gpt-4o) | ?? | ?? | ?? | ?? |
*TC = tool-calling strategy (the function-calling strategy reported in the paper)
- Clone this repository:
git clone https://github.com/sierra-research/tau-bench && cd ./tau-bench- Install from source (which also installs required packages):
pip install -e .- Set up your OpenAI / Anthropic / Google / Mistral / AnyScale API keys as environment variables.
OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
GOOGLE_API_KEY=...
MISTRAL_API_KEY=...Run a tool-calling agent on the τ-retail environment:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --user-model gpt-4o --user-model-provider openai --user-strategy llm --max-concurrency 10Set max concurrency according to your API limit(s).
To run specific tasks, use the --task-ids flag. For example:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --user-model gpt-4o --user-model-provider openai --user-strategy llm --max-concurrency 10 --task-ids 2 4 6This command will run only the tasks with IDs 2, 4, and 6.
By default, we use gpt-4o as the user simulator with strategy llm. You can use other models by setting the --user-model flag, or other strategies by setting the --user-strategy flag. For example, run a tool-calling agent with a claude user simulator:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --max-concurrency 10 --user-model claude-3-5-sonnet-20240620 --user-model-provider anthropic --user-strategy llmOther strategies:
To run react user simulator:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --max-concurrency 10 --user-model gpt-4o --user-model-provider openai --user-strategy reactExample of a react user response:
Thought:
I should provide my name and zip code as I wasn't given an email address to use.
User Response:
Sure, my name is Yusuf Rossi, and my zip code is 19122.To run verify user simulator:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --max-concurrency 10 --user-model gpt-4o --user-model-provider openai --user-strategy verifyThis strategy uses a subsequent LLM verification step to check if the user simulator's response is satisfactory. If not, the user simulator will be prompted to generate a new response.
To run reflection user simulator:
python run.py --agent-strategy tool-calling --env retail --model gpt-4o --model-provider openai --max-concurrency 10 --user-model gpt-4o --user-model-provider openai --user-strategy reflectionThis strategy uses a subsequent LLM verification step to check if the user simulator's response is satisfactory. If not, the user simulator will be prompted to reflect on its response and generate a new response.
Often times, it is difficult and time consuming to manually identify specific error locations in trajectories as they can be long and the constraints can be complex. We have provided an auto error identification tool that can do the following:
- Fault assignment: determine the entity that is responsible for the fault (user, agent, environment)
- Fault type classification: classify the type of fault (goal_partially_completed, used_wrong_tool, used_wrong_tool_argument, took_unintended_action)
Both of the labels are accompanied with a description.
To run the auto error identification, run:
python auto_error_identification.py --env <airline/retail> --platform openai --results-path <the path to your results file here> --max-concurrency 16 --output-path test-auto-error-identification --max-num-failed-results 10Please note that this feature utilizes an LLM, which may lead to inaccurate error identifications.
*Notice: If an error is raised due to the structure of your results file, you may have to rerun the benchmark to produce a new results file. We have recently rewritten the benchmark to be more type-safe and extensible.
τ-bench might be expensive to run. We have provided a set of historical trajectories for the airline and retail environments in ./historical_trajectories.
If you would like to contribute your historical trajectories to this benchmark, please submit a PR!
See ./LICENSE.
Please submit issues or pull requests if you find problems with the benchmark.
@misc{yao2024tau,
title={$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains},
author={Shunyu Yao and Noah Shinn and Pedram Razavi and Karthik Narasimhan},
year={2024},
eprint={2406.12045},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2406.12045},
}