This repo includes a performance analysis comparing Commonbase API to OpenAI's API. It includes:
- a script to generate live benchmarks comparing Commonbase performance to OpenAI
- benchmarks ran by us on Sep 14, 2023
- a python notebook showing the analysis of the recorded benchbmarks.
bun installin case you are less adventurous and don't yet have bun, here you go: Install Bun
Copy the .env.example file to .env and fill in the values.
Sign up on Commonbase and copy the API key from the onboarding flow.
Copy the Project ID from Settings in the web app.
Create a new OpenAI API key from OpenAI API keys
bun providers.ts > providers-$(date +%s).csvThe results of our benchmarks and the analysis of the results can be found in the analysis folder.
We ran two different benchmarks for the gpt-4 and gpt-3.5-turbo model.
- Single token completion (max_tokens=1)
- 128 tokens completion (max_tokens=128)
The used prompts can be found in the providers.ts file.