-
Use builder.pynb to create the benchmark. The script relies on randomness, so commit the generated benchmark to the repository.
-
Use completions.py to geenerate completions.
python3 completions.py --input benchmark.jsonl --output completions.jsonl --model-name /home/arjun/models/starcoderbase --batch-size 50 --num-completions 1 --max-tokens 8192 -
Use executions.py to execution completions. (Needs an update.)
python3 executions.py --input completions.jsonl --output executions.jsonl -
Use pass1.ipynb to look at the results. (Needs an update.)
This repository was archived by the owner on Jun 7, 2025. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
I cannot remember the point of this repository. I'd delete it, but it looks like @cassanof did some work on it.
License
arjunguha/longbench
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
I cannot remember the point of this repository. I'd delete it, but it looks like @cassanof did some work on it.