Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, Sida I. Wang
- Download the MBPP, Spider, and NL2Bash datasets to
data/and follow their instructions for necessary preprocessing steps. - Download our collected Codex data. We have included the pre-executed result with the data; see also
execution.pyif you'd like to execute automatically collected code locally. - Install the
condaenvironment by
conda env create -f env.ymlSuppose that the collected Codex data is located at data/mbr-exec/, the following code returns the execution accuracy of selected MBPP test split code among 5 samples which are collected with temperature 0.3 (from Codex), using the mbr_exec(ours) method and random seed 0:
from sample_selectors import select_mbpp
select_mbpp(('test', 0.3, 'mbr_exec', 'data/mbr-exec/mbpp/', 5, 0))See also the code for more details.
MIT