SWE-smith & multimodal base support #1092
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR introduces multimodal support and initial SWE-smith changes while refining repository reset procedures. Key changes include:
- Updating tests to validate the new “multimodal” subset for bench instances.
- Adding the XMLFunctionCallingParser and expanding instance source support with SWE-smith.
- Adjusting Docker image naming conventions and simplifying reset commands in the environment.
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_batch_instance.py | Adds “multimodal” subset case in bench instance tests. |
| sweagent/tools/parsing.py | Introduces XMLFunctionCallingParser for XML-based tool responses. |
| sweagent/run/hooks/swe_bench_evaluate.py | Extends subset mapping to include “multimodal”. |
| sweagent/run/batch_instances.py | Updates image name formatting and adds SWE-smith instance support. |
| sweagent/environment/swe_env.py | Revises reset repository commands by removing the -x flag. |
Comments suppressed due to low confidence (3)
tests/test_batch_instance.py:35
- [nitpick] The test now iterates over the multimodal subset; please ensure that separate test cases or assertions are in place to validate multimodal-specific behavior.
for subset in ["lite", "verified", "full", "multimodal"]:
sweagent/run/batch_instances.py:162
- [nitpick] Verify that the lowercasing of the Docker image name is intentional and does not conflict with case-sensitive tag requirements.
image_name = f"swebench/sweb.eval.x86_64.{id_docker_compatible}:latest".lower()
sweagent/environment/swe_env.py:160
- [nitpick] Ensure that the revised git reset commands (including the removal of the '-x' flag in 'git clean') reliably achieve a fully clean state and do not leave behind untracked changes.
git checkout {self.repo.base_commit}
Comment on lines
+213
to
+214
| FN_REGEX_PATTERN = r"<function=([^>]+)>\n(.*?)</function>" | ||
| FN_PARAM_REGEX_PATTERN = r"<parameter=([^>]+)>(.*?)</parameter>" |
There was a problem hiding this comment.
[nitpick] Consider pre-compiling the regex patterns FN_REGEX_PATTERN and FN_PARAM_REGEX_PATTERN to improve performance if this parser is invoked frequently.
Suggested change
| FN_REGEX_PATTERN = r"<function=([^>]+)>\n(.*?)</function>" | |
| FN_PARAM_REGEX_PATTERN = r"<parameter=([^>]+)>(.*?)</parameter>" | |
| FN_REGEX_PATTERN = re.compile(r"<function=([^>]+)>\n(.*?)</function>") | |
| FN_PARAM_REGEX_PATTERN = re.compile(r"<parameter=([^>]+)>(.*?)</parameter>") |
Copilot uses AI. Check for mistakes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
-xfromgit cleanin reset