8000 No such file or directory: 'eval_sequences.json' · Issue #19 · Robot-VLAs/RoboVLMs · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

No such file or directory: 'eval_sequences.json' #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
leaf0793 opened this issue Feb 12, 2025 · 3 comments
Open

No such file or directory: 'eval_sequences.json' #19

leaf0793 opened this issue Feb 12, 2025 · 3 comments

Comments

@leaf0793
Copy link

After finishing the evaluation, the line 515 with open("eval_sequences.json", "r") as f: is this json the same as configs/data/calvin/eval_sequences.json ? Should i replace the eval_sequences.json" with "configs/data/calvin/eval_sequences.json "
@ducido
Copy link
ducido commented Mar 21, 2025 •
edited
Loading

@leaf0793 Hi, Do you experience weird results when evaluating?

I got a zero score for all task, like:

1/1000: 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |   
2/1000: 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 
3/1000: 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 
4/1000: 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 
....

Results for Epoch None:
Average successful sequence length: 0.0
Success rates for i instructions in a row:
1: 0.0%
2: 0.0%
3: 0.0%
4: 0.0%
5: 0.0%
rotate_blue_block_right: 0 / 1 |  SR: 0.0%
turn_off_led: 0 / 1 |  SR: 0.0%
lift_pink_block_slider: 0 / 1 |  SR: 0.0%
...

I used kosmos_ph_calvin_abc model and the only things I changed in the code was set os.environ["PYOPENGL_PLATFORM"] = "egl" instead of osmesa.

@lixinghang12
Copy link

After finishing the evaluation, the line 515 with open("eval_sequences.json", "r") as f: is this json the same as configs/data/calvin/eval_sequences.json ? Should i replace the eval_sequences.json" with "configs/data/calvin/eval_sequences.json "

Yes, you should configure the path of eval_sequences.json

@lixinghang12
Copy link

@leaf0793 Hi, Do you experience weird results when evaluating?

I got a zero score for all task, like:

1/1000: 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |   
2/1000: 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 
3/1000: 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 
4/1000: 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 
....

Results for Epoch None:
Average successful sequence length: 0.0
Success rates for i instructions in a row:
1: 0.0%
2: 0.0%
3: 0.0%
4: 0.0%
5: 0.0%
rotate_blue_block_right: 0 / 1 |  SR: 0.0%
turn_off_led: 0 / 1 |  SR: 0.0%
lift_pink_block_slider: 0 / 1 |  SR: 0.0%
...

I used kosmos_ph_calvin_abc model and the only things I changed in the code was set os.environ["PYOPENGL_PLATFORM"] = "egl" instead of osmesa.

You may need to run eval/calvin/env_test.py to visualize whether the simulator has been rendered correctlywith "egl".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@leaf0793 @lixinghang12 @ducido
0