8000 Wrong generation length for HFPolicy with stop strings · Issue #499 · NVIDIA-NeMo/RL · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Wrong generation length for HFPolicy with stop strings #499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
KiddoZhu opened this issue Jun 10, 2025 · 0 comments
Open

Wrong generation length for HFPolicy with stop strings #499

KiddoZhu opened this issue Jun 10, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@KiddoZhu
Copy link
Collaborator

When custom stop strings are used in HFPolicy, the generation length computed by fsdp1_policy_worker.py#L716-L721 always treats the first padded EOS token as a generated EOS token. For a batch of two samples

[token1, stop token, padded EOS, padded EOS]
[token2, token3, token4, token5]

The current implementation will compute generation lengths as 3 and 4 respectively. The ground truths are 2 and 4 respectively.

This issue mainly affects run_multi_turn_rollout with HFPolicy and batch size > 1. Some potential fixes I can think of:

  • Using different tokens for padding and EOS in HFPolicy.
  • Computing the generation lengths based on non-zero logprobs (rejected by @SahilJain314 ).
@KiddoZhu KiddoZhu added the bug Something isn't working label Jun 10, 2025
@KiddoZhu KiddoZhu mentioned this issue Jun 10, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@KiddoZhu
0