8000 Some questions of results on MPI dataset · Issue #1 · jianggy/MPI · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Some questions of results on MPI dataset #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
IamJunhaHwang opened this issue Feb 6, 2025 · 0 comments
Open

Some questions of results on MPI dataset #1

IamJunhaHwang opened this issue Feb 6, 2025 · 0 comments

Comments

@IamJunhaHwang
Copy link
IamJunhaHwang commented Feb 6, 2025 •
edited
Loading

Hello,

I sincerely appreciate your valuable research and the effort you put into the paper Evaluating and Inducing Personality in Pre-trained Language Models. I found it incredibly insightful.

I am currently working on reproducing the results using the code and data you generously provided. While doing so, I came across a few questions regarding the results on the MPI dataset (as shown in Table 2 of the paper).

  1. Are the results reported in Table 2 from a single trial, or are they the average of multiple trials?

  2. In the latest version of Transformers library, setting temperature = 0 causes the error Temperature needs to be > 0. Could you share an alternative solution or the generate_config settings you used in the experiment?

  3. During the reproduction process, I used GPT-Neo 2.7B with a low temperature setting of 0.001, and I found the model generated biased responses. For instance, when counting the responses, I observed the following distribution: {'A': 9, 'B': 0, 'C': 109, 'D': 2, 'E': 0, 'UNK': 0}. Did you encounter a similar issue during your experiments? If possible, could you also share the response counts from your experiment?

  4. Could you share the version of Transformers library you used in your experiments?

I have great respect for your work and understand you have a busy schedule, but if possible, I would appreciate your feedback.
Thank you again for your time and consideration. I look forward to hearing from you.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@IamJunhaHwang
0