8000 Certain unicode characters in Pixtral cause "OverflowError: Error in model execution: out of range integral type conversion attempted" · Issue #62 · mistralai/mistral-common · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Certain unicode characters in Pixtral cause "OverflowError: Error in model execution: out of range integral type conversion attempted" #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
han-steve opened this issue Oct 27, 2024 · 0 comments
Open

Certain unicode characters in Pixtral cause "OverflowError: Error in model execution: out of range integral type conversion attempted" #62

han-steve opened this issue Oct 27, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@han-steve
Copy link
han-steve commented Oct 27, 2024 •
edited
Loading

Python -VV

Python 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0]

Pip Freeze

mistral_common==1.4.4
vllm==0.6.2

Reproduction Steps

  1. Initialize Pixtral following the vllm tutorial
from vllm import LLM
from vllm.sampling_params import SamplingParams
llm = LLM(model="mistralai/Pixtral-12B-2409", tokenizer_mode="mistral")
  1. Create a message with text that contains a unicode character like \u2705.
messages = [{"role": "user", "content": [{"type":"text", "text": "\u2705"}]}]
  1. Invoke Pixtral using it
llm.chat(messages, sampling_params=SamplingParams(max_tokens=8192))
  1. Observe the error
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1630, in execute_model
[rank0]:     model_input.async_callback()
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/utils.py", line 1102, in weak_bound
[rank0]:     unbound(inst, *args, **kwargs)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1014, in _process_model_outputs
[rank0]:     self.output_processor.process_outputs(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/output_processor/single_step.py", line 96, in process_outputs
[rank0]:     return self._process_sequence_group_outputs(sequence_group, outputs[0],
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/output_processor/single_step.py", line 124, in _process_sequence_group_outputs
[rank0]:     new_char_count = self.detokenizer.decode_sequence_inplace(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/transformers_utils/detokenizer.py", line 115, in decode_sequence_inplace
[rank0]:     seq.read_offset) = convert_prompt_ids_to_tokens(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/transformers_utils/detokenizer.py", line 224, in convert_prompt_ids_to_tokens
[rank0]:     new_tokens = tokenizer.convert_ids_to_tokens(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 227, in convert_ids_to_tokens
[rank0]:     tokens = [self.tokenizer.id_to_byte_piece(id) for id in ids]
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 227, in <listcomp>
[rank0]:     tokens = [self.tokenizer.id_to_byte_piece(id) for id in ids]
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py", line 280, in id_to_byte_piece
[rank0]:     return self._model.decode_single_token_bytes(token_id - self.num_special_tokens)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/tiktoken/core.py", line 272, in decode_single_token_bytes
[rank0]:     return self._core_bpe.decode_single_token_bytes(token)
[rank0]: OverflowError: out of range integral type conversion attempted

[rank0]: The above exception was the direct cause of the following exception:

[rank0]: Traceback (most recent call last):
[rank0]:   File "<stdin>", line 1, in <module>
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 580, in chat
[rank0]:     return self.generate(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/utils.py", line 1047, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 388, in generate
[rank0]:     outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 877, in _run_engine
[rank0]:     step_outputs = self.llm_engine.step()
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1264, in step
[rank0]:     outputs = self.model_executor.execute_model(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 130, in execute_model
[rank0]:     output = self.driver_worker.execute_model(execute_model_req)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 327, in execute_model
[rank0]:     output = self.model_runner.execute_model(
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 146, in _wrapper
[rank0]:     raise type(err)(f"Error in model execution: "
[rank0]: OverflowError: Error in model execution: out of range integral type conversion attempted

Expected Behavior

There should not be an error even though there are unicode characters. Note that other characters like \u2018 don't cause this issue.

Additional Context

No response

Suggested Solutions

No response
@han-steve han-steve added the bug Something isn't working label Oct 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@han-steve
0