Help serving vllm with kuberay #11957

torsteinelv · 2025-01-11T14:25:02Z

torsteinelv
Jan 11, 2025

Hello,

have been trying to get this setup working for a long long time and now im about to give up. I have been googling and reading on others issues and deployment, and I see people ask of help on kuberay's github page and the are recommended to ask on vllms page , I hope someone will be able to assist.

Currently I have 3 node 1 gpu in each of them and im able to get the deployment started on 3 nodes but get internal server error when i try to make a request to the openai api. Was hoping someone could help me here :)

current setup:

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: vllm-test
  namespace: ray-operator
spec:
  serveConfigV2: |
    applications:
    - name: llm
      route_prefix: /
      import_path: ray-serve.vllm_engine:build_app
      runtime_env:
        working_dir: "https://github.com/torsteinelv/ray-serve/archive/refs/heads/main.zip"
        pip: ["ray[serve]", "requests"]
      deployments:
      - name: VLLMDeployment
        ray_actor_options:
          num_cpus: 2
      args:
        model: microsoft/Phi-3.5-vision-instruct
        dtype: float16
        pipeline-parallel-size: 3
        accelerator: GPU
        trust_remote_code: True
        enforce-eager: True
        gpu-memory-utilization: 0.8
        max-model-len: 1024
        cpu_offload_gb: 2
  rayClusterConfig:
    headGroupSpec:
      serviceType: LoadBalancer
      rayStartParams:
        dashboard-host: '0.0.0.0'
      template:
        spec:
          volumes:
            - name: model-storage
              persistentVolumeClaim:
                claimName: vllm-test-models
            - name: shm-volume
              emptyDir:
                medium: Memory
          containers:
          - name: ray-head
            image: ghcr.io/sasha0552/vllm:v0.6.6.post1
            resources:
              limits:
                cpu: "4"
                memory: "16Gi"
                nvidia.com/gpu: "1"
              requests:
                cpu: "1"
                memory: "1Gi"
                nvidia.com/gpu: "1"
            ports:
            - containerPort: 6379
              name: gcs-server
            - containerPort: 8265
              name: dashboard
            - containerPort: 10001
              name: client
            - containerPort: 8000
              name: serve
            env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: hf_api_token
            volumeMounts:
              - mountPath: /models
                name: model-storage
              - mountPath: /dev/shm
                name: shm-volume
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: ray.io/node-type
                    operator: In
                    values:
                    - worker
                topologyKey: "kubernetes.io/hostname"
    workerGroupSpecs:
    - replicas: 2
      minReplicas: 0
      maxReplicas: 4
      groupName: gpu-group
      rayStartParams: {}
      template:
        spec:
          volumes:
            - name: model-storage
              persis
8000
tentVolumeClaim:
                claimName: vllm-test-models
            - name: shm-volume
              emptyDir:
                medium: Memory
          containers:
          - name: llm
            image: ghcr.io/sasha0552/vllm:v0.6.6.post1
            env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: hf_api_token
            resources:
              limits:
                cpu: "4"
                memory: "16Gi"
                nvidia.com/gpu: "1"
              requests:
                cpu: "1"
                memory: "1Gi"
                nvidia.com/gpu: "1"
            volumeMounts:
              - mountPath: /models
                name: model-storage
              - mountPath: /dev/shm
                name: shm-volume
          tolerations:
            - key: "nvidia.com/gpu"
              operator: "Exists"
              effect: "NoSchedule"
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: ray.io/node-type
                    operator: In
                    values:
                    - worker
                topologyKey: "kubernetes.io/hostname"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-test-models
  namespace: ray-operator
spec:
  storageClassName: csi-cephfs-sc
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi

code

from typing import Dict, Optional, List
import logging

from fastapi import FastAPI
from starlette.requests import Request
from starlette.responses import StreamingResponse, JSONResponse

from ray import serve

from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.entrypoints.openai.cli_args import make_arg_parser
from vllm.entrypoints.openai.protocol import (
    ChatCompletionRequest,
    ChatCompletionResponse,
    ErrorResponse,
)
from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
from vllm.entrypoints.openai.serving_engine import LoRAModulePath, PromptAdapterPath
from vllm.utils import FlexibleArgumentParser
from vllm.entrypoints.logger import RequestLogger

logger = logging.getLogger("ray.serve")

app = FastAPI()


@serve.deployment(
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 10,
        "target_ongoing_requests": 5,
    },
    max_ongoing_requests=10,
)
@serve.ingress(app)
class VLLMDeployment:
    def __init__(
        self,
        engine_args: AsyncEngineArgs,
        response_role: str,
        lora_modules: Optional[List[LoRAModulePath]] = None,
        prompt_adapters: Optional[List[PromptAdapterPath]] = None,
        request_logger: Optional[RequestLogger] = None,
        chat_template: Optional[str] = None,
    ):
        logger.info(f"Starting with engine args: {engine_args}")
        self.openai_serving_chat = None
        self.engine_args = engine_args
        self.response_role = response_role
        self.lora_modules = lora_modules
        self.prompt_adapters = prompt_adapters
        self.request_logger = request_logger
        self.chat_template = chat_template
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)

    @app.post("/v1/chat/completions")
    async def create_chat_completion(
        self, request: ChatCompletionRequest, raw_request: Request
    ):
        """OpenAI-compatible HTTP endpoint.

        API reference:
            - https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html
        """
        if not self.openai_serving_chat:
            model_config = await self.engine.get_model_config()
            # Determine the name of the served model for the OpenAI client.
            if self.engine_args.served_model_name is not None:
                served_model_names = self.engine_args.served_model_name
            else:
                served_model_names = [self.engine_args.model]
            self.openai_serving_chat = OpenAIServingChat(
                self.engine,
                model_config,
                served_model_names,
                self.response_role,
                lora_modules=self.lora_modules,
                prompt_adapters=self.prompt_adapters,
                request_logger=self.request_logger,
                chat_template=self.chat_template,
            )
        logger.info(f"Request: {request}")
        generator = await self.openai_serving_chat.create_chat_completion(
            request, raw_request
        )
        if isinstance(generator, ErrorResponse):
            return JSONResponse(
                content=generator.model_dump(), status_code=generator.code
            )
        if request.stream:
            return StreamingResponse(content=generator, media_type="text/event-stream")
        else:
            assert isinstance(generator, ChatCompletionResponse)
            return JSONResponse(content=generator.model_dump())


def parse_vllm_args(cli_args: Dict[str, str]):
    """Parses vLLM args based on CLI inputs.

    Currently uses argparse because vLLM doesn't expose Python models for all of the
    config options we want to support.
    """
    arg_parser = FlexibleArgumentParser(
        description="vLLM OpenAI-Compatible RESTful API server."
    )

    parser = make_arg_parser(arg_parser)
    arg_strings = []
    for key, value in cli_args.items():
        logger.info(f"Processing argument: --{key} with value: {value}")
        
        if value is True:  # Håndter boolske flagg satt til True
            arg_strings.append(f"--{key}")
        elif value not in (None, "None"):  # Ignorer None eller 'None' som streng
            arg_strings.extend([f"--{key}", str(value)])
        else:
            arg_strings.append(f"--{key}")

    parsed_args = parser.parse_args(args=arg_strings)
    
    return parsed_args


def build_app(cli_args: Dict[str, str]) -> serve.Application:
    """Builds the Serve app based on CLI arguments.

    See https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#command-line-arguments-for-the-server
    for the complete set of arguments.

    Supported engine arguments: https://docs.vllm.ai/en/latest/models/engine_args.html.
    """  # noqa: E501
    if "accelerator" in cli_args.keys():
        accelerator = cli_args.pop("accelerator")
    else:
        accelerator = "GPU"
    parsed_args = parse_vllm_args(cli_args)
    engine_args = AsyncEngineArgs.from_cli_args(parsed_args)
    engine_args.worker_use_ray = True

    tp = engine_args.tensor_parallel_size
    logger.info(f"Tensor parallelism = {tp}")
    pg_resources = []
    pg_resources.append({"CPU": 1})  # for the deployment replica
    for i in range(tp):
        pg_resources.append({"CPU": 1, accelerator: 1})  # for the vLLM actors

    # We use the "STRICT_PACK" strategy below to ensure all vLLM actors are placed on
    # the same Ray node.
    return VLLMDeployment.bind(
        engine_args,
        parsed_args.response_role,
        parsed_args.lora_modules,
        parsed_args.prompt_adapters,
        cli_args.get("request_logger"),
        parsed_args.chat_template,
    )

error:

INFO 2025-01-11 03:24:22,491 llm_VLLMDeployment 2dqbcyec -- Started initializing replica.
INFO 2025-01-11 03:24:22,492 llm_VLLMDeployment 2dqbcyec -- Starting with engine args: AsyncEngineArgs(model='microsoft/Phi-3.5-vision-instruct', served_model_name=None, tokenizer='microsoft/Phi-3.5-vision-instruct', task='auto', skip_tokenizer_init=False, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='float16', kv_cache_dtype='auto', quantization_param_path=None, seed=0, max_model_len=1024, worker_use_ray=True, distributed_executor_backend=None, pipeline_parallel_size=3, tensor_parallel_size=1, max_parallel_loading_workers=None, block_size=None, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=True, swap_space=4, cpu_offload_gb=2.0, gpu_memory_utilization=0.8, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, revision=None, code_revision=None, rope_scaling=None, rope_theta=None, hf_overrides=None, tokenizer_revision=None, quantization=None, enforce_eager=True, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, fully_sharded_loras=False, lora_extra_vocab_size=256, long_lora_scaling_factors=None, lora_dtype='auto', max_cpu_loras=None, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, ray_workers_use_nsight=False, num_gpu_blocks_override=None, num_lookahead_slots=0, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, scheduler_delay_factor=0.0, enable_chunked_prefill=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, speculative_model=None, speculative_model_quantization=None, speculative_draft_tensor_parallel_size=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, qlora_adapter_name_or_path=None, disable_logprobs_during_spec_decoding=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, worker_cls='auto', kv_transfer_config=None, generation_config=None, disable_log_requests=False)
INFO 2025-01-11 03:26:05,246 llm_VLLMDeployment 2dqbcyec -- Finished initializing replica.
INFO 2025-01-11 03:26:05,462 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 03:26:05,469 llm_VLLMDeployment 2dqbcyec 28832e3a-87e5-43c8-a9b4-e7784ed04813 -- GET / 404 15.3ms
INFO 2025-01-11 03:26:30,377 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:26:30,672 llm_VLLMDeployment 2dqbcyec 6f171a9c-9e57-42c4-8392-ab18991a3262 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:26:30,
8000
867 llm_VLLMDeployment 2dqbcyec 6f171a9c-9e57-42c4-8392-ab18991a3262 -- POST /v1/chat/completions 500 493.2ms
INFO 2025-01-11 03:26:31,273 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:26:31,289 llm_VLLMDeployment 2dqbcyec 94c1f8f1-00d5-4950-a35c-d856bfe43cef -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:26:31,294 llm_VLLMDeployment 2dqbcyec 94c1f8f1-00d5-4950-a35c-d856bfe43cef -- POST /v1/chat/completions 500 23.3ms
INFO 2025-01-11 03:26:32,185 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:26:32,203 llm_VLLMDeployment 2dqbcyec f98e1298-7a24-4577-a6bf-85a797ec6263 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:26:32,211 llm_VLLMDeployment 2dqbcyec f98e1298-7a24-4577-a6bf-85a797ec6263 -- POST /v1/chat/completions 500 28.2ms
INFO 2025-01-11 03:27:04,610 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:27:04,627 llm_VLLMDeployment 2dqbcyec cea64e16-5db2-4670-8833-3eee1e6f21ec -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:27:04,632 llm_VLLMDeployment 2dqbcyec cea64e16-5db2-4670-8833-3eee1e6f21ec -- POST /v1/chat/completions 500 24.1ms
INFO 2025-01-11 03:27:05,056 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:27:05,076 llm_VLLMDeployment 2dqbcyec 5675a6b5-f7c6-4c18-b4bc-2d4fd6f5fecd -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:27:05,081 llm_VLLMDeployment 2dqbcyec 5675a6b5-f7c6-4c18-b4bc-2d4fd6f5fecd -- POST /v1/chat/completions 500 29.8ms
INFO 2025-01-11 03:27:05,882 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:27:05,907 llm_VLLMDeployment 2dqbcyec dca2788e-d2c9-4f15-967e-4eec35e3aad7 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:27:05,933 llm_VLLMDeployment 2dqbcyec dca2788e-d2c9-4f15-967e-4eec35e3aad7 -- POST /v1/chat/completions 500 53.5ms
INFO 2025-01-11 03:27:35,390 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 03:27:35,396 llm_VLLMDeployment 2dqbcyec f8e6cc65-fc15-4587-a5fb-f577565c83ab -- GET / 404 8.6ms
INFO 2025-01-11 03:27:46,133 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 03:27:46,137 llm_VLLMDeployment 2dqbcyec 6d7165c0-0687-45fd-aa22-90f6d189ff44 -- GET / 404 5.0ms
INFO 2025-01-11 03:28:48,411 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 03:28:48,416 llm_VLLMDeployment 2dqbcyec d889f79a-903b-4988-91cb-631c57f07c75 -- GET / 404 9.7ms
INFO 2025-01-11 06:19:08,632 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 06:19:08,640 llm_VLLMDeployment 2dqbcyec 63c3d3f3-069b-4247-a86e-24662f824ad2 -- GET / 404 12.0ms
INFO 2025-01-11 06:19:18,520 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 06:19:18,524 llm_VLLMDeployment 2dqbcyec de4da3e5-e057-47c7-a149-2030a0f4c719 -- GET / 404 6.2ms
INFO 2025-01-11 06:19:25,200 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 06:19:25,246 llm_VLLMDeployment 2dqbcyec ecf2c282-b83d-4652-a084-657034db315f -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 06:19:25,262 llm_VLLMDeployment 2dqbcyec ecf2c282-b83d-4652-a084-657034db315f -- POST /v1/chat/completions 500 63.5ms
INFO 2025-01-11 06:19:25,761 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 06:19:25,779 llm_VLLMDeployment 2dqbcyec f1485b1b-f16b-477f-955d-e0e0a805b417 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 06:19:25,787 llm_VLLMDeployment 2dqbcyec f1485b1b-f16b-477f-955d-e0e0a805b417 -- POST /v1/chat/completions 500 28.5ms
INFO 2025-01-11 06:19:26,794 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 06:19:26,807 llm_VLLMDeployment 2dqbcyec fde34a5a-a56a-433f-a567-1b3a16c8d903 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlett
8000
e/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 06:19:26,814 llm_VLLMDeployment 2dqbcyec fde34a5a-a56a-433f-a567-1b3a16c8d903 -- POST /v1/chat/completions 500 22.2ms

mdklause · 2025-01-21T15:54:20Z

mdklause
Jan 21, 2025

To fix this TypeError: You can just add chat_template_content_format='auto' in the declaration of OpenAIServingChat

2 replies

10000

torsteinelv Mar 13, 2025
Author

thanks alot this helped me get to the next problem, I cannot get arround this and I see there are other peope having the same problem, it can be a change for a newer version of vllm. Currently Im trying with vllm v0.6.6.post1, I beleave it works if I will go with and older version.

INFO 2025-03-12 22:57:58,688 llm_VLLMDeployment 2je6s29h e02a48fd-822a-48aa-a529-b5ba84ea1ff3 -- Request: messages=[{'content': '### Task:\nYou are an autocompletion system. Continue the text in `<text>` based on the **completion type** in `<type>` and the given language.  \n\n### **Instructions**:\n1. Analyze `<text>` for context and meaning.  \n2. Use `<type>` to guide your output:  \n   - **General**: Provide a natural, concise continuation.  \n   - **Search Query**: Complete as if generating a realistic search query.  \n3. Start as if you are directly continuing `<text>`. Do **not** repeat, paraphrase, or respond as a model. Simply complete the text.  \n4. Ensure the continuation:\n   - Flows naturally from `<text>`.  \n   - Avoids repetition, overexplaining, or unrelated ideas.  \n5. If unsure, return: `{ "text": "" }`.  \n\n### **Output Rules**:\n- Respond only in JSON format: `{ "text": "<your_completion>" }`.\n\n### **Examples**:\n#### Example 1:  \nInput:  \n<type>General</type>  \n<text>The sun was setting over the horizon, painting the sky</text>  \nOutput:  \n{ "text": "with vibrant shades of orange and pink." }\n\n#### Example 2:  \nInput:  \n<type>Search Query</type>  \n<text>Top-rated restaurants in</text>  \nOutput:  \n{ "text": "New York City for Italian cuisine." }  \n\n---\n### Context:\n<chat_history>\nUSER: test\nASSISTANT: \n</chat_history>\n<type>search query</type>  \n<text>test</text>  \n#### Output:\n', 'role': 'user'}] model='test' frequency_penalty=0.0 logit_bias=None logprobs=False top_logprobs=0 max_tokens=None max_completion_tokens=None n=1 presence_penalty=0.0 response_format=None seed=None stop=[] stream=False stream_options=None temperature=None top_p=None tools=None tool_choice='none' parallel_tool_calls=False user=None best_of=None use_beam_search=False top_k=None min_p=None repetition_penalty=None length_penalty=1.0 stop_token_ids=[] include_stop_str_in_output=False ignore_eos=False min_tokens=0 skip_special_tokens=True spaces_between_special_tokens=True truncate_prompt_tokens=None prompt_logprobs=None echo=False add_generation_prompt=True continue_final_message=False add_special_tokens=False documents=None chat_template=None chat_template_kwargs=None guided_json=None guided_regex=None guided_choice=None guided_grammar=None guided_decoding_backend=None guided_whitespace_pattern=None priority=0 request_id='781b28c4316c405d97bff03d93ed9154' logits_processors=None
ERROR 2025-03-12 22:57:58,690 llm_VLLMDeployment 2je6s29h e02a48fd-822a-48aa-a529-b5ba84ea1ff3 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 474, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 882, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 648, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 585, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1610, in call_user_method
    result, sync_gen_consumed = await self._call_func_or_gen(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1328, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 502, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-03-12_12-59-05_244804_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 86, in create_chat_completion
    generator = await self.openai_serving_chat.create_chat_completion(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 112, in create_chat_completion
    error_check_ret = await self._check_model(request)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 206, in _check_model
    if self._is_model_supported(request.model):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 665, in _is_model_supported
    return any(model.name == model_name for model in self.base_model_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 665, in <genexpr>
    return any(model.name == model_name for model in self.base_model_paths)
               ^^^^^^^^^^
AttributeError: 'str' object has no attribute 'name'
INFO 2025-03-12 22:57:58,793 llm_VLLMDeployment 2je6s29h e02a48fd-822a-48aa-a529-b5ba84ea1ff3 -- POST /v1/chat/completions 500 171.7ms
INFO 2025-03-12 22:57:59,067 llm_VLLMDeployment 2je6s29h 0f5253d5-60b0-4838-a944-9f636911d96a -- Request: messages=[{'content': 'test', 'role': 'user'}] model='test' frequency_penalty=0.0 logit_bias=None logprobs=False top_logprobs=0 max_tokens=None max_completion_tokens=None n=1 presence_penalty=0.0 response_format=None seed=None stop=[] stream=True stream_options=None temperature=None top_p=None tools=None tool_choice='none' parallel_tool_calls=False user=None best_of=None use_beam_search=False top_k=None min_p=None repetition_penalty=None length_penalty=1.0 stop_token_ids=[] include_stop_str_in_output=False ignore_eos=False min_tokens=0 skip_special_tokens=True spaces_between_special_tokens=True truncate_prompt_tokens=None prompt_logprobs=None echo=False add_generation_prompt=True continue_final_message=False add_special_tokens=False documents=None chat_template=None chat_template_kwargs=None guided_json=None guided_regex=None guided_choice=None guided_grammar=None guided_decoding_backend=None guided_whitespace_pattern=None priority=0 request_id='a5f5d43d9b1a40bc9b35d8209367e187' logits_processors=None
ERROR 2025-03-12 22:57:59,068 llm_VLLMDeployment 2je6s29h 0f5253d5-60b0-4838-a944-9f636911d96a -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 474, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 882, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 648, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 585, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1610, in call_user_method
    result, sync_gen_consumed = await self._call_func_or_gen(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1328, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 502, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-03-12_12-59-05_244804_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 86, in create_chat_completion
    generator = await self.openai_serving_chat.create_chat_completion(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 112, in create_chat_completion
    error_check_ret = await self._check_model(request)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 206, in _check_model
    if self._is_model_supported(request.model):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 665, in _is_model_supported
    return any(model.name == model_name for model in self.base_model_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 665, in <genexpr>
    return any(model.name == model_name for model in self.base_model_paths)
               ^^^^^^^^^^
AttributeError: 'str' object has no attribute 'name'
INFO 2025-03-12 22:57:59,073 llm_VLLMDeployment 2je6s29h 0f5253d5-60b0-4838-a944-9f636911d96a -- POST /v1/chat/completions 500 14.5ms

omrishiv Jun 8, 2025

@torsteinelv did you ever figure this out? I'm working through a similar issue right now

torsteinelv · 2025-06-09T15:32:54Z

torsteinelv
Jun 9, 2025
Author

Yes i did it like this

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: vllm
  namespace: vllm
spec:
  replicas: 0
  leaderWorkerTemplate:
    size: 3
    restartPolicy: RecreateGroupOnPodRestart
    leaderTemplate:
      metadata:
        labels:
          role: leader
      spec:
        nodeSelector:
          kubernetes.io/hostname: k8s-w3
        containers:
          - name: vllm-leader
            image: docker.io/vllm/vllm-openai:v0.8.1
            env:
              - name: VLLM_USE_V1
                value: "0"
              - name: HUGGING_FACE_HUB_TOKEN
                valueFrom:
                  secretKeyRef:
                    name: hf-token-secret
                    key: token
            command:
              - sh
              - -c
              - >
                bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
                python3 -m vllm.entrypoints.openai.api_server --port 8080 --model unsloth/Phi-3-medium-4k-instruct-bnb-4bit --quantization bitsandbytes --load-format bitsandbytes --max-model-len 1024 --enforce-eager --trust-remote-code --dtype float16 --gpu-memory-utilization 0.8 --tensor-parallel-size 1 --pipeline_parallel_size 3
            resources:
              limits:
                nvidia.com/gpu: "1"
                ephemeral-storage: 10Gi
              requests:
                ephemeral-storage: 10Gi
            ports:
              - containerPort: 8080
            readinessProbe:
              tcpSocket:
                port: 8080
              initialDelaySeconds: 600
              periodSeconds: 10
            volumeMounts:
              - mountPath: /dev/shm
                name: dshm
        volumes:
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 10Gi
    workerTemplate:
      metadata:
        labels:
          role: worker
      spec:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 1
                preference:
                  matchExpressions:
                    - key: kubernetes.io/hostname
                      operator: In
                      values:
                        - k8s-w1
                        - k8s-w2
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                role: worker
        containers:
          - name: vllm-worker
            image: ghcr.io/sasha0552/vllm:v0.8.1
            command:
              - sh
              - -c
              - |
                pip install -U ray==2.43.0 ray[default]==2.43.0 && \
                echo "Forced Ray version to 2.43.0" && \
                sleep 15 && \
                bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS) && \
                tail -f /dev/null
            resources:
              limits:
                nvidia.com/gpu: "1"
                ephemeral-storage: 10Gi
              requests:
                ephemeral-storage: 10Gi
            env:
              - name: HUGGING_FACE_HUB_TOKEN
                valueFrom:
                  secretKeyRef:
                    name: hf-token-secret
                    key: token
            volumeMounts:
              - mountPath: /dev/shm
                name: dshm
        volumes:
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-leader
  namespace: vllm
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    leaderworkerset.sigs.k8s.io/name: vllm
    role: leader
  type: LoadBalancer

3 replies

omrishiv Jun 9, 2025

Thank you! I actually did manage to sort mine out without LWS. I’ll follow up with my solution

torsteinelv Jun 9, 2025
Author

Yea i did also but more easy and i guess better with lws

omrishiv Jun 10, 2025

As I know I'll eventually run into this issue again if I don't document it, vLLM ~0.6 version needs
base_model_paths = [BaseModelPath(name=self.engine_args.model, model_path=self.engine_args.model)] for the base_model_paths for OpenAI.

Uh oh!

Help serving vllm with kuberay #11957

Uh oh!

Uh oh!

torsteinelv Jan 11, 2025

Replies: 2 comments · 5 replies

Uh oh!

mdklause Jan 21, 2025

Uh oh!

Uh oh!

torsteinelv Mar 13, 2025 Author

Uh oh!

omrishiv Jun 8, 2025

Uh oh!

torsteinelv Jun 9, 2025 Author

Uh oh!

omrishiv Jun 9, 2025

Uh oh!

torsteinelv Jun 9, 2025 Author

Uh oh!

omrishiv Jun 10, 2025

torsteinelv
Jan 11, 2025

Replies: 2 comments 5 replies

mdklause
Jan 21, 2025

torsteinelv Mar 13, 2025
Author

torsteinelv
Jun 9, 2025
Author

torsteinelv Jun 9, 2025
Author