Run in headless mode. See multi-node data parallel documentation for more details.

Default: False

How many API server processes to run. Defaults to data_parallel_size if not specified.

Read CLI options from a config file. Must be a YAML with the following options: Content from docs.vllm.ai is not included.https://docs.vllm.ai/en/latest/configuration/serve_args.html

Disable logging statistics.

Default: False

Log aggregate rather than per-engine statistics when using data parallelism.

Default: False

If set, the engine will raise an error if environment validation fails.

Default: False

Shutdown timeout in seconds. 0 = abort, >0 = wait.

Default: 0

Possible choices: flashinfer, triton Select GDN prefill backend.

Enable logging request information, dependent on log level:

You can set the minimum log level via VLLM_LOGGING_LEVEL.

Default: False

Possible choices: auto, openai, string

Default: auto

Default: False

Default: assistant

Default: False

Default: False

Default: False

Default: False

Default: ""

Default: False

Default: False

Default: False

Default: False

Default: False

Default: True

Default: False

Default: False

Host name.

Port number.

Default: 8000

Unix domain socket path. If set, host and port arguments are ignored.

Possible choices: critical, debug, error, info, trace, warning Log level for uvicorn. Default: info

Disable uvicorn access log.

Default: False

Comma-separated list of endpoint paths to exclude from uvicorn access logs. This is useful to reduce log noise from high-frequency endpoints like health checks. Example: "/health,/metrics,/ping". When set, access logs for requests to these paths will be suppressed while keeping logs for other endpoints.

Allow credentials.

Default: False

Allowed origins.

Default: ['*']

Allowed methods.

Default: ['*']

Allowed headers.

Default: ['*']

If provided, the server will require one of these keys to be presented in the header.

The file path to the SSL key file.

The file path to the SSL cert file.

The CA certificates file.

Refresh SSL Context when SSL certificate files change

Default: False

Whether client certificate is required (see stdlib ssl module’s).

Default: 0

SSL cipher suites for HTTPS (TLS 1.2 and below only). Example: 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305'

FastAPI root_path when app is behind a path based routing proxy.

Additional ASGI middleware to apply to the app. We accept multiple --middleware arguments. The value should be an import path. If a function is provided, vLLM will add it to the server using @app.middleware('http'). If a class is provided, vLLM will add it to the server using app.add_middleware().

Default: []

If specified, API server will add X-Request-Id header to responses.

Default: False

Disable FastAPI’s OpenAPI schema, Swagger UI, and ReDoc endpoint.

Default: False

Maximum size (bytes) of an incomplete HTTP event (header or body) for h11 parser. Helps mitigate header abuse. Default: 4194304 (4 MB).

Default: 4194304

Maximum number of HTTP headers allowed in a request for h11 parser. Helps mitigate header abuse. Default: 256.

Default: 256

Enable offline FastAPI documentation for air-gapped environments. Uses vendored static assets bundled with vLLM.

Default: False

Name or path of the Hugging Face model to use. It is also used as the content for model_name tag in metrics output when served_model_name is not specified.

Default: Qwen/Qwen3-0.6B

Possible choices: auto, draft, generate, pooling The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types. Default: auto

Possible choices: auto, classify, embed, none Convert the model using adapters defined in [vllm.model_executor.models.adapters][]. The most common use case is to adapt a text generation model to be used for pooling tasks. Default: auto

Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used.

Possible choices: auto, deepseek_v32, hf, mistral, slow Tokenizer mode:

Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer.

Default: False

Possible choices: auto, bfloat16, float, float16, float32, half Data type for model weights and activations:

Random seed for reproducibility. We must set the global seed because otherwise, different tensor parallel workers would sample different tokens, leading to inconsistent results.

Default: 0

Name or path of the Hugging Face config to use. If unspecified, model name or path will be used.

Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments.

Default: ""

If set, only media URLs that belong to this domain can be used for multi-modal inputs.

The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.

The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.

The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.

Model context length (prompt and output). If unspecified, will be automatically derived from the model config. When passing via --max-model-len, supports k/m/g/K/M/G in human-readable format. Examples:

Method used to quantize the weights. If None, we first check the quantization_config attribute in the model config file. If that is None, we assume the model weights are not quantized and use dtype to determine the data type of the weights.

Whether to allow deprecated quantization methods.

Default: False

Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility.

Default: False

Whether to return routed experts.

Default: False

Maximum number of log probabilities to return when logprobs is specified in SamplingParams. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output_length * vocab_size) logprobs are allowed to be returned and it may cause OOM.

Default: 20

Possible choices: processed_logits, processed_logprobs, raw_logits, raw_logprobs Indicates the content returned in the logprobs and prompt_logprobs. Supported mode: 1) raw_logprobs, 2) processed_logprobs, 3) raw_logits, 4) processed_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top_k/top_p. Default: raw_logprobs

Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored.

Default: False

Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be useful for preventing potential numerical issues. This defaults to True, so users must opt in to cascade attention by setting this to False. Even when this is set to False, cascade attention will only be used when the heuristic tells that it’s beneficial.

Default: True

Skip initialization of tokenizer and detokenizer. Expects valid prompt_token_ids and None for prompt from the input. The generated output will contain token ids.

Default: False

If True, enables passing text embeddings as inputs via the prompt_embeds key. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users!

Default: False

The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the --model argument. Noted that this name(s) will also be used in model_name tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one.

Possible choices: auto, hf, mistral The format of the model config to load:

The token to use as HTTP bearer authorization for remote files . If True, will use the token generated when running hf auth login (stored in ~/.cache/huggingface/token).

If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config.

Default: {}

Pooler config which controls the behaviour of output pooling in pooling models. Should either be a valid JSON string or JSON keys passed individually.

The folder path to the generation config. Defaults to "auto", the generation config will be loaded from model path. If set to "vllm", no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If max_new_tokens is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests.

Default: auto

Overrides or sets generation config. e.g. {"temperature": 0.5}. If used with --generation-config auto, the override parameters will be merged with the default config from the model. If used with --generation-config vllm, only the override parameters are used. Should either be a valid JSON string or JSON keys passed individually.

Default: {}

Enable sleep mode for the engine (only cuda and hip platforms are supported).

Default: False

Possible choices: auto, terratorch, transformers, vllm Which implementation of the model to use:

Override dtype for attention

One or more logits processors' fully-qualified class names or class definitions

IOProcessor plugin name to load at model startup

The format of the model weights to load:

Default: auto

Directory to download and load the weights, default to the default cache directory of Hugging Face.

Specifies the loading strategy for safetensors weights.

on-demand loading and is highly efficient for models on local storage.

This is recommended for models on network filesystems (e.g., Lustre, NFS) as it avoids inefficient random reads, significantly speeding up model initialization. However, it uses more CPU RAM.

workers load them, speeding up the model loading phase. Useful on network or high-latency storage.

into torchao tensor subclasses. This is used when the checkpoint was quantized using torchao and saved using safetensors. Needs torchao >= 0.14.0

Default: lazy

Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format.

Default: {}

The list of patterns to ignore when loading the model. Default to "original/*/" to avoid repeated loading of llama’s checkpoints.

Default: ['original/**/*']

Whether to enable tqdm for showing progress bar when loading model weights.

Default: True

pt_load_map_location: the map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to {"": "cuda"}. Another supported format is mapping from different devices like from GPU 1 to GPU 0: {"cuda:1": "cuda:0"}. Note that when passed from command line, the strings in dictionary needs to be double quoted for json parsing. For more details, see original doc for map_location in Content from pytorch.org is not included.https://pytorch.org/docs/stable/generated/torch.load.html

Default: cpu

Attention backend to use. Use "auto" or None for automatic selection.

Select the reasoning parser depending on the model that you’re using. This is used to parse the reasoning content into OpenAI API format.

Default: ""

Path to a dynamically reasoning parser plugin that can be dynamically loaded and registered.

Default: ""

Possible choices: external_launcher, mp, ray, uni Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline_parallel_size and tensor_parallel_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, an error will be raised. To use "mp" you must also set nnodes, and to use "ray" you must manually set distributed_executor_backend to "ray".

+ Note that tpu only support Ray for distributed inference.

Number of pipeline parallel groups.

Default: 1

distributed master address for multi-node distributed inference when distributed_executor_backend is mp.

Default: 127.0.0.1

distributed master port for multi-node distributed inference when distributed_executor_backend is mp.

Default: 29501

num of nodes for multi-node distributed inference when distributed_executor_backend is mp.

Default: 1

distributed node rank for multi-node distributed inference when distributed_executor_backend is mp.

Default: 0

Timeout in seconds for distributed operations (e.g., init_process_group). If set, this value is passed to torch.distributed.init_process_group as the timeout parameter. If None, PyTorch’s default timeout is used (600s for NCCL). Increase this for multi-node setups where model downloads may be slow.

Number of tensor parallel groups.

Default: 1

Number of decode context parallel groups, because the world size does not change by dcp, it simply reuse the GPUs of TP group, and tp_size needs to be divisible by dcp_size.

Default: 1

Possible choices: a2a, ag_rs Communication backend for Decode Context Parallel (DCP).

Interleave size of kv_cache storage while using DCP. dcp_kv_cache_interleave_size has been replaced by cp_kv_cache_interleave_size, and will be deprecated when PCP is fully supported.

Default: 1

Interleave size of kv_cache storage while using DCP or PCP. For total_cp_rank = pcp_rank * dcp_world_size + dcp_rank, and total_cp_world_size = pcp_world_size * dcp_world_size. store interleave_size tokens on total_cp_rank i, then store next interleave_size tokens on total_cp_rank i+1. Interleave_size=1: token-level alignment, where token i is stored on total_cp_rank i %% total_cp_world_size. Interleave_size=block_size: block-level alignment, where tokens are first populated to the preceding ranks. Tokens are then stored in (rank i+1, block j) only after (rank i, block j) is fully occupied. Block_size should be greater than or equal to cp_kv_cache_interleave_size. Block_size should be divisible by cp_kv_cache_interleave_size.

Default: 1

Number of prefill context parallel groups.

Default: 1

Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size.

Default: 1

Data parallel rank of this instance. When set, enables external load balancer mode.

Starting data parallel rank for secondary nodes.

Number of data parallel replicas to run on this node.

Address of data parallel cluster head-node.

Port for data parallel RPC communication.

Backend for data parallel, either "mp" or "ray".

Default: mp

Whether to use "hybrid" DP LB mode. Applies only to online serving and when data_parallel_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank.

Default: False

Whether to use "external" DP LB mode. Applies only to online serving and when data_parallel_size > 0. This is useful for a "one-pod-per-rank" wide-EP setup in Kubernetes. Set implicitly when --data-parallel-rank is provided explicitly to vllm serve.

Default: False

Use expert parallelism instead of tensor parallelism for MoE layers.

Default: False

Skip non-local expert weights during model loading when expert parallelism is active. Each rank only reads its own expert shard from disk, which can drastically reduce storage I/O for MoE models with per-expert weight tensors (e.g. DeepSeek, Mixtral, Kimi-K2.5). Has no effect on 3D fused-expert checkpoints (e.g. GPT-OSS) or non-MoE models.

Default: False

Possible choices: allgather_reducescatter, deepep_high_throughput, deepep_low_latency, flashinfer_all2allv, flashinfer_nvlink_one_sided, flashinfer_nvlink_two_sided, mori, naive, nixl_ep, pplx All2All backend for MoE expert parallel communication. Available options:

Enable dual batch overlap for the model executor.

Default: False

Number of ubatch size.

Default: 0

Enable elastic expert parallelism with stateless NCCL groups for DP/EP.

Default: False

The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch.

Default: 32

The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch.

Default: 512

Forces the dp synchronization logic in vllm/v1/worker/dp_utils.py to use Gloo instead of NCCL for its all reduce. Defaults to True when async scheduling is enabled, False otherwise.

Enable expert parallelism load balancing for MoE layers.

Default: False

Expert parallelism configuration. Should either be a valid JSON string or JSON keys passed individually.

Default:

EPLBConfig(window_size=1000, step_interval=3000, num_redundant_experts=0, log_balancedness=False, log_balancedness_interval=1, use_async=False, policy='default')

Possible choices: linear, round_robin The expert placement strategy for MoE layers:

Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models.

Whether to profile Ray workers with nsight, see Content from docs.ray.io is not included.https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler.

Default: False

Disable the custom all-reduce kernel and fall back to NCCL.

Default: False

The full name of the worker class to use. If "auto", the worker class will be determined based on the platform.

Default: auto

The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective_rpc calls.

Default: ""

Size of a contiguous cache block in number of tokens. Accepts None (meaning "use default"). After construction, always int.

The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50%% GPU memory utilization. If unspecified, will use the default value of 0.9. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance.

Default: 0.9

Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu_memory_utilization. However, users may want to manually specify the kv cache memory size. kv_cache_memory_bytes allows more fine-grain control of how much memory gets used when compared with using gpu_memory_utilization. Note that kv_cache_memory_bytes (when not-None) ignores gpu_memory_utilization Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. Examples: - '1k' → 1,000 - '1K' → 1,024 - '25.6k' → 25,600

Possible choices: auto, bfloat16, float16, fp8, fp8_ds_mla, fp8_e4m3, fp8_e5m2, fp8_inc Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8_e4m3) and fp8_e5m2. ROCm (AMD GPU) supports fp8 (=fp8_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8_inc). Some models (namely DeepSeekV3.2) default to fp8, set to bfloat16 to use bfloat16 instead, this is an invalid option for models that do not default to fp8. Default: auto

Number of GPU blocks to use. This overrides the profiled num_gpu_blocks if specified. Does nothing if None. Used for testing preemption.

Whether to enable prefix caching.

Possible choices: sha256, sha256_cbor, xxhash, xxhash_cbor Set the hash algorithm for prefix caching:

This enables dynamic calculation of k_scale and v_scale when kv_cache_dtype is fp8. If False, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0.

Default: False

This feature is work in progress and no prefill optimization takes place with this flag enabled currently. In some KV sharing setups, e.g. YOCO (Content from arxiv.org is not included.https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n)

Default: False

Possible choices: auto, float16, float32 The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config. Default: auto

Possible choices: auto, float16, float32 The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba_cache_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba_cache_dtype. Default: auto

Size of a contiguous cache block in number of tokens for mamba cache. Can be set only when prefix caching is enabled. Value must be a multiple of 8 to align with causal_conv1d kernel.

Possible choices: align, all, none The cache strategy for Mamba layers.

Size of the KV cache offloading buffer in GiB. When TP > 1, this is the total buffer size summed across all TP ranks. By default, this is set to None, which means no KV offloading is enabled. When set, vLLM will enable KV cache offloading to CPU using the kv_offloading_backend.

Possible choices: lmcache, native The backend to use for KV cache offloading. Supported backends include 'native' (vLLM native CPU offloading), 'lmcache'. KV offloading is only activated when kv_offloading_size is set. Default: native

Possible choices: auto, prefetch, uva The backend for weight offloading. Options:

The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. This uses UVA (Unified Virtual Addressing) for zero-copy access.

Default: 0

The set of parameter name segments to target for CPU offloading. Unmatched parameters are not offloaded. If this set is empty, parameters are offloaded non-selectively until the memory limit defined by cpu_offload_gb is reached. Examples:

This allows distinguishing parameters like "w2_weight" and "w2_weight_scale".

Default: set()

Group every N layers together. Offload last offload_num_in_group layers of each group. Default is 0 (disabled). Example: group_size=8, num_in_group=2 offloads layers 6,7,14,15,22,23,…​ Unlike cpu_offload_gb, this uses explicit async prefetching to hide transfer latency.

Default: 0

Number of layers to offload per group. Must be <= offload_group_size. Default is 1.

Default: 1

Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1.

Default: 1

The set of parameter name segments to target for prefetch offloading. Unmatched parameters are not offloaded. If this set is empty, ALL parameters of each offloaded layer are offloaded. Uses segment matching: "w13_weight" matches "mlp.experts.w13_weight" but not "mlp.experts.w13_weight_scale".

Default: set()

If True, disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting --limit-mm-per-prompt to 0 for every modality.

Default: False

The maximum number of input items and options allowed per prompt for each modality. Defaults to 999 for each modality. Legacy format (count only): {"image": 16, "video": 2} Configurable format (with options): {"video": {"count": 1, "num_frames": 32, "width": 512, "height": 512}, "image": {"count": 5, "width": 512, "height": 512}} Mixed format (combining both): {"image": 16, "video": {"count": 1, "num_frames": 32, "width": 512, "height": 512}} Should either be a valid JSON string or JSON keys passed individually.

Default: {}

If True, enables passing multimodal embeddings: for LLM class, this refers to tensor inputs under multi_modal_data; for the OpenAI-compatible server, this refers to chat messages with content "type": "*_embeds". When enabled with --limit-mm-per-prompt set to 0 for a modality, precomputed embeddings skip count validation for that modality, saving memory by not loading encoder modules while still enabling embeddings as an input. Limits greater than 0 still apply to embeddings. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users!

Default: False

Additional args passed to process media inputs, keyed by modalities. For example, to set num_frames for video, set --media-io-kwargs '{"video": {"num_frames": 40} }' Should either be a valid JSON string or JSON keys passed individually.

Default: {}

Arguments to be forwarded to the model’s processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from transformers.AutoProcessor.from_pretrained. The available overrides depend on the model that is being run. For example, for Phi-3-Vision: {"num_crops": 4}. Should either be a valid JSON string or JSON keys passed individually.

The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs. This cache is duplicated for each API process and engine core process, resulting in a total memory usage of mm_processor_cache_gb * (api_server_count + data_parallel_size). Set to 0 to disable this cache completely (not recommended).

Default: 4

Possible choices: lru, shm Type of cache to use for the multi-modal preprocessor/mapper. If shm, use shared memory FIFO cache. If lru, use mirrored LRU cache. Default: lru

Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when mm_processor_cache_type is "shm".

Default: 128

When enabled, skips the language component of the model. This is usually only valid in disaggregated Encoder process.

Default: False

Possible choices: data, weights Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP).

Optional override for the multi-modal encoder attention backend when using vision transformers. Accepts any value from vllm.v1.attention.backends.registry.AttentionBackendEnum (e.g. FLASH_ATTN).

Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string.

Default: False

When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization. This reduces engine startup time but shifts the responsibility to users for estimating the peak memory usage of the activation of multimodal encoder and embedding cache.

Default: False

Sets pruning rate for video pruning via Efficient Video Sampling. Value sits in range [0;1) and determines fraction of media tokens from each video to be pruned.

If True, enable handling of LoRA adapters.

Max number of LoRAs in a single batch.

Default: 1

Possible choices: 1, 8, 16, 32, 64, 128, 256, 320, 512 Max LoRA rank. Default: 16

Data type for LoRA. If auto, will default to base model dtype.

Default: auto

If True, LoRA support for the tower (vision encoder) and connector of multimodal models will be enabled. This is an experimental feature and currently only supports some MM models such as the Qwen VL series. The default is False.

Default: False

Maximum number of LoRAs to store in CPU memory. Must be >= than max_loras.

By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster.

Default: False

Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default_mm_loras because we currently only support one lora adapter per prompt. When run in offline mode, the lora IDs for n modalities will be automatically assigned to 1-n with the names of the modalities in alphabetic order. Should either be a valid JSON string or JSON keys passed individually.

Whether to construct lora kernel grid by the number of active LoRA adapters. When set to True, separate cuda graphs will be captured for different counts of active LoRAs (powers of 2 up to max_loras), which can improve performance for variable LoRA usage patterns at the cost of increased startup time and memory usage. Only takes effect when cudagraph_specialize_lora is True.

Default: False

Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use --show-hidden-metrics-for-version=0.7 as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release.

Target URL to which OpenTelemetry traces will be sent.

Possible choices: all, model, worker, None, model,worker, model,all, worker,model, worker,all, all,model, all,worker It makes sense to set this only if --otlp-traces-endpoint is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact.

+ Note that collecting detailed timing information for each request can be expensive.

Enable KV cache residency metrics (lifetime, idle time, reuse gaps). Uses sampling to minimize overhead. Requires log stats to be enabled (i.e., --disable-log-stats not set).

Default: False

Sampling rate for KV cache metrics (0.0, 1.0]. Default 0.01 = 1%% of blocks.

Default: 0.01

Enable CUDA graph metrics (number of padded/unpadded tokens, runtime cudagraph dispatch modes, and their observed frequencies at every logging interval).

Default: False

Enable layerwise NVTX tracing. This traces the execution of each layer or module in the model and attach information such as input/output shapes to nvtx range markers. Noted that this doesn’t work with CUDA graphs enabled.

Default: False

Enable Model FLOPs Utilization (MFU) metrics.

Default: False

Enable detailed logging of iteration details. If set, vllm EngineCore will log iteration details This includes number of context/generation requests and tokens and the elapsed cpu time for the iteration.

Default: False

Maximum number of tokens that can be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in EngineArgs.create_engine_config. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. Examples: - '1k' → 1,000 - '1K' → 1,024 - '25.6k' → 25,600

Maximum number of sequences to be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in EngineArgs.create_engine_config.

For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently.

Default: 1

For chunked prefill, the maximum number of prompts longer than long_prefill_token_threshold that will be prefilled concurrently. Setting this less than max_num_partial_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency.

Default: 1

For chunked prefill, a request is considered long if the prompt is longer than this number of tokens.

Default: 0

Possible choices: fcfs, priority The scheduling policy to use:

If True, prefill requests can be chunked based on the remaining max_num_batched_tokens. The default value here is mainly for convenience when testing. In real usage, this should be set in EngineArgs.create_engine_config.

If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next.

Default: False

The scheduler class to use. "vllm.v1.core.sched.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom_class".

If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention. If set to None, the default value will be determined based on the environment and starting configuration.

If set to False, disable async scheduling. Async scheduling helps to avoid gaps in GPU utilization, leading to better latency and throughput.

The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending.

Default: 1

Sizes to capture cudagraph. - None (default): capture sizes are inferred from vllm config. - list[int]: capture sizes are specified as given.

The maximum cudagraph capture size. If cudagraph_capture_sizes is specified, this will be set to the largest size in that list (or checked for consistency if specified). If cudagraph_capture_sizes is not specified, the list of sizes is generated automatically following the pattern: [1, 2, 4] + list(range(8, 256, 8)) + list( range(256, max_cudagraph_capture_size + 1, 16)) If not specified, max_cudagraph_capture_size is set to min(max_num_seqs*2, 512) by default. This voids OOM in tight memory scenarios with small max_num_seqs, and prevents capture of many large graphs (>512) that would greatly increase startup time with limited performance benefit.

If True, run FlashInfer autotuning during kernel warmup.

Possible choices: aiter, auto, cutlass, deep_gemm, flashinfer_cutedsl, flashinfer_cutlass, flashinfer_trtllm, marlin, triton Backend for MoE expert computation kernels. Available options:

Speculative decoding configuration. Should either be a valid JSON string or JSON keys passed individually.

The configurations for distributed KV cache transfer. Should either be a valid JSON string or JSON keys passed individually.

The configurations for event publishing. Should either be a valid JSON string or JSON keys passed individually.

The configurations for distributed EC cache transfer. Should either be a valid JSON string or JSON keys passed individually.

torch.compile and cudagraph capture configuration for the model. As a shorthand, one can append compilation arguments via -cc.parameter=argument such as -cc.mode=3 (same as -cc='{"mode":3}'). You can specify the full compilation config like so: {"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]} Should either be a valid JSON string or JSON keys passed individually.

Default:

{'mode': None, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'splitting_ops': None, 'compile_mm_encoder': False, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': None, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}

Attention configuration. Should either be a valid JSON string or JSON keys passed individually.

Default:

AttentionConfig(backend=None, flash_attn_version=None, use_prefill_decode_attention=False, flash_attn_max_num_splits_for_cuda_graph=32, use_cudnn_prefill=False, use_trtllm_ragged_deepseek_prefill=False, use_trtllm_attention=None, disable_flashinfer_prefill=True, disable_flashinfer_q_quantization=False, use_prefill_query_quantization=False)

Kernel configuration. Should either be a valid JSON string or JSON keys passed individually.

Default: KernelConfig(enable_flashinfer_autotune=None, moe_backend='auto')

Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable.

Default: {}

Structured outputs configuration. Should either be a valid JSON string or JSON keys passed individually.

Default:

StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False)

Profiling configuration. Should either be a valid JSON string or JSON keys passed individually.

Default:

ProfilerConfig(profiler=None, torch_profiler_dir='', torch_profiler_with_stack=False, torch_profiler_with_flops=False, torch_profiler_use_gzip=True, torch_profiler_dump_cuda_time_total=True, torch_profiler_record_shapes=False, torch_profiler_with_memory=False, ignore_frontend=False, delay_iterations=0, max_iterations=0, warmup_iterations=0, active_iterations=5, wait_iterations=0)

The optimization level. These levels trade startup time cost for performance, with -O0 having the best startup time and -O3 having the best performance. -O2 is used by default. See OptimizationLevel for full description.

Default: 2

Possible choices: balanced, interactivity, throughput Performance mode for runtime behavior, 'balanced' is the default. 'interactivity' favors low end-to-end per-request latency at small batch sizes (fine-grained CUDA graphs, latency-oriented kernels). 'throughput' favors aggregate tokens/sec at high concurrency (larger CUDA graphs, more aggressive batching, throughput-oriented kernels). Default: balanced

The configurations for weight transfer during RL training. Should either be a valid JSON string or JSON keys passed individually.