Structured output generation on IBM Z with llguidance backend now works correctly.

Previously, when using the llguidance structured decoding backend on IBM Z (s390x), JSON Schema-constrained generation produced invalid output or became stuck generating whitespace indefinitely. This issue is now resolved.

Sarvam-105B model now loads successfully.

Previously, the RedHatAI/sarvam-105b-FP8-Dynamic model failed to load due to a RoPE validation error caused by a Transformers v5 incompatibility. This issue is now resolved.

Prithvi TerraTorch model warmup fails on arm64 GH200 due to tensor size limit.

When deploying the ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11 model on arm64 NVIDIA GH200 accelerators, the vLLM engine core fails during warmup with the error "upsample_bilinear2d_nhwc only supports output tensors with less than INT_MAX elements". A PyTorch CUDA kernel limitation on arm64 causes this issue when the TerraTorch UPerNet decoder warmup tensor exceeds the signed 32-bit integer maximum.

To work around this issue, deploy the model with limited concurrency:

--max-num-seqs=8 --max-num-batched-tokens=8

This configuration does not affect performance for geospatial models because each request uses only one token.

DeepSeek-V4-Flash-NVFP4-FP8 model crashes during initialization.

The RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8 quantized model fails to load with a KeyError for scale_fmt in the quantization configuration. The DeepseekV4Attention initialization expects a scale_fmt key that is not present in the model configuration.

Mistral Small 3.1 tool calling returns no tool calls with tool_choice=auto.

When using the Mistral Small 3.1 model with tool calling and tool_choice=auto, the model does not return tool calls in the response.

Qwen2.5-VL FP8 tool calling fails because the model does not support tools.

The Qwen2.5-VL FP8 model fails tool calling smoke tests because the model architecture does not support tool calling functionality.

Mistral Small FP8-dynamic fails to start due to missing backend attribute.

The Mistral Small FP8-dynamic model fails to start with an error indicating that CachedMistralCommonBackend is missing the is_fast attribute.

Gemma 4 12B model evaluation crashes due to batch token limit.

When running LM Evaluation Harness on the Gemma 4 12B model, the evaluation crashes with an error indicating that max_num_batched_tokens is too small for the prefix-LM multimodal model.

To work around this issue, explicitly set --max-num-batched-tokens to any value greater than 2496:

--max-num-batched-tokens 4096

Granite 4.0-h-small model crashes on first inference with ROCm.

When deploying the ibm-granite/granite-4.0-h-small model on AMD ROCm accelerators, the vLLM server crashes during the first inference request with a RuntimeError: PassManager::run failed error. This occurs because the Triton JIT compiler fails to compile Mamba chunk_scan kernels on the first run. After the pod restarts, the model works correctly due to the warm compilation cache.

Qwen 3.5 and 3.6 models fail on CPU deployments.

Qwen 3.5 and 3.6 models fail to run correctly on CPU-only deployments due to an upstream vLLM issue with GDN attention operations.

Intel Gaudi deployments fail due to habana-torch-plugin version rejection.

When deploying models on Intel Gaudi accelerators, the vllm-gaudi configuration validation rejects the habana-torch-plugin version 1.23.0.post1 because it does not conform to PEP 440 versioning requirements.

Prefix caching with models requiring transformers 5.x crashes the inference scheduler.

Deploying models whose architecture was introduced in transformers 5.0 or later with prefix caching enabled (precise-prefix-cache-scorer) causes the inference scheduler to crash. The tokenizer sidecar (odh-llm-d-kv-cache-rhel9) ships transformers 4.57.6, which does not recognize architecture types added in transformers 5.x. Affected models include Gemma 4, GLM-5, Qwen3.5 MoE, and others added after the 4.x series.

To work around this issue, disable precise-prefix-cache-scorer in the EndpointPickerConfig when using these models.

This issue is fixed in 3.5 stable. The UDS tokenizer sidecar is replaced by a vLLM render sidecar that stays in sync with the serving image’s transformers version.

AMD ROCm deployments on MI300X encounter memory access faults during CUDA graph capture.

When running models such as meta-llama/Llama-3.2-1B-Instruct or ibm-granite/granite-3.2-2b-instruct on AMD MI300X accelerators, GPU memory access faults occur with the error "Memory access fault by GPU node-2". This issue is caused by an incompatibility between vLLM CUDA Graph FULL mode and MI300X VF memory behavior under ROCm.

To work around this issue, set cudagraph_mode=PIECEWISE in your vLLM configuration:

--compilation-config '{"cudagraph_mode":"PIECEWISE"}'

Phi-4 Vision model fails to load with SigLIP2 compatibility error.

The RedHatAI/Phi-4-reasoning-vision-15B model requires a filter_out_non_signature_kwargs() function that is missing from the transformers library bundled with vLLM. The vLLM API server crashes at startup and cannot serve the model.

Sarvam-105B model fails to load with RoPE validation error.

The RedHatAI/sarvam-105b-FP8-Dynamic model configuration calls validate_rope(ignore_keys=…​), but the ignore_keys parameter was removed in Transformers v5. The vLLM API server crashes during model initialization.

AMD Zen CPU deployments crash due to zentorch plugin incompatibility.

The zentorch vLLM plugin supports vLLM versions 0.15.0 through 0.18.0, but Red Hat AI Inference 3.5.0-ea.1 ships vLLM 0.19.1. The engine crashes during model loading with a segfault.

To work around this issue, unset the VLLM_PLUGINS environment variable:

unset VLLM_PLUGINS

Structured output generation fails on IBM Z when using the llguidance backend.

When using the llguidance structured decoding backend on IBM Z (s390x), JSON Schema-constrained generation may produce invalid output or become stuck generating whitespace indefinitely.

Gemma 4 12B model deployment might fail due to unsupported gemma4_unified architecture in the bundled Transformers version.

Gemma 4 12B models use the gemma4_unified architecture, which is not supported by the version of Transformers included in the Red Hat AI Inference container image. Deployment of the google/gemma-4-12B-it model fails during vLLM initialization with the error "The checkpoint you are trying to load has model type gemma4_unified but Transformers does not recognize this architecture." Other Gemma 4 variants deploy successfully because they use different model architectures that are already supported.