Speculative decoding

Red Hat AI Inference 3.5

Speculative decoding with Red Hat AI Inference

Red Hat AI Documentation Team

Abstract

Learn about speculative decoding and how to deploy and train custom speculator models using the Speculators library with Red Hat AI Inference.

Chapter 1. About Speculators

Speculators is a unified library for building, training, and storing speculative decoding algorithms for large language model (LLM) inference, including frameworks such as vLLM.

Important

Speculators is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Speculative decoding is an optimization technique that improves inference performance for the LLM you are trying to serve. Red Hat AI Inference supports Eagle 3, a speculative decoding algorithm that uses a small, single-layer draft model and a full-sized 'verifier' model, which is the LLM you are serving. The Eagle 3 speculator model auto-regressively predicts several tokens, and then the verifier model processes these tokens in parallel. As the verifier model can accept multiple tokens per forward pass, effective throughput increases. When the verifier model rejects a token, it samples a corrected token from its own distribution, ensuring the output matches what it would produce alone.

Speculative decoding provides the following advantages:

  • Latency decreases through parallel token validation.
  • Eagle 3 speculator models require minimal processing due to their small size.
  • Output quality matches what the verifier model would produce alone.

Additional resources

Chapter 2. Deploying speculator models

Deploy a trained EAGLE-3 speculator model to accelerate inference using speculative decoding with Red Hat AI Inference.

Prerequisites

Procedure

  1. Log in to the Red Hat container registry: 

    podman login registry.redhat.io

  2. Pull the AI Inference container image: 

    podman pull registry.redhat.io/{rhaii-registry-namespace}/vllm-cuda-rhel9:{rhaiis-version}

  3. Set your Hugging Face token as an environment variable: 

    export HF_TOKEN=<your_huggingface_token>
Note

This example uses the RedHatAI/Qwen3-8B-speculator.eagle3 pre-trained model. For other available speculator models, see the Content from huggingface.co is not included.Red Hat AI speculator models collection. If you encounter an access error, verify that you have accepted a license agreement on Hugging Face before downloading.

  1. Start the inference server with a speculator model: 

    podman run --rm -it \
    --device nvidia.com/gpu=all \
    --shm-size=2g \
    -p 8000:8000 \
    --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
    --env "HF_HUB_OFFLINE=0" \
    registry.redhat.io/{rhaii-registry-namespace}/vllm-cuda-rhel9:{rhaiis-version} \
      --model RedHatAI/Qwen3-8B-speculator.eagle3 \
      --host 0.0.0.0 \
      --port 8000

    vLLM reads the speculator configuration from the model and loads both the draft model and the verifier model.

Verification

  1. In a separate terminal, send a request to the model: 

    curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "RedHatAI/Qwen3-8B-speculator.eagle3",
      "messages": [
        {"role": "user", "content": "Hello, how are you?"}
      ],
      "max_tokens": 50
    }'

    Example output

    {
    "id": "chatcmpl-8dc33cb67b69b432",
    "object": "chat.completion",
    "created": 1776107449,
    "model": "RedHatAI/Qwen3-8B-speculator.eagle3",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "<think>\nOkay, the user greeted me with \"Hello, how are you?\" I need to respond in a friendly and helpful manner..."
        },
        "finish_reason": "length"
      }
    ],
    "usage": {
      "prompt_tokens": 14,
      "total_tokens": 64,
      "completion_tokens": 50
    }
}

Additional resources

Chapter 3. Converting models with Speculators

Convert an existing Eagle 3 speculator model to the Speculators format for use with Red Hat AI Inference. Use this procedure when you have an externally-trained Eagle 3 checkpoint that is not already in the Speculators format.

Prerequisites

Note

This example uses the meta-llama/Meta-Llama-3.1-8B-Instruct model, which requires accepting a license agreement. Before running this procedure, request access at Content from huggingface.co is not included.meta-llama/Llama-3.1-8B-Instruct on Hugging Face.

Procedure

  1. Pull the Red Hat AI Model Optimization Toolkit container image: 

    $ podman pull registry.redhat.io/rhaii-early-access/model-opt-cuda-rhel9:3.5.0-ea.1

  2. Verify the Speculators version installed in the container: 

    $ podman run --rm -it \
  registry.redhat.io/rhaii-early-access/model-opt-cuda-rhel9:3.5.0-ea.1 \
  pip show speculators | grep Version

    Example output

    Version: 0.4.0a1

  3. Create a working directory and clone the upstream Speculators repository: 

    $ mkdir model-opt && \
cd model-opt && \
git clone https://github.com/vllm-project/speculators.git

  4. Check out the Speculators branch that matches the version installed in the container: 

    $ cd speculators && \
git checkout v0.4.0+rhaiis

  5. Create or append your HF_TOKEN Hugging Face token to the private.env file and source it: 

    $ echo "export HF_TOKEN=<YOUR_HF_TOKEN>" > private.env
$ source private.env

  6. If your system has SELinux enabled, configure SELinux to allow device access: 

    $ sudo setsebool -P container_use_devices 1

  7. Run the Content from github.com is not included.apply_eagle3_eagle.sh convert example using the Red Hat AI Model Optimization Toolkit container: 

    $ podman run --rm \
  -v "$(pwd):/opt/app-root/model-opt" \
  --device nvidia.com/gpu=0 \
  --ipc=host \
  -e HF_TOKEN=$HF_TOKEN \
  registry.redhat.io/rhaii-early-access/model-opt-cuda-rhel9:3.5.0-ea.1 \
  bash /opt/app-root/model-opt/speculators/examples/convert/eagle3/apply_eagle3_eagle.sh

    The script downloads the Eagle 3 checkpoint, converts it to the Speculators format, and validates the result.

Verification

  • Verify that the output includes Validation succeeded.
  • Confirm that the converted model directory exists in your working directory, for example eagle3-llama-3.1-8b-instruct-converted.

Example output

2026-04-17 13:58:49.830 | INFO     | speculators.convert.eagle.eagle3_converter:convert:41 - Converting Eagle-3 checkpoint: yuhuili/EAGLE3-LLaMA3.1-Instruct-8B
Fetching 2 files: 100%|██████████| 2/2 [00:06<00:00,  3.04s/it]
2026-04-17 13:59:01.127 | SUCCESS  | speculators.convert.eagle.eagle3_converter:convert:88 - Saved to: eagle3-llama-3.1-8b-instruct-converted
2026-04-17 13:59:03.888 | SUCCESS  | speculators.convert.eagle.eagle3_converter:_validate_converted_checkpoint:220 - Validation succeeded

Additional resources

Legal Notice

Copyright © Red Hat.
Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.
The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.
All other trademarks are the property of their respective owners.