Skip to content
vLLM
step3p5
Initializing search
GitHub
Home
User Guide
Developer Guide
Benchmarking
API Reference
CLI Reference
Community
vLLM
GitHub
Home
User Guide
User Guide
Getting Started
Getting Started
Quickstart
Installation
Installation
GPU
CPU
TPU
Examples
Examples
Applications
Applications
API Server
Chatbot
Rag
Basic
Basic
Offline Inference
Online Serving
Deployment
Deployment
Async LLM Streaming
Helm Charts
LLM Engine Example
Sagemaker-Entrypoint
Disaggregated
Disaggregated
Disaggregated Encoder
Disaggregated Serving
Ec Both Encoder
Disaggregated Prefill V1
Flexkv Connector
KV Load Failure Recovery Test
LMCache Examples
Mooncake Connector
Features
Features
Automatic Prefix Caching
Batch Invariance
Context Extension
Data Parallel
Kv Events
Logging Configuration
Custom Logits Processors
LoRA
Offline Inference with the OpenAI Batch file format
Pause Resume
Profiling
Prompt Embed
Reset Kv
Sharded State
Speculative Decoding
Structured reads on DiffusionGemma
Structured Outputs
Tensorize vLLM Model
Torchrun
Generate
Generate
Batched Chat Completions Online
Multimodal
Qwen 1M Offline
Trace Replay Offline
Observability
Observability
Monitoring Dashboards
Metrics
Setup OpenTelemetry POC
Prometheus and Grafana
Pooling
Pooling
Classify
Embed
Plugin
Reward
Score
Token Classify
Token Embed
Ray Serving
Ray Serving
Batch LLM Inference
Elastic Ep
Multi-Node-Serving
Ray Serve Deepseek
Run Cluster
Reasoning
Reasoning
OpenAI Chat Completion Tool Calls With Reasoning
OpenAI Chat Completion With Reasoning
OpenAI Chat Completion With Reasoning Streaming
OpenAI Responses Client
RL
RL
Rdt vLLM Serve
Rdt Weight Source
RLHF Async New APIs
RLHF Http IPC
RLHF Http NCCL
RLHF IPC Fsdp Ep
RLHF M2N
RLHF NCCL Fsdp Ep
RLHF Sharded Rdt Small Ep
RLHF Sparse NCCL
Routed Experts E2E
Skip Loading Weights In Engine Init
Scale Out
Scale Out
Init
Example Mm Serve
Token Generation Client
Speech To Text
Speech To Text
OpenAI
Realtime
Tool Calling
Tool Calling
Chat With Tools Offline
OpenAI Chat Completion Client With Tools
OpenAI Chat Completion Client With Tools Required
OpenAI Chat Completion Client With Tools Xlam
OpenAI Chat Completion Client With Tools Xlam Streaming
OpenAI Responses Client With Mcp Tools
OpenAI Responses Client With Tools
General
General
vLLM V1
Frequently Asked Questions
Production Metrics
Reproducibility
Security
Troubleshooting
Usage Stats Collection
Inference and Serving
Inference and Serving
Offline Inference
Online Serving
Online Serving
Derenderer APIs
Generative Scoring
OpenAI-Compatible Server
Renderer APIs
Speech to Text APIs
Tokens In <> Tokens Out API
Trace Replay
Context Parallel Deployment
Data Parallel Deployment
Troubleshooting distributed deployments
Expert Parallel Deployment
Parallelism and Scaling
Integrations
Integrations
Claude Code
Codex
LangChain
LlamaIndex
Deployment
Deployment
Using Docker
Using Kubernetes
Using Nginx
Frameworks
Frameworks
Anyscale
AnythingLLM
AutoGen
BentoML
Cerebrium
Chatbox
Crusoe
Dify
dstack
Haystack
Helm
Hugging Face Inference Endpoints
LiteLLM
Lobe Chat
LWS
Modal
Nebius Serverless AI
Open WebUI
Retrieval-Augmented Generation
RunPod
SkyPilot
Streamlit
NVIDIA Triton
Integrations
Integrations
AIBrix
NVIDIA Dynamo
KAITO
KServe
Kthena
KubeAI
KubeRay
Llama Stack
llm-d
llmaz
Production stack
Training
Training
Async Reinforcement Learning
What is Layerwise (Re)loading?
Prompt Token ID Logprobs (Teacher Scoring)
Reinforcement Learning from Human Feedback
Sampling Mask (Distribution Replay)
Transformers Reinforcement Learning
Weight Checker
Weight Transfer
Weight Transfer
Base Classes and Custom Engines
IPC Engine
NCCL M2N Engine
ModelExpress Engine
NCCL Engine
Sharded RDT Engine
Configuration
Configuration
Conserving Memory
Engine Arguments
Environment Variables
Model Resolution
Optimization and Tuning
Server Arguments
TPU
Models
Models
Supported Models
Generative Models
Pooling Models
Pooling Models
Classification Usages
Embedding Usages
Reward Usages
Scoring Usages
Specific Model Examples
Token Classification Usages
Token Embedding Usages
Extensions
Extensions
Loading model weights with fastsafetensors
Loading Model Weights with InstantTensor
Loading models with Run:ai Model Streamer
Loading models with CoreWeave's Tensorizer
Hardware Supported Models
Hardware Supported Models
CPU - Intel® Xeon®
XPU - Intel® GPUs
TPU
Features
Features
Automatic Prefix Caching
Batch Invariance
Context Extension
Cross-Encoder Output Reuse
Custom Arguments
Custom Logits Processors
Disaggregated Encoder
Disaggregated Prefilling (experimental)
CPU EC Connector Usage Guide
Engram: conditional memory via n-gram lookups
IndexCache
Initialized engine snapshots
Interleaved Thinking
KV Offloading Usage Guide
LoRA Adapters
MooncakeConnector Usage Guide
ECMooncakeConnector Usage Guide
MooncakeStoreConnector Usage Guide
MoRIIOConnector Usage Guide
Multimodal Inputs
NixlConnector Compatibility Matrix
NixlConnector Usage Guide
Per-Request Metrics
Preload
Prompt Embedding Inputs
Reasoning Outputs
Sleep Mode
Structured Outputs
Tool Calling
Text watermarking
Quantization
Quantization
AutoAWQ
b12x Linear and MoE Backends
BitsAndBytes
FP8 ViT Encoder Attention
GGUF
GPTQModel
Intel Quantization Support
NVIDIA Model Optimizer
Online Quantization
Quantized KV Cache
AMD Quark
TorchAO
LLM Compressor
LLM Compressor
FP8 W8A8
INT4 W4A16
INT8 W4A8
INT8 W8A8
Speculative Decoding
Speculative Decoding
Per-Request Acceptance Metrics
Adaptive Verification
Draft Models
Dynamic Speculative Decoding
EAGLE Draft Models
Hidden State Extraction
LiLiCorr
MLP Draft Models
MTP (Multi-Token Prediction)
N-Gram Speculation
Parallel Draft Models
vLLM-Project/Speculators
Suffix Decoding
Developer Guide
Developer Guide
General
General
Deprecation Policy
Dockerfile
Editing Agent Instructions
Incremental Compilation Workflow
JIT Kernel Warmup
Labels
Profiling vLLM
Releasing vLLM
Vulnerability Management
Model Implementation
Model Implementation
Basic Model
Registering a Model
Unit Testing
Multi-Modal Support
Speech-to-Text (Transcription/Translation) Support
CI
CI
CI Failures
Nightly Builds of vLLM Wheels
Update PyTorch version on vLLM OSS CI/CD
Design Documents
Design Documents
Plugins
Plugins
Endpoint Plugins
IO Processor Plugins
LoRA Resolver Plugins
Plugin System
Architecture Overview
Attention Backend Feature Support
CUDA Graphs
Vision Encoder (ViT) CUDA Graphs
CustomOp
Dual Batch Overlap
How to debug the vLLM-torch.compile integration
Fused MoE Modular Kernel
Fusion torch.compile passes
HiSparse local KV offload architecture
Integration with Hugging Face
Hybrid KV Cache Manager
Logits Processors
Metrics
Multi-Modal Data Processing
Model Runner V2 Design Document
Fused MoE Kernel Features
Python Multiprocessing
NIXL KV Cache Lease Renewal
NIXL push-mode KV transfer
Optimization Levels
Paged Attention
Automatic Prefix Caching
torch.compile integration
torch.compile with Multimodal Encoders
vLLM IR: Functional Intermediate Representation
Benchmarking
Benchmarking
Benchmark CLI
Parameter Sweeps
Performance Dashboard
API Reference
API Reference
vllm
vllm
collect_env
connections
env_override
envs
exceptions
forward_context
logger
logits_process
logprobs
model_inspection
outputs
pooling_params
sampling_params
scalar_type
scripts
sequence
tasks
version
assets
benchmarks
compilation
config
cute_utils
device_allocator
distributed
engine
entrypoints
inputs
ir
kernels
logging_utils
lora
model_executor
model_executor
custom_op
parameter
utils
determinism
hw_agnostic
kernels
layers
model_loader
models
models
AXK1
adapters
afmoe
aimv2
apertus
aria
audioflamingo3
bagel
bailing_moe
bailing_moe_linear
bailing_moe_mtp
bailing_moe_v3
bailing_moe_v3_mtp
bailing_moe_v3_vl
bee
bert
bert_with_rope
blip
blip2
bloom
chatglm
clip
cohere2_moe
cohere2_vision
cohere_asr
cohere_compass
cohere_eagle
colbert
colmodernvbert
colpali
colqwen3
colqwen3_5
commandr
config
conformer_encoder
cosmos3
cosmos3_edge
dbrx
deepencoder
deepencoder2
deepseek_eagle
deepseek_eagle3
deepseek_mtp
deepseek_ocr
deepseek_ocr2
deepseek_v2
deepseek_vl2
diffusion_gemma
diffusion_gemma_sampler
dots_ocr
eagle2_5_vl
ernie45
ernie45_moe
ernie45_vl
ernie45_vl_moe
ernie_mtp
exaone
exaone4
exaone4_5
exaone4_5_mtp
exaone_moe
exaone_moe_mtp
extract_hidden_states
falcon
falcon_h1
fireredasr2
funasr
funaudiochat
gemma
gemma2
gemma3
gemma3_mm
gemma3n
gemma3n_audio_utils
gemma3n_mm
gemma4
gemma4_dspark
gemma4_mm
gemma4_mtp
gemma4_unified
glm4
glm4_1v
glm4_moe
glm4_moe_lite
glm4_moe_lite_mtp
glm4_moe_mtp
glm4v
glm_ocr
glm_ocr_mtp
glmasr
glmasr_utils
gpt2
gpt_j
gpt_oss
granite
granite4_vision
granite_speech
granite_speech_plus
granitemoe
granitemoehybrid
granitemoeshared
h2ovl
hrm_text
hy_v3
hy_v3_mtp
hyperclovax
hyperclovax_vision_v2
idefics2_vision_model
idefics3
interfaces
interfaces_base
intern_vit
internlm2
interns1
interns1_pro
interns1_vit
interns2_mobius
interns2_preview
internvl
iquest_loopcoder
isaac
jamba
jina
jina_vl
k2_horizon
kanana_v
keye
keye_vl1_5
kimi_audio
kimi_k25
kimi_k25_vit
kimi_vl
laguna
laguna_dflash
lfm2
lfm2_moe
lfm2_siglip2
lfm2_vl
lightonocr
lilicorr
llama
llama4
llama4_eagle
llama_eagle
llama_eagle3
llava
llava_next
llava_next_video
llava_onevision
llava_onevision2
longcat_flash
longcat_flash_mtp
longcat_flash_ngram
mamba
mamba2
medusa
midashenglm
mimo
mimo_audio
mimo_mtp
mimo_v2
mimo_v2_mtp
mimo_v2_omni
minicpm
minicpm3
minicpm_eagle
minicpmo
minicpmv
minicpmv4_6
minimax_m2
mistral
mistral3
mistral_eagle
mistral_large_3
mistral_large_3_eagle
mixtral
mllama4
mlp_speculator
modernbert
module_mapping
molmo
molmo2
moondream3
moonvit
moss_audio
moss_transcribe_diarize
muse_glimmer
nano_nemotron_vl
nemotron
nemotron_h
nemotron_h_mtp
nemotron_nas
nemotron_parse
nemotron_vl
nvlm_d
olmo_hybrid
olmoe
openai_privacy_filter
opencua
openpangu
openpangu_mtp
openpangu_vl
openvla
opt
orion
ovis
ovis2_5
paddleocr_vl
paligemma
parakeet
param2moe
phi3
phi3v
phi4mm
phi4mm_audio
phi4mm_utils
phi4siglip
phimoe
pixtral
plamo3
qianfan_ocr
qwen2
qwen2_5_omni_thinker
qwen2_5_vl
qwen2_audio
qwen2_moe
qwen2_rm
qwen2_vl
qwen3
qwen3_5
qwen3_5_mtp
qwen3_asr
qwen3_asr_forced_aligner
qwen3_asr_realtime
qwen3_dflash
qwen3_dflash2
qwen3_dspark
qwen3_eagle3
qwen3_moe
qwen3_next
qwen3_next_mtp
qwen3_omni_moe_thinker
qwen3_vl
qwen3_vl_moe
radio
registry
rnj1
roberta
rvl
sarvam
siglip
siglip2navit
skyworkr1v
smolvlm
solar
stablelm
step1
step3_text
step3_vl
step3p5
step3p5_mtp
step3p7
step_vl
telechat2
teleflm
terratorch
ultravox
unlimited_ocr
utils
vision
voxtral
voxtral_realtime
voyage
whisper
whisper_causal
whisper_utils
zamba2
transformers
offloader
warmup
models
multimodal
parser
platforms
plugins
profiler
ray
reasoning
renderers
snapshot
tilelang_utils
tokenizers
tool_parsers
tracing
transformers_utils
triton_utils
usage
utils
v1
CLI Reference
CLI Reference
vllm
vllm
chat
complete
preload
run-batch
serve
bench
bench
latency
mm-processor
serve
startup
throughput
sweep
sweep
plot
plot_pareto
serve
serve_workload
startup
launch
launch
render
Community
Community
Contact Us
Meetups
Reviewers
Sponsors
Governance
Governance
Collaboration Policy
Committers
Governance Process
Blog
Forum
Slack
Home
API Reference
vllm
model_executor
models
vllm.model_executor.models.step3p5
¶
Inference-only Jurassic model.
Back to top