vllm.model_executor.models.config
¶
Classes:
-
ColQwen3_5Config–Apply the attention contract declared by a ColQwen3.5 checkpoint.
-
D3ForSequenceClassificationConfig–Apply the contract declared by a Decision 3.0
decision_config.json. -
DiffusionGemmaDiffusionOptions–DiffusionGemma
DiffusionConfig.model_options. Unset options are read -
DiffusionGemmaModelForBlockDiffusionConfig– -
Gemma4Config– -
HybridAttentionMambaModelConfig– -
JinaEmbeddingsV5ModelConfig–Config handler for Jina Embeddings V5 embedding models.
-
KimiK3ForConditionalGenerationConfig–Route MXFP4-checkpointed Kimi-K3 MoE experts to the MXFP4 interface.
-
LlamaNemotronVLConfig–Config handler for LlamaNemotronVL embedding models.
-
MambaModelConfig– -
MiniMaxM3SparseForConditionalGenerationConfig– -
NemotronDiffusionOptions–Nemotron Labs Diffusion
DiffusionConfig.model_options. -
NemotronHForCausalLMConfig– -
NemotronLabsDiffusionForBlockDiffusionConfig– -
Qwen3_5ForConditionalGenerationConfig– -
Qwen4ExpForConditionalGenerationConfig–Apply the Qwen3.5 hybrid-cache contract to Qwen4Exp.
-
Qwen4ExpMTPConfig–Preserve MRoPE for a VL target and use 1D RoPE for a text target.
-
UnlimitedOCRForCausalLMConfig–
ColQwen3_5Config
¶
Bases: Qwen3_5ForConditionalGenerationConfig
Apply the attention contract declared by a ColQwen3.5 checkpoint.
Source code in vllm/model_executor/models/config.py
D3ForSequenceClassificationConfig
¶
Bases: Qwen3_5ForConditionalGenerationConfig
Apply the contract declared by a Decision 3.0 decision_config.json.
Source code in vllm/model_executor/models/config.py
DiffusionGemmaDiffusionOptions
¶
Bases: DiffusionModelOptions
DiffusionGemma DiffusionConfig.model_options. Unset options are read
from the model's generation_config.json.
Attributes:
-
confidence_threshold(float | None) –Confidence above which a position is committed.
-
entropy_bound(float | None) –Entropy budget for the EntropyBound sampler.
-
stability_threshold(int | None) –Number of previous steps a prediction must match before it is
-
t_max(float | None) –Temperature at the first denoising step.
-
t_min(float | None) –Temperature at the last denoising step.
Source code in vllm/model_executor/models/config.py
confidence_threshold = None
class-attribute
instance-attribute
¶
Confidence above which a position is committed.
entropy_bound = Field(default=None, gt=0)
class-attribute
instance-attribute
¶
Entropy budget for the EntropyBound sampler.
stability_threshold = Field(default=None, ge=0)
class-attribute
instance-attribute
¶
Number of previous steps a prediction must match before it is committed.
t_max = None
class-attribute
instance-attribute
¶
Temperature at the first denoising step.
t_min = None
class-attribute
instance-attribute
¶
Temperature at the last denoising step.
DiffusionGemmaModelForBlockDiffusionConfig
¶
Bases: VerifyAndUpdateConfig
Methods:
-
verify_and_update_config–Set up the diffusion config and defaults for DiffusionGemma.
Source code in vllm/model_executor/models/config.py
verify_and_update_config(vllm_config)
classmethod
¶
Set up the diffusion config and defaults for DiffusionGemma.
Auto-creates DiffusionConfig from the HF config when the user
didn't pass --diffusion-config. Diffusion sampling params are
read straight from generation_config.json at sampler-build time
(see DiffusionGemma's custom_sampler), not injected here.
Source code in vllm/model_executor/models/config.py
Gemma4Config
¶
Bases: VerifyAndUpdateConfig
Methods:
-
verify_and_update_config–Configure attention for heterogeneous head dimensions.
Source code in vllm/model_executor/models/config.py
563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 | |
verify_and_update_config(vllm_config)
staticmethod
¶
Configure attention for heterogeneous head dimensions.
Gemma4 uses different head dimensions for sliding window vs full attention layers. The default FA3 on Hopper cannot handle head_dim > 256.
On SM90 with FP8 KV cache, use FA3 for supported layers and let the generic FlashAttention selector upgrade larger head dimensions to FA4. The multimodal-prefix composite routes image masks to Triton and causal requests to this per-layer FA3/FA4 selection. For other configurations, force FA4 for all layers to avoid the mixed FA3+FA4 penalty. When FA4 is not available, fall back to Triton, or to FlashInfer for a plain Gemma4 with an NVFP4 KV cache, which Triton cannot read.
Source code in vllm/model_executor/models/config.py
HybridAttentionMambaModelConfig
¶
Bases: VerifyAndUpdateConfig
Methods:
-
verify_and_update_config–Perform early validation and setup for hybrid attention/mamba models.
Source code in vllm/model_executor/models/config.py
verify_and_update_config(vllm_config)
classmethod
¶
Perform early validation and setup for hybrid attention/mamba models.
Block size alignment with mamba page sizes is handled later by Platform.update_block_size_for_backend(), which runs after model layers are constructed and the attention backend is known.
Parameters:
-
(vllm_config¶VllmConfig) –vLLM Config
Source code in vllm/model_executor/models/config.py
JinaEmbeddingsV5ModelConfig
¶
Bases: VerifyAndUpdateConfig
Config handler for Jina Embeddings V5 embedding models.
Methods:
-
verify_and_update_model_config–Enable the bidirectional encoder backbone for -nano checkpoints.
Source code in vllm/model_executor/models/config.py
verify_and_update_model_config(model_config)
staticmethod
¶
Enable the bidirectional encoder backbone for -nano checkpoints.
The V5 family ships more than one backbone under a single
architectures entry: the -small variants are Qwen3 decoders, while
-nano is a bidirectional EuroBERT encoder. Upstream ships a separate
configuration_*.py per repository, so the config carries no backbone
field and the encoder variants are only identifiable by
is_decoder=False. For encoder checkpoints, set is_causal=False so the
Llama backbone uses EncoderOnlyAttention; JinaEmbeddingsV5Model then
dispatches to its encoder implementation.
Source code in vllm/model_executor/models/config.py
KimiK3ForConditionalGenerationConfig
¶
Bases: VerifyAndUpdateConfig
Route MXFP4-checkpointed Kimi-K3 MoE experts to the MXFP4 interface.
Kimi-K3 ships its routed experts as compressed-tensors
mxfp4-pack-quantized (quant_method="compressed-tensors"), which
lands them on CompressedTensorsW4A4Mxfp4MoEMethod and its narrow kernel
selection. Rewriting quant_method to "mxfp4" selects Mxfp4Config
(hence Mxfp4MoEMethod) with its full backend set, while any non-MXFP4
checkpoint is left untouched. Covers both the main model and the MTP draft.
model_arch_config.quantization_config is a separate dict, snapshotted in
ModelConfig.__init__ before this hook runs, and it is what
_verify_quantization reads when resolving the quant method. Patch it
alongside the hf configs so the rewrite lands before resolution; otherwise
the main model still resolves to compressed-tensors.
Methods:
-
verify_and_update_config–Default the kv-cache dtype when the text backbone carries DSA:
Source code in vllm/model_executor/models/config.py
verify_and_update_config(vllm_config)
classmethod
¶
Default the kv-cache dtype when the text backbone carries DSA: per-tensor fp8 for FLASHINFER_MLA_SPARSE, the default on SM10x, and "fp8_ds_mla" otherwise.
Source code in vllm/model_executor/models/config.py
LlamaNemotronVLConfig
¶
Bases: VerifyAndUpdateConfig
Config handler for LlamaNemotronVL embedding models.
Source code in vllm/model_executor/models/config.py
MambaModelConfig
¶
Bases: VerifyAndUpdateConfig
Methods:
-
verify_and_update_config–Enable FULL_AND_PIECEWISE cuda graph mode by default (required
Source code in vllm/model_executor/models/config.py
verify_and_update_config(vllm_config)
classmethod
¶
Enable FULL_AND_PIECEWISE cuda graph mode by default (required to get good performance for mamba layers in V1).
Parameters:
-
(vllm_config¶VllmConfig) –vLLM Config
Source code in vllm/model_executor/models/config.py
MiniMaxM3SparseForConditionalGenerationConfig
¶
Bases: MiniMaxM3SparseConfig
Methods:
-
verify_and_update_model_config–Reject IndexCache overrides set on the outer VL config, where the
Source code in vllm/model_executor/models/config.py
verify_and_update_model_config(model_config)
classmethod
¶
Reject IndexCache overrides set on the outer VL config, where the ROCm text model never reads them.
Source code in vllm/model_executor/models/config.py
NemotronDiffusionOptions
¶
Bases: DiffusionModelOptions
Nemotron Labs Diffusion DiffusionConfig.model_options.
Attributes:
-
confidence_threshold(float | None) –Probability threshold for
selection_policy="confidence_threshold". -
selection_policy(Literal['low_confidence', 'leftmost', 'confidence_threshold'] | None) –Which masked positions to unmask each denoising step.
-
temperature(float | None) –Default sampling temperature; explicit request temperatures take
Source code in vllm/model_executor/models/config.py
confidence_threshold = Field(default=None, gt=0, lt=1)
class-attribute
instance-attribute
¶
Probability threshold for selection_policy="confidence_threshold".
If not set, read from the model's generation_config.json
(diffusion_confidence_threshold), defaulting to 0.9.
selection_policy = None
class-attribute
instance-attribute
¶
Which masked positions to unmask each denoising step.
low_confidence: the most confident predictions first (LLaDA),
top-k per step with k from the even transfer schedule.
leftmost: strictly left-to-right.
confidence_threshold: unmask every position whose chosen-token
probability exceeds confidence_threshold (always at least the single
most confident one, so each step makes progress).
If not set, read from the model's generation_config.json
(diffusion_selection_policy), defaulting to confidence_threshold.
temperature = Field(default=None, ge=0)
class-attribute
instance-attribute
¶
Default sampling temperature; explicit request temperatures take precedence. 0 means greedy. If unset, the model's generation defaults apply.
NemotronHForCausalLMConfig
¶
Bases: VerifyAndUpdateConfig
Methods:
-
update_mamba_ssm_cache_dtype–Update mamba_ssm_cache_dtype for NemotronH models when set to 'auto'
Attributes:
-
DEFAULT_MAMBA_SSM_CACHE_DTYPE(MambaDType) –Only
float32is known to have no accuracy issues by default.
Source code in vllm/model_executor/models/config.py
DEFAULT_MAMBA_SSM_CACHE_DTYPE = 'float32'
class-attribute
instance-attribute
¶
Only float32 is known to have no accuracy issues by default.
update_mamba_ssm_cache_dtype(*, cache_config, hf_config)
classmethod
¶
Update mamba_ssm_cache_dtype for NemotronH models when set to 'auto'
(or not explicitly set), to the value specified in the HF config, or to
float32 if not specified.
Source code in vllm/model_executor/models/config.py
NemotronLabsDiffusionForBlockDiffusionConfig
¶
Bases: VerifyAndUpdateConfig
Methods:
-
verify_and_update_config–Set up the diffusion config and defaults for the Nemotron
Source code in vllm/model_executor/models/config.py
841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 | |
verify_and_update_config(vllm_config)
classmethod
¶
Set up the diffusion config and defaults for the Nemotron Ministral masked block-diffusion LM.
Auto-creates DiffusionConfig from the HF config (canvas_length ==
block_size) when the user didn't pass --diffusion-config.
max_denoising_steps defaults to canvas_length at
sampler-build time when generation_config.json doesn't provide it.
Source code in vllm/model_executor/models/config.py
Qwen3_5ForConditionalGenerationConfig
¶
Bases: VerifyAndUpdateConfig
Methods:
-
verify_and_update_config–Update mamba_ssm_cache_dtype for Qwen3.5 models when set to 'auto'
Source code in vllm/model_executor/models/config.py
verify_and_update_config(vllm_config)
staticmethod
¶
Update mamba_ssm_cache_dtype for Qwen3.5 models when set to 'auto' (or not explicitly set), to the value specified in the HF config's mamba_ssm_dtype field. Warn if the user explicitly overrides it to a different value.
Source code in vllm/model_executor/models/config.py
Qwen4ExpForConditionalGenerationConfig
¶
Bases: Qwen3_5ForConditionalGenerationConfig
Apply the Qwen3.5 hybrid-cache contract to Qwen4Exp.
Source code in vllm/model_executor/models/config.py
Qwen4ExpMTPConfig
¶
Bases: Qwen4ExpForConditionalGenerationConfig
Preserve MRoPE for a VL target and use 1D RoPE for a text target.
Source code in vllm/model_executor/models/config.py
UnlimitedOCRForCausalLMConfig
¶
Bases: VerifyAndUpdateConfig
Methods:
-
verify_and_update_config–Configure Unlimited-OCR attention backends for R-SWA and vision.
Source code in vllm/model_executor/models/config.py
277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 | |
verify_and_update_config(vllm_config)
staticmethod
¶
Configure Unlimited-OCR attention backends for R-SWA and vision.
Backend selection — controlled by the standard --attention-config
CLI argument (priority order):
-
--attention-config '{"backend": "FLASH_ATTN"}'→ FA4 + rswa_mask_mod. Exact token-level R-SWA.flash_attn_versionis forced to 4 if not already set (R-SWA mask_mod requires FA4; FA3 cannot express it). Raises if FA4 is not available on this device. -
--attention-config '{"backend": "FLEX_ATTENTION"}'→ FlexAttention R-SWA via Triton block mask. -
--attention-config '{"backend": "TRITON_ATTN"}'→ Triton unified attention with an R-SWA decode mask. -
--attention-config '{"backend": "auto"}'(or omitted) → Auto-detect: FA4 if available (H20/H100 SM90), else TritonAttention.
Regardless of backend, prefix caching is disabled for this model: R-SWA decode-phase KV is not a pure causal function of the prefix (so decode blocks are not reusable), and single-turn image-led OCR prompts rarely hit the prefix cache.
Example — force FlexAttention even on a machine with FA4::
vllm serve baidu/Unlimited-OCR \
--attention-config '{"backend": "FLEX_ATTENTION"}'
Source code in vllm/model_executor/models/config.py
278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 | |
_indexer_sparse_logits_supported(vllm_config)
¶
Whether to enable candidate-only indexer scoring by default.
Source code in vllm/model_executor/models/config.py
_parse_diffusion_model_options(vllm_config, options_cls)
¶
Validate --diffusion-config model_options against options_cls.
Source code in vllm/model_executor/models/config.py
_set_deepseek_indexer_logits_dtype(vllm_config)
¶
Default DeepSeek indexer logits to BF16 where the MQA logits kernels also reduce in BF16 (SM100) and to FP32 elsewhere.