vllm.model_inspection
¶
Model inspection utilities for vLLM.
Classes:
-
LayerBackend–The backend one or more layers of a model run on.
Functions:
-
collect_layer_backends–The backends the layers of
modelselected, one entry per distinct -
format_layer_backends–A table with one row per entry: its kind, how many layers run it, the
-
format_model_inspection–Format a model into a transformers-style hierarchical string.
LayerBackend
dataclass
¶
The backend one or more layers of a model run on.
Attributes:
-
backend(str) –The attention backend or kernel class, or
unquantized. -
details(tuple[tuple[str, str], ...]) –What the backend runs, e.g.
(("w", "mxfp4"), ("a", "mxfp8")). -
kind(str) –What the layers are:
attention,linearormoe. -
modules(tuple[str, ...]) –Names of the layers, as in
model.named_modules().
Source code in vllm/model_inspection.py
backend
instance-attribute
¶
The attention backend or kernel class, or unquantized.
details
instance-attribute
¶
What the backend runs, e.g. (("w", "mxfp4"), ("a", "mxfp8")).
kind
instance-attribute
¶
What the layers are: attention, linear or moe.
modules
instance-attribute
¶
Names of the layers, as in model.named_modules().
_describe_kernel(method)
¶
The kernel a quant method (or compressed-tensors scheme) selected, whatever attribute it keeps it in, with the precision it runs if known.
Source code in vllm/model_inspection.py
_describe_layer(module)
¶
(kind, backend, details) for a layer that selects a backend.
Source code in vllm/model_inspection.py
_format_index_ranges(indices)
¶
Format indices into range notation (e.g., [0,1,2,4,5,6] -> '0-2, 4-6').
Source code in vllm/model_inspection.py
_format_layer_names(names)
¶
Collapse layer names: layers.[0-47].attn.{q_proj,k_proj}.
Source code in vllm/model_inspection.py
_format_module_tree(module, name='', indent=0)
¶
Format a module tree with indentation, grouping identical layers.
Produces output like
(layers): ModuleList( (0-27, 29-47): 47 x LlamaDecoderLayer( ... ) (28, 48): 2 x DifferentDecoderLayer( ... ) )
Source code in vllm/model_inspection.py
_get_child_signature(child)
¶
Get a signature for a child module to detect duplicates.
Source code in vllm/model_inspection.py
_get_module_info(module)
¶
Get info string for a module.
Source code in vllm/model_inspection.py
collect_layer_backends(model)
¶
The backends the layers of model selected, one entry per distinct
(kind, backend, details), ordered by kind.
Source code in vllm/model_inspection.py
format_layer_backends(backends)
¶
A table with one row per entry: its kind, how many layers run it, the backend with its details, and the layer names collapsed.