Skip to content

vllm.model_executor.models.transformers.base

Transformers modeling backend base class.

Classes:

Attributes:

  • VLLM_ATTN_ATTR –

    Attribute in which to store the vLLM Attention instance that serves a Transformers

VLLM_ATTN_ATTR = 'attn' module-attribute

Attribute in which to store the vLLM Attention instance that serves a Transformers attention module. "attn" is chosen deliberately to match the mappings generated by maybe_remap_kv_scale_name.

Base

Bases: Module, VllmModel, SupportsQuant, SupportsLoRA, SupportsPP, SupportsEagle, SupportsEagle3, EagleModelMixin

Methods:

Attributes:

Source code in vllm/model_executor/models/transformers/base.py
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
 299
 300
 301
 302
 303
 304
 305
 306
 307
 308
 309
 310
 311
 312
 313
 314
 315
 316
 317
 318
 319
 320
 321
 322
 323
 324
 325
 326
 327
 328
 329
 330
 331
 332
 333
 334
 335
 336
 337
 338
 339
 340
 341
 342
 343
 344
 345
 346
 347
 348
 349
 350
 351
 352
 353
 354
 355
 356
 357
 358
 359
 360
 361
 362
 363
 364
 365
 366
 367
 368
 369
 370
 371
 372
 373
 374
 375
 376
 377
 378
 379
 380
 381
 382
 383
 384
 385
 386
 387
 388
 389
 390
 391
 392
 393
 394
 395
 396
 397
 398
 399
 400
 401
 402
 403
 404
 405
 406
 407
 408
 409
 410
 411
 412
 413
 414
 415
 416
 417
 418
 419
 420
 421
 422
 423
 424
 425
 426
 427
 428
 429
 430
 431
 432
 433
 434
 435
 436
 437
 438
 439
 440
 441
 442
 443
 444
 445
 446
 447
 448
 449
 450
 451
 452
 453
 454
 455
 456
 457
 458
 459
 460
 461
 462
 463
 464
 465
 466
 467
 468
 469
 470
 471
 472
 473
 474
 475
 476
 477
 478
 479
 480
 481
 482
 483
 484
 485
 486
 487
 488
 489
 490
 491
 492
 493
 494
 495
 496
 497
 498
 499
 500
 501
 502
 503
 504
 505
 506
 507
 508
 509
 510
 511
 512
 513
 514
 515
 516
 517
 518
 519
 520
 521
 522
 523
 524
 525
 526
 527
 528
 529
 530
 531
 532
 533
 534
 535
 536
 537
 538
 539
 540
 541
 542
 543
 544
 545
 546
 547
 548
 549
 550
 551
 552
 553
 554
 555
 556
 557
 558
 559
 560
 561
 562
 563
 564
 565
 566
 567
 568
 569
 570
 571
 572
 573
 574
 575
 576
 577
 578
 579
 580
 581
 582
 583
 584
 585
 586
 587
 588
 589
 590
 591
 592
 593
 594
 595
 596
 597
 598
 599
 600
 601
 602
 603
 604
 605
 606
 607
 608
 609
 610
 611
 612
 613
 614
 615
 616
 617
 618
 619
 620
 621
 622
 623
 624
 625
 626
 627
 628
 629
 630
 631
 632
 633
 634
 635
 636
 637
 638
 639
 640
 641
 642
 643
 644
 645
 646
 647
 648
 649
 650
 651
 652
 653
 654
 655
 656
 657
 658
 659
 660
 661
 662
 663
 664
 665
 666
 667
 668
 669
 670
 671
 672
 673
 674
 675
 676
 677
 678
 679
 680
 681
 682
 683
 684
 685
 686
 687
 688
 689
 690
 691
 692
 693
 694
 695
 696
 697
 698
 699
 700
 701
 702
 703
 704
 705
 706
 707
 708
 709
 710
 711
 712
 713
 714
 715
 716
 717
 718
 719
 720
 721
 722
 723
 724
 725
 726
 727
 728
 729
 730
 731
 732
 733
 734
 735
 736
 737
 738
 739
 740
 741
 742
 743
 744
 745
 746
 747
 748
 749
 750
 751
 752
 753
 754
 755
 756
 757
 758
 759
 760
 761
 762
 763
 764
 765
 766
 767
 768
 769
 770
 771
 772
 773
 774
 775
 776
 777
 778
 779
 780
 781
 782
 783
 784
 785
 786
 787
 788
 789
 790
 791
 792
 793
 794
 795
 796
 797
 798
 799
 800
 801
 802
 803
 804
 805
 806
 807
 808
 809
 810
 811
 812
 813
 814
 815
 816
 817
 818
 819
 820
 821
 822
 823
 824
 825
 826
 827
 828
 829
 830
 831
 832
 833
 834
 835
 836
 837
 838
 839
 840
 841
 842
 843
 844
 845
 846
 847
 848
 849
 850
 851
 852
 853
 854
 855
 856
 857
 858
 859
 860
 861
 862
 863
 864
 865
 866
 867
 868
 869
 870
 871
 872
 873
 874
 875
 876
 877
 878
 879
 880
 881
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
class Base(
    nn.Module,
    VllmModel,
    SupportsQuant,
    SupportsLoRA,
    SupportsPP,
    SupportsEagle,
    SupportsEagle3,
    EagleModelMixin,
):
    hf_to_vllm_mapper: WeightsMapper
    supports_aux_hidden_states_over_pp = True

    def __init__(self, *, vllm_config: "VllmConfig", prefix: str = ""):
        nn.Module.__init__(self)
        logger.info("Using Transformers modeling backend.")

        self.vllm_config = vllm_config
        self.config = vllm_config.model_config.hf_config
        self.text_config = self.config.get_text_config()
        self.cache_config = vllm_config.cache_config
        self.compilation_config = vllm_config.compilation_config
        self.device_config = vllm_config.device_config
        self.model_config = vllm_config.model_config
        self.parallel_config = vllm_config.parallel_config
        self.quant_config = vllm_config.quant_config

        self.pp_group = get_pp_group()
        self.tp_group = get_tp_group()

        # Attrs for weight loading (see self.load_weights)
        self.ignore_unexpected_prefixes: list[str] = []
        """Ignore unexpected weights whose qualname starts with these prefixes."""
        self.ignore_unexpected_suffixes: list[str] = []
        """Ignore unexpected weights whose qualname ends with these suffixes."""
        self.packed_modules_mapping: dict[str, list[str]] = {}
        """Fused module -> constituent projections, populated by `recursive_replace`
        for the quantization machinery and loaders (e.g. bitsandbytes)."""
        self.fusers: dict[str, list[BaseFuser]] = {}
        """Module qualname -> the fusers applied to it, populated
        by `recursive_replace` for `_create_attention_instances`."""
        self.attention_fusers: dict[int, tuple[str, AttentionFuser]] = {}
        """`layer_idx` -> the qualname and fuser of the module computing that
        layer's attention, populated by `recursive_replace` for
        `_create_attention_instances`."""

        # Attrs for Eagle3 (see self.set_aux_hidden_state_layers)
        self._layer_names: dict[int, str] = {}
        """Mapping from layer index to layer name for Eagle3."""
        self._aux_hidden_states: list[torch.Tensor] = []
        """Eagle3 aux hidden states recorded by hooks during the forward pass."""

        self._pre_decoder_layer_replace: Callable[[nn.Module, str], None]
        """Called on each decoder layer before `recursive_replace` replaces it."""
        self._pre_decoder_layer_replace = lambda layer, prefix: None

        if self.quant_config:
            quant_method_name = self.quant_config.get_name()
            # Check for unsupported quantization methods.
            if quant_method_name in ("mxfp4", "gpt_oss_mxfp4"):
                raise NotImplementedError(
                    "Transformers modeling backend does "
                    "not support MXFP4 quantization yet."
                )

        if self.model_config.uses_mrope:
            # CUDA graphs replay their capture-time input addresses, see `forward`
            max_tokens = vllm_config.scheduler_config.max_num_batched_tokens
            mrope_positions = torch.zeros(3 * max_tokens, dtype=torch.long)
            self.register_buffer("mrope_positions", mrope_positions, persistent=False)

        self._patch_config()
        self._decorate_for_torch_compile()
        # Init on "meta" to delay allocating GPU tensors
        with (
            self._mark_model_components(vllm_config),
            init_on_device_without_buffers("meta"),
        ):
            self.model: PreTrainedModel = self._create_model()

        # LoRA matches embeddings by the last part of their name, which varies by model
        input_embeddings = set(self.model.get_input_embeddings().modules())
        embedding_modules = {
            name.rpartition(".")[2]: "input_embeddings"
            for name, module in self.model.named_modules()
            if module in input_embeddings and isinstance(module, nn.Embedding)
        }
        self.embedding_modules = self.embedding_modules | embedding_modules

        # Create weight name to module qualname mapper
        self._create_hf_to_vllm_mapper()
        # Remove layers not on this pipeline parallel rank
        self.start_layer, self.end_layer = self._pp_indices()
        self.pipeline_parallel()
        # Skip the device sync of rope updates where `inv_freq` can be fixed
        maybe_disable_dynamic_rope_update(self.model.get_decoder(), vllm_config)
        # Substitute remaining layers with vLLM's layers as needed
        self.recursive_replace()
        # Create attention instances for KV cache allocation
        self._create_attention_instances(prefix)

        # Initialize any parameters that have not had their modules replaced
        self.init_parameters(self.model)

        # Upcast weights Transformers always keeps in fp32
        self.keep_in_fp32(self.model)

        # Later pipeline ranks pass their received hidden states to the first layer
        self._intermediate_hidden_states: torch.Tensor | None = None
        self._copy_intermediate_hidden_states = False
        if not self.pp_group.is_first_rank and (layers := self._decoder_layers):
            hidden_size = self.text_config.hidden_size
            # Run eagerly, compiling here would trace the probe
            with set_forward_context(None, vllm_config, skip_compiled=True):
                mutates = decoder_mutates_inputs_embeds(
                    self.model,
                    layers[self.start_layer],
                    hidden_size,
                    self.model_config.uses_mrope,
                )
            if mutates:
                max_tokens = vllm_config.scheduler_config.max_num_batched_tokens
                # Piecewise CUDA graphs require a stable address for the saved input
                self.register_buffer(
                    "_intermediate_hidden_states_buffer",
                    torch.zeros(1, max_tokens, hidden_size),
                    persistent=False,
                )
                self._copy_intermediate_hidden_states = True
            layers[self.start_layer].register_forward_pre_hook(
                self._inject_intermediate_hidden_states, with_kwargs=True
            )

        # Pipeline parallel intermediate tensors
        self.make_empty_intermediate_tensors = make_empty_intermediate_tensors_factory(
            ["hidden_states"], self.text_config.hidden_size
        )

    def _patch_config(self):
        """Patch the config to ensure that the model is created correctly:

        - Sets the attention implementation to "vllm" so the attention instances from
        `_create_attention_instances` are used
        - Sets the dtype to the default torch dtype set by vLLM because Transformers
        uses the config dtype when creating the model
        """
        self.text_config._attn_implementation = VLLM_ATTN_IMPL
        self.config.dtype = torch.get_default_dtype()

    @contextmanager
    def _mark_model_components(self, vllm_config: "VllmConfig"):
        """Mark language model and tower submodules as `self.model` is created.

        Nothing to do in `Base`, `MultiModalMixin` will override."""
        yield

    @cached_property
    def _from_config_kwargs(self) -> dict[str, Any]:
        """The kwargs used to create `self.model`."""
        return dict(
            config=self.config,
            dtype=self.model_config.dtype,
            trust_remote_code=self.model_config.trust_remote_code,
        )

    def _create_model(self) -> "PreTrainedModel":
        """Create the Transformers model that `self.model` wraps."""
        return AutoModel.from_config(**self._from_config_kwargs)

    def _find_encoder_classes(
        self, model: "PreTrainedModel"
    ) -> dict[str, type["PreTrainedModel"]]:
        """Find the encoder class of each modality `model` has one for.

        Text models have none. Multi-modal ones override this.
        """
        return {}

    @cached_property
    def _pre_trained_model_classes(self) -> PreTrainedModelClasses:
        """The decoder class and the encoder class of each modality that has one.

        Both come from a single throwaway model, since building one is not cheap.
        """
        with torch.device("meta"):
            model: PreTrainedModel = AutoModel.from_config(**self._from_config_kwargs)
        model_classes = PreTrainedModelClasses(
            decoder=type(model.get_decoder()),
            encoders=self._find_encoder_classes(model),
        )
        del model

        logger.debug("Identified model classes as: %s", model_classes)
        return model_classes

    def _decorate_cls_for_torch_compile(
        self,
        cls: type["PreTrainedModel"],
        dynamic_arg_dims: DynamicArgDims | None,
        enable_if: Callable[["VllmConfig"], bool],
        is_encoder: bool,
    ):
        """Decorate `cls` to indicate to vLLM that it supports torch compile.

        Args:
            cls: The PreTrainedModel class to decorate.
            dynamic_arg_dims: A mapping from argument name to the dynamic dimensions
                of the argument. If None, default dynamic arg dims will be used. See
                [`support_torch_compile`][vllm.compilation.decorators.support_torch_compile]
                for more details.
            enable_if: A function which takes in the vLLM config and returns whether
                torch compile should be enabled for this class.
            is_encoder: Whether the class being decorated is an encoder.

        """
        logger.debug(
            "Decorating `%s` as %s for torch compile with dynamic_arg_dims of %s",
            cls.__name__,
            "encoder" if is_encoder else "decoder",
            dynamic_arg_dims,
        )

        support_torch_compile(
            dynamic_arg_dims=dynamic_arg_dims,
            enable_if=enable_if,
            is_encoder=is_encoder,
            use_aot_compile=False,
        )(cls)

    def _decorate_for_torch_compile(self):
        """Decorate the model's decoder class to indicate to vLLM that it
        supports torch compile if `can_enable_torch_compile` is True."""
        self._decorate_cls_for_torch_compile(
            cls=self._pre_trained_model_classes.decoder,
            # Applied to a PreTrainedModel so the batch dimension will exist
            dynamic_arg_dims=DynamicArgDims(
                input_ids=1,  # shape: [1, seq_len]
                inputs_embeds=1,  # shape: [1, seq_len, hidden_size]
                position_ids=-1,  # shape: [1, seq_len] or [3, 1, seq_len] for mrope
            ),
            enable_if=can_enable_torch_compile,
            is_encoder=False,
        )

    def _create_hf_to_vllm_mapper(self):
        """Create a WeightsMapper to map checkpoint weight names to module qualnames.

        This handles:

        - Transformers weight renaming from `WeightRenaming`
        - Checkpoints saved with a base model prefix that is not `model`
        - Checkpoints saved with no base model prefix
        - Any quantization config specific mappings
        """
        orig_to_new_renaming: list[WeightRenaming] = []
        orig_to_new_regex: dict[re.Pattern, str | None] = {}

        for mapping in self._weight_conversions():
            # Handle weights which have been renamed in Transformers
            if isinstance(mapping, WeightRenaming):
                orig_to_new_renaming.append(mapping)
            # TODO: Handle WeightConverter to enable layer merging

        # Handle unexpected weights which should be ignored
        if self.model._keys_to_ignore_on_load_unexpected is not None:
            for key in self.model._keys_to_ignore_on_load_unexpected:
                orig_to_new_regex[re.compile(key)] = None

        # Standardise base model prefix
        bmp = self.model.base_model_prefix
        expected_bmp = r"model.\1"
        # Handle checkpoints saved with different base model prefix
        if bmp and bmp != "model":
            different_bmp_pattern = re.compile(rf"^{bmp}\.(.+)")
            orig_to_new_regex[different_bmp_pattern] = expected_bmp
        # Handle direct children of self.model which were saved without the model prefix
        direct_children = chain(
            self.model.named_children(),
            self.model.named_parameters(recurse=False),
            self.model.named_buffers(recurse=False),
        )
        model_children = "|".join(name for name, _ in direct_children)
        missing_bmp_pattern = re.compile(rf"^(?!model\.)(({model_children}).*)")
        orig_to_new_regex[missing_bmp_pattern] = expected_bmp
        # Handle weights saved as direct children of self.model which no longer are
        unexpected_bmp_pattern = re.compile(rf"^(model\.)((?!{model_children}).+)")
        orig_to_new_regex[unexpected_bmp_pattern] = r"\2"
        # Handle lm_head which was saved inside the base model
        nested_lm_head_pattern = re.compile(r"^model\.(.+\.)*(lm_head.+)")
        orig_to_new_regex[nested_lm_head_pattern] = r"\2"

        self.hf_to_vllm_mapper = WeightsMapper(
            orig_to_new_renaming=orig_to_new_renaming,
            orig_to_new_regex=orig_to_new_regex,
        )

        # Apply mapping to quantization config if needed
        self._maybe_apply_model_mapping()

    def _weight_conversions(self) -> list[WeightTransform]:
        """The Transformers weight conversions that apply to `self.model`."""
        mappings = get_model_conversion_mapping(self.model)
        # Ensure layers that are part of the model, but not AutoModel, get their mapping
        instantiated = {type(m).__name__ for m in self.model.modules()}
        for arch in self.config.architectures or []:
            if arch not in instantiated:
                mappings.extend(get_checkpoint_conversion_mapping(arch) or [])
        return mappings

    def _tp_plan(self) -> dict:
        """The tensor parallel plan of `self.model`, relative to `self.model`."""
        return self.model.tp_plan or {}

    def _get_tie_word_embeddings(self):
        """Check if the model has tied word embeddings."""
        # Models created with Transformers v4 and v5 will store this in different places
        tie_word_embeddings_v4 = getattr(self.text_config, "tie_word_embeddings", False)
        tie_word_embeddings_v5 = getattr(self.config, "tie_word_embeddings", False)
        return tie_word_embeddings_v4 or tie_word_embeddings_v5

    def pipeline_parallel(self):
        """Apply the model's pipeline parallelization plan."""
        if self.pp_group.world_size <= 1:
            return

        module, names = self._pp_split
        if not self.model.supports_pp_plan:
            tip = get_feature_request_tip(
                self.model_config.model, self.model_config.trust_remote_code
            )
            logger.warning_once(
                "%s does not define a pipeline parallel plan. The Transformers "
                "modeling backend will infer the split from the layers of %s in order "
                "of declaration and keep parameter-free modules on every rank. This "
                "may fail if the model's structure is non-standard. %s",
                type(self.model).__name__,
                type(module).__name__,
                tip,
            )

        if (layers := self._decoder_layers) is None:
            raise ValueError(f"Could not identify the decoder layers of {type(module)}")
        # attrgetter in case the module is nested (e.g. "text_model.layers")
        module_list_idx = next(
            i for i, name in enumerate(names) if attrgetter(name)(module) is layers
        )

        # Layers before module list
        input_embeddings = set(self.model.get_input_embeddings().modules())
        keep_input_embeddings = (
            self._get_tie_word_embeddings() and self.pp_group.is_last_rank
        ) or spec_decode_needs_target_embed(self.vllm_config)
        for name in names[:module_list_idx]:
            if self.pp_group.is_first_rank:
                break
            child = attrgetter(name)(module)
            is_input_embedding = not input_embeddings.isdisjoint(child.modules())
            if is_input_embedding and keep_input_embeddings:
                continue
            missing = (
                PPMissingScalar()
                if isinstance(child, nn.Embedding) and not is_input_embedding
                else PPMissingLayer()
            )
            # attrsetter in case the module is nested (e.g. "text_model.embed_tokens")
            attrsetter(name)(module, missing)

        # Module list
        start_layer, end_layer = self._pp_indices()
        for i in range(len(layers)):
            if start_layer <= i and i < end_layer:
                continue
            layers[i] = PPMissingLayer()

        # Layers after module list
        for name in names[module_list_idx + 1 :]:
            # Modules that should be on last rank
            if not self.pp_group.is_last_rank:
                # attrsetter in case the module is nested (e.g. "text_model.norm")
                attrsetter(name)(module, PPMissingLayer())

    @cached_property
    def _pp_split(self) -> tuple[nn.Module, list[str]]:
        """The module the pipeline parallel split is applied to, and the names (in
        order) of its children that the split assigns to ranks."""
        if self.model.supports_pp_plan:
            return self.model, list(self.model._pp_plan.keys())
        module = self.model.get_decoder()
        has_parameters = lambda m: next(m.parameters(), None) is not None
        return module, [n for n, c in module.named_children() if has_parameters(c)]

    @cached_property
    def _decoder_layers(self) -> nn.ModuleList | None:
        """The decoder layers, i.e. the only `ModuleList` in `self._pp_split`, if it
        has `num_hidden_layers` entries."""
        module, names = self._pp_split

        def get(name: str) -> nn.Module | None:
            # A plan may name a module the model lacks (e.g. `MtpModel`'s "norm")
            try:
                return attrgetter(name)(module)
            except AttributeError:
                return None

        module_lists = [m for n in names if isinstance(m := get(n), nn.ModuleList)]
        if len(module_lists) != 1:
            return None
        # Some models have multiple attention layers per entry
        layers = module_lists[0]
        return layers if len(layers) == self.text_config.num_hidden_layers else None

    def _inject_intermediate_hidden_states(self, module, args, kwargs):
        """Pass the hidden states from the previous pipeline rank to `module` unchanged.

        The decoder still runs its pre-layer code for masks and rotary embeddings, but
        anything it does to the hidden states was already done on the first rank."""
        if (hidden_states := self._intermediate_hidden_states) is None:
            return None
        if self._copy_intermediate_hidden_states:
            # Sliced here so the token count stays dynamic under torch.compile
            buffer = self._intermediate_hidden_states_buffer
            hidden_states = buffer[:, : hidden_states.shape[1]]
        if args:
            return (hidden_states, *args[1:]), kwargs
        return args, kwargs | {"hidden_states": hidden_states}

    def _offloader(self) -> BaseOffloader:
        """The offloader for the decoder layers."""
        return get_offloader()

    def _vocab_embeddings(self) -> set[nn.Embedding]:
        """The `nn.Embedding`s in `self.model` which hold a vocab table."""

        def vocab_sizes(config: PreTrainedConfig):
            for key, value in vars(config).items():
                if isinstance(value, PreTrainedConfig):
                    yield from vocab_sizes(value)
                elif "vocab_size" in key and isinstance(value, int):
                    yield value

        sizes = set(vocab_sizes(self.config))
        # `get_input_embeddings` may return a module which composes the `nn.Embedding`,
        # or a `PPMissingLayer` if the embeddings are not on this pipeline stage
        embeddings = {
            module
            for module in self.model.get_input_embeddings().modules()
            if isinstance(module, nn.Embedding)
        }
        embeddings.update(
            module
            for module in self.model.modules()
            if isinstance(module, nn.Embedding) and module.num_embeddings in sizes
        )
        return embeddings

    def recursive_replace(self):
        """Recursively replace modules in the model as needed.

        Currently, this replaces:

        - GLUs with a fused `MergedColumnParallelLinear` + `...AndMul`
        - Attention QKV projections with a fused `QKVParallelLinear` + split
        - `nn.Linear` with vLLM's tensor parallel linear classes
        - `nn.Conv2d` / `nn.Conv3d` with vLLM's `Conv2d` / `Conv3d`
        - Vocab `nn.Embedding`s with vLLM's `VocabParallelEmbedding`
        - RMSNorm (detected from their dataflow) with vLLM's `RMSNorm`or `GemmaRMSNorm`
        - `nn.LayerNorm` with vLLM's `LayerNorm`
        """
        tp_plan = self._tp_plan()

        if not tp_plan and self.tp_group.world_size > 1:
            tip = get_feature_request_tip(
                self.model_config.model, self.model_config.trust_remote_code
            )
            logger.warning_once(
                "%s does not define a tensor parallel plan. The Transformers modeling "
                "backend will shard the model the best it can during graph fusion and "
                "replicate the rest. This may be suboptimal or fail if the model does "
                "not fuse cleanly. %s",
                type(self.model).__name__,
                tip,
            )

        # Prefix the patterns because we always start from `self.model`
        tp_plan = {maybe_prefix("model", k): v for k, v in tp_plan.items()}
        vocab_embeddings = self._vocab_embeddings()

        orig_to_new_stacked: dict[str, tuple[str, ShardId]] = {}

        def register_fusion(fuser: BaseFuser, prefix: str, module: nn.Module):
            """Register a fused layer's mappings just before it is built."""
            self.fusers.setdefault(prefix, []).append(fuser)

            if (
                isinstance(fuser, AttentionFuser)
                and (index := fuser.layer_index(module)) is not None
            ):
                if index in self.attention_fusers:
                    other, _ = self.attention_fusers[index]
                    raise ValueError(
                        f"Layer {index} dispatches attention from more than one "
                        f"module ({other} and {prefix}), which one vLLM attention "
                        "layer cannot serve."
                    )
                self.attention_fusers[index] = (prefix, fuser)

            orig_to_new_stacked.update(fuser.orig_to_new_stacked(prefix))

            packed_modules_mapping = fuser.packed_modules_mapping
            self.packed_modules_mapping.update(packed_modules_mapping)
            if self.quant_config is not None:
                self.quant_config.packed_modules_mapping.update(packed_modules_mapping)

        offloader = self._offloader()
        if (decoder_layers := self._decoder_layers) is None and not isinstance(
            offloader, NoopOffloader
        ):
            logger.warning_once(
                "Could not identify the decoder layers of %s, so weight offloading "
                "will not be applied to them.",
                type(self.model).__name__,
            )
        # Decoder layers are detected as they are built, after their pre-replace hook
        fusers = Fusers(self.model, self.vllm_config, skip=decoder_layers)

        def _replace_decoder_layers(layers: nn.ModuleList, prefix: str):
            """Build this rank's decoder layers one at a time and offload each as
            soon as it is built, as `make_layers` does."""
            # Record the layers for Eagle3
            for name, _ in layers.named_children():
                self._layer_names[int(name)] = f"{prefix}.{name}".removeprefix("model.")
            # MTP weights should not be loaded into the base model
            num_layers = self.text_config.num_hidden_layers
            names = (
                "n_predict",  # Override from SpeculativeConfig
                "num_nextn_predict_layers",  # Most models
                "mtp_num_hidden_layers",  # Qwen 3.5
            )
            n_predict = getattr_iter(self.text_config, names, 0) or 0
            self.ignore_unexpected_prefixes.extend(
                f"{prefix}.{i}." for i in range(num_layers, num_layers + n_predict)
            )

            def build_layers():
                """Must mimic what `__init__` does to the whole model's parameters,
                because each layer is offloaded as soon as it is yielded."""
                for i in range(*self._pp_indices()):
                    layer_prefix = f"{prefix}.{i}"
                    self._pre_decoder_layer_replace(layers[i], layer_prefix)
                    fusers.add(layers[i])
                    _replace_child(layers, str(i), layers[i], prefix)
                    self.init_parameters(layers[i])
                    self.keep_in_fp32(layers[i], layer_prefix.removeprefix("model."))
                    yield layers[i]

            offloader.wrap_modules(build_layers())

        def _recursive_replace(module: nn.Module, prefix: str):
            if module is decoder_layers:
                _replace_decoder_layers(module, prefix)
                return
            for child_name, child_module in module.named_children():
                _replace_child(module, child_name, child_module, prefix)

        def _replace_child(
            module: nn.Module, child_name: str, child_module: nn.Module, prefix: str
        ):
            new_module = child_module
            qual_name = maybe_prefix(prefix, child_name)
            # Replace modules as needed
            if isinstance(child_module, nn.Linear):
                generator = (p for p in tp_plan if re.match(p, qual_name))
                pattern = next(generator, None)
                # Some weight loaders expect all linear layers to inherit
                # LinearBase, so we set a default style which causes any
                # unspecified layers to be replaced with ReplicatedLinear
                style = tp_plan.get(pattern, "replicate")
                new_module = replace_linear_class(
                    child_module, style, self.quant_config, prefix=qual_name
                )
            elif isinstance(child_module, (nn.Conv2d, nn.Conv3d)):
                new_module = replace_conv_class(child_module)
            elif child_module in vocab_embeddings:
                new_module = replace_embedding_class(
                    child_module, self.quant_config, prefix=qual_name
                )
            elif isinstance(child_module, nn.LayerNorm):
                new_module = replace_layernorm_class(child_module)
            elif child_module_fusers := fusers[child_module]:
                for fuser in child_module_fusers:
                    register_fusion(fuser, qual_name, child_module)
                    new_module = fuser.fuse(new_module, qual_name, self.vllm_config)
                    logger.info_once(fuser.info(child_name))
                _recursive_replace(new_module, prefix=qual_name)
            elif not isinstance(child_module, MoERunner):
                # MoERunner can contain aliases of shared experts and gates,
                # so we don't want to recurse into it and break weight loading.
                _recursive_replace(child_module, prefix=qual_name)

            if new_module is not child_module:
                setattr(module, child_name, new_module)
                log_replacement(qual_name, child_module, new_module)

        _recursive_replace(self.model, prefix="model")

        self.hf_to_vllm_mapper |= WeightsMapper(orig_to_new_stacked=orig_to_new_stacked)

    def _pp_indices(self) -> tuple[int, int]:
        """The range of decoder layers on this pipeline parallel rank."""
        return get_pp_indices(
            self.text_config.num_hidden_layers,
            self.pp_group.rank_in_group,
            self.pp_group.world_size,
        )

    def _create_attention_instances(self, prefix: str = ""):
        """Create `Attention` instances to inform KV cache allocation."""
        text_config = self.text_config
        attn_cls = self._get_attn_cls()

        # kv_lora_rank indicates that this is an MLA model
        if getattr(text_config, "kv_lora_rank", None) is not None:
            if attn_cls is MLAAttention:
                text_config._attn_implementation = VLLM_MLA_ATTN_IMPL
            else:
                # MLA model not using MLAAttention: recompute head_size for full attn
                qk_nope_head_dim = getattr(text_config, "qk_nope_head_dim", 0)
                qk_rope_head_dim = getattr(text_config, "qk_rope_head_dim", 0)
                if qk_head_dim := qk_nope_head_dim + qk_rope_head_dim:
                    self.model_config.model_arch_config.head_size = qk_head_dim

        logits_soft_cap = getattr(text_config, "attn_logit_softcapping", None)
        layer_types = getattr(text_config, "layer_types", None)

        start, end = self._pp_indices()

        for i in range(start, end):
            if i not in self.attention_fusers:
                layer = (
                    f"{i} ({layer_types[i]})"
                    if layer_types and i < len(layer_types)
                    else str(i)
                )
                raise ValueError(
                    f"Layer {layer} does not dispatch through the Transformers "
                    "attention interface and vLLM has no other way to handle it."
                )
            attn_prefix, attn_fuser = self.attention_fusers[i]
            attn_module = self.get_submodule(attn_prefix)

            # `[i]` is the whole-model config unless the checkpoint is
            # heterogeneous, in which case it is this layer's own geometry.
            arch_config = self.model_config.model_arch_config[i]
            num_heads = self.model_config.get_num_attention_heads(
                self.parallel_config, arch_config
            )
            head_size = arch_config.head_size
            if (scale := attn_fuser.scale(attn_module)) is None:
                # Default to Llama scale if AttentionFuser couldn't identify it
                scale = head_size**-0.5

            kwargs = dict(
                num_heads=num_heads,
                scale=scale,
                cache_config=self.cache_config,
                quant_config=self.quant_config,
                prefix=maybe_prefix(prefix, f"{attn_prefix}.{VLLM_ATTN_ATTR}"),
            )

            if attn_cls is MLAAttention:
                mla_fuser = next(
                    (f for f in self.fusers[attn_prefix] if isinstance(f, MLAFuser)),
                    None,
                )
                if mla_fuser is None:
                    raise ValueError(
                        f"Layer {i} ({attn_prefix}) did not fuse as MLA, but the rest "
                        "of the model did, so its latent KV cache cannot be allocated."
                    )
                dims = get_mla_dims(self.model_config)
                kwargs.update(
                    qk_nope_head_dim=dims.qk_nope_head_dim,
                    qk_rope_head_dim=dims.qk_rope_head_dim,
                    v_head_dim=dims.v_head_dim,
                    q_lora_rank=dims.q_lora_rank,
                    kv_lora_rank=dims.kv_lora_rank,
                    kv_b_proj=attn_module.get_submodule(mla_fuser.kv_b_proj_name),
                )
            else:
                kwargs.update(
                    head_size=head_size,
                    num_kv_heads=self.model_config.get_num_kv_heads(
                        self.parallel_config, arch_config
                    ),
                    logits_soft_cap=logits_soft_cap,
                )

                # Handle interleaved sliding window attention
                if layer_types and layer_types[i] == "sliding_attention":
                    kwargs["per_layer_sliding_window"] = text_config.sliding_window
                # Handle attention sinks
                if (sinks := attn_fuser.sinks(attn_module)) is not None:
                    kwargs["sinks"] = sinks

            attn_instance = attn_cls(**kwargs)

            # Attach the instance to the module it serves, which is the only way it
            # is in the module tree at all: a plain dict is invisible to
            # `nn.Module.__setattr__`. Being in the tree runs its
            # `process_weights_after_loading` and moves its KV-scale buffers with
            # `model.to(device)`, and lets the attention interface read it off the
            # module rather than index by `module.layer_idx`, an `int` that bakes
            # layer identity into the traced graph and so costs one compiled
            # artifact per layer.
            setattr(attn_module, VLLM_ATTN_ATTR, attn_instance)

    def _get_attn_cls(self) -> type[AttentionLayerBase]:
        """Return the `Attention` class to use for this model's layers."""
        # Some checkpoints make a decoder bidirectional in the config (gte-Qwen2)
        is_bidirectional = getattr(self.text_config, "is_causal", True) is False
        # A VLM's text model may be bidirectional over the multimodal prefix
        is_prefix_lm = self.model_config.is_multimodal_model
        has_encoder = has_bidirectional_attention(self.model) and not is_prefix_lm
        if is_bidirectional or has_encoder:
            return EncoderOnlyAttention
        if self.model_config.use_mla:
            if any(
                isinstance(fuser, MLAFuser)
                for fusers in self.fusers.values()
                for fuser in fusers
            ):
                return MLAAttention
            logger.warning_once(
                "This model uses MLA but `MLAFuser` failed to match and/or fuse any "
                "MLA attention layers. Falling back to full attention with a padded "
                "`value` head dimension."
            )
            os.environ["VLLM_MLA_DISABLE"] = "1"
        return Attention

    def init_parameters(self, module: nn.Module, dtype: torch.dtype | None = None):
        """If a `parameter` is on the `meta` device, then its parent
        `module` is the original module created by:

        ```python
        with torch.device("meta"):
            self.model: "PreTrainedModel" = AutoModel.from_config(...)
        ```
        """
        device = self.device_config.device

        def _init_parameters(module: nn.Module):
            for name, param in module.named_parameters(recurse=False):
                # Already on device, nothing to do
                if param.device != torch.device("meta"):
                    continue
                # Already a vLLM parameter, nothing to do
                if hasattr(param, "weight_loader"):
                    continue
                param_dtype = dtype or param.dtype
                data = torch.empty_like(param.data, dtype=param_dtype, device=device)
                param = nn.Parameter(data, requires_grad=param.requires_grad)
                setattr(module, name, param)
            for child in module.children():
                _init_parameters(child)

        _init_parameters(module)

    def keep_in_fp32(self, module: nn.Module, prefix: str = ""):
        """Honor `_keep_in_fp32_modules_strict` as `from_pretrained` would.

        `module` is `self.model` or one of its submodules, whose qualname from
        `self.model` is `prefix`."""
        if self.model_config.dtype not in (torch.float16, torch.bfloat16):
            return
        fragments = getattr(self.model, "_keep_in_fp32_modules_strict", None)
        if not fragments:
            return
        pattern = re.compile("|".join(re.escape(f) for f in fragments))
        for name, tensor in named_state(module):
            name = maybe_prefix(prefix, name)
            if not hasattr(tensor, "weight_loader") and pattern.search(name):
                tensor.data = tensor.data.to(torch.float32)

    def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
        return self.model.get_input_embeddings()(input_ids)

    def forward(
        self,
        input_ids: torch.Tensor | None,
        positions: torch.Tensor,
        intermediate_tensors: IntermediateTensors | None = None,
        inputs_embeds: torch.Tensor | None = None,
        **kwargs,
    ) -> torch.Tensor | IntermediateTensors:
        if not self.pp_group.is_first_rank:
            assert intermediate_tensors is not None
            input_ids = None
            inputs_embeds = intermediate_tensors["hidden_states"][None, ...]
            if self._copy_intermediate_hidden_states:
                buffer = self._intermediate_hidden_states_buffer
                buffer[:, : inputs_embeds.shape[1]].copy_(inputs_embeds)
            self._intermediate_hidden_states = inputs_embeds

        # Add batch dimension before entering Transformers model
        if input_ids is not None and input_ids.ndim == 1:
            # [seq_len] -> [1, seq_len]
            input_ids = input_ids[None, ...]
        if inputs_embeds is not None and inputs_embeds.ndim == 2:
            # [seq_len, hidden_size] -> [1, seq_len, hidden_size]
            inputs_embeds = inputs_embeds[None, ...]
        if positions.ndim == 1:
            # [seq_len] -> [1, seq_len]
            positions = positions[None, ...]
        elif self.model_config.uses_mrope:
            # [3, seq_len] -> [3, 1, seq_len], contiguous and at a fixed address
            mrope_positions = self.mrope_positions[: positions.numel()].view(3, 1, -1)
            positions = mrope_positions.copy_(positions[:, None])

        # Transformers models expect either input_ids or inputs_embeds, but not both
        if input_ids is not None and inputs_embeds is not None:
            input_ids = None

        self._aux_hidden_states.clear()
        outputs = self.model(
            input_ids=input_ids,
            inputs_embeds=inputs_embeds,
            use_cache=False,
            position_ids=positions,
            return_dict=False,
            **kwargs,
        )
        # Unset so modules which call the layers directly (e.g. MTP) skip the hook
        self._intermediate_hidden_states = None

        # Remove batch dimension after exiting Transformers model
        hidden_states = outputs[0][0, ...]
        aux_hidden_states = [x[0, ...] for x in self._aux_hidden_states]

        if not self.pp_group.is_last_rank:
            return IntermediateTensors(
                {
                    "hidden_states": hidden_states,
                    **self.pack_local_aux_hidden_states(aux_hidden_states),
                }
            )

        if aux_hidden_states:
            return hidden_states, aux_hidden_states
        return hidden_states

    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
        loader = AutoWeightsLoader(
            self,
            ignore_unexpected_prefixes=self.ignore_unexpected_prefixes,
            ignore_unexpected_suffixes=self.ignore_unexpected_suffixes,
        )
        return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)

    @staticmethod
    def check_version(min_version: str, feature: str):
        installed = Version(transformers.__version__)
        required = Version(min_version)
        if installed < required:
            raise ImportError(
                f"Transformers modeling backend requires transformers>={required} "
                f"for {feature}, but got {installed}"
            )

    def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
        self._set_aux_hidden_state_layers(layers)
        for layer in self.aux_hidden_state_layers:
            # The input to a layer is the output of the layer before it
            module = self.model.get_submodule(self._layer_names[max(layer - 1, 0)])
            if isinstance(module, PPMissingLayer):
                continue
            if layer == 0:
                module.register_forward_pre_hook(
                    self._record_layer_input, with_kwargs=True
                )
            else:
                module.register_forward_hook(self._record_layer_output)

    def _record_layer_input(self, module, args, kwargs) -> None:
        hidden_states = args[0] if args else kwargs["hidden_states"]
        self._aux_hidden_states.append(hidden_states)

    def _record_layer_output(self, module, args, output) -> None:
        hidden_states = output[0] if isinstance(output, tuple) else output
        self._aux_hidden_states.append(hidden_states)

    def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
        num_layers = self.text_config.num_hidden_layers
        return (2, num_layers // 2, num_layers - 3)

_aux_hidden_states = [] instance-attribute

Eagle3 aux hidden states recorded by hooks during the forward pass.

_decoder_layers cached property

The decoder layers, i.e. the only ModuleList in self._pp_split, if it has num_hidden_layers entries.

_from_config_kwargs cached property

The kwargs used to create self.model.

_layer_names = {} instance-attribute

Mapping from layer index to layer name for Eagle3.

_pp_split cached property

The module the pipeline parallel split is applied to, and the names (in order) of its children that the split assigns to ranks.

_pre_decoder_layer_replace = lambda layer, prefix: None instance-attribute

Called on each decoder layer before recursive_replace replaces it.

_pre_trained_model_classes cached property

The decoder class and the encoder class of each modality that has one.

Both come from a single throwaway model, since building one is not cheap.

attention_fusers = {} instance-attribute

layer_idx -> the qualname and fuser of the module computing that layer's attention, populated by recursive_replace for _create_attention_instances.

fusers = {} instance-attribute

Module qualname -> the fusers applied to it, populated by recursive_replace for _create_attention_instances.

ignore_unexpected_prefixes = [] instance-attribute

Ignore unexpected weights whose qualname starts with these prefixes.

ignore_unexpected_suffixes = [] instance-attribute

Ignore unexpected weights whose qualname ends with these suffixes.

packed_modules_mapping = {} instance-attribute

Fused module -> constituent projections, populated by recursive_replace for the quantization machinery and loaders (e.g. bitsandbytes).

_create_attention_instances(prefix='')

Create Attention instances to inform KV cache allocation.

Source code in vllm/model_executor/models/transformers/base.py
def _create_attention_instances(self, prefix: str = ""):
    """Create `Attention` instances to inform KV cache allocation."""
    text_config = self.text_config
    attn_cls = self._get_attn_cls()

    # kv_lora_rank indicates that this is an MLA model
    if getattr(text_config, "kv_lora_rank", None) is not None:
        if attn_cls is MLAAttention:
            text_config._attn_implementation = VLLM_MLA_ATTN_IMPL
        else:
            # MLA model not using MLAAttention: recompute head_size for full attn
            qk_nope_head_dim = getattr(text_config, "qk_nope_head_dim", 0)
            qk_rope_head_dim = getattr(text_config, "qk_rope_head_dim", 0)
            if qk_head_dim := qk_nope_head_dim + qk_rope_head_dim:
                self.model_config.model_arch_config.head_size = qk_head_dim

    logits_soft_cap = getattr(text_config, "attn_logit_softcapping", None)
    layer_types = getattr(text_config, "layer_types", None)

    start, end = self._pp_indices()

    for i in range(start, end):
        if i not in self.attention_fusers:
            layer = (
                f"{i} ({layer_types[i]})"
                if layer_types and i < len(layer_types)
                else str(i)
            )
            raise ValueError(
                f"Layer {layer} does not dispatch through the Transformers "
                "attention interface and vLLM has no other way to handle it."
            )
        attn_prefix, attn_fuser = self.attention_fusers[i]
        attn_module = self.get_submodule(attn_prefix)

        # `[i]` is the whole-model config unless the checkpoint is
        # heterogeneous, in which case it is this layer's own geometry.
        arch_config = self.model_config.model_arch_config[i]
        num_heads = self.model_config.get_num_attention_heads(
            self.parallel_config, arch_config
        )
        head_size = arch_config.head_size
        if (scale := attn_fuser.scale(attn_module)) is None:
            # Default to Llama scale if AttentionFuser couldn't identify it
            scale = head_size**-0.5

        kwargs = dict(
            num_heads=num_heads,
            scale=scale,
            cache_config=self.cache_config,
            quant_config=self.quant_config,
            prefix=maybe_prefix(prefix, f"{attn_prefix}.{VLLM_ATTN_ATTR}"),
        )

        if attn_cls is MLAAttention:
            mla_fuser = next(
                (f for f in self.fusers[attn_prefix] if isinstance(f, MLAFuser)),
                None,
            )
            if mla_fuser is None:
                raise ValueError(
                    f"Layer {i} ({attn_prefix}) did not fuse as MLA, but the rest "
                    "of the model did, so its latent KV cache cannot be allocated."
                )
            dims = get_mla_dims(self.model_config)
            kwargs.update(
                qk_nope_head_dim=dims.qk_nope_head_dim,
                qk_rope_head_dim=dims.qk_rope_head_dim,
                v_head_dim=dims.v_head_dim,
                q_lora_rank=dims.q_lora_rank,
                kv_lora_rank=dims.kv_lora_rank,
                kv_b_proj=attn_module.get_submodule(mla_fuser.kv_b_proj_name),
            )
        else:
            kwargs.update(
                head_size=head_size,
                num_kv_heads=self.model_config.get_num_kv_heads(
                    self.parallel_config, arch_config
                ),
                logits_soft_cap=logits_soft_cap,
            )

            # Handle interleaved sliding window attention
            if layer_types and layer_types[i] == "sliding_attention":
                kwargs["per_layer_sliding_window"] = text_config.sliding_window
            # Handle attention sinks
            if (sinks := attn_fuser.sinks(attn_module)) is not None:
                kwargs["sinks"] = sinks

        attn_instance = attn_cls(**kwargs)

        # Attach the instance to the module it serves, which is the only way it
        # is in the module tree at all: a plain dict is invisible to
        # `nn.Module.__setattr__`. Being in the tree runs its
        # `process_weights_after_loading` and moves its KV-scale buffers with
        # `model.to(device)`, and lets the attention interface read it off the
        # module rather than index by `module.layer_idx`, an `int` that bakes
        # layer identity into the traced graph and so costs one compiled
        # artifact per layer.
        setattr(attn_module, VLLM_ATTN_ATTR, attn_instance)

_create_hf_to_vllm_mapper()

Create a WeightsMapper to map checkpoint weight names to module qualnames.

This handles:

  • Transformers weight renaming from WeightRenaming
  • Checkpoints saved with a base model prefix that is not model
  • Checkpoints saved with no base model prefix
  • Any quantization config specific mappings
Source code in vllm/model_executor/models/transformers/base.py
def _create_hf_to_vllm_mapper(self):
    """Create a WeightsMapper to map checkpoint weight names to module qualnames.

    This handles:

    - Transformers weight renaming from `WeightRenaming`
    - Checkpoints saved with a base model prefix that is not `model`
    - Checkpoints saved with no base model prefix
    - Any quantization config specific mappings
    """
    orig_to_new_renaming: list[WeightRenaming] = []
    orig_to_new_regex: dict[re.Pattern, str | None] = {}

    for mapping in self._weight_conversions():
        # Handle weights which have been renamed in Transformers
        if isinstance(mapping, WeightRenaming):
            orig_to_new_renaming.append(mapping)
        # TODO: Handle WeightConverter to enable layer merging

    # Handle unexpected weights which should be ignored
    if self.model._keys_to_ignore_on_load_unexpected is not None:
        for key in self.model._keys_to_ignore_on_load_unexpected:
            orig_to_new_regex[re.compile(key)] = None

    # Standardise base model prefix
    bmp = self.model.base_model_prefix
    expected_bmp = r"model.\1"
    # Handle checkpoints saved with different base model prefix
    if bmp and bmp != "model":
        different_bmp_pattern = re.compile(rf"^{bmp}\.(.+)")
        orig_to_new_regex[different_bmp_pattern] = expected_bmp
    # Handle direct children of self.model which were saved without the model prefix
    direct_children = chain(
        self.model.named_children(),
        self.model.named_parameters(recurse=False),
        self.model.named_buffers(recurse=False),
    )
    model_children = "|".join(name for name, _ in direct_children)
    missing_bmp_pattern = re.compile(rf"^(?!model\.)(({model_children}).*)")
    orig_to_new_regex[missing_bmp_pattern] = expected_bmp
    # Handle weights saved as direct children of self.model which no longer are
    unexpected_bmp_pattern = re.compile(rf"^(model\.)((?!{model_children}).+)")
    orig_to_new_regex[unexpected_bmp_pattern] = r"\2"
    # Handle lm_head which was saved inside the base model
    nested_lm_head_pattern = re.compile(r"^model\.(.+\.)*(lm_head.+)")
    orig_to_new_regex[nested_lm_head_pattern] = r"\2"

    self.hf_to_vllm_mapper = WeightsMapper(
        orig_to_new_renaming=orig_to_new_renaming,
        orig_to_new_regex=orig_to_new_regex,
    )

    # Apply mapping to quantization config if needed
    self._maybe_apply_model_mapping()

_create_model()

Create the Transformers model that self.model wraps.

Source code in vllm/model_executor/models/transformers/base.py
def _create_model(self) -> "PreTrainedModel":
    """Create the Transformers model that `self.model` wraps."""
    return AutoModel.from_config(**self._from_config_kwargs)

_decorate_cls_for_torch_compile(cls, dynamic_arg_dims, enable_if, is_encoder)

Decorate cls to indicate to vLLM that it supports torch compile.

Parameters:

  • cls

    (type[PreTrainedModel]) –

    The PreTrainedModel class to decorate.

  • dynamic_arg_dims

    (DynamicArgDims | None) –

    A mapping from argument name to the dynamic dimensions of the argument. If None, default dynamic arg dims will be used. See support_torch_compile for more details.

  • enable_if

    (Callable[[VllmConfig], bool]) –

    A function which takes in the vLLM config and returns whether torch compile should be enabled for this class.

  • is_encoder

    (bool) –

    Whether the class being decorated is an encoder.

Source code in vllm/model_executor/models/transformers/base.py
def _decorate_cls_for_torch_compile(
    self,
    cls: type["PreTrainedModel"],
    dynamic_arg_dims: DynamicArgDims | None,
    enable_if: Callable[["VllmConfig"], bool],
    is_encoder: bool,
):
    """Decorate `cls` to indicate to vLLM that it supports torch compile.

    Args:
        cls: The PreTrainedModel class to decorate.
        dynamic_arg_dims: A mapping from argument name to the dynamic dimensions
            of the argument. If None, default dynamic arg dims will be used. See
            [`support_torch_compile`][vllm.compilation.decorators.support_torch_compile]
            for more details.
        enable_if: A function which takes in the vLLM config and returns whether
            torch compile should be enabled for this class.
        is_encoder: Whether the class being decorated is an encoder.

    """
    logger.debug(
        "Decorating `%s` as %s for torch compile with dynamic_arg_dims of %s",
        cls.__name__,
        "encoder" if is_encoder else "decoder",
        dynamic_arg_dims,
    )

    support_torch_compile(
        dynamic_arg_dims=dynamic_arg_dims,
        enable_if=enable_if,
        is_encoder=is_encoder,
        use_aot_compile=False,
    )(cls)

_decorate_for_torch_compile()

Decorate the model's decoder class to indicate to vLLM that it supports torch compile if can_enable_torch_compile is True.

Source code in vllm/model_executor/models/transformers/base.py
def _decorate_for_torch_compile(self):
    """Decorate the model's decoder class to indicate to vLLM that it
    supports torch compile if `can_enable_torch_compile` is True."""
    self._decorate_cls_for_torch_compile(
        cls=self._pre_trained_model_classes.decoder,
        # Applied to a PreTrainedModel so the batch dimension will exist
        dynamic_arg_dims=DynamicArgDims(
            input_ids=1,  # shape: [1, seq_len]
            inputs_embeds=1,  # shape: [1, seq_len, hidden_size]
            position_ids=-1,  # shape: [1, seq_len] or [3, 1, seq_len] for mrope
        ),
        enable_if=can_enable_torch_compile,
        is_encoder=False,
    )

_find_encoder_classes(model)

Find the encoder class of each modality model has one for.

Text models have none. Multi-modal ones override this.

Source code in vllm/model_executor/models/transformers/base.py
def _find_encoder_classes(
    self, model: "PreTrainedModel"
) -> dict[str, type["PreTrainedModel"]]:
    """Find the encoder class of each modality `model` has one for.

    Text models have none. Multi-modal ones override this.
    """
    return {}

_get_attn_cls()

Return the Attention class to use for this model's layers.

Source code in vllm/model_executor/models/transformers/base.py
def _get_attn_cls(self) -> type[AttentionLayerBase]:
    """Return the `Attention` class to use for this model's layers."""
    # Some checkpoints make a decoder bidirectional in the config (gte-Qwen2)
    is_bidirectional = getattr(self.text_config, "is_causal", True) is False
    # A VLM's text model may be bidirectional over the multimodal prefix
    is_prefix_lm = self.model_config.is_multimodal_model
    has_encoder = has_bidirectional_attention(self.model) and not is_prefix_lm
    if is_bidirectional or has_encoder:
        return EncoderOnlyAttention
    if self.model_config.use_mla:
        if any(
            isinstance(fuser, MLAFuser)
            for fusers in self.fusers.values()
            for fuser in fusers
        ):
            return MLAAttention
        logger.warning_once(
            "This model uses MLA but `MLAFuser` failed to match and/or fuse any "
            "MLA attention layers. Falling back to full attention with a padded "
            "`value` head dimension."
        )
        os.environ["VLLM_MLA_DISABLE"] = "1"
    return Attention

_get_tie_word_embeddings()

Check if the model has tied word embeddings.

Source code in vllm/model_executor/models/transformers/base.py
def _get_tie_word_embeddings(self):
    """Check if the model has tied word embeddings."""
    # Models created with Transformers v4 and v5 will store this in different places
    tie_word_embeddings_v4 = getattr(self.text_config, "tie_word_embeddings", False)
    tie_word_embeddings_v5 = getattr(self.config, "tie_word_embeddings", False)
    return tie_word_embeddings_v4 or tie_word_embeddings_v5

_inject_intermediate_hidden_states(module, args, kwargs)

Pass the hidden states from the previous pipeline rank to module unchanged.

The decoder still runs its pre-layer code for masks and rotary embeddings, but anything it does to the hidden states was already done on the first rank.

Source code in vllm/model_executor/models/transformers/base.py
def _inject_intermediate_hidden_states(self, module, args, kwargs):
    """Pass the hidden states from the previous pipeline rank to `module` unchanged.

    The decoder still runs its pre-layer code for masks and rotary embeddings, but
    anything it does to the hidden states was already done on the first rank."""
    if (hidden_states := self._intermediate_hidden_states) is None:
        return None
    if self._copy_intermediate_hidden_states:
        # Sliced here so the token count stays dynamic under torch.compile
        buffer = self._intermediate_hidden_states_buffer
        hidden_states = buffer[:, : hidden_states.shape[1]]
    if args:
        return (hidden_states, *args[1:]), kwargs
    return args, kwargs | {"hidden_states": hidden_states}

_mark_model_components(vllm_config)

Mark language model and tower submodules as self.model is created.

Nothing to do in Base, MultiModalMixin will override.

Source code in vllm/model_executor/models/transformers/base.py
@contextmanager
def _mark_model_components(self, vllm_config: "VllmConfig"):
    """Mark language model and tower submodules as `self.model` is created.

    Nothing to do in `Base`, `MultiModalMixin` will override."""
    yield

_offloader()

The offloader for the decoder layers.

Source code in vllm/model_executor/models/transformers/base.py
def _offloader(self) -> BaseOffloader:
    """The offloader for the decoder layers."""
    return get_offloader()

_patch_config()

Patch the config to ensure that the model is created correctly:

  • Sets the attention implementation to "vllm" so the attention instances from _create_attention_instances are used
  • Sets the dtype to the default torch dtype set by vLLM because Transformers uses the config dtype when creating the model
Source code in vllm/model_executor/models/transformers/base.py
def _patch_config(self):
    """Patch the config to ensure that the model is created correctly:

    - Sets the attention implementation to "vllm" so the attention instances from
    `_create_attention_instances` are used
    - Sets the dtype to the default torch dtype set by vLLM because Transformers
    uses the config dtype when creating the model
    """
    self.text_config._attn_implementation = VLLM_ATTN_IMPL
    self.config.dtype = torch.get_default_dtype()

_pp_indices()

The range of decoder layers on this pipeline parallel rank.

Source code in vllm/model_executor/models/transformers/base.py
def _pp_indices(self) -> tuple[int, int]:
    """The range of decoder layers on this pipeline parallel rank."""
    return get_pp_indices(
        self.text_config.num_hidden_layers,
        self.pp_group.rank_in_group,
        self.pp_group.world_size,
    )

_tp_plan()

The tensor parallel plan of self.model, relative to self.model.

Source code in vllm/model_executor/models/transformers/base.py
def _tp_plan(self) -> dict:
    """The tensor parallel plan of `self.model`, relative to `self.model`."""
    return self.model.tp_plan or {}

_vocab_embeddings()

The nn.Embeddings in self.model which hold a vocab table.

Source code in vllm/model_executor/models/transformers/base.py
def _vocab_embeddings(self) -> set[nn.Embedding]:
    """The `nn.Embedding`s in `self.model` which hold a vocab table."""

    def vocab_sizes(config: PreTrainedConfig):
        for key, value in vars(config).items():
            if isinstance(value, PreTrainedConfig):
                yield from vocab_sizes(value)
            elif "vocab_size" in key and isinstance(value, int):
                yield value

    sizes = set(vocab_sizes(self.config))
    # `get_input_embeddings` may return a module which composes the `nn.Embedding`,
    # or a `PPMissingLayer` if the embeddings are not on this pipeline stage
    embeddings = {
        module
        for module in self.model.get_input_embeddings().modules()
        if isinstance(module, nn.Embedding)
    }
    embeddings.update(
        module
        for module in self.model.modules()
        if isinstance(module, nn.Embedding) and module.num_embeddings in sizes
    )
    return embeddings

_weight_conversions()

The Transformers weight conversions that apply to self.model.

Source code in vllm/model_executor/models/transformers/base.py
def _weight_conversions(self) -> list[WeightTransform]:
    """The Transformers weight conversions that apply to `self.model`."""
    mappings = get_model_conversion_mapping(self.model)
    # Ensure layers that are part of the model, but not AutoModel, get their mapping
    instantiated = {type(m).__name__ for m in self.model.modules()}
    for arch in self.config.architectures or []:
        if arch not in instantiated:
            mappings.extend(get_checkpoint_conversion_mapping(arch) or [])
    return mappings

init_parameters(module, dtype=None)

If a parameter is on the meta device, then its parent module is the original module created by:

with torch.device("meta"):
    self.model: "PreTrainedModel" = AutoModel.from_config(...)
Source code in vllm/model_executor/models/transformers/base.py
def init_parameters(self, module: nn.Module, dtype: torch.dtype | None = None):
    """If a `parameter` is on the `meta` device, then its parent
    `module` is the original module created by:

    ```python
    with torch.device("meta"):
        self.model: "PreTrainedModel" = AutoModel.from_config(...)
    ```
    """
    device = self.device_config.device

    def _init_parameters(module: nn.Module):
        for name, param in module.named_parameters(recurse=False):
            # Already on device, nothing to do
            if param.device != torch.device("meta"):
                continue
            # Already a vLLM parameter, nothing to do
            if hasattr(param, "weight_loader"):
                continue
            param_dtype = dtype or param.dtype
            data = torch.empty_like(param.data, dtype=param_dtype, device=device)
            param = nn.Parameter(data, requires_grad=param.requires_grad)
            setattr(module, name, param)
        for child in module.children():
            _init_parameters(child)

    _init_parameters(module)

keep_in_fp32(module, prefix='')

Honor _keep_in_fp32_modules_strict as from_pretrained would.

module is self.model or one of its submodules, whose qualname from self.model is prefix.

Source code in vllm/model_executor/models/transformers/base.py
def keep_in_fp32(self, module: nn.Module, prefix: str = ""):
    """Honor `_keep_in_fp32_modules_strict` as `from_pretrained` would.

    `module` is `self.model` or one of its submodules, whose qualname from
    `self.model` is `prefix`."""
    if self.model_config.dtype not in (torch.float16, torch.bfloat16):
        return
    fragments = getattr(self.model, "_keep_in_fp32_modules_strict", None)
    if not fragments:
        return
    pattern = re.compile("|".join(re.escape(f) for f in fragments))
    for name, tensor in named_state(module):
        name = maybe_prefix(prefix, name)
        if not hasattr(tensor, "weight_loader") and pattern.search(name):
            tensor.data = tensor.data.to(torch.float32)

pipeline_parallel()

Apply the model's pipeline parallelization plan.

Source code in vllm/model_executor/models/transformers/base.py
def pipeline_parallel(self):
    """Apply the model's pipeline parallelization plan."""
    if self.pp_group.world_size <= 1:
        return

    module, names = self._pp_split
    if not self.model.supports_pp_plan:
        tip = get_feature_request_tip(
            self.model_config.model, self.model_config.trust_remote_code
        )
        logger.warning_once(
            "%s does not define a pipeline parallel plan. The Transformers "
            "modeling backend will infer the split from the layers of %s in order "
            "of declaration and keep parameter-free modules on every rank. This "
            "may fail if the model's structure is non-standard. %s",
            type(self.model).__name__,
            type(module).__name__,
            tip,
        )

    if (layers := self._decoder_layers) is None:
        raise ValueError(f"Could not identify the decoder layers of {type(module)}")
    # attrgetter in case the module is nested (e.g. "text_model.layers")
    module_list_idx = next(
        i for i, name in enumerate(names) if attrgetter(name)(module) is layers
    )

    # Layers before module list
    input_embeddings = set(self.model.get_input_embeddings().modules())
    keep_input_embeddings = (
        self._get_tie_word_embeddings() and self.pp_group.is_last_rank
    ) or spec_decode_needs_target_embed(self.vllm_config)
    for name in names[:module_list_idx]:
        if self.pp_group.is_first_rank:
            break
        child = attrgetter(name)(module)
        is_input_embedding = not input_embeddings.isdisjoint(child.modules())
        if is_input_embedding and keep_input_embeddings:
            continue
        missing = (
            PPMissingScalar()
            if isinstance(child, nn.Embedding) and not is_input_embedding
            else PPMissingLayer()
        )
        # attrsetter in case the module is nested (e.g. "text_model.embed_tokens")
        attrsetter(name)(module, missing)

    # Module list
    start_layer, end_layer = self._pp_indices()
    for i in range(len(layers)):
        if start_layer <= i and i < end_layer:
            continue
        layers[i] = PPMissingLayer()

    # Layers after module list
    for name in names[module_list_idx + 1 :]:
        # Modules that should be on last rank
        if not self.pp_group.is_last_rank:
            # attrsetter in case the module is nested (e.g. "text_model.norm")
            attrsetter(name)(module, PPMissingLayer())

recursive_replace()

Recursively replace modules in the model as needed.

Currently, this replaces:

  • GLUs with a fused MergedColumnParallelLinear + ...AndMul
  • Attention QKV projections with a fused QKVParallelLinear + split
  • nn.Linear with vLLM's tensor parallel linear classes
  • nn.Conv2d / nn.Conv3d with vLLM's Conv2d / Conv3d
  • Vocab nn.Embeddings with vLLM's VocabParallelEmbedding
  • RMSNorm (detected from their dataflow) with vLLM's RMSNormor GemmaRMSNorm
  • nn.LayerNorm with vLLM's LayerNorm
Source code in vllm/model_executor/models/transformers/base.py
def recursive_replace(self):
    """Recursively replace modules in the model as needed.

    Currently, this replaces:

    - GLUs with a fused `MergedColumnParallelLinear` + `...AndMul`
    - Attention QKV projections with a fused `QKVParallelLinear` + split
    - `nn.Linear` with vLLM's tensor parallel linear classes
    - `nn.Conv2d` / `nn.Conv3d` with vLLM's `Conv2d` / `Conv3d`
    - Vocab `nn.Embedding`s with vLLM's `VocabParallelEmbedding`
    - RMSNorm (detected from their dataflow) with vLLM's `RMSNorm`or `GemmaRMSNorm`
    - `nn.LayerNorm` with vLLM's `LayerNorm`
    """
    tp_plan = self._tp_plan()

    if not tp_plan and self.tp_group.world_size > 1:
        tip = get_feature_request_tip(
            self.model_config.model, self.model_config.trust_remote_code
        )
        logger.warning_once(
            "%s does not define a tensor parallel plan. The Transformers modeling "
            "backend will shard the model the best it can during graph fusion and "
            "replicate the rest. This may be suboptimal or fail if the model does "
            "not fuse cleanly. %s",
            type(self.model).__name__,
            tip,
        )

    # Prefix the patterns because we always start from `self.model`
    tp_plan = {maybe_prefix("model", k): v for k, v in tp_plan.items()}
    vocab_embeddings = self._vocab_embeddings()

    orig_to_new_stacked: dict[str, tuple[str, ShardId]] = {}

    def register_fusion(fuser: BaseFuser, prefix: str, module: nn.Module):
        """Register a fused layer's mappings just before it is built."""
        self.fusers.setdefault(prefix, []).append(fuser)

        if (
            isinstance(fuser, AttentionFuser)
            and (index := fuser.layer_index(module)) is not None
        ):
            if index in self.attention_fusers:
                other, _ = self.attention_fusers[index]
                raise ValueError(
                    f"Layer {index} dispatches attention from more than one "
                    f"module ({other} and {prefix}), which one vLLM attention "
                    "layer cannot serve."
                )
            self.attention_fusers[index] = (prefix, fuser)

        orig_to_new_stacked.update(fuser.orig_to_new_stacked(prefix))

        packed_modules_mapping = fuser.packed_modules_mapping
        self.packed_modules_mapping.update(packed_modules_mapping)
        if self.quant_config is not None:
            self.quant_config.packed_modules_mapping.update(packed_modules_mapping)

    offloader = self._offloader()
    if (decoder_layers := self._decoder_layers) is None and not isinstance(
        offloader, NoopOffloader
    ):
        logger.warning_once(
            "Could not identify the decoder layers of %s, so weight offloading "
            "will not be applied to them.",
            type(self.model).__name__,
        )
    # Decoder layers are detected as they are built, after their pre-replace hook
    fusers = Fusers(self.model, self.vllm_config, skip=decoder_layers)

    def _replace_decoder_layers(layers: nn.ModuleList, prefix: str):
        """Build this rank's decoder layers one at a time and offload each as
        soon as it is built, as `make_layers` does."""
        # Record the layers for Eagle3
        for name, _ in layers.named_children():
            self._layer_names[int(name)] = f"{prefix}.{name}".removeprefix("model.")
        # MTP weights should not be loaded into the base model
        num_layers = self.text_config.num_hidden_layers
        names = (
            "n_predict",  # Override from SpeculativeConfig
            "num_nextn_predict_layers",  # Most models
            "mtp_num_hidden_layers",  # Qwen 3.5
        )
        n_predict = getattr_iter(self.text_config, names, 0) or 0
        self.ignore_unexpected_prefixes.extend(
            f"{prefix}.{i}." for i in range(num_layers, num_layers + n_predict)
        )

        def build_layers():
            """Must mimic what `__init__` does to the whole model's parameters,
            because each layer is offloaded as soon as it is yielded."""
            for i in range(*self._pp_indices()):
                layer_prefix = f"{prefix}.{i}"
                self._pre_decoder_layer_replace(layers[i], layer_prefix)
                fusers.add(layers[i])
                _replace_child(layers, str(i), layers[i], prefix)
                self.init_parameters(layers[i])
                self.keep_in_fp32(layers[i], layer_prefix.removeprefix("model."))
                yield layers[i]

        offloader.wrap_modules(build_layers())

    def _recursive_replace(module: nn.Module, prefix: str):
        if module is decoder_layers:
            _replace_decoder_layers(module, prefix)
            return
        for child_name, child_module in module.named_children():
            _replace_child(module, child_name, child_module, prefix)

    def _replace_child(
        module: nn.Module, child_name: str, child_module: nn.Module, prefix: str
    ):
        new_module = child_module
        qual_name = maybe_prefix(prefix, child_name)
        # Replace modules as needed
        if isinstance(child_module, nn.Linear):
            generator = (p for p in tp_plan if re.match(p, qual_name))
            pattern = next(generator, None)
            # Some weight loaders expect all linear layers to inherit
            # LinearBase, so we set a default style which causes any
            # unspecified layers to be replaced with ReplicatedLinear
            style = tp_plan.get(pattern, "replicate")
            new_module = replace_linear_class(
                child_module, style, self.quant_config, prefix=qual_name
            )
        elif isinstance(child_module, (nn.Conv2d, nn.Conv3d)):
            new_module = replace_conv_class(child_module)
        elif child_module in vocab_embeddings:
            new_module = replace_embedding_class(
                child_module, self.quant_config, prefix=qual_name
            )
        elif isinstance(child_module, nn.LayerNorm):
            new_module = replace_layernorm_class(child_module)
        elif child_module_fusers := fusers[child_module]:
            for fuser in child_module_fusers:
                register_fusion(fuser, qual_name, child_module)
                new_module = fuser.fuse(new_module, qual_name, self.vllm_config)
                logger.info_once(fuser.info(child_name))
            _recursive_replace(new_module, prefix=qual_name)
        elif not isinstance(child_module, MoERunner):
            # MoERunner can contain aliases of shared experts and gates,
            # so we don't want to recurse into it and break weight loading.
            _recursive_replace(child_module, prefix=qual_name)

        if new_module is not child_module:
            setattr(module, child_name, new_module)
            log_replacement(qual_name, child_module, new_module)

    _recursive_replace(self.model, prefix="model")

    self.hf_to_vllm_mapper |= WeightsMapper(orig_to_new_stacked=orig_to_new_stacked)

PPMissingScalar

Bases: PPMissingLayer

A PPMissingLayer that returns a scalar zero instead of its input.

For layers whose input is not hidden states (e.g. learned position embeddings), where passing the input through would give the wrong shape.

Source code in vllm/model_executor/models/transformers/base.py
class PPMissingScalar(PPMissingLayer):
    """A `PPMissingLayer` that returns a scalar zero instead of its input.

    For layers whose input is not hidden states (e.g. learned position embeddings),
    where passing the input through would give the wrong shape."""

    def __init__(self):
        super().__init__()
        self.register_buffer("zero", torch.zeros(()), persistent=False)

    def forward(self, *args, **kwargs) -> torch.Tensor:
        return self.zero

PreTrainedModelClasses

Bases: NamedTuple

Attributes:

  • encoders (dict[str, type[PreTrainedModel]]) –

    Modality -> encoder class, for each modality that has one.

Source code in vllm/model_executor/models/transformers/base.py
class PreTrainedModelClasses(NamedTuple):
    decoder: type["PreTrainedModel"]
    encoders: dict[str, type["PreTrainedModel"]]
    """Modality -> encoder class, for each modality that has one."""

encoders instance-attribute

Modality -> encoder class, for each modality that has one.