vllm.model_executor.model_loader.base_loader
¶
Classes:
-
BaseModelLoader–Base class for model loaders.
Functions:
-
log_model_inspection–Log model structure if VLLM_LOG_MODEL_INSPECTION=1.
-
log_online_quantization–Log the online-quantized layer count and types, when applicable.
-
log_online_quantization_time–Log online quantization processing time, when applicable.
BaseModelLoader
¶
Bases: ABC
Base class for model loaders.
Methods:
-
create_model–Create a model with the given configurations.
-
download_model–Download a model so that it can be immediately loaded.
-
get_all_weights–Iterate over all weights by checkpoint name, for filtered
-
get_external_weight_memory–Get weights memory from external process;
-
load_model–Load a model with the given configurations.
-
load_weights–Load weights into a model. This standalone API allows
Source code in vllm/model_executor/model_loader/base_loader.py
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | |
create_model(vllm_config, model_config, prefix='')
¶
Create a model with the given configurations.
Source code in vllm/model_executor/model_loader/base_loader.py
download_model(model_config)
abstractmethod
¶
get_all_weights(model_config, model)
¶
Iterate over all weights by checkpoint name, for filtered partial reloads (e.g. FT expert reload after scale_down).
Source code in vllm/model_executor/model_loader/base_loader.py
get_external_weight_memory(vllm_config)
¶
Get weights memory from external process; 0 when the weights are not external.
load_model(vllm_config, model_config, prefix='')
¶
Load a model with the given configurations.
Source code in vllm/model_executor/model_loader/base_loader.py
load_weights(model, model_config)
abstractmethod
¶
Load weights into a model. This standalone API allows inplace weights loading for an already-initialized model
Source code in vllm/model_executor/model_loader/base_loader.py
_get_online_quantization_config(vllm_config)
¶
Return the online quantization config when one is configured.
Source code in vllm/model_executor/model_loader/base_loader.py
log_model_inspection(model)
¶
Log model structure if VLLM_LOG_MODEL_INSPECTION=1.
Source code in vllm/model_executor/model_loader/base_loader.py
log_online_quantization(vllm_config)
¶
Log the online-quantized layer count and types, when applicable.
Source code in vllm/model_executor/model_loader/base_loader.py
log_online_quantization_time(vllm_config)
¶
Log online quantization processing time, when applicable.