vllm.v1.attention.ops.rocm_gqa_verify
¶
Decode and speculative-decode verify attention for ROCM_AITER_FA (gfx950) on
GQA 8, head size 64 layers with an FP8 KV cache.
With an FP8 e4m3 KV cache, 16- or 32-token pages and every decode request having the
same number of query tokens, 1 to 8 (plain decode, or 1 to 7 draft tokens; head size
64, GQA 8, no sinks / soft cap / ALiBi), the backend calls fast_verify instead of
AITER paged_attention_v1 / unified_attention. It runs a FlyDSL split-KV kernel
(BF16 QK / PV, K / V upcast exactly from FP8):
- full-attention layers: a length-balanced grid, where a fixed one or two workgroups
per CU share the step's KV chunks evenly (mapped on device from
seq_lens, once per step by the metadata builder), so the kernel tracks total KV bytes instead of the longest request while there are well under two requests per CU (each request takes at least one workgroup); - sliding-window layers: vLLM's window (keys with
q_pos - k_pos < sliding_windoware visible, the same mask asunified_attention's1 + window_size[0]), with equal splits per request.
The kernel reads the block table and sequence lengths on device only, launches fixed
grids for a given batch size and allocates its partials with torch.empty, so it is
CUDA-graph capturable. Zero-length padded requests (FULL-graph padding) produce
zeros.
Functions:
-
fast_verify–Attention for the decode requests of a step, if the batch fits.
-
layout_supported–Host-only dtype / page size / stride checks; no device reads.
-
map_buffers–Persistent storage for
verify_map(full CUDA graphs keep reading it). -
supports_layer–Whether an attention layer has the shape and features the kernels handle.
-
uniform_query_len–Whether every decode request has the same 1 to 8 query tokens (a batch with
-
verify_map–The full-attention layers' work map for a step's decode
seq_lens, shared by
fast_verify(query, key_cache, value_cache, output, attn_metadata, num_decodes, num_decode_tokens, decode_max_query_len, softmax_scale, sliding_window, layer, maps=None)
¶
Attention for the decode requests of a step, if the batch fits.
Runs when every decode request has the same number of query tokens, 1 to 8, the cache layout is supported and (full attention) the batch fits the balanced grid; returns False, having done nothing, otherwise.
Parameters:
-
(query¶Tensor) –[tokens, heads, 64]bf16 query; the decode rows come first. -
(key_cache¶Tensor) –[num_blocks, 16 or 32, kv_heads, 64]FP8 view of the K cache. -
(value_cache¶Tensor) –the same for V.
-
(output¶Tensor) –[tokens, heads, 64]bf16, the decode rows are written. -
(attn_metadata¶AiterFlashAttentionMetadata) –the backend's metadata (
seq_lens,block_table). -
(num_decodes¶int) –number of decode requests.
-
(num_decode_tokens¶int) –number of decode tokens.
-
(decode_max_query_len¶int) –longest decode query.
-
(softmax_scale¶float) –attention scale.
-
(sliding_window¶int) –vLLM's window in tokens, or -1 / 0 for full attention.
-
(layer¶Module) –the attention layer, for its
_k_scale/_v_scale. -
(maps¶tuple[Tensor, Tensor] | None, default:None) –the step's
verify_mapfor full-attention layers, if prebuilt.
Returns:
-
bool–Whether the kernels ran.
Source code in vllm/v1/attention/ops/rocm_gqa_verify.py
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 | |
layout_supported(query, key_cache, value_cache, output, seq_lens, block_table)
¶
Host-only dtype / page size / stride checks; no device reads.
A failing check warns once: the backend then uses its AITER kernels for every decode step.
Source code in vllm/v1/attention/ops/rocm_gqa_verify.py
map_buffers(max_requests, device)
¶
Persistent storage for verify_map (full CUDA graphs keep reading it).
Source code in vllm/v1/attention/ops/rocm_gqa_verify.py
supports_layer(num_heads, num_kv_heads, head_size, alibi_slopes, sinks, logits_soft_cap)
¶
Whether an attention layer has the shape and features the kernels handle.
Evaluated once per layer. Warns once and returns False when FlyDSL cannot be imported.
Source code in vllm/v1/attention/ops/rocm_gqa_verify.py
uniform_query_len(num_decodes, num_decode_tokens, decode_max_query_len)
¶
Whether every decode request has the same 1 to 8 query tokens (a batch with
mixed query lengths has fewer than decode_max_query_len * num_decodes).
Source code in vllm/v1/attention/ops/rocm_gqa_verify.py
verify_map(seq_lens, kv_heads, out=None)
¶
The full-attention layers' work map for a step's decode seq_lens, shared by
every layer with kv_heads KV heads per rank; None if the batch is too large
for the balanced grid. out takes buffers from map_buffers.