Skip to content

vllm.v1.attention.ops.flydsl_kernels.gqa_verify

Speculative-decode verify (full or sliding-window attention) on gfx950 in FlyDSL.

Up to 8 verify tokens x GQA 8 (64 query columns) per KV head (for fewer tokens the unused columns duplicate the last token and their tiles are trimmed), one wave per KV head, one workgroup per (request, split); BF16 Q, FP8 e4m3 paged KV [pages, 16 or 32, kv_heads, 64], split-KV FP32 partials. QK and PV run on BF16 MFMAs (K and V upcast exactly from FP8).

Transposed form per wave: S^T = K Q^T (keys x query columns) and O^T = V^T P^T on 16x16x32 BF16 MFMAs. * K is the A operand straight from HBM: lane (g, i) loads 16 contiguous bytes (dims 16g..16g+15) of key i of a 16-key half chunk; dims are permuted consistently in Q (k-step ks holds dims 16g + 8ks + 0..7). * Softmax runs along keys = MFMA rows: in-thread over the 8 rows of a lane, across the four 16-lane groups with two permlane swaps, and only when some score clears the running max by 2^8 (lazy rescale, exact: P <= 256). * P^T's accumulator registers are the PV B operand once the 32 keys of a chunk are relabelled (k-position 8g + i <-> key i < 4 ? 4g + i : 12 + 4g + i); V is written to wave-private LDS in that order with contiguous ds_write_b128 and read back as the V^T A operand with ds_read_b64_tr_b8 (16-byte chunks XOR-swizzled by row so the transposed reads are bank-conflict free). * Page ids are loaded one chunk group ahead of use, as vector loads in order with K and V. * K / V go through buffer loads (32-bit byte offsets), or 64-bit global loads when a layer's pool reaches 2^31 bytes. * NS register load slots refilled right after use keep ~NS chunks in flight per wave.

Functions:

  • balance_map –

    Balanced-grid map, computed on device: wgmap [G, 4] int32 = (request, first

  • gqa_verify_balanced –

    Full-attention verify on the length-balanced grid, merged into out.

  • gqa_verify_window –

    Sliding-window verify with equal splits per request, merged into out.

  • pick_balanced –

    (groups, wg_per_cu, waves_per_eu) of the balanced grid for a batch, or None

  • pick_window_splits –

    Splits per request of the sliding-window verify (a power of two). A window

_allreduce_groups(x_f32, use_max)

Reduce an f32 across lanes l, l^16, l^32, l^48 (the four 16-lane groups).

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def _allreduce_groups(x_f32, use_max):
    """Reduce an f32 across lanes l, l^16, l^32, l^48 (the four 16-lane groups)."""
    v = _raw(x_f32)
    for op in ("permlane32_swap", "permlane16_swap"):
        bits = arith.bitcast(T.i32, v)
        a, b = _swap(op, bits)
        fa = arith.bitcast(T.f32, a)
        fb = arith.bitcast(T.f32, b)
        v = arith.maximumf(fa, fb) if use_max else arith.addf(fa, fb)
    return v

_cvt8(lo_word, hi_word)

8 FP8 e4m3 bytes (two i32) -> v8bf16 in byte order (exact upcast, scale 1).

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def _cvt8(lo_word, hi_word):
    """8 FP8 e4m3 bytes (two i32) -> v8bf16 in byte order (exact upcast, scale 1)."""
    one = _f32(1.0)
    parts = []
    for w in (lo_word, hi_word):
        for sel in (False, True):
            pr = _rocdl.cvt_scalef32_pk_bf16_fp8(T.vec(2, T.bf16), _raw(w), one, sel)
            parts.append(fx.Vector(pr).bitcast(fx.Int32)[0])
    return _raw(fx.Vector.from_elements(parts, fx.Int32).bitcast(fx.BFloat16))

_global_load16(addr)

16 bytes (vector<4 x i32>) at the 64-bit global byte address addr.

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def _global_load16(addr):
    """16 bytes (vector<4 x i32>) at the 64-bit global byte address `addr`."""
    ptr = llvm.IntToPtrOp(ir.Type.parse("!llvm.ptr<1>"), _raw(addr)).result
    return llvm.LoadOp(T.vec(4, T.i32), ptr, alignment=16).result

_partials(q, k, v, seq_lens, block_table, scale, k_scale, splits, *, waves_per_eu, wg_per_cu, query_len, wgmap=None, window=0)

FP32 split-KV partials plus max / denom (log2 domain).

Sliding window (wgmap None): equal splits, partials [tokens, heads, splits(, 64)]. Full attention (wgmap from balance_map): splits is the workgroup count G and the partials are [G, query_len, heads(, 64)], indexed by workgroup.

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def _partials(
    q,
    k,
    v,
    seq_lens,
    block_table,
    scale,
    k_scale,
    splits,
    *,
    waves_per_eu,
    wg_per_cu,
    query_len,
    wgmap=None,
    window=0,
):
    """FP32 split-KV partials plus max / denom (log2 domain).

    Sliding window (`wgmap` None): equal splits, partials `[tokens, heads, splits(,
    64)]`. Full attention (`wgmap` from `balance_map`): `splits` is the workgroup
    count G and the partials are `[G, query_len, heads(, 64)]`, indexed by workgroup.
    """
    import torch

    n = seq_lens.numel()
    heads, kvh = q.shape[1], k.shape[2]
    balanced = wgmap is not None
    assert balanced != bool(window)

    # buffer offsets are 32-bit byte offsets: a pool past them takes the 64-bit loads
    def _span(t):  # strided-view span in elements
        return 1 + sum((n - 1) * st for n, st in zip(t.shape, t.stride()))

    addr64 = max(_span(k), _span(v)) * k.element_size() >= 2**31
    st = (splits, query_len, heads) if balanced else (q.shape[0], heads, splits)
    part = torch.empty((*st, HD), dtype=torch.float32, device=q.device)
    maxi = torch.empty(st, dtype=torch.float32, device=q.device)
    den = torch.empty(st, dtype=torch.float32, device=q.device)
    _run_compiled(
        _launcher(
            kvh,
            k.stride(0),
            k.stride(1),
            k.stride(2),
            float(scale),
            waves_per_eu,
            wg_per_cu,
            addr64,
            balanced,
            int(query_len),
            int(window),
            int(k.shape[1]),
        ),
        q,
        k.view(torch.uint8),
        v.view(torch.uint8),
        block_table,
        seq_lens,
        k_scale,
        part,
        maxi,
        den,
        wgmap if balanced else seq_lens,
        int(splits),
        int(block_table.stride(0)),
        int(q.stride(0)),
        int(splits if balanced else n * splits),
        fx.Stream(torch.cuda.current_stream(q.device).cuda_stream),
    )
    return part, maxi, den

_sload(view, index)

Wave-uniform i32 element of a buffer view (scalar load into an SGPR).

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def _sload(view, index):
    """Wave-uniform i32 element of a buffer view (scalar load into an SGPR)."""
    return fx.rocdl.s_buffer_load(fx.get_iter(view), index)

_swap(op_name, x)

Partner values across lane groups.

permlane{32,16}_swap of x with itself -> (vdst, vsrc) pair, i32.

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def _swap(op_name, x):
    """Partner values across lane groups.

    permlane{32,16}_swap of x with itself -> (vdst, vsrc) pair, i32.
    """
    i32 = ir.IntegerType.get_signless(32)
    pair = ir.Type.parse("!llvm.struct<(i32, i32)>")
    raw = _raw(x)
    sw = getattr(_rocdl, op_name)(pair, raw, raw, False, True)
    return llvm.extractvalue(i32, sw, [0]), llvm.extractvalue(i32, sw, [1])

balance_map(seq_lens, groups, out=None)

Balanced-grid map, computed on device: wgmap [G, 4] int32 = (request, first chunk, end chunk, -) and wgstart [n + 1] int32. out optionally gives buffers of at least those sizes to fill.

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def balance_map(seq_lens, groups, out=None):
    """Balanced-grid map, computed on device: `wgmap` `[G, 4]` int32 = (request, first
    chunk, end chunk, -) and `wgstart` `[n + 1]` int32. `out` optionally gives
    buffers of at least those sizes to fill.
    """
    import torch

    n = seq_lens.numel()
    assert n < groups
    if out is None:
        wgmap = torch.empty((groups, 4), dtype=torch.int32, device=seq_lens.device)
        wgstart = torch.empty(n + 1, dtype=torch.int32, device=seq_lens.device)
    else:
        wgmap, wgstart = out[0][:groups], out[1][: n + 1]
    _balance_map[(triton.cdiv(groups, MAP_BLOCK_G),)](
        seq_lens,
        wgmap,
        wgstart,
        n,
        groups,
        NS,
        max(16, triton.next_power_of_2(n)),
        MAP_BLOCK_G,
        num_warps=MAP_WARPS,
    )
    return wgmap, wgstart

gqa_verify_balanced(q, k, v, out, seq_lens, block_table, scale, k_scale, v_scale, *, query_len, grid, maps)

Full-attention verify on the length-balanced grid, merged into out.

grid is pick_balanced's (groups, wg_per_cu, waves_per_eu); maps the balance_map of seq_lens for those groups.

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def gqa_verify_balanced(
    q,
    k,
    v,
    out,
    seq_lens,
    block_table,
    scale,
    k_scale,
    v_scale,
    *,
    query_len,
    grid,
    maps,
):
    """Full-attention verify on the length-balanced grid, merged into `out`.

    `grid` is `pick_balanced`'s (groups, wg_per_cu, waves_per_eu); `maps` the
    `balance_map` of `seq_lens` for those groups.
    """
    groups, wg_per_cu, waves_per_eu = grid
    wgmap, wgstart = maps
    partial, maximum, denom = _partials(
        q,
        k,
        v,
        seq_lens,
        block_table,
        scale,
        k_scale,
        groups,
        waves_per_eu=waves_per_eu,
        wg_per_cu=wg_per_cu,
        query_len=query_len,
        wgmap=wgmap,
    )
    _reduce_balanced[(q.shape[0], q.shape[1])](
        partial,
        maximum,
        denom,
        out,
        v_scale,
        wgstart,
        q.shape[1],
        16,
        out.stride(0),
        out.stride(1),
        query_len,
        num_warps=2,
    )

gqa_verify_window(q, k, v, out, seq_lens, block_table, scale, k_scale, v_scale, *, query_len, window, num_cus)

Sliding-window verify with equal splits per request, merged into out.

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def gqa_verify_window(
    q,
    k,
    v,
    out,
    seq_lens,
    block_table,
    scale,
    k_scale,
    v_scale,
    *,
    query_len,
    window,
    num_cus,
):
    """Sliding-window verify with equal splits per request, merged into `out`."""
    splits = pick_window_splits(seq_lens.numel(), k.shape[2], num_cus)
    partial, maximum, denom = _partials(
        q,
        k,
        v,
        seq_lens,
        block_table,
        scale,
        k_scale,
        splits,
        waves_per_eu=0,
        wg_per_cu=2,
        query_len=query_len,
        window=window,
    )
    _reduce_splits[(q.shape[0], q.shape[1])](
        partial,
        maximum,
        denom,
        out,
        v_scale,
        q.shape[1],
        splits,
        out.stride(0),
        out.stride(1),
        num_warps=2,
    )

pick_balanced(num_seqs, kv_heads, num_cus)

(groups, wg_per_cu, waves_per_eu) of the balanced grid for a batch, or None when the batch has too many requests for it (each needs its own workgroup).

Fixed per batch size: CUDA-graph safe. A workgroup runs one wave per KV head. One workgroup per CU suits 4 or more KV heads below 24 requests; two per CU (waves_per_eu 2: twice the loads in flight) from 24 requests up, and from 3 requests up with 2 or fewer KV heads (no waves_per_eu hint: the LDS pin already allows only one wave per SIMD).

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def pick_balanced(
    num_seqs: int, kv_heads: int, num_cus: int
) -> tuple[int, int, int] | None:
    """(groups, wg_per_cu, waves_per_eu) of the balanced grid for a batch, or None
    when the batch has too many requests for it (each needs its own workgroup).

    Fixed per batch size: CUDA-graph safe. A workgroup runs one wave per KV head. One
    workgroup per CU suits 4 or more KV heads below 24 requests; two per CU
    (waves_per_eu 2: twice the loads in flight) from 24 requests up, and from 3
    requests up with 2 or fewer KV heads (no waves_per_eu hint: the LDS pin already
    allows only one wave per SIMD).
    """
    if kv_heads <= 2:
        two = num_seqs >= TWO_WG_PER_CU_MIN_REQUESTS_FEW_HEADS
        pick = (2, 0) if two else (1, 1)
    else:
        pick = (2, 2) if num_seqs >= TWO_WG_PER_CU_MIN_REQUESTS else (1, 1)
    groups = pick[0] * num_cus
    return (groups, *pick) if num_seqs < groups else None

pick_window_splits(num_seqs, kv_heads, num_cus)

Splits per request of the sliding-window verify (a power of two). A window spans few chunks: one or two requests split it finely, larger batches use fewer, longer splits once the grid covers the CUs.

Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
def pick_window_splits(num_seqs: int, kv_heads: int, num_cus: int) -> int:
    """Splits per request of the sliding-window verify (a power of two). A window
    spans few chunks: one or two requests split it finely, larger batches use fewer,
    longer splits once the grid covers the CUs.
    """
    if num_seqs <= 2:
        return 64 // num_seqs
    if num_seqs * 16 <= num_cus:
        return 16
    return 8 if num_seqs * kv_heads * 2 <= num_cus else 4