NIXL KV Cache Lease Renewal¶
In disaggregated prefill/decode deployments, the Prefill instance (P) must hold KV cache blocks in GPU memory after completing a prefill, waiting for the Decode instance (D) to read them via RDMA. A mechanism is needed to determine when those blocks can safely be freed when D isn't able to retrieve them. This mechanism was introduced in PR #41383.
Motivation¶
The single-timeout problem¶
The original design used a single, large timeout (VLLM_NIXL_ABORT_REQUEST_TIMEOUT, default 480s) to control how long P retains KV blocks. When D crashed or disconnected, P would hold onto potentially several GBs of "dead" blocks for up to 8 minutes before reclaiming them. During this window, subsequent requests hitting P would find reduced cache capacity and experience degraded performance.
The overloading problem¶
Simply lowering the timeout introduces a different failure mode. Under traffic surges, requests can sit in D's waiting queue for a long time before being scheduled. If the fixed timeout on P is too short, blocks get freed before D ever has a chance to read them --- causing unnecessary recomputation and wasted prefill work.
Solution: lease renewal via heartbeats¶
The lease renewal mechanism addresses both problems simultaneously. P grants a short initial lease (default 30s) when prefill completes. While a request is queued or in-flight on D, D periodically sends heartbeats to P extending the lease. If D crashes and stops heartbeating, P reclaims blocks within seconds of the last heartbeat rather than waiting minutes. If D is merely overloaded, the heartbeats keep the blocks alive for as long as needed.
How It Works¶
Lease lifecycle¶
When P finishes a prefill, it pins the KV blocks with an initial lease duration (kv_lease_duration, default 30s). From that point, the blocks are held until either:
- D completes the KV transfer --- P receives a read-completion notification and frees the blocks immediately.
- D keeps heartbeating --- each heartbeat extends the lease by
lease_duration * 2/3(~20s), keeping blocks alive indefinitely while D is healthy. - No heartbeat arrives --- the lease expires and P reclaims the blocks.
Piggybacking on NIXL notifications¶
Rather than introducing a new transport channel, heartbeats reuse NIXL's existing notification system (send_notif / get_new_notifs). The notification medium is backend-specific, with automatic fallback from IB/RoCE to TCP already handled by NIXL. Each single heartbeat message sent from D to a particular P renews all requests pinned in P on behalf of that D --- in other words, a single batched message per iteration renews the lease of multiple requests.
Scheduler-side tracking (D)¶
A critical insight is that heartbeating must start as soon as a request enters D's scheduler --- not when it gets scheduled for execution. Under heavy load, a request may sit in the waiting queue for much longer than the initial lease duration, and the gap between arrival and scheduling is unbounded.
To achieve this, D's connector (NixlConnectorScheduler) hooks into the scheduler via on_new_request(). When a request with do_remote_prefill=True arrives, the connector immediately starts tracking it for heartbeats. Requests are grouped by remote_engine_id for efficient batching. On each scheduler step, heartbeat metadata is packaged into NixlConnectorMetadata and sent to the worker, throttled by a heartbeat interval of lease_duration // 6 (~5s). A newly tracked request is heartbeated at the next step without waiting for the interval, as a router may have held it for most of its lease.
Tracking stops when either the KV transfer completes (via update_connector_output) or the request finishes/aborts (via request_finished).
Timing and simplicity¶
Heartbeat sending and processing happen in the forward loop, not in a background thread. This means timing is not millisecond-precise --- a long model forward pass will delay heartbeats. However, the lease durations are configured with sufficient margin: with default settings, the heartbeat interval (~5s) and lease extension (~20s) are at least an order of magnitude larger than a typical forward pass. This avoids lock complexity between threads while keeping the design simple and extensible.
Stale read protection¶
Heartbeats only start once the request has entered D's scheduler. A request that a router or proxy holds outside D after P finished (for example while D is at capacity) renews nothing, so P's lease runs out and P recycles the blocks for new prefills. Nothing in the transport notices: a NIXL read of recycled blocks succeeds and loads other requests' KV cache (and, for hybrid models, their SSM state), which surfaces later as corrupted output or NaN logits.
To close this gap, P exports the lease deadline in kv_transfer_params
(remote_blocks_expiry_time, in P's perf_counter domain) together with
remote_lease_extension, the number of seconds each heartbeat renews the lease
by. Before issuing a read, D's worker rebases the deadline with the clock offset
measured at handshake and declines the read if the deadline is within
the safety margin of now: 5s, or the heartbeat interval
(kv_lease_duration / 6) for leases shorter than 30s, so that heartbeats can
keep a short lease alive.
The heartbeats D itself sent for the request are taken into account with an
exact model of P's renewal rule: D's scheduler ships the exported deadline and
renewal amount along with each heartbeat batch (HeartbeatInfo.leases), and
the worker applies every heartbeat it sends to a per-request lease deadline
the way P's _handle_heartbeat would. A heartbeat that goes out while the
modelled lease is still alive renews it to send time + extension; one that
goes out after the lease ran out renews nothing, and neither does anything after
it. The send time plus the safety margin stands in for the arrival time on P, so
the model can only lag the real lease, never run ahead of it (a pause between
heartbeats is harmless as long as the current lease, initial or renewed, covers
it). A heartbeat round that failed to reach one of the remote ranks is not
counted: this avoids relying on the scheduler-side aggregation that keeps the
blocks allocated until every rank's lease has ended (consequence: with PP > 1
notification-only agents, a permanently unreachable rank means no round is ever
recorded for that engine, and heartbeated waits longer than the lease are
declined).
The same check runs again when the read is observed complete: if the lease may have run out while the transfer was in flight, the KV is discarded as if the read had been declined (completion is observed no earlier than it happened, so this only errs on the side of declining).
A declined read is reported to the scheduler as failed_recving, counted in
vllm:nixl_num_declined_expired_reads on the reading instance (the P-side
vllm:nixl_num_kv_expired_reqs counts the leases that ran out), and the request
follows kv_load_failure_policy: with the default fail it finishes with an
error, with recompute D prefills it locally. Deployments whose router may hold
requests longer than kv_lease_duration should raise the lease, fix the router,
or set "kv_load_failure_policy": "recompute".
What this does not cover:
- Peers that export no deadline or no renewal amount (older versions, fixed-TTL blocks, or a router that drops the fields), or for which no clock offset is known, are read as before.
- A single heartbeat stall longer than
extension - margin(15s with the defaults) marks the modelled lease as lapsed for good, even though P may have renewed it if the late heartbeat still arrived in time; the request is then declined however long it is heartbeated afterwards. The shared margin trades this (rare, conservative) case for simplicity. - A heartbeat notification that takes longer than the safety margin to reach P. The model credits the renewal at send time; if P only processes the heartbeat after the lease ran out, the blocks are gone and the model does not know. This is the assumption the heartbeat protocol itself rests on (a heartbeat delayed past the lease already lets P free the blocks); a confirmation round-trip from D to P before the read would remove it and is a possible follow-up.
- Request ids containing
,: the heartbeat payload is comma-separated, so P cannot match them and never renews their leases (pre-existing). The model never credits a renewal for such ids, so their reads are declined once the initial lease is over. - The clock offset is estimated once per handshake (and again whenever the remote re-registers). Both hosts' monotonic clocks are assumed to be rate-disciplined (NTP/chrony); an undisciplined drift that exceeds the margin over the lifetime of the pairing would defeat the check.
- Blocks released for other reasons: P frees them as soon as the expected
consumers' completion notifications arrive, so replayed or duplicated
kv_transfer_paramsread blocks the lease model considers live; and a P restarted behind the same address and engine id holds neither the lease nor the data. Both are outside the lease model.
Happy Path¶
sequenceDiagram
participant R as Routing Proxy
participant P as Prefill Instance
participant D as Decode Instance
R->>P: Request (do_remote_decode=True)
P->>P: Run prefill
P->>P: Grant lease (30s)
P->>R: Response (with kv_transfer_params)
R->>D: Request (do_remote_prefill=True)
note over D: Request enters waiting queue
D->>D: on_new_request() starts tracking
loop Every ~5s (heartbeat interval)
D->>P: Heartbeat (extend lease)
P->>P: Lease extended by ~20s
end
note over D: Request scheduled for execution
D->>P: KV transfer (RDMA read)
P-->D: Transfer complete
D->>D: Stop heartbeating
P->>P: Free KV blocks
Decode Instance Crash¶
sequenceDiagram
participant R as Routing Proxy
participant P as Prefill Instance
participant D as Decode Instance
R->>P: Request (do_remote_decode=True)
P->>P: Run prefill (holds onto KVs with lease)
P->>R: Response
R->>D: Request (do_remote_prefill=True)
D->>P: Heartbeat (extend lease)
D->>P: Heartbeat (extend lease)
note over D: D crashes
note over P: No heartbeat received
P->>P: Lease expires (~20s, not 480s)
P->>P: Free KV blocks
Worker-side sending and receiving¶
On D (sending): During start_load_kv() (called every forward pass), the worker reads metadata.heartbeat_by_engine and sends batched heartbeat notifications to each remote P engine. If D hasn't yet handshaked with P for a given engine (common for requests still in the waiting queue), it triggers a proactive handshake in a background thread.
The heartbeat is deferred to the next step once the handshake completes --- the early handshake also speeds up the eventual KV transfer.
On P (receiving): In _get_new_notifs(), P's worker checks incoming NIXL notifications. Messages starting with "HB:" are routed to _handle_heartbeat(), which extends the lease expiry for each referenced request using max(old_expiry, now + lease_extension). This ensures leases are never accidentally shortened.
Bidirectional KV Transfer¶
For multi-turn conversations, bidirectional KV transfer allows D to cache KV blocks that P can pull from on subsequent turns. Since the timing of the next conversational turn is client-dependent (not controlled by the system), the heartbeat-based lease mechanism does not apply here. Instead, a separate decoder_kv_blocks_ttl (default 480s) provides a simple fixed timeout for blocks cached on D. If the client takes too long to continue the conversation, the blocks expire. D communicates back the expiry time so P can know when blocks are expired and recompute. Because the deadline is a perf_counter value produced on D and the two engines run in separate processes (with unrelated clocks), P estimates the clock offset to D from the handshake round-trip and applies it before comparing the deadline against its own perf_counter. Future work may extend a symmetric heartbeat mechanism to this case.
Key Design Decisions¶
-
Per-request leasing, not per-instance. P has no notion of which D its KV blocks belong to --- block ownership is only resolved after prefill completes and the router selects a D. Leasing at the request level avoids coupling P/D selection in the load balancer. In practice, D batches lease extensions toward the same P by grouping requests with the same
remote_engine_id. -
NIXL notifications as transport. Heartbeats reuse the existing
send_notif/get_new_notifssystem rather than adding ZMQ connections or API changes. The notification medium is backend-specific with IB/RoCE-to-TCP fallback already handled, making heartbeats work across any NIXL-supported transport. -
No background thread. Heartbeat sending and processing happen in the forward loop (
start_load_kv/get_finished). This avoids lock complexity between threads. Lease durations provide sufficient margin over forward-pass latency (seconds vs. milliseconds). -
Proactive handshake. When D needs to heartbeat a P engine it hasn't connected to yet (common for requests still in the waiting queue), it triggers an early handshake in a background thread. This also speeds up the eventual KV transfer.
-
Heterogeneous TP support. When P TP > D TP (e.g., P TP=4, D TP=2), a single D worker pulls from multiple P workers. Heartbeats must be sent to all P workers for a given engine. Conversely, when D TP > P TP, a single P receives notifications from multiple Ds, which simply refreshes the TTL multiple times with no downside.
Configuration¶
The lease mechanism is controlled through kv_connector_extra_config in --kv-transfer-config:
| Parameter | Default | Description |
|---|---|---|
kv_lease_duration |
30s | Initial lease duration on P. Heartbeat interval and extension amount are derived automatically (interval = duration // 6, extension = duration * 2 // 3). |
decoder_kv_blocks_ttl |
480s | TTL for KV blocks cached on D in bidirectional transfer mode. Simple fixed timeout, not renewed via heartbeats. |
vllm serve <MODEL> \
--kv-transfer-config '{
"kv_connector": "NixlConnector",
"kv_role": "kv_producer",
"kv_connector_extra_config": {"kv_lease_duration": 60}
}'
kv_load_failure_policy (top-level --kv-transfer-config, default fail) decides what
happens to a request whose remote read was declined because the lease had run out: fail
ends it with an error, recompute prefills it locally.
For full NixlConnector configuration details, see the NixlConnector Usage Guide.