Skip to content

vllm.v1.kv_offload.cpu.swap_blocks_triton

Triton kernel + tuned constants for the swap_blocks_batch fast path.

Functions:

  • swap_blocks_batch –

    Triton implementation of swap_blocks_batch for small CPU->GPU batches.

swap_blocks_batch(src_addrs, dst_addrs, sizes, is_src_access_order_any=False, *, bytes_per_chunk)

Triton implementation of swap_blocks_batch for small CPU->GPU batches.

Source code in vllm/v1/kv_offload/cpu/swap_blocks_triton.py 
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
def swap_blocks_batch(
    src_addrs: torch.Tensor,
    dst_addrs: torch.Tensor,
    sizes: torch.Tensor,
    is_src_access_order_any: bool = False,
    *,
    bytes_per_chunk: int,
) -> None:
    """Triton implementation of ``swap_blocks_batch`` for small CPU->GPU batches."""
    n = src_addrs.numel()
    # Too few descriptors to amortize Triton's launch cost.
    if n < MIN_N:
        ops.swap_blocks_batch(
            src_addrs,
            dst_addrs,
            sizes,
            is_src_access_order_any=is_src_access_order_any,
        )
        return
    _swap_blocks_kernel[(min(NUM_SMS, n),)](
        src_addrs.to("cuda", non_blocking=True),
        dst_addrs.to("cuda", non_blocking=True),
        sizes.to("cuda", non_blocking=True),
        n,
        BYTES_PER_CHUNK=bytes_per_chunk,
    )