Skip to content

vllm.distributed.kv_transfer.kv_connector.v1.lmcache_integration.utils

Functions:

apply_mm_hashes_to_token_ids(token_ids, mm_hashes, mm_positions)

Overwrite token_ids in-place for multimodal placeholders using efficient slice assignments.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/lmcache_integration/utils.py 
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
def apply_mm_hashes_to_token_ids(
    token_ids: torch.Tensor,
    mm_hashes: list[str],
    mm_positions: list["PlaceholderRange"],
) -> torch.Tensor:
    """Overwrite token_ids in-place for multimodal placeholders using
    efficient slice assignments.
    """
    n = token_ids.size(0)
    for hash_str, placeholder in zip(mm_hashes, mm_positions):
        start, length = placeholder.offset, placeholder.length
        if start >= n:
            continue
        end = min(start + length, n)
        token_ids[start:end] = hex_hash_to_int16(hash_str)
    return token_ids

extract_mm_features(request, modify=False)

Normalize multimodal information from a Request into parallel lists.

This helper reads either

1) request.mm_features (objects each exposing .identifier and .mm_position), or 2) legacy fields request.mm_hashes and request.mm_positions.

It returns two equally sized lists: the multimodal hash identifiers and their corresponding positions. If the request contains no multimodal info, it returns ([], []).

Parameters:

  • request

    (Request) –

    The source object.

  • modify

    (bool, default: False ) –

    Controls copy semantics for the legacy-path return values. - If True and legacy fields are used, shallow-copies are returned so the caller can mutate the lists without affecting request. - If False, the original legacy sequences are returned as-is (zero-copy); treat them as read-only.

Returns:

  • list[str] –

    tuple[list[str], list[PlaceholderRange]]: (mm_hashes, mm_positions).

  • list[PlaceholderRange] –

    May be ([], []) when no multimodal data is present.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/lmcache_integration/utils.py 
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
def extract_mm_features(
    request: Union["Request", "NewRequestData"], modify: bool = False
) -> tuple[list[str], list["PlaceholderRange"]]:
    """Normalize multimodal information from a Request into parallel lists.

    This helper reads either:
      1) `request.mm_features` (objects each exposing `.identifier` and
      `.mm_position`), or
      2) legacy fields `request.mm_hashes` and `request.mm_positions`.

    It returns two equally sized lists: the multimodal hash identifiers and
    their corresponding positions. If the request contains no multimodal info,
    it returns `([], [])`.

    Args:
        request (Request): The source object.
        modify (bool):
            Controls copy semantics for the legacy-path return values.
            - If True and legacy fields are used, shallow-copies are returned so
              the caller can mutate the lists without affecting `request`.
            - If False, the original legacy sequences are returned as-is
              (zero-copy); treat them as read-only.

    Returns:
        tuple[list[str], list[PlaceholderRange]]: (`mm_hashes`, `mm_positions`).
        May be `([], [])` when no multimodal data is present.

    """
    if getattr(request, "mm_features", None):
        mm_hashes, mm_positions = zip(
            *((f.identifier, f.mm_position) for f in request.mm_features)
        )
        return (list(mm_hashes), list(mm_positions))
    elif getattr(request, "mm_hashes", None):
        if modify:
            return (
                request.mm_hashes.copy(),  # type: ignore
                request.mm_positions.copy(),  # type: ignore
            )
        else:
            return (request.mm_hashes, request.mm_positions)  # type: ignore
    else:
        return ([], [])

hex_hash_to_int16(s)

Convert a hex hash string to a 16-bit integer.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/lmcache_integration/utils.py 
61
62
63
def hex_hash_to_int16(s: str) -> int:
    """Convert a hex hash string to a 16-bit integer."""
    return int(s, 16) & 0xFFFF

lmcache_get_or_create_config()

Get the LMCache configuration from the environment variable LMCACHE_CONFIG_FILE. If the environment variable is not set, this function will return the default configuration.

This function is thread-safe and implements singleton pattern, ensuring the configuration is loaded only once.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/lmcache_integration/utils.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
def lmcache_get_or_create_config() -> V1Config:
    """Get the LMCache configuration from the environment variable
    `LMCACHE_CONFIG_FILE`. If the environment variable is not set, this
    function will return the default configuration.

    This function is thread-safe and implements singleton pattern,
    ensuring the configuration is loaded only once.
    """
    global _config_instance

    # Double-checked locking for thread-safe singleton
    if _config_instance is None:
        with _config_lock:
            if _config_instance is None:  # Check again within lock
                LMCacheEngineConfig = V1Config  # type: ignore[assignment]

                if "LMCACHE_CONFIG_FILE" not in os.environ:
                    logger.warning(
                        "No LMCache configuration file is set. Trying to read"
                        " configurations from the environment variables."
                    )
                    logger.warning(
                        "You can set the configuration file through "
                        "the environment variable: LMCACHE_CONFIG_FILE"
                    )
                    _config_instance = LMCacheEngineConfig.from_env()
                else:
                    config_file = os.environ["LMCACHE_CONFIG_FILE"]
                    logger.info("Loading LMCache config file %s", config_file)
                    _config_instance = LMCacheEngineConfig.from_file(config_file)
                    # Update config from environment variables
                    _config_instance.update_config_from_env()
    return _config_instance