/ 
Langertha-0.503
/ Langertha::Role::RuntimeKnobs

NAME

Langertha::Role::RuntimeKnobs - Role for a self-hosted engine with per-request prefix-cache runtime knobs

VERSION

version 0.503

prefix_cache_salt

Optional prefix-cache salt string, emitted as cache_salt on the vLLM and SGLang wires. Cache isolation, not a performance knob: only requests sharing a salt reuse each other's KV blocks, so a random per-request salt reduces cache reuse. vLLM's cache_salt requires vLLM v1.x; older servers silently ignore unknown fields. See Langertha::Runtime::Knobs.

cache_prompt

Optional llama.cpp cache_prompt boolean: reuse the prompt cache for this request. Emitted as a JSON boolean on the llama.cpp wire.

n_cache_reuse

Optional llama.cpp n_cache_reuse integer. Counter-intuitive: 0 means reuse all cached tokens; higher values limit reuse. See Langertha::Runtime::Knobs.

id_slot

Optional llama.cpp id_slot integer: pin this request to a specific server slot (useful when the relevant KV cache lives on one slot).

priority

Optional SGLang priority integer: request scheduling priority. Higher values are served first.

return_cached_tokens_details

Optional SGLang return_cached_tokens_details boolean: ask the server to report cached prompt tokens (usage.prompt_tokens_details.cached_tokens on the response). Emitted as a JSON boolean on the SGLang wire.

extra_key

Optional SGLang extra_key string: an additional cache-key component beyond the prompt text, used to partition the prefix cache.

knob_wire_format

knob_wire_format => 'vllm'

The per-engine enum naming which self-hosted runtime-knob dialect this engine speaks — vllm | sglang | llamacpp. Drives the value-object dispatch in "knobs_kwargs". There is no shared default: each engine must set its tag explicitly (override _build_knob_wire_format), because the three servers accept disjoint field sets and no generic 'openai' knob dialect exists. Accessing the attribute on an engine that forgot the tag croaks with a message naming the role and the missing tag.

knobs_kwargs_for

my %kwargs = $engine->knobs_kwargs_for( prefix_cache_salt => 'tenant-a' );
my %kwargs = $engine->knobs_kwargs_for( %$controls );

Returns the body kwargs to merge into a chat request for the runtime knobs, serialized for "knob_wire_format" via Langertha::Runtime::Knobs. %args may carry any of the canonical knob names (prefix_cache_salt, cache_prompt, n_cache_reuse, id_slot, priority, return_cached_tokens_details, extra_key — the same names as the attributes, so the whole controls hash from chat_f can be passed wholesale); keys it does not carry fall back to the engine attributes, so a per-request control (chat_f, karr #46) beats the configured attribute on a per-key basis. Empty list when neither a per-request value nor an attribute is set, or when none of the set knobs is supported on the engine's wire.

knobs_kwargs

my %kwargs = $engine->knobs_kwargs;

Returns the body kwargs to merge into a chat request for the configured runtime knobs, serialized for "knob_wire_format" via Langertha::Runtime::Knobs. Empty list when nothing is set, or when none of the set knobs is supported on the engine's wire. Delegates to "knobs_kwargs_for" with no per-request overrides.

SEE ALSO

SUPPORT

Issues

Please report bugs and feature requests on GitHub at https://github.com/Getty/langertha/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <getty@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.