/ 
Langertha-0.505
/ Langertha::Engine::SGLang

NAME

Langertha::Engine::SGLang - SGLang inference server

VERSION

version 0.505

SYNOPSIS

use Langertha::Engine::SGLang;

# 1. Simple chat
my $sglang = Langertha::Engine::SGLang->new(
    url   => 'http://localhost:30000/v1',
    model => 'Qwen/Qwen2.5-7B-Instruct',
);

print $sglang->simple_chat('Say something nice');

# 2. Streaming
$sglang->simple_chat_stream(sub {
    print shift->content;
}, 'Write a haiku about Perl');

# 3. MCP tool calling (requires a tool-call-parser-compatible model)
use Future::AsyncAwait;

my $sglang = Langertha::Engine::SGLang->new(
    url         => 'http://localhost:30000/v1',
    model       => 'Qwen/Qwen2.5-7B-Instruct',
    mcp_servers => [$mcp],
);

my $response = await $sglang->chat_with_tools_f('Add 7 and 15');

# 4. Multimodal input (vision-capable models served by SGLang)
use Langertha::Content::Image;

my $img  = Langertha::Content::Image->from_url('https://example.com/cat.jpg');
my $resp = await $sglang->simple_chat_f({
    role    => 'user',
    content => [ 'What is in this image?', $img ],
});

# 5. Prometheus /metrics scraping (Runtime::MetricsPoll)
my $records = await $sglang->poll_metrics_f('sglang:');

# 6. Embeddings (server launched with an embedding model)
my $embedder = Langertha::Engine::SGLang->new(
    url   => 'http://localhost:30000/v1',
    model => 'Alibaba-NLP/gte-Qwen2-1.5B-instruct',
);
my $vector  = $embedder->simple_embedding('Some text to embed');
my $vectors = $embedder->simple_embedding([ 'first', 'second' ]);

DESCRIPTION

Adapter for SGLang's OpenAI-compatible endpoint. SGLang is typically exposed as /v1/chat/completions with optional tool-calling support depending on model/backend setup.

Extends Langertha::Engine::OpenAIBase (which composes Langertha::Role::OpenAICompatible, Langertha::Role::OpenAPI, Langertha::Role::Models, Langertha::Role::Temperature, Langertha::Role::ResponseSize, Langertha::Role::SystemPrompt, Langertha::Role::ResponseFormat, Langertha::Role::Streaming, Langertha::Role::Chat, Langertha::Role::ReasoningEffort, and Langertha::Role::PromptCache); SGLang itself additionally composes Langertha::Role::Tools (MCP tool calling), Langertha::Role::Embedding (OpenAI-compatible /v1/embeddings) and Langertha::Role::Runtime::MetricsPoll (Prometheus /metrics scrape).

Supports chat, streaming, tool calling, embeddings, structured output, multimodal input, and Prometheus /metrics scraping. Transcription is not exposed on the OpenAI-compatible surface SGLang serves.

Only url is required. Use the full /v1 base URL. No API key is required for local setups.

See https://docs.sglang.ai/ for installation and configuration details.

REASONING

"reasoning_wire_format" in Langertha::Role::ReasoningEffort defaults to chat_template_kwargs ("to_chat_template_kwargs" in Langertha::Reasoning). What goes on the wire:

reasoning_effort none             ->  chat_template_kwargs => { enable_thinking => false }
reasoning_effort low .. max       ->  chat_template_kwargs => { enable_thinking => true },
                                      reasoning_effort     => $effort
thinking_enabled => 0             ->  same as reasoning_effort none
thinking_enabled => 1 (no effort) ->  chat_template_kwargs => { enable_thinking => true }
(unset)                           ->  nothing; the template's default applies

SGLang maps a reasoning_effort itself onto both the thinking and the enable_thinking template kwargs, and rejects effort strings it does not know; the effort is sent as given, not clamped to a per-model ladder. none sends no reasoning_effort (older SGLang builds reject it), only enable_thinking => false.

Only enable_thinking is sent. To turn thinking off on a template that reads thinking (DeepSeek-V3.x, Kimi), pass it per request; it is merged with the reasoning switch, the caller's keys winning:

my $response = await $sglang->chat_f(
    messages             => \@messages,
    reasoning_effort     => 'none',
    chat_template_kwargs => { thinking => JSON->false },
);

EMBEDDINGS

Composes Langertha::Role::Embedding. SGLang serves /v1/embeddings when launched with an embedding model (decoder-style models also need --is-embedding). The request carries embedding_model if you set it, else model if you set it, else no model field at all: the server embeds with the model it serves. A string returns one vector, an ArrayRef of strings one vector per input, in input order.

CAPABILITIES

Advertised flags (derived from composed roles via Langertha::Role::Capabilities):

A forced tool_choice (required or a named tool) together with a response_format of json_schema, json_object or structural_tag croaks before the request is sent: the SGLang server rejects that combination with HTTP 400. tool_choice auto with a response_format is sent.

SEE ALSO

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <getty@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.