/ 
Langertha-0.505
/ Langertha::Engine::LlamaCpp

NAME

Langertha::Engine::LlamaCpp - llama.cpp server

VERSION

version 0.505

SYNOPSIS

use Langertha::Engine::LlamaCpp;

my $llama = Langertha::Engine::LlamaCpp->new(
    url           => 'http://localhost:8080/v1',
    system_prompt => 'You are a helpful assistant',
);

print $llama->simple_chat('Hello!');

my $embedding = $llama->simple_embedding('Some text');

DESCRIPTION

Provides access to llama.cpp's built-in HTTP server, which exposes an OpenAI-compatible API. Composes Langertha::Role::OpenAICompatible.

Only url is required. The URL must include the /v1 path prefix (e.g., http://localhost:8080/v1). Since llama.cpp serves exactly one model (loaded at server startup), no model name or API key is needed.

Supports chat, streaming, embeddings, and MCP tool calling.

"embedding_dimensions" in Langertha::Role::Embedding is not sent (it carps once): llama.cpp's server does not parse a dimensions field, drops it silently and returns the full vector.

See https://github.com/ggml-org/llama.cpp/blob/master/examples/server/README.md for server setup.

REASONING

"reasoning_wire_format" in Langertha::Role::ReasoningEffort defaults to chat_template_kwargs ("to_chat_template_kwargs" in Langertha::Reasoning). What goes on the wire:

reasoning_effort none             ->  chat_template_kwargs => { enable_thinking => false }
reasoning_effort low .. max       ->  chat_template_kwargs => { enable_thinking => true },
                                      reasoning_effort     => $effort
thinking_enabled => 0             ->  same as reasoning_effort none
thinking_enabled => 1 (no effort) ->  chat_template_kwargs => { enable_thinking => true }
(unset)                           ->  nothing; the template's default applies

my $llama = Langertha::Engine::LlamaCpp->new(
    url              => 'http://localhost:8080/v1',
    thinking_enabled => 0,          # Qwen3 & co: no thinking
);

The server must run with --jinja for the template to see the kwargs. llama.cpp honors chat_template_kwargs since about build b5780 and a top-level reasoning_effort since about b10121/b10434 (those build numbers are not validated); older builds ignore the field, and a template that does not read enable_thinking ignores the switch. The effort is sent as given, not clamped to a per-model ladder; none sends no reasoning_effort.

Only enable_thinking is sent. Templates that read thinking instead (DeepSeek-V3.x, Kimi) need it passed per request, which is merged with the reasoning switch, the caller's keys winning:

my $response = await $llama->chat_f(
    messages             => \@messages,
    reasoning_effort     => 'none',
    chat_template_kwargs => { thinking => JSON->false },
);

THIS API IS WORK IN PROGRESS

SEE ALSO

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <getty@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.