/ 
App-karr-0.500
/ App::karr::Encoding

NAME

App::karr::Encoding - The character/octet boundary for karr

VERSION

version 0.500

SYNOPSIS

use App::karr::Encoding qw( decode_argv enable_std_utf8 yaml_dump );

enable_std_utf8();
decode_argv();

print yaml_dump( { title => "Fix \x{fc}nicode \x{2014} \x{e4}rger" } );

DESCRIPTION

karr holds one rule: everything inside the program is a Perl character string, and bytes exist only at the outer edges. This module is the only place that crosses that line, so every edge crosses it the same way.

The edges, and who guards them:

  • @ARGV — "decode_argv", called from bin/karr and bin/karr-foundation.

  • STDOUT/STDERR — "enable_std_utf8", likewise called from the two scripts. An in-process caller that captures output (a test, say) has to put the same layer on its capture handle, because reopening STDOUT drops the layer the script installed.

  • Git refs — "write_ref" in App::karr::Git and "read_ref" in App::karr::Git call "to_octets" and "from_octets". Blobs hold UTF-8 octets; everything above read_ref sees characters.

  • Files — Path::Tiny's slurp_utf8/spew_utf8, which are already character-level. Nothing extra is needed, and nothing extra may be added: an Encode::encode in front of a spew_utf8 is a double encode.

  • YAML — "yaml_dump" and "yaml_load". YAML::XS::Dump emits octets and YAML::XS::Load expects them, which is the opposite of the rule above, so those two functions are never called directly. (DumpFile and LoadFile are character-level and are used unwrapped.)

  • JSON — "json_encode" and "json_decode". The encode_json and decode_json functions are octet-level for the same reason and are likewise not used directly.

  • %ENV — "to_octets_for_env" and "from_octets_from_env". Perl's %ENV is a byte boundary: assigning a character string warns Wide character in setenv, and $ENV{NAME} reads back whatever bytes were stored. The crossing is named here so neither side is handled ad hoc at the call site.

Legacy boards

karr up to and including 0.402 mixed the two levels, and every board written by those versions has UTF-8 octets encoded a second time in its task frontmatter, its config, and its activity log. Task bodies are unaffected: they never passed through Dump. "repair_mojibake" undoes exactly that second encoding, and "board_encoding_version" in App::karr::Git decides when to apply it — see App::karr::Cmd::Repair for the migration.

SEE ALSO

karr, App::karr, App::karr::Git, App::karr::Task, App::karr::Cmd::Repair

BOARD_ENCODING_VERSION

The encoding contract version this code writes, stored per board in refs/karr/meta/encoding. A board without that ref predates the contract and is read through "repair_mojibake".

to_octets

my $bytes = to_octets($characters);

Encodes a character string to UTF-8 octets. undef passes through. This is the character-to-octet edge: the only place karr calls Encode::encode directly, so nothing outside this module ever needs to.

from_octets

my $characters = from_octets($bytes);

Decodes UTF-8 octets to a character string -- the octet-to-character edge, and the only place karr calls Encode::decode directly. A payload that is not valid UTF-8 is returned unchanged rather than being lossily substituted: karr ref blobs and command-line arguments are UTF-8 by contract, and passing a non-conforming payload through keeps the byte-in/byte-out behaviour karr had before this boundary existed, instead of quietly replacing bytes with U+FFFD.

to_octets_for_env

my $bytes = to_octets_for_env($characters);

The character-to-octet edge for %ENV. undef passes through. Assigning a character string to $ENV{NAME} makes Perl emit Wide character in setenv and emit bytes whose encoding depends on the IO layers in scope; encoding explicitly here makes the warning impossible to emit and the bytes the child process receives well-defined. Equivalent to "to_octets"; the dedicated name marks the call site as an ENV crossing, the way "decode_argv" marks argv.

from_octets_from_env

my $characters = from_octets_from_env($bytes);

The octet-to-character edge for %ENV. %ENV stores bytes; reading a non-ASCII value back through $ENV{NAME} gives octets, not characters. A payload that is not valid UTF-8 is returned unchanged, for the same reason "from_octets" does. Equivalent to "from_octets"; the dedicated name marks the call site as an ENV crossing.

decode_argv

decode_argv();

Decodes @ARGV in place from UTF-8. Arguments that are not valid UTF-8 are left as they arrived.

enable_std_utf8

enable_std_utf8();

Puts a :encoding(UTF-8) layer on STDOUT and STDERR so command bodies can print character strings.

STDIN is deliberately left alone. Only App::karr::Cmd::Restore reads it, and it decodes its own payload, so a layer here would decode it twice.

yaml_dump

my $characters = yaml_dump($data);

YAML::XS::Dump at the character level, and the only place karr calls it directly: Dump itself emits octets, which is why the result is passed through "from_octets" before any caller sees it.

yaml_load

my $data = yaml_load($characters);

YAML::XS::Load at the character level, and the only place karr calls it directly: Load expects octets, which is why the character string is turned to them with "to_octets" first.

json_encode

my $characters = json_encode($data);

encode_json at the character level, and the only place karr calls it directly, with canonical key ordering so the --json payload an agent parses is byte-stable across runs.

json_decode

my $data = json_decode($characters);

decode_json at the character level, and the only place karr calls it directly.

repair_mojibake

my $fixed = repair_mojibake($data);

Undoes one round of UTF-8 double encoding, walking hashes and arrays and returning a fresh structure. Blessed references and hash keys are passed through untouched.

A string is only rewritten when it cannot be anything but double-encoded:

  • pure ASCII is never touched, so an ASCII board is bit-identical afterwards and the repair is safe to run over a whole board;

  • a string containing a codepoint above U+00FF cannot be a byte string misread as characters, so it is already correct and is left alone;

  • what remains must additionally form valid UTF-8 when read back as bytes. Ordinary Latin-1 text almost never does — "\x{fc}ber" is fc 62, which is not valid UTF-8 — whereas "\x{c3}\x{bc}ber" is c3 bc 65 72, which decodes to "\x{fc}ber".

The residual ambiguity is a string whose non-ASCII characters happen to spell valid UTF-8, such as a literal "\x{c3}\x{a9}". That is why the repair is bounded to boards that predate refs/karr/meta/encoding instead of running forever.

SUPPORT

Issues

Please report bugs and feature requests on GitHub at https://github.com/Getty/karr/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <getty@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.