/ 
Locale-Simple-0.200
/ Locale::Simple::Scraper

NAME

Locale::Simple::Scraper - scraper to find translation tokens in a directory

VERSION

version 0.200

SYNOPSIS

use Locale::Simple::Scraper 'scrape';

# Parse @ARGV-style options; emits a .pot template to STDOUT.
scrape(
    '--ignore', 'node_modules',
    '--ignore', 'build',
    '--output', 'po',
);

DESCRIPTION

Locale::Simple::Scraper walks the current working directory, parses recognised source files with Parser::MGC, extracts every l()/ln()/lp()/lnp()/ld()/ldn()/ldp()/ldnp() call it can resolve statically, and writes a .pot (gettext template) or one of a few alternative output formats.

Usually invoked through locale-simple scrape; the scrape function is exported so build tools can drive it directly.

EXPORTED FUNCTIONS

scrape(@argv)

Run the scraper. @argv is parsed with Getopt::Long. Reads files from the current working directory recursively and prints to STDOUT. --help prints the options and returns. A bad call (unknown option, unexpected argument, unknown --output format) dies with a Locale::Simple::Scraper::BadCall object whose message field holds the reason, before anything is written.

usage()

Returns the text of the options, as locale-simple scrape --help prints it.

OPTIONS

--js=EXT, --pl=EXT, --py=EXT, --php=EXT, --go=EXT, --rs=EXT, --tx=EXT

Comma-separated list of extra file extensions to recognise as the given language. Repeatable; the lists add up. The built-in defaults (.js; .pl/.pm/.t; .py; .tx; .php; .go; .rs) are always included.

--ignore=REGEX

Skip any file whose path matches this regex. Repeatable.

--only=REGEX

Process only files whose path matches one of the given regexes. Repeatable; combines with --ignore.

--output=FORMAT

Output format — po (default), perl (Data::Dumper, keys sorted), json (through JSON::PP, keys sorted) or yaml (needs the optional YAML module).

--md5

Hash filenames with MD5 before writing them — useful when paths may contain sensitive information. Requires Digest::MD5.

--no_line_numbers

Omit #: source-location comments from the .pot.

SUPPORTED CALLS

Recognised call shapes (identical across Perl, Python and JavaScript):

l( msgid, ... )
ln( msgid, msgid_plural, n, ... )
lp( msgctxt, msgid, ... )
lnp( msgctxt, msgid, msgid_plural, n, ... )
ld( domain, msgid, ... )
ldn( domain, msgid, msgid_plural, n, ... )
ldp( domain, msgctxt, msgid, ... )
ldnp( domain, msgctxt, msgid, msgid_plural, n, ... )

Strings must be statically resolvable — literal strings and static concatenation ("a" . "b" in Perl/PHP, "a" + "b" in JS/Python, "a" ~ "b" in Xslate Kolon). PHP also reads heredoc and nowdoc literals; calls qualified with the PHP namespace are found as well.

Escapes are read the way the language reads them, so the msgid is the text the program sees: "Line\n" gives a msgid ending in a newline, which the .pot writes as \n. Single quotes in Perl, Xslate and PHP know only \\ and \'. A character named by its code point is stored as UTF-8. A literal with an escape the language rejects is skipped, and so is a Perl literal with a case change such as \U.

A literal that interpolates is not constant either. In Perl double quotes every $ is a variable ($name, $1, $::name, ${...}, $% as in "$%.2f", $ total after a space; a final $ does not compile), and so is an @ before a name, ::, ', $, +, -, a digit or {. PHP double quotes and heredocs interpolate $name, {$...} and ${...}. An escaped \$ or \@ is text. Python reads plain, raw (r"...") and u"..." strings in single or triple quotes, raw ones keeping their backslashes, and joins adjacent literals ("a" "b"); f-strings and bytes are not msgids. A JavaScript template literal is a msgid while it holds no ${...}. A call with a string that is not constant is skipped and reported on STDERR; the calls in its arguments are still extracted.

That holds for the msgid, the plural msgid, the context and the domain. The count and the arguments after them can be any code, such as $name, $obj->count, count($items), items.len() or n as i64. Such an argument ends at the comma or closing bracket outside any string literal or bracket pair, and the translation calls inside it are extracted as well.

Go exports its helpers capitalised: L, Ln, Lp, Lnp, Ld, Ldn, Ldp, Ldnp (the deferred markers keep their names). They are found with any package prefix or bare after a dot import; strings are "..." and backtick raw strings, never '...' (a rune in Go). The argument shapes are those above.

Rust calls are macros: the names carry a bang and the deferred markers are lower case (n_!, nn_!, np_!, nnp_!, nd_!, ndn_!, ndp_!, ndnp_!), a path before the macro is fine. Only the round brackets are read as the argument list, not {} or []. Strings are "..." and raw strings with any number of hashes; there is no concatenation operator and no '...' (chars and lifetimes). Comments are not skipped, as in every other language, so a commented-out call is extracted. Runtime interpolation is not extracted; keep msgids constant and pass dynamic data as sprintf arguments.

SEE ALSO

Locale::Simple, Locale::Simple::Scraper::Parser.

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/locale-simple/issues.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <getty@cpan.org>

COPYRIGHT AND LICENSE

This software is Copyright (c) 2026 by Torsten Raudssus https://raudssus.de/.

This is free software, licensed under:

The MIT (X11) License