NAME

perllocale - Perl locale handling (internationlization)

DESCRIPTION

Perl supports language-specific notions of data such as "is this a letter", "what is the upper-case equivalent of this letter", and "which of these letters comes first". These are important issues, especially for languages other than English - but also for English: it would be very naÔve to think that A-Za-z defines all the "letters". Perl is also aware that some character other than '.' may be preferred as a decimal point, and that output date representations may be language-specific.

Perl can understand language-specific data via the standardized (ISO C, XPG4, POSIX 1.c) method called "the locale system". The locale system is controlled per application using a pragma, one function call, and several environment variables.

NOTE: This feature is new in Perl 5.004, and does not apply unless an application specifically requests it - see "Backward compatibility".

PREPARING TO USE LOCALES

If Perl applications are to be able to understand and present your data correctly according a locale of your choice, all of the following must be true:

• Your operating system must support the locale system. If it does, you should find that the setlocale function is a documented part of its C library.

• Definitions for the locales which you use must be installed. You, or your system administrator, must make sure that this is the case. The available locales, the location in which they are kept, and the manner in which they are installed, vary from system to system. Some systems provide only a few, hard-wired, locales, and do not allow more to be added; others allow you to add "canned" locales provided by the system supplier; still others allow you or the system administrator to define and add arbitrary locales. (You may have to ask your supplier to provide canned locales whch are not delivered with your operating system.) Read your system documentation for further illumination.

• Perl must believe that the locale system is supported. If it does, perl -V:d_setlocale will say that the value for d_setlocale is define.

If you want a Perl application to process and present your data according to a particular locale, the application code should include the use locale pragma ("The use locale Pragma") where appropriate, and at least one of the following must be true:

• The locale-determining environment variables (see ENVIRONMENT) must be correctly set up, either by yourself, or by the person who set up your system account, at the time the application is started.

• The application must set its own locale using the method described in "The setlocale function".

USING LOCALES

The use locale pragma

By default, Perl ignores the current locale. The use locale pragma tells Perl to use the current locale for some operations:

• The comparison operators (lt, le, cmp, ge, and gt) use LC_COLLATE. The sort function is also affected if it is used without an explicit comparison function because it uses cmp by default.

  Note: The eq and ne operators are unaffected by the locale: they always perform a byte-by-byte comparison of their scalar arguments. If you really want to know if two strings - which eq may consider different - are equal as far as collation is concerned, use something like

  !("space and case ignored" cmp "SpaceAndCaseIgnored")

  (which would be true if the collation locale specified a dictionary-like ordering).

  Editor's note: I am right about eq and ne, aren't I?

• Regular expressions and case-modification functions (uc, lc, ucfirst, and lcfirst) use LC_CTYPE

• The formatting functions (printf and sprintf) use LC_NUMERIC

• The POSIX date formatting function (strftime) uses LC_TIME.

LC_COLLATE, LC_CTYPE, and so on, are discussed further in "LOCALE CATEGORIES".

The default behaviour returns with no locale or on reaching the end of the enclosing block.

Note that the result of any operation that uses locale information is tainted (see perlsec.pod), since locales can be created by unprivileged users on some systems.

The setlocale function

You can switch locales as often as you wish at runtime with the POSIX::setlocale function:

# This functionality not usable prior to Perl 5.004
require 5.004;

# Import locale-handling tool set from POSIX module.
# This example uses: setlocale -- the function call
#                    LC_CTYPE -- explained below
use POSIX qw(locale_h);

# query and save the old locale.
$old_locale = setlocale(LC_CTYPE);

setlocale(LC_CTYPE, "fr_CA.ISO8859-1");
# LC_CTYPE now in locale "French, Canada, codeset ISO 8859-1"

setlocale(LC_CTYPE, "");
# LC_CTYPE now reset to default defined by LC_ALL/LC_CTYPE/LANG
# environment variables.  See below for documentation.

# restore the old locale
setlocale(LC_CTYPE, $old_locale);

The first argument of setlocale gives the category, the second the locale. The category tells in what aspect of data processing you want to apply locale-specific rules. Category names are discussed in "LOCALE CATEGORIES" and ENVIRONMENT. The locale is the name of a collection of customization information corresponding to a paricular combination of language, country or territory, and codeset. Read on for hints on the naming of locales: not all systems name locales as in the example.

If no second argument is provided, the function returns a string naming the current locale for the category. You can use this value as the second argument in a subsequent call to setlocale. If a second argument is given and it corresponds to a valid locale, the locale for the category is set to that value, and the function returns the now-current locale value. You can use this in a subsequent call to setlocale. (In some implementations, the return value may sometimes differ from the value you gave as the second argument - think of it as an alias for the value that you gave.)

As the example shows, if the second argument is an empty string, the category's locale is returned to the default specified by the corresponding environment variables. Generally, this results in a return to the default which was in force when Perl started up: changes to the environment made by the application after start-up may or may not be noticed, depending on the implementation of your system's C library.

If the second argument does not correspond to a valid locale, the locale for the category is not changed, and the function returns undef.

For further information about the categories, consult setlocale(3). For the locales available in your system, also consult setlocale(3) and see whether it leads you to the list of the available locales (search for the SEE ALSO section). If that fails, try the following command lines:

locale -a

nlsinfo

ls /usr/lib/nls/loc

ls /usr/lib/locale

ls /usr/lib/nls

and see whether they list something resembling these

en_US.ISO8859-1         de_DE.ISO8859-1         ru_RU.ISO8859-5
en_US                   de_DE                   ru_RU
en                      de                      ru
english                 german                  russian
english.iso88591        german.iso88591         russian.iso88595

Sadly, even though the calling interface for setlocale has been standardized, the names of the locales have not. The form of the name is usually language_country/territory.codeset, but the latter parts are not always present.

Two special locales are worth particular mention: "C" and "POSIX". Currently these are effectively the same locale: the difference is mainly that the first one is defined by the C standard and the second by the POSIX standard. What they define is the default locale in which every program starts in the absence of locale information in its environment. (The default default locale, if you will.) Its language is (American) English and its character codeset ASCII.

NOTE: Not all systems have the "POSIX" locale (not all systems are POSIX-conformant), so use "C" when you need explicitly to specify this default locale.

The localeconv function

The POSIX::localeconv function allows you to get particulars of the locale-dependent numeric formatting information specified by the current LC_NUMERIC and LC_MONETARY locales. (If you just want the name of the current locale for a particular category, use POSIX::setlocale with a single parameter - see "The setlocale function".)

use POSIX qw(locale_h);
use locale;

# Get a reference to a hash of locale-dependent info
$locale_values = localeconv();

# Output sorted list of the values
for (sort keys %$locale_values) {
        printf "%-20s = %s\n", $_, $locale_values->{$_}
}

localeconv takes no arguments, and returns a reference to a hash. The keys of this hash are formatting variable names such as decimal_point and thousands_sep; the values are the corresponding values. See "localeconv" in POSIX (3) for a longer example, which lists all the categories an implementation might be expected to provide; some provide more and others fewer, however.

Editor's note: I can't work out whether POSIX::localeconv correctly obeys use locale and no locale. In my opinion, it should, if only to be consistent with other locale stuff - although it's hardly a show-stopper if it doesn't. Could someone check, please?

Here's a simple-minded example program which rewrites its command line parameters as integers formatted correctly in the current locale:

# See comments in previous example
require 5.004;
use POSIX qw(locale_h);
use locale;

# Get some of locale's numeric formatting parameters
my ($thousands_sep, $grouping) =
    @{localeconv()}{'thousands_sep', 'grouping'};

# Apply defaults if values are missing
$thousands_sep = ',' unless $thousands_sep;
$grouping = 3 unless $grouping;

# Format command line params for current locale
for (@ARGV)
{
    $_ = int; # Chop non-integer part
    1 while
        s/(\d)(\d{$grouping}($|$thousands_sep))/$1$thousands_sep$2/;
    print "$_ ";
}
print "\n";

Editor's note: Like all the examples, this needs testing on systems which, unlike mine, have non-toy implementations of locale handling.

LOCALE CATEGORIES

The subsections which follow descibe basic locale categories. As well as these, there are some combination categories which allow the manipulation of of more than one basic category at a time. See "ENVIRONMENT VARIABLES" for a discussion of these.

Category LC_COLLATE: Collation

When in the scope of use locale, Perl looks to the LC_COLLATE environment variable to determine the application's notions on the collation (ordering) of characters. ('B' follows 'A' in Latin alphabets, but where do '¡' and 'Ÿ' belong?)

Here is a code snippet that will tell you what are the alphanumeric characters in the current locale, in the locale order:

use locale;
print +(sort grep /\w/, map { chr() } 0..255), "\n";

Editor's note: The original example had setlocale(LC_COLLATE, "") prior to print .... I think this is wrong: as soon as you utter use locale, the default behaviour of sort (well, cmp, really) becomes locale-aware. The locale it's aware of is the current locale which, unless you've changed it yourself, is the default locale defined by your environment.

Compare this with the characters that you see and their order if you state explicitly that the locale should be ignored:

no locale;
print +(sort grep /\w/, map { chr() } 0..255), "\n";

This machine-native collation (which is what you get unless use locale has appeared earlier in the same block) must be used for sorting raw binary data, whereas the locale-dependent collation of the first example is useful for written text.

NOTE: In some locales some characters may have no collation value at all - for example, if '-' is such a character, 'relocate' and 're-locate' may be considered to be equal to each other, and so sort to the same position.

Category LC_CTYPE: Character Types

When in the scope of use locale, Perl obeys the LC_CTYPE locale setting. This controls the application's notion of which characters are alphabetic. This affects Perl's \w regular expression metanotation, which stands for alphanumeric characters - that is, alphabetic and numeric characters. (Consult perlre for more information about regular expressions.) Thanks to LC_CTYPE, depending on your locale setting, characters like '�', 'Š', 'þ', and '¯' may be understood as \w characters.

LC_CTYPE also affects the POSIX character-class test functions - isalpha, islower and so on. For example, if you move from the "C" locale to a 7-bit Scandinavian one, you may find - possibly to your surprise -that "|" moves from the ispunct class to isalpha.

Editor's note: I can't work out whether the POSIX::is... stuff correctly obeys use locale and no locale. In my opinion, they should. Could someone check, please?

Note: A broken or malicious LC_CTYPE locale definition may result in clearly ineligible characters being considered to be alphanumeric by your application. For strict matching of (unaccented) letters and digits - for example, in command strings - locale-aware applications should use \w inside a no locale block.

Category LC_NUMERIC: Numeric Formatting

When in the scope of use locale, Perl obeys the LC_NUMERIC locale information which controls application's idea of how numbers should be formatted for human readability by the printf, fprintf, and write functions. String to numeric conversion by the POSIX::strtod function is also affected. In most impementations the only effect is to change the character used for the decimal point - perhaps from '.' to ',': these functions aren't aware of such niceties as thousands separation and so on. (See "The localeconv function" if you care about these things.)

Editor's note: I can't work out whether POSIX::strtod correctly obeys use locale and no locale. In my opinion, it should - although it's hardly a show-stopper if it doesn't. Could someone check, please?

Note that output produced by print is never affected by the current locale: it is independent of whether use locale or no locale is in effect, and corresponds to what you'd get from printf in the "C" locale. The same is true for Perl's internal conversions between numeric and string formats:

use POSIX qw(strtod);
use locale;
$n = 5/2;   # Assign numeric 2.5 to $n

$a = " $n"; # Locale-independent conversion to string

print "half five is $n\n";       # Locale-independent output

printf "half five is %g\n", $n;  # Locale-dependent output

print "DECIMAL POINT IS COMMA\n" # Locale-dependent conversion
    if $n == (strtod("2,5"))[0];

Category LC_MONETARY: Formatting of monetary amounts

The C standard defines the LC_MONETARY category, but no function that is affected by its contents. (Those with experience of standards committees will recognise that the working group decided to punt on the issue.) Consequently, Perl takes no notice of it. If you really want to use LC_MONETARY, you can query its contents - see "The localeconv function" - and use the information that it returns in your application's own formating of currency amounts. However, you may well find that the information, though voluminous and complex, does not quite meet your requirements: currency formatting is a hard nut to crack.

LC_TIME

The output produced by POSIX::strftime, which builds a formatted human-readable date/time string, is affected by the current LC_TIME locale. Thus, in a French locale, the output produced by the %B format element (full month name) for the first month of the year would be "janvier". Here's how to get a list of the long month names in the current locale:

use POSIX qw(strftime);
use locale;
for (0..11)
{
    $long_month_name[$_] = strftime("%B", 0, 0, 0, 1, $_, 96);
}

Editor's note: Unchecked in "alien" locales: my system can't do French...

Other categories

The remaining locale category, LC_MESSAGES (possibly supplemented by others in particular implementations) is not currently used by Perl - except possibly to affect the behaviour of library functions called by extensions which are not part of the standard Perl distribution.

ENVIRONMENT

PERL_BADLANG

A string that controls whether Perl warns in its startup about failed locale settings. This can happen if the locale support in the operating system is lacking (broken) is some way. If this string has an integer value differing from zero, Perl will not complain.

NOTE: This is just hiding the warning message. The message tells about some problem in your system's locale support and you should investigate what the problem is.

The following environment variables are not specific to Perl: They are part of the standardized (ISO C, XPG4, POSIX 1.c) setlocale method to control an application's opinion on data.

LC_ALL

LC_ALL is the "override-all" locale environment variable. If it is set, it overrides all the rest of the locale environment variables.

LC_CTYPE

In the absence of LC_ALL, LC_CTYPE chooses the character type locale. In the absence of both LC_ALL and LC_CTYPE, LANG chooses the character type locale.

LC_COLLATE

In the absence of LC_ALL, LC_COLLATE chooses the collation (sorting) locale. In the absence of both LC_ALL and LC_COLLATE, LANG chooses the collation locale.

LC_MONETARY

In the absence of LC_ALL, LC_MONETARY chooses the montary formatting locale. In the absence of both LC_ALL and LC_MONETARY, LANG chooses the monetary formatting locale.

LC_NUMERIC

In the absence of LC_ALL, LC_NUMERIC chooses the numeric format locale. In the absence of both LC_ALL and LC_NUMERIC, LANG chooses the numeric format.

LC_TIME

In the absence of LC_ALL, LC_TIME chooses the date and time formatting locale. In the absence of both LC_ALL and LC_TIME, LANG chooses the date and time formatting locale.

LANG

LANG is the "catch-all" locale environment variable. If it is set, it is used as the last resort after the overall LC_ALL and the category-specific LC_....

NOTES

Backward compatibility

Versions of Perl prior to 5.004 ignored locale information, generally behaving as if something similar to the "C" locale (see "The setlocale function") was always in force, even if the program environment suggested otherwise. By default, Perl still behaves this way so as to maintain backward compatibility. If you want a Perl application to pay attention to locale information, you must use the use locale pragma (see "The use locale Pragma") to instruct it to do so.

Sort speed

Comparing and sorting by locale is usually slower than the default sorting; factors of 2 to 4 have been observed. It will also consume more memory: while a Perl scalar variable is participating in any string comparison or sorting operation and obeying the locale collation rules it will take about 3-15 (the exact value depends on the operating system and the locale) times more memory than normally. These downsides are dictated more by the operating system implementation of the locale system than by Perl.

I18N:Collate

In Perl 5.003 (and later development releases prior to 5.003_06), per-locale collation was possible using the I18N::Collate library module. This is now mildly obsolete and should be avoided in new applications. The LC_COLLATE functionality is integrated into the Perl core language and one can use locale-specific scalar data completely normally - there is no need to juggle with the scalar references of I18N::Collate.

An imperfect standard

Internationalization, as defined in the C and POSIX standards, can be criticized as incomplete, ungainly, and having too large a granularity. (Locales apply to a whole process, when it would arguably be more useful to have them apply to a single thread, window group, or whatever.) They also have a tendency, like standards groups, to divide the world into nations, when we all know that the world can equally well be divided into bankers, bikers, gamers, and so on. But, for now, it's the only standard we've got. This may be construed as a bug.

Freely available locale definitions

There is a large collection of locale definitions at ftp://dkuug.dk/i18n/WG15-collection. You should be aware that they are unsupported, and are not claimed to be fit for any purpose. If your system allows the installation of arbitrary locales, you may find them useful as they are, or as a basis for the development of your own locales.

i18n and l10n

Internationalization is often abbreviated as i18n because its first and last letters are separated by eighteen others. You can also talk of localization (l10n), the process of tailoring an internationalizated application for use in a particular locale.

BUGS

Broken systems

In certain system environments the operating system's locale support is broken and cannot be fixed or used by Perl. Such deficiencies can and will result in mysterious hangs and/or Perl core dumps. One example is IRIX before release 6.2, in which the LC_COLLATE support simply does not work. When confronted with such a system, please report in excruciating detail to perlbug@perl.com, and complain to your vendor: maybe some bug fixes exist for these problems in your operating system. Sometimes such bug fixes are called an operating system upgrade.

Rendering of this documentation

This manual page contains non-ASCII characters, which should all be rendered as accented letters, and which should make some sort of sense in context. If this is not the case, your system is probably not using the ISO 8859-1 character set which was used to write them, and/or your formatting, display, and printing software are not correctly mapping them to your host's character set. If this annoys you, and if you can convince yourself that it is due to a bug in one of Perl's various pod2... utilities, by all means report it as a Perl bug. Otherwise, pausing only to curse anyone who ever invented yet another character set, see if you can make it handle ISO 8859-1 sensibly.

SEE ALSO

"isalnum" in POSIX (3), "isalpha" in POSIX (3), "isdigit" in POSIX (3), "isgraph" in POSIX (3), "islower" in POSIX (3), "isprint" in POSIX (3), "ispunct" in POSIX (3), "isspace" in POSIX (3), "isupper" in POSIX (3), "isxdigit" in POSIX (3), "localeconv" in POSIX (3), "setlocale" in POSIX (3), "strtod" in POSIX (3)

Editor's note: That looks horrible after going through pod2man. But I do want to call out all thse sectins by name. What should I have done?

HISTORY

Perl 5.003's perli18n.pod heavily hacked by Dominic Dunlop.

Last update: Mon Dec 16 14:13:10 WET 1996