Unicode Support

This file contains various utility functions for manipulating UTF8-encoded strings. For the uninitiated, this is a method of representing arbitrary Unicode characters as a variable number of bytes, in such a way that characters in the ASCII range are unmodified, and a zero byte never appears within non-zero characters.

Adds the UTF-8 representation of the Unicode codepoint uv to the end of the string d; d should be have at least UTF8_MAXBYTES+1 free bytes available. The return value is the pointer to the byte after the end of the new character. In other words,

d = uvuni_to_utf8_flags(d, uv, flags);

or, in most cases,

d = uvuni_to_utf8(d, uv);

(which is equivalent to)

d = uvuni_to_utf8_flags(d, uv, 0);

is the recommended Unicode-aware way of saying

*(d++) = uv;

Tests if some arbitrary number of bytes begins in a valid UTF-8 character. Note that an INVARIANT (i.e. ASCII on non-EBCDIC machines) character is a valid UTF-8 character. The actual number of bytes in the UTF-8 character will be returned if it is valid, otherwise 0.

Returns true if first len bytes of the given string form a valid UTF-8 string, false otherwise. Note that 'a valid UTF-8 string' does not mean 'a string that contains code points above 0x7F encoded in UTF-8' because a valid ASCII string is a valid UTF-8 string.

See also is_utf8_string_loclen() and is_utf8_string_loc().

Like is_utf8_string() but stores the location of the failure (in the case of "utf8ness failure") or the location s+len (in the case of "utf8ness success") in the ep.

See also is_utf8_string_loclen() and is_utf8_string().

Like is_utf8_string() but stores the location of the failure (in the case of "utf8ness failure") or the location s+len (in the case of "utf8ness success") in the ep, and the number of UTF-8 encoded characters in the el.

See also is_utf8_string_loc() and is_utf8_string().

Bottom level UTF-8 decode routine. Returns the Unicode code point value of the first character in the string s which is assumed to be in UTF-8 encoding and no longer than curlen; retlen will be set to the length, in bytes, of that character.

If s does not point to a well-formed UTF-8 character, the behaviour is dependent on the value of flags: if it contains UTF8_CHECK_ONLY, it is assumed that the caller will raise a warning, and this function will silently just set retlen to -1 and return zero. If the flags does not contain UTF8_CHECK_ONLY, warnings about malformations will be given, retlen will be set to the expected length of the UTF-8 character in bytes, and zero will be returned.

The flags can also contain various flags to allow deviations from the strict UTF-8 encoding (see utf8.h).

Most code should use utf8_to_uvchr() rather than call this directly.

Returns the native character value of the first character in the string s which is assumed to be in UTF-8 encoding; retlen will be set to the length, in bytes, of that character.

If s does not point to a well-formed UTF-8 character, zero is returned and retlen is set, if possible, to -1.

Returns the Unicode code point of the first character in the string s which is assumed to be in UTF-8 encoding; retlen will be set to the length, in bytes, of that character.

This function should only be used when the returned UV is considered an index into the Unicode semantic tables (e.g. swashes).

If s does not point to a well-formed UTF-8 character, zero is returned and retlen is set, if possible, to -1.

Return the length of the UTF-8 char encoded string s in characters. Stops at e (inclusive). If e < s or if the scan would end up past e, croaks.

Returns the number of UTF-8 characters between the UTF-8 pointers a and b.

WARNING: use only if you *know* that the pointers point inside the same UTF-8 buffer.

Return the UTF-8 pointer s displaced by off characters, either forward or backward.

WARNING: do not use the following unless you *know* off is within the UTF-8 data pointed to by s *and* that on entry s is aligned on the first byte of character or just after the last byte of a character.

Converts a string s of length len from UTF-8 into native byte encoding. Unlike bytes_to_utf8, this over-writes the original string, and updates len to contain the new length. Returns zero on failure, setting len to -1.

If you need a copy of the string, see bytes_from_utf8.

Converts a string s of length len from UTF-8 into native byte encoding. Unlike utf8_to_bytes but like bytes_to_utf8, returns a pointer to the newly-created string, and updates len to contain the new length. Returns the original string if no conversion occurs, len is unchanged. Do nothing if is_utf8 points to 0. Sets is_utf8 to 0 if s is converted or consisted entirely of characters that are invariant in utf8 (i.e., US-ASCII on non-EBCDIC machines).

Converts a string s of length len from the native encoding into UTF-8. Returns a pointer to the newly-created string, and sets len to reflect the new length.

A NUL character will be written after the end of the string.

If you want to convert to UTF-8 from encodings other than the native (Latin1 or EBCDIC), see sv_recode_to_utf8().

The "p" contains the pointer to the UTF-8 string encoding the character that is being converted.

The "ustrp" is a pointer to the character buffer to put the conversion result to. The "lenp" is a pointer to the length of the result.

The "swashp" is a pointer to the swash to use.

Both the special and normal mappings are stored lib/unicore/To/Foo.pl, and loaded by SWASHNEW, using lib/utf8_heavy.pl. The special (usually, but not always, a multicharacter mapping), is tried first.

The "special" is a string like "utf8::ToSpecLower", which means the hash %utf8::ToSpecLower. The access to the hash is through Perl_to_utf8_case().

The "normal" is a string like "ToLower" which means the swash %utf8::ToLower.

Convert the UTF-8 encoded character at p to its uppercase version and store that in UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least UTF8_MAXBYTES_CASE+1 bytes since the uppercase version may be longer than the original character.

The first character of the uppercased version is returned (but note, as explained above, that there may be more.)

Convert the UTF-8 encoded character at p to its titlecase version and store that in UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least UTF8_MAXBYTES_CASE+1 bytes since the titlecase version may be longer than the original character.

The first character of the titlecased version is returned (but note, as explained above, that there may be more.)

Convert the UTF-8 encoded character at p to its lowercase version and store that in UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least UTF8_MAXBYTES_CASE+1 bytes since the lowercase version may be longer than the original character.

The first character of the lowercased version is returned (but note, as explained above, that there may be more.)

Convert the UTF-8 encoded character at p to its foldcase version and store that in UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least UTF8_MAXBYTES_CASE+1 bytes since the foldcase version may be longer than the original character (up to three characters).

The first character of the foldcased version is returned (but note, as explained above, that there may be more.)

Adds the UTF-8 representation of the Native codepoint uv to the end of the string d; d should be have at least UTF8_MAXBYTES+1 free bytes available. The return value is the pointer to the byte after the end of the new character. In other words,

d = uvchr_to_utf8(d, uv);

is the recommended wide native character-aware way of saying

*(d++) = uv;

Returns the native character value of the first character in the string s which is assumed to be in UTF-8 encoding; retlen will be set to the length, in bytes, of that character.

Allows length and flags to be passed to low level routine.

Build to the scalar dsv a displayable version of the string spv, length len, the displayable version being at most pvlim bytes long (if longer, the rest is truncated and "..." will be appended).

The flags argument can have UNI_DISPLAY_ISPRINT set to display isPRINT()able characters as themselves, UNI_DISPLAY_BACKSLASH to display the \\[nrfta\\] as the backslashed versions (like '\n') (UNI_DISPLAY_BACKSLASH is preferred over UNI_DISPLAY_ISPRINT for \\). UNI_DISPLAY_QQ (and its alias UNI_DISPLAY_REGEX) have both UNI_DISPLAY_BACKSLASH and UNI_DISPLAY_ISPRINT turned on.

The pointer to the PV of the dsv is returned.

Build to the scalar dsv a displayable version of the scalar sv, the displayable version being at most pvlim bytes long (if longer, the rest is truncated and "..." will be appended).

The flags argument is as in pv_uni_display().

The pointer to the PV of the dsv is returned.

Return true if the strings s1 and s2 differ case-insensitively, false if not (if they are equal case-insensitively). If u1 is true, the string s1 is assumed to be in UTF-8-encoded Unicode. If u2 is true, the string s2 is assumed to be in UTF-8-encoded Unicode. If u1 or u2 are false, the respective string is assumed to be in native 8-bit encoding.

If the pe1 and pe2 are non-NULL, the scanning pointers will be copied in there (they will point at the beginning of the next character). If the pointers behind pe1 or pe2 are non-NULL, they are the end pointers beyond which scanning will not continue under any circumstances. If the byte lengths l1 and l2 are non-zero, s1+l1 and s2+l2 will be used as goal end pointers that will also stop the scan, and which qualify towards defining a successful match: all the scans that define an explicit length must reach their goal pointers for a match to succeed).

For case-insensitiveness, the "casefolding" of Unicode is used instead of upper/lowercasing both the characters, see http://www.unicode.org/unicode/reports/tr21/ (Case Mappings).

cpanm Env

perl -MCPAN -e shell
install Env