NAME

Char::Eutf2 - Run-time routines for Char/UTF2.pm

SYNOPSIS

use Char::Eutf2;

  Char::Eutf2::split(...);
  Char::Eutf2::tr(...);
  Char::Eutf2::chop(...);
  Char::Eutf2::index(...);
  Char::Eutf2::rindex(...);
  Char::Eutf2::lc(...);
  Char::Eutf2::lc_;
  Char::Eutf2::lcfirst(...);
  Char::Eutf2::lcfirst_;
  Char::Eutf2::uc(...);
  Char::Eutf2::uc_;
  Char::Eutf2::ucfirst(...);
  Char::Eutf2::ucfirst_;
  Char::Eutf2::fc(...);
  Char::Eutf2::fc_;
  Char::Eutf2::ignorecase(...);
  Char::Eutf2::capture(...);
  Char::Eutf2::chr(...);
  Char::Eutf2::chr_;
  Char::Eutf2::glob(...);
  Char::Eutf2::glob_;

# "no Char::Eutf2;" not supported

ABSTRACT

This module is a run-time routines of the Char/UTF2.pm. Because the Char/UTF2.pm automatically uses this module, you need not use directly.

BUGS AND LIMITATIONS

I have tested and verified this software using the best of my ability. However, a software containing much regular expression is bound to contain some bugs. Thus, if you happen to find a bug that's in Char::UTF2 software and not your own program, you can try to reduce it to a minimal test case and then report it to the following author's address. If you have an idea that could make this a more useful tool, please let everyone share it.

HISTORY

This Char::Eutf2 module first appeared in ActivePerl Build 522 Built under MSWin32 Compiled at Nov 2 1999 09:52:28

AUTHOR

INABA Hitoshi <ina@cpan.org>

This project was originated by INABA Hitoshi. For any questions, use <ina@cpan.org> so we can share this file.

LICENSE AND COPYRIGHT

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

EXAMPLES

Split string

@split = Char::Eutf2::split(/pattern/,$string,$limit);
@split = Char::Eutf2::split(/pattern/,$string);
@split = Char::Eutf2::split(/pattern/);
@split = Char::Eutf2::split('',$string,$limit);
@split = Char::Eutf2::split('',$string);
@split = Char::Eutf2::split('');
@split = Char::Eutf2::split();
@split = Char::Eutf2::split;

This subroutine scans a string given by $string for separators, and splits the
string into a list of substring, returning the resulting list value in list
context or the count of substring in scalar context. Scalar context also causes
split to write its result to @_, but this usage is deprecated. The separators
are determined by repeated pattern matching, using the regular expression given
in /pattern/, so the separators may be of any size and need not be the same
string on every match. (The separators are not ordinarily returned; exceptions
are discussed later in this section.) If the /pattern/ doesn't match the string
at all, Char::Eutf2::split returns the original string as a single substring, If it
matches once, you get two substrings, and so on. You may supply regular
expression modifiers to the /pattern/, like /pattern/i, /pattern/x, etc. The
//m modifier is assumed when you split on the pattern /^/.

If $limit is specified and positive, the subroutine splits into no more than that
many fields (though it may split into fewer if it runs out of separators). If
$limit is negative, it is treated as if an arbitrarily large $limit has been
specified If $limit is omitted or zero, trailing null fields are stripped from
the result (which potential users of pop would do wel to remember). If $string
is omitted, the subroutine splits the $_ string. If /pattern/ is also omitted or
is the literal space, " ", the subroutine split on whitespace, /\s+/, after
skipping any leading whitespace.

A /pattern/ of /^/ is secretly treated if it it were /^/m, since it isn't much
use otherwise.

String of any length can be split:

@chars  = Char::Eutf2::split(//,  $word);
@fields = Char::Eutf2::split(/:/, $line);
@words  = Char::Eutf2::split(" ", $paragraph);
@lines  = Char::Eutf2::split(/^/, $buffer);

A pattern capable of matching either the null string or something longer than
the null string (for instance, a pattern consisting of any single character
modified by a * or ?) will split the value of $string into separate characters
wherever it matches the null string between characters; nonnull matches will
skip over the matched separator characters in the usual fashion. (In other words,
a pattern won't match in one spot more than once, even if it matched with a zero
width.) For example:

print join(":" => Char::Eutf2::split(/ */, "hi there"));

produces the output "h:i:t:h:e:r:e". The space disappers because it matches
as part of the separator. As a trivial case, the null pattern // simply splits
into separate characters, and spaces do not disappear. (For normal pattern
matches, a // pattern would repeat the last successfully matched pattern, but
Char::Eutf2::split's pattern is exempt from that wrinkle.)

The $limit parameter splits only part of a string:

my ($login, $passwd, $remainder) = Char::Eutf2::split(/:/, $_, 3);

We encourage you to split to lists of names like this to make your code
self-documenting. (For purposes of error checking, note that $remainder would
be undefined if there were fewer than three fields.) When assigning to a list,
if $limit is omitted, Perl supplies a $limit one larger than the number of
variables in the list, to avoid unneccessary work. For the split above, $limit
would have been 4 by default, and $remainder would have received only the third
field, not all the rest of the fields. In time-critical applications, it behooves
you not to split into more fields than you really need. (The trouble with
powerful languages it that they let you be powerfully stupid at times.)

We said earlier that the separators are not returned, but if the /pattern/
contains parentheses, then the substring matched by each pair of parentheses is
included in the resulting list, interspersed with the fields that are ordinarily
returned. Here's a simple example:

Char::Eutf2::split(/([-,])/, "1-10,20");

which produces the list value:

(1, "-", 10, ",", 20)

With more parentheses, a field is returned for each pair, even if some pairs
don't match, in which case undefined values are returned in those positions. So
if you say:

Char::Eutf2::split(/(-)|(,)/, "1-10,20");

you get the value:

(1, "-", undef, 10, undef, ",", 20)

The /pattern/ argument may be replaced with an expression to specify patterns
that vary at runtime. As with ordinary patterns, to do run-time compilation only
once, use /$variable/o.

As a special case, if the expression is a single space (" "), the subroutine
splits on whitespace just as Char::Eutf2::split with no arguments does. Thus,
Char::Eutf2::split(" ") can be used to emulate awk's default behavior. In contrast,
Char::Eutf2::split(/ /) will give you as many null initial fields as there are
leading spaces. (Other than this special case, if you supply a string instead
of a regular expression, it'll be interpreted as a regular expression anyway.)
You can use this property to remove leading and trailing whitespace from a
string and to collapse intervaning stretches of whitespace into a single
space:

$string = join(" ", Char::Eutf2::split(" ", $string));

The following example splits an RFC822 message header into a hash containing
$head{'Date'}, $head{'Subject'}, and so on. It uses the trick of assigning a
list of pairs to a hash, because separators altinate with separated fields, It
users parentheses to return part of each separator as part of the returned list
value. Since the split pattern is guaranteed to return things in pairs by virtue
of containing one set of parentheses, the hash assignment is guaranteed to
receive a list consisting of key/value pairs, where each key is the name of a
header field. (Unfortunately, this technique loses information for multiple lines
with the same key field, such as Received-By lines. Ah well)

$header =~ s/\n\s+/ /g; # Merge continuation lines.
%head = ("FRONTSTUFF", Char::Eutf2::split(/^(\S*?):\s*/m, $header));

The following example processes the entries in a Unix passwd(5) file. You could
leave out the chomp, in which case $shell would have a newline on the end of it.

open(PASSWD, "/etc/passwd");
while (<PASSWD>) {
    chomp; # remove trailing newline.
    ($login, $passwd, $uid, $gid, $gcos, $home, $shell) =
        Char::Eutf2::split(/:/);
    ...
}

Here's how process each word of each line of each file of input to create a
word-frequency hash.

while (<>) {
    for my $word (Char::Eutf2::split()) {
        $count{$word}++;
    }
}

The inverse of Char::Eutf2::split is join, except that join can only join with the
same separator between all fields. To break apart a string with fixed-position
fields, use unpack.

Processing long $string (over 32766 octets) requires Perl 5.010001 or later.

Transliteration

$tr = Char::Eutf2::tr($variable,$bind_operator,$searchlist,$replacementlist,$modifier);
$tr = Char::Eutf2::tr($variable,$bind_operator,$searchlist,$replacementlist);

This is the transliteration (sometimes erroneously called translation) operator,
which is like the y/// operator in the Unix sed program, only better, in
everybody's humble opinion.

This subroutine scans a UTF-8 string character by character and replaces all
occurrences of the characters found in $searchlist with the corresponding character
in $replacementlist. It returns the number of characters replaced or deleted.
If no UTF-8 string is specified via =~ operator, the $_ variable is translated.
$modifier are:

---------------------------------------------------------------------------
Modifier   Meaning
---------------------------------------------------------------------------
c          Complement $searchlist.
d          Delete found but unreplaced characters.
s          Squash duplicate replaced characters.
r          Return transliteration and leave the original string untouched.
---------------------------------------------------------------------------

To use with a read-only value without raising an exception, use the /r modifier.

print Char::Eutf2::tr('bookkeeper','=~','boep','peob','r'); # prints 'peekkoobor'

Chop string

$chop = Char::Eutf2::chop(@list);
$chop = Char::Eutf2::chop();
$chop = Char::Eutf2::chop;

This subroutine chops off the last character of a string variable and returns the
character chopped. The Char::Eutf2::chop subroutine is used primary to remove the newline
from the end of an input recoed, and it is more efficient than using a
substitution. If that's all you're doing, then it would be safer to use chomp,
since Char::Eutf2::chop always shortens the string no matter what's there, and chomp
is more selective. If no argument is given, the subroutine chops the $_ variable.

You cannot Char::Eutf2::chop a literal, only a variable. If you Char::Eutf2::chop a list of
variables, each string in the list is chopped:

@lines = `cat myfile`;
Char::Eutf2::chop(@lines);

You can Char::Eutf2::chop anything that is an lvalue, including an assignment:

Char::Eutf2::chop($cwd = `pwd`);
Char::Eutf2::chop($answer = <STDIN>);

This is different from:

$answer = Char::Eutf2::chop($tmp = <STDIN>); # WRONG

which puts a newline into $answer because Char::Eutf2::chop returns the character
chopped, not the remaining string (which is in $tmp). One way to get the result
intended here is with substr:

$answer = substr <STDIN>, 0, -1;

But this is more commonly written as:

Char::Eutf2::chop($answer = <STDIN>);

In the most general case, Char::Eutf2::chop can be expressed using substr:

$last_code = Char::Eutf2::chop($var);
$last_code = substr($var, -1, 1, ""); # same thing

Once you understand this equivalence, you can use it to do bigger chops. To
Char::Eutf2::chop more than one character, use substr as an lvalue, assigning a null
string. The following removes the last five characters of $caravan:

substr($caravan, -5) = '';

The negative subscript causes substr to count from the end of the string instead
of the beginning. To save the removed characters, you could use the four-argument
form of substr, creating something of a quintuple Char::Eutf2::chop;

$tail = substr($caravan, -5, 5, '');

This is all dangerous business dealing with characters instead of graphemes. Perl
doesn't really have a grapheme mode, so you have to deal with them yourself.

Index string

$byte_pos = Char::Eutf2::index($string,$substr,$byte_offset);
$byte_pos = Char::Eutf2::index($string,$substr);

This subroutine searches for one string within another. It returns the byte position
of the first occurrence of $substring in $string. The $byte_offset, if specified,
says how many bytes from the start to skip before beginning to look. Positions are
based at 0. If the substring is not found, the subroutine returns one less than the
base, ordinarily -1. To work your way through a string, you might say:

$byte_pos = -1;
while (($byte_pos = Char::Eutf2::index($string, $lookfor, $byte_pos)) > -1) {
    print "Found at $byte_pos\n";
    $byte_pos++;
}

Reverse index string

$byte_pos = Char::Eutf2::rindex($string,$substr,$byte_offset);
$byte_pos = Char::Eutf2::rindex($string,$substr);

This subroutine works just like Char::Eutf2::index except that it returns the byte
position of the last occurrence of $substring in $string (a reverse Char::Eutf2::index).
The subroutine returns -1 if $substring is not found. $byte_offset, if specified,
is the rightmost byte position that may be returned. To work your way through a
string backward, say:

$byte_pos = length($string);
while (($byte_pos = Char::UTF2::rindex($string, $lookfor, $byte_pos)) >= 0) {
    print "Found at $byte_pos\n";
    $byte_pos--;
}

Lower case string

$lc = Char::Eutf2::lc($string);
$lc = Char::Eutf2::lc_;

This subroutine returns a lowercased version of UTF-8 $string (or $_, if
$string is omitted). This is the internal subroutine implementing the \L escape
in double-quoted strings.

You can use the Char::Eutf2::fc subroutine for case-insensitive comparisons via Char::UTF2
software.

Lower case first character of string

$lcfirst = Char::Eutf2::lcfirst($string);
$lcfirst = Char::Eutf2::lcfirst_;

This subroutine returns a version of UTF-8 $string with the first character
lowercased (or $_, if $string is omitted). This is the internal subroutine
implementing the \l escape in double-quoted strings.

Upper case string

$uc = Char::Eutf2::uc($string);
$uc = Char::Eutf2::uc_;

This subroutine returns an uppercased version of UTF-8 $string (or $_, if
$string is omitted). This is the internal subroutine implementing the \U escape
in interpolated strings. For titlecase, use Char::Eutf2::ucfirst instead.

You can use the Char::Eutf2::fc subroutine for case-insensitive comparisons via Char::UTF2
software.

Upper case first character of string

$ucfirst = Char::Eutf2::ucfirst($string);
$ucfirst = Char::Eutf2::ucfirst_;

This subroutine returns a version of UTF-8 $string with the first character
titlecased and other characters left alone (or $_, if $string is omitted).
Titlecase is "Camel" for an initial capital that has (or expects to have)
lowercase characters following it, not uppercase ones. Exsamples are the first
letter of a sentence, of a person's name, of a newspaper headline, or of most
words in a title. Characters with no titlecase mapping return the uppercase
mapping instead. This is the internal subroutine implementing the \u escape in
double-quoted strings.

To capitalize a string by mapping its first character to titlecase and the rest
to lowercase, use:

$titlecase = Char::Eutf2::ucfirst(substr($word,0,1)) . Char::Eutf2::lc(substr($word,1));

or

$string =~ s/(\w)(\w*)/\u$1\L$2/g;

Do not use:

$do_not_use = Char::Eutf2::ucfirst(Char::Eutf2::lc($word));

or "\u\L$word", because that can produce a different and incorrect answer with
certain characters. The titlecase of something that's been lowercased doesn't
always produce the same thing titlecasing the original produces.

Because titlecasing only makes sense at the start of a string that's followed
by lowercase characters, we can't think of any reason you might want to titlecase
every character in a string.

See also P.287 A Case of Mistaken Identity
in Chapter 6: Unicode
of ISBN 978-0-596-00492-7 Programming Perl 4th Edition.

Fold case string

P.860 fc
in Chapter 27: Functions
of ISBN 978-0-596-00492-7 Programming Perl 4th Edition.

$fc = Char::Eutf2::fc($string);
$fc = Char::Eutf2::fc_;

New to Char::UTF2 software, this subroutine returns the full Unicode-like casefold of
UTF-8 $string (or $_, if omitted). This is the internal subroutine implementing
the \F escape in double-quoted strings.

Just as title-case is based on uppercase but different, foldcase is based on
lowercase but different. In ASCII there is a one-to-one mapping between only
two cases, but in other encoding there is a one-to-many mapping and between three
cases. Because that's too many combinations to check manually each time, a fourth
casemap called foldcase was invented as a common intermediary for the other three.
It is not a case itself, but it is a casemap.

To compare whether two strings are the same without regard to case, do this:

Char::Eutf2::fc($a) eq Char::Eutf2::fc($b)

The reliable way to compare string case-insensitively was with the /i pattern
modifier, because Char::UTF2 software has always used casefolding semantics for
case-insensitive pattern matches. Knowing this, you can emulate equality
comparisons like this:

sub fc_eq ($$) {
    my($a,$b) = @_;
    return $a =~ /\A\Q$b\E\z/i;
}

Make ignore case string

@ignorecase = Char::Eutf2::ignorecase(@string);

This subroutine is internal use to m/ /i, s/ / /i, split / /i, and qr/ /i.

Make capture number

$capturenumber = Char::Eutf2::capture($string);

This subroutine is internal use to m/ /, s/ / /, split / /, and qr/ /.

Make character

$chr = Char::Eutf2::chr($code);
$chr = Char::Eutf2::chr_;

This subroutine returns a programmer-visible character, character represented by
that $code in the character set. For example, Char::Eutf2::chr(65) is "A" in either
ASCII or UTF-8, not Unicode. For the reverse of Char::Eutf2::chr, use Char::UTF2::ord.

Filename expansion (globbing)

@glob = Char::Eutf2::glob($string);
@glob = Char::Eutf2::glob_;

This subroutine returns the value of $string with filename expansions the way a
DOS-like shell would expand them, returning the next successive name on each
call. If $string is omitted, $_ is globbed instead. This is the internal
subroutine implementing the <*> and glob operator.
This subroutine function when the pathname ends with chr(0x5C) on MSWin32.

For ease of use, the algorithm matches the DOS-like shell's style of expansion,
not the UNIX-like shell's. An asterisk ("*") matches any sequence of any
character (including none). A question mark ("?") matches any one character or
none. A tilde ("~") expands to a home directory, as in "~/.*rc" for all the
current user's "rc" files, or "~jane/Mail/*" for all of Jane's mail files.

Note that all path components are case-insensitive, and that backslashes and
forward slashes are both accepted, and preserved. You may have to double the
backslashes if you are putting them in literally, due to double-quotish parsing
of the pattern by perl.

The Char::Eutf2::glob subroutine grandfathers the use of whitespace to separate multiple
patterns such as <*.c *.h>. If you want to glob filenames that might contain
whitespace, you'll have to use extra quotes around the spacy filename to protect
it. For example, to glob filenames that have an "e" followed by a space followed
by an "f", use either of:

@spacies = <"*e f*">;
@spacies = Char::Eutf2::glob('"*e f*"');
@spacies = Char::Eutf2::glob(q("*e f*"));

If you had to get a variable through, you could do this:

@spacies = Char::Eutf2::glob("'*${var}e f*'");
@spacies = Char::Eutf2::glob(qq("*${var}e f*"));

Another way on MSWin32

# relative path
@relpath_file = split(/\n/,`dir /b wildcard\\here*.txt 2>NUL`);

# absolute path
@abspath_file = split(/\n/,`dir /s /b wildcard\\here*.txt 2>NUL`);

# on COMMAND.COM
@relpath_file = split(/\n/,`dir /b wildcard\\here*.txt`);
@abspath_file = split(/\n/,`dir /s /b wildcard\\here*.txt`);