NAME

ShiftJIS::Regexp - Shift_JIS-oriented regexps in the byte-oriented perl

SYNOPSIS

use ShiftJIS::Regexp qw(:all);

match('‚ ‚¨‚P‚Q', '\p{InHiragana}{2}\p{IsDigit}{2}');
match('‚ ‚¢‚¢‚¤‚¤‚¤', '^‚ ‚¢+‚¤{3}$');
replace($str, 'A', '‚`', 'g');

DESCRIPTION

This module provides some functions to use Shift_JIS-oriented regexps in the byte-oriented perl.

The legal Shift_JIS character in this module must match the following regexp:

[\x00-\x7F\xA1-\xDF]|[\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC]

FUNCTIONS

re(PATTERN)

re(PATTERN, MODIFIER)

Returns regexp parsable by the byte-oriented perl.

PATTERN is specified as a string.

MODIFIER is specified as a string.

i  do case-insensitive pattern matching (only for ascii alphabets)
I  do case-insensitive pattern matching
   (greek, cyrillic, fullwidth latin)
j  do hiragana-katakana-insensitive pattern matching

s  treat string as single line
m  treat string as multiple lines
x  ignore whitespace (i.e. [ \n\r\t\f], but not comments!)
   unless backslashed or inside a character class

o  once parsed (not compiled!) and the result is cached internally.

re('^ƒRƒ“ƒsƒ…�[ƒ^�[?$') matches 'ƒRƒ“ƒsƒ…�[ƒ^�[' or 'ƒRƒ“ƒsƒ…�[ƒ^'.

re('^‚ç‚‚¾$','j') matches '‚ç‚‚¾', 'ƒ‰ƒNƒ_', '‚çƒN‚¾', etc.

o modifier

while(<DATA>){
  print replace($_, '(perl)', '<strong>$1</strong>', 'igo');
}
   is more efficient than

while(<DATA>){
  print replace($_, '(perl)', '<strong>$1</strong>', 'ig');
}

because in the latter case the pattern is parsed every time
whenever the function is called.

match(STRING, PATTERN)

match(STRING, PATTERN, MODIFIER)

emulation of m// operator for the Shift_JIS encoding.

PATTERN is specified as a string.

MODIFIER is specified as a string.

i  do case-insensitive pattern matching (only for ascii alphabets)
I  do case-insensitive pattern matching
   (greek, cyrillic, fullwidth latin)
j  do hiragana-katakana-insensitive pattern matching

s  treat string as single line
m  treat string as multiple lines
x  ignore whitespace (i.e. [ \n\r\t\f], but not comments!)
   unless backslashed or inside a character class
g  match globally
z  tell the function the pattern matches zero-length substring
      (sorry, due to the poor auto-detection)

o  once parsed (not compiled!) and the result is cached internally.

replace(STRING or SCALAR REF, PATTERN, REPLACEMENT)

replace(STRING or SCALAR REF, PATTERN, REPLACEMENT, MODIFIER)

emulation of s/// operator for the Shift_JIS encoding.

If a reference of scalar variable is specified as the first argument, returns the number of substitutions made. If a string is specified as the first argument, returns the substituted string and the specified string is unaffected.

my $d = '\p{IsDigit}';
my $str = '‹à‚P‚T‚R‚O‚O‚O‚O‰~';
1 while replace(\$str, "($d)($d$d$d)(?!$d)", '$1�C$2');
print $str; # ‹à‚P�C‚T‚R‚O�C‚O‚O‚O‰~

MODIFIER is specified as a string.

i  do case-insensitive pattern matching (only for ascii alphabets)
I  do case-insensitive pattern matching
   (greek, cyrillic, fullwidth latin)
j  do hiragana-katakana-insensitive pattern matching

s  treat string as single line  treat string as single line
m  treat string as multiple lines
x  ignore whitespace (i.e. [ \n\r\t\f], but not comments!)
   unless backslashed or inside a character class
g  match globally
z  tell the function the pattern matches zero-length substring
      (sorry, due to the poor auto-detection)

o  once parsed (not compiled!) and the result is cached internally.

jsplit(PATTERN or ARRAY REF of [PATTERN, MODIFIER], STRING)

jsplit(PATTERN or ARRAY REF of [PATTERN, MODIFIER], STRING, LIMIT)

This function emulates CORE::split.

If not in list context, these functions do only return the number of fields found, but do not split into the @_ array.

PATTERN is specified as a string.

But ' ' as PATTERN has no special meaning; when you want to split the string on whitespace, you can use splitspace() function.

jsplit('�^', '‚ ‚¢‚¤�^‚¦‚¨ƒ�^');

If you want to pass pattern with modifiers, specify an arrayref of [PATTERN, MODIFIER] as the first argument.

jsplit([ '‚ ', 'jo' ], '01234‚ ‚¢‚¤‚¦‚¨ƒAƒCƒEƒGƒI');

MODIFIER is specified as a string.

i  do case-insensitive pattern matching (only for ascii alphabets)
I  do case-insensitive pattern matching
   (greek, cyrillic, fullwidth latin)
j  do hiragana-katakana-insensitive pattern matching

s  treat string as single line
m  treat string as multiple lines
x  ignore whitespace (i.e. [ \n\r\t\f], but not comments!)
   unless backslashed or inside a character class

o  once parsed (not compiled!) and the result is cached internally.

splitspace(STRING)

splitspace(STRING, LIMIT)

This function emulates CORE::split ' ', STRING and returns the array given by split on whitespace including IDEOGRAPHIC SPACE. Leading whitespace characters do not produce any field.

splitchar(STRING)

splitchar(STRING, LIMIT)

This function emulates CORE::split //, STRING and returns the array given by split of the specified string into characters.

REGEXPS

  regexp          meaning

  ^               match the start of the string
                  match the start of any line with 'm' modifier

  $               match the end of the string
                  match the end of any line with 'm' modifier

  .               match any character except \n
                  match any character with 's' modifier

  \A              only at beginning of string
  \Z              at end of string, or before newline at the end
  \z              only at end of string (eq. '(?!\n)\Z')

  \C              match a single C char (octet), i.e. [\0-\xFF] in perl.
  \j              match any character, i.e. [\0-\x{FCFC}] in this module.
  \J              match any character except \n, i.e. [^\n] in this module.

    * \j and \J are extensions by this module. e.g.

       match($_, '(\j{5})\z') returns last five chars including \n at the end
       match($_, '(\J{5})\Z') returns last five chars excluding \n at the end

  \a              alarm      (BEL)
  \b              backspace  (BS) * within character classes *
  \t              tab        (HT, TAB)
  \n              newline    (LF, NL)
  \f              form feed  (FF)
  \r              return     (CR)
  \e              escape     (ESC)

  \0              null       (NUL)

  \ooo            octal single-byte character
  \xhh            hexadecimal single-byte character
  \x{hhhh}        hexadecimal double-byte character
  \c[             control character

     e.g. \012 \123 \x5c \x5C \x{824F} \x{9Fae} \cA \cZ \c^ \c?

  regexp           equivalent character class

  \d               [\d]              [0-9]
! \D               [\D]              [^0-9]
  \w               [\w]              [0-9A-Z_a-z]
! \W               [\W]              [^0-9A-Z_a-z]
  \s               [\s]              [\t\n\r\f ]
! \S               [\S]              [^\t\n\r\f ]

  \p{IsDigit}      [[:digit:]]       [0-9‚O-‚X]
! \P{IsDigit}      [[:^digit:]]      [^0-9‚O-‚X]
  \p{IsUpper}      [[:upper:]]       [A-Z‚`-‚y]
! \P{IsUpper}      [[:^upper:]]      [^A-Z‚`-‚y]
  \p{IsLower}      [[:lower:]]       [a-z‚�-‚š]
! \P{IsLower}      [[:^lower:]]      [^a-z‚�-‚š]
  \p{IsAlpha}      [[:alpha:]]       [A-Za-z‚`-‚y‚�-‚š]
! \P{IsAlpha}      [[:^alpha:]]      [^A-Za-z‚`-‚y‚�-‚š]
  \p{IsAlnum}      [[:alnum:]]       [0-9A-Za-z‚O-‚X‚`-‚y‚�-‚š]
! \P{IsAlnum}      [[:^alnum:]]      [^0-9A-Za-z‚O-‚X‚`-‚y‚�-‚š]

  \p{IsWord}       [[:word:]]
         [0-9A-Z_a-z‚O-‚X‚`-‚y‚�-‚šƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘¦-ß‚Ÿ-‚ñƒ@-ƒ–�J�K�R-�[ˆŸ-˜r˜Ÿ-ê¤]
! \P{IsWord}       [[:^word:]]
         [^0-9A-Z_a-z‚O-‚X‚`-‚y‚�-‚šƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘¦-ß‚Ÿ-‚ñƒ@-ƒ–�J�K�R-�[ˆŸ-˜r˜Ÿ-ê¤]

  \p{IsPunct}      [[:punct:]]
               [!-/:-@[-`{-~¡-¥�A-�I�L-�Q�\-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü„Ÿ-„¾]
! \P{IsPunct}      [[:^punct:]]
               [^!-/:-@[-`{-~¡-¥�A-�I�L-�Q�\-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü„Ÿ-„¾]
  \p{IsSpace}      [[:space:]]       [\t\n\r\f \x{8140}]
! \P{IsSpace}      [[:^space:]]      [^\t\n\r\f \x{8140}]
! \p{IsGraph}      [[:graph:]]       [^\0- \x7F\x{8140}]
  \P{IsGraph}      [[:^graph:]]      [\0- \x7F\x{8140}]
! \p{IsPrint}      [[:print:]]       [^\0- \x0B\x0E-\x1F\x7F]
  \P{IsPrint}      [[:^print:]]      [\x00-\x08\x0B\x0E-\x1F\x7F]
  \p{IsCntrl}      [[:cntrl:]]       [\x00-\x1F]
! \P{IsCntrl}      [[:^cntrl:]]      [^\x00-\x1F]

  \p{IsAscii}      [[:ascii:]]       [\x00-\x7F]
! \P{IsAscii}      [[:^ascii:]]      [^\x00-\x7F]
  \p{IsHankaku}    [[:hankaku:]]     [\xA1-\xDF]
! \P{IsHankaku}    [[:^hankaku:]]    [^\xA1-\xDF]
! \p{IsZenkaku}    [[:zenkaku:]]     [\x{8140}-\x{FCFC}]
  \P{IsZenkaku}    [[:^zenkaku:]]    [^\x{8140}-\x{FCFC}]

  \p{IsX0201}      [[:x0201:]]       [\x00-\x7F\xA1-\xDF]
! \P{IsX0201}      [[:^x0201:]]      [^\x00-\x7F\xA1-\xDF]
  \p{IsX0208}      [[:x0208:]]
         [\x{8140}-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü‚O-‚X‚`-‚y‚�-‚š‚Ÿ-‚ñƒ@-ƒ–ƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘„Ÿ-„¾ˆŸ-˜r˜Ÿ-ê¤]

! \P{IsX0208}      [[:^x0208:]]
         [^\x{8140}-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü‚O-‚X‚`-‚y‚�-‚š‚Ÿ-‚ñƒ@-ƒ–ƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘„Ÿ-„¾ˆŸ-˜r˜Ÿ-ê¤]

  \p{InLatin}      [[:latin:]]       [A-Za-z]
! \P{InLatin}      [[:^latin:]]      [^A-Za-z]
  \p{InFullLatin}  [[:fulllatin:]]   [‚`-‚y‚�-‚š]
! \P{InFullLatin}  [[:^fulllatin:]]  [^‚`-‚y‚�-‚š]
  \p{InGreek}      [[:greek:]]       [ƒŸ-ƒ¶ƒ¿-ƒÖ]
! \P{InGreek}      [[:^greek:]]      [^ƒŸ-ƒ¶ƒ¿-ƒÖ]
  \p{InCyrillic}   [[:cyrillic:]]    [„@-„`„p-„‘]
! \P{InCyrillic}   [[:^cyrillic:]]   [^„@-„`„p-„‘]
  \p{InHalfKana}   [[:halfkana:]]    [¦-ß]
! \P{InHalfKana}   [[:^halfkana:]]   [^¦-ß]
  \p{InHiragana}   [[:hiragana:]]    [‚Ÿ-‚ñ�J�K�T�U]
! \P{InHiragana}   [[:^hiragana:]]   [^‚Ÿ-‚ñ�J�K�T�U]
  \p{InKatakana}   [[:katakana:]]    [ƒ@-ƒ–�[�R�S]
! \P{InKatakana}   [[:^katakana:]]   [^ƒ@-ƒ–�[�R�S]
  \p{InFullKana}   [[:fullkana:]]    [‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S]
! \P{InFullKana}   [[:^fullkana:]]   [^‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S]
  \p{InKana}       [[:kana:]]        [¦-ß‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S]
! \P{InKana}       [[:^kana:]]       [^¦-ß‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S]
  \p{InKanji1}     [[:kanji1:]]      [ˆŸ-˜r]
! \P{InKanji1}     [[:^kanji1:]]     [^ˆŸ-˜r]
  \p{InKanji2}     [[:kanji2:]]      [˜Ÿ-ê¤]
! \P{InKanji2}     [[:^kanji2:]]     [^˜Ÿ-ê¤]
  \p{InKanji}      [[:kanji:]]       [�V-�ZˆŸ-˜r˜Ÿ-ê¤]
! \P{InKanji}      [[:^kanji:]]      [^�V-�ZˆŸ-˜r˜Ÿ-ê¤]
  \p{InBoxDrawing} [[:boxdrawing:]]  [„Ÿ-„¾]
! \P{InBoxDrawing} [[:^boxdrawing:]] [^„Ÿ-„¾]

  * On \p{Prop} or \P{Prop} expressions, 'Is' or 'In' can be omitted
    like \p{Digit} or \P{Kanji}.
   (the omission of 'In' is an extension by this module)

  * Character classes marked with <!> contain undefined codepoints
    in JIS (JIS X 0201 or JIS X 0208).

Character class

Ranges in character class are supported.

The order of Shift_JIS characters is: 0x00 .. 0x7F, 0xA1 .. 0xDF, 0x8140 .. 0x9FFC, 0xE040 .. 0xFCFC.

So [\0-\x{fcfc}] matches any one Shift_JIS character.

In character classes, any character or byte sequence that does not match any one Shift_JIS character, e.g. re('[\xA0-\xFF]'), is croaked.

Character classes that match non-Shift_JIS substring are not supported (use \C or alternation).

Character equivalents

Since the version 0.13, the POSIX character equivalent classes [=cc=] are supported. e.g. [[=‚ =]] is identical to [‚Ÿƒ@§‚ ƒA±]; [[=P=]] to [pP‚�‚o]; [[=4=]] to [4‚S]. They are used in a character class, like [[=cc=]], [[=p=][=e=][=r=][=l=]].

As cc in [=cc=], any character literal or meta chatacter (\xhh, \x{hhhh}) that belongs to the character equivalents can be used. e.g. [=‚ =], [=ƒA=], [=\x{82A0}=], [=\xB1=], etc. have identical meanings.

[[=‚©=]] matches '‚©', 'ƒJ', '¶', '‚ª', 'ƒK', '¶Þ', 'ƒ•' ('¶Þ' is a two-character string, but one collation element, HALFWIDTH FORM FOR KATAKANA LETTER GA.

[[===]] matches EQUALS SIGN or FULLWIDTH EQUALS SIGN; [[=[=]] matches LEFT SQUARE BRACKET or FULLWIDTH LEFT SQUARE BRACKET; [[=]=]] matches RIGHT SQUARE BRACKET or FULLWIDTH RIGHT SQUARE BRACKET; [[=\=]] matches YEN SIGN or FULLWIDTH YEN SIGN.

Code embedded in regexp (Perl 5.005 or later)

Parsing (?{ ... }) or (??{ ... }) assertions is carried out without any special care of double-byte characters.

(?{ ... }) assertions are disallowed in match() or replace() functions by perl due to security concerns. Use them via re() function inside your scope.

use ShiftJIS::Regexp qw(:all);

use re 'eval';

$::res = 0;
$_ = 'ƒ|' x 8;

my $regex = re(q/
     \j*?
     (?{ $cnt = 0 })
     (
       ƒ| (?{ local $cnt = $cnt + 1; })
     )*  
     ƒ|ƒ|ƒ|
     (?{ $::res = $cnt })
   /, 'x');

/$regex/;
print $::res; # 5

CAVEATS

A legal Shift_JIS character in this module must match the following regexp:

[\x00-\x7F\xA1-\xDF]|[\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC]

Any string from external resource should be checked by issjis() function of ShiftJIS::String, excepting you know it is surely encoded in Shift_JIS.

Use of an illegal Shift_JIS string may lead to odd results.

Some Shift_JIS double-byte characters have one of [\x40-\x7E] as the trail byte.

@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

The Perl lexer doesn't take any care to these characters, so they sometimes make trouble. e.g. the quoted literal "•\" causes fatal error, since its trail byte 0x5C backslashes the closing quote.

Such a problem doesn't arise when the string is gotten from any external resource. But writing the script containing Shift_JIS double-byte characters needs the greatest care.

The use of single-quoted heredoc, << '', or \xhh meta characters is recommended in order to define a Shift_JIS string literal.

The safe ASCII-graphic characters, [\x21-\x3F], are:

!"#$%&'()*+,-./0123456789:;<=>?

They are preferred as the delimiter of quote-like operators.

BUGS

The \U, \L, \Q, \E, and interpolation are not considered. If necessary, use them in "" (or qq//) operators in the argument list.

The regexps of the word boundary, \b and \B, don't work correctly.

Never pass any regexp containing '(?i)' on perl below 5.005. Pass 'i' modifier as the second argument. (On Perl 5.005 or later, '(?i)' is allowed because '(?-i:RE)' prevents it from wrong matching)

e.g.

match('ƒG', '(?i)ƒg') returns true on Perl below 5.005 (Wrong).
match('ƒG', '(?i)ƒg') returns false on Perl 5.005 or later (Good).
match('ƒG', 'ƒg', 'i') returns false, ok.
# The trail byte of 'ƒG' is 'G' and that of 'ƒg' is 'g';

The i, I and j modifiers are invalid to \p{}, \P{}, and POSIX [: :]. (e.g. \p{IsLower}, [:lower:], etc). So use re('\p{IsAlpha}') instead of re('\p{IsLower}', 'iI').

The look-behind assertion like (?<=[A-Z]) is not prevented from matching trail byte of the previous double byte character: e.g. match("ƒAƒCƒE", '(?<=[A-Z])(\p{InKana})') returns ('ƒC').

Use of not greedy regexp, which can match empty string, such as .?? and \d*?, as the PATTERN in jsplit(), may cause failure to the emulation of CORE::split.

AUTHOR

Tomoyuki SADAHIRO

bqw10602@nifty.com
http://homepage1.nifty.com/nomenclator/perl/
This program is free software; you can redistribute it and/or 
modify it under the same terms as Perl itself.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

SYNOPSIS

DESCRIPTION

FUNCTIONS

REGEXPS

CAVEATS

BUGS

AUTHOR

SEE ALSO

Module Install Instructions