NAME

ShiftJIS::Collate - collation of ShiftJIS strings

SYNOPSIS

use ShiftJIS::Collate;

@sorted = ShiftJIS::Collate->new(%tailoring)->sort(@not_sorted);

DESCRIPTION

This module provides some functions to compare and sort strings in the ShiftJIS encoding using the collation of Japanese character strings.

This module is an implementation of JIS X 4061-1996 and the collation rules are based on that standard. See "Informations for Conformance".

Constructor and Tailoring

The new method returns a collator object.

$Collator = ShiftJIS::Collate->new(
   kanji => $kanji_class,
   katakana_before_hiragana => $bool,
   level => $collationLevel,
   overrideCJK => \&overrideCJK,
   position_in_bytes => $bool,
   preprocess => \&preprocess,
   upper_before_lower => $bool,
);
# if %tailoring is false (empty),
# $Collator should do the default collation.

katakana_before_hiragana

By default, hiragana is before katakana.

If the parameter is true, this is reversed.

kanji

Set the kanji class. See "Kanji Classes".

Level 1: 'saisho' (minimal)
Level 2: 'kihon' (basic)
Level 3: 'kakucho' (extended)

This module does not provide collation of 'kakucho' kanji class since the repertory Shift_JIS does not define all the Unicode CJK unified ideographs.

The kanji class is specified as 1 or 2. If omitted, class 2 is applied.

level

Set the maximum level. See "Collation Levels". Any higher levels than the specified one are ignored.

Level 1: alphabetic ordering
Level 2: diacritic ordering
Level 3: case ordering
Level 4: script ordering
Level 5: width ordering

The collation level is specified as a number between 1 and 5. If omitted, level 4 is applied.

overrideCJK

By default, mapping of CJK Unified Ideographs uses the JIS codepoint order.

The mapping of CJK Unified Ideographs may be overrided by this parameter.

ex. CJK Unified Ideographs in the Unicode codepoint order.

overrideCJK => sub {
  my $c = shift;               # get Shift_JIS kanji
  my $s = your_sjis_to_utf16_converter($c); # convert
  my $n = unpack('n', $s);     # convert utf16 to short
  [ $n, 50, 20, 10, 1 ];       # return collation element
},

position_in_bytes

By default, the index method returns its results in characters.

If this parameter is true, it returns the results in bytes.

preprocess

If specified, the coderef is used to preprocess before the formation of sort keys.

upper_before_lower

By default, lowercase is before uppercase.

If the parameter is true, this is reversed.

Other Methods

$result = $Collator->cmp($a, $b)

Returns 1 (when $a is greater than $b) or 0 (when $a is equal to $b) or -1 (when $a is lesser than $b).

@sorted = $Collator->sort(@not_sorted)

Sorts a list of strings by tanjun shogo: 'the simple collation'.

@sorted = $Collator->sortYomi(@not_sorted)

Sorts a list of references to arrays of (spell, reading) by yomi-hyoki shogo: 'the collation using readings and spells'.

@sorted = $Collator->sortDaihyo(@not_sorted)

Sorts a list of references to arrays of (spell, reading) by kan'i-daihyo-yomi shogo: 'the simplified representative reading collation'.

$sortKey = $Collator->getSortKey($string)

Returns a sort key.

You compare the sort keys using a binary comparison and get the result of the comparison of the strings.

$Collator->getSortKey($a) cmp $Collator->getSortKey($b)

   is equivalent to

$Collator->cmp($a, $b)

$position = $Collator->index($string, $substring)

($position, $length) = $Collator->index($string, $substring)

If $substring matches a part of $string, returns the position of the first occurrence of the matching part in scalar context; in list context, returns a two-element list of the position and the length of the matching part.

Notice that the length of the matching part may differ from the length of $substring.

If $substring does not match any part of $string, returns -1 in scalar context and an empty list in list context.

e.g. you say

use ShiftJIS::Collate;
use ShiftJIS::String qw(substr);

my $Col = ShiftJIS::Collate->new( level => $level );
my $str = "* ‚Ð‚ç‚ª‚È‚ÆƒJƒ^ƒJƒi‚ÍƒŒƒxƒ‹‚R‚Å‚Í“™‚µ‚¢‚©‚È�B";
my $sub = "‚©‚È";
my $match;
if(my @tmp = $Col->index($str, $sub)){
  $match = substr($str, $tmp[0], $tmp[1]);
}

If $level is 1, you get "‚ª‚È"; if $level is 2 or 3, you get "ƒJƒi"; if $level is 4 or 5, you get "‚©‚È".

If your substr function is not oriented to Shift_JIS, specify true as position_in_bytes. See "Constructor and Tailoring".

NOTE

Collation Levels

The following criteria are considered in order until the collation order is determined. By default, Levels 1 to 4 are applied and Level 5 is ignored (as JIS does).

Level 1: alphabetic ordering.

The character class early appeared in the following list is smaller.

Space characters, Symbols and Punctuations, Digits, Greek Letters,
Cyrillic Letters, Latin letters, Kana letters, Kanji ideographs,
and Geta mark.

In the class, alphabets are collated alphabetically; kana letters are AIUEO-betically (in the Gozyuon order); kanji are in the JIS X 0208 order.

Characters that do not belong to any character class are ignored and skipped for collation.

Level 2: diacritic ordering.

In kana, the order is as shown the following list.

A voiceless kana, the voiced, then the semi-voiced (if exists).
 (eg. Ka before Ga; Ha before Ba before Pa)

Level 3: case ordering.

A small Latin is lesser than the corresponding Capital.

In kana, the order is as shown the following list. see "Replacement of PROLONGED SOUND MARK and ITERATION MARKs".

Replaced PROLONGED SOUND MARK (U+30FC);
Small Kana;
Replaced ITERATION MARK (U+309D, U+309E, U+30FD or U+30FE);
then Normal Kana.

Level 4: script ordering.

Hiragana is lesser than katakana.

Level 5: width ordering.

A character that belongs to the block Halfwidth and Fullwidth Forms is greater than the corresponding normal character.

BN: According to the JIS standard, the level 5 should be ignored.

Kanji Classes

There are three kanji classes. This modules provides the Classes 1 and 2.

Class 1: the 'saisho' (minimum) kanji class: It comprises five kanji-like chars, i.e. U+3003, U+3005, U+4EDD, U+3006, U+3007. Any kanji except U+4EDD are ignored on collation.
Class 2: the 'kihon' (basic) kanji class: It comprises JIS levels 1 and 2 kanji in addition to the minimum kanji class. Sorted in the JIS order. Any kanji excepting those defined by JIS X 0208 are ignored on collation.
Class 3: the 'kakucho' (extended) kanji class: All the CJK Unified Ideographs in addition to the minimum kanji class. Sorted in the unicode order.

Replacement of PROLONGED SOUND MARK and ITERATION MARKs

        RFC1345 UCS
	[*5]    U+309D  HIRAGANA ITERATION MARK
	[+5]    U+309E  HIRAGANA VOICED ITERATION MARK
	[-6]    U+30FC  KATAKANA-HIRAGANA PROLONGED SOUND MARK
	[*6]    U+30FD  KATAKANA ITERATION MARK
	[+6]    U+30FE  KATAKANA VOICED ITERATION MARK

To represent Japanese characters, RFC 1345 Mnemonic characters enclosed by brackets are used below.

These characters, if replaced, are secondary equal to the replacing kana, while ternary not equal to.

KATAKANA-HIRAGANA PROLONGED SOUND MARK

The PROLONGED MARK is repleced to normal vowel or nasal katakana corresponding to the preceding kana if exists.

  eg.	[Ka][-6] to [Ka][A6]
	[bi][-6] to [bi][I6]
	[Pi][YU][-6] to [Pi][YU][U6]
	[N6][-6] to [N6][N6]

HIRAGANA- and KATAKANA ITERATION MARKs

The ITERATION MARKs (VOICELESS) are repleced to normal kana corresponding to the preceding kana if exists.

  eg.	[Ka][*6] to [Ka][Ka]
	[Do][*5] to [Do][to]
	[n5][*5] to [n5][n5]
	[Pu][*6] to [Pu][Hu]
	[Pi][YU][*6] to [Pi][YU][Yu]

HIRAGANA- and KATAKANA VOICED ITERATION MARKs

The VOICED ITERATION MARKs are repleced to the voiced kana corresponding to the preceding kana if exists.

  eg.	[ha][+5] to [ha][ba]
	[Pu][+5] to [Pu][bu]
	[Ko][+6] to [Ko][Go]
	[U6][+6] to [U6][Vu]

Cases of no replacement

Otherwise, no replacement occurs. Especially in the cases when these marks follow any character except kana.

The unreplaced characters are primary greater than any kana.

  eg.	CJK Ideograph followed by PROLONGED SOUND MARK
	Digit followed by ITERATION MARK
	[A6][+6] ([A6] has no voiced variant)

Example

For example, the Japanese string [Pa][-6][Ru] (Perl in kana) has three collation elements: KATAKANA PA, PROLONGED SOUND MARK replaced by KATAKANA A, and KATAKANA RU.

   [Pa][-6][Ru] is converted to [Pa][A6][Ru] by replacement.
		primary equal to [ha][a5][ru].
		secondary equal to [pa][a5][ru], greater than [ha][a5][ru].
		tertiary equal to [pa][-6][ru], lesser than [Pa][A6][Ru].
		quartenary greater than [pa][-6][ru].

Informations for Conformance

  [according to the article 6.2, JIS X 4061]

(1) charset: Shift_JIS.

(2) No limit of the number of characters in the string considered
    to collate.

(3) No character class is added.

(4) The following characters are added as collation elements.

    IDEOGRAPHIC SPACE in the space class.

    ACUTE ACCENT, GRAVE ACCENT, DIAERESIS, CIRCUMFLEX ACCENT
    in the class of descriptive symbols.

    APOSTROPHE, QUOTATION MARK in the class of parentheses.

    HYPHEN-MINUS in the class of mathematical symbols.

(5) Collation of Latin alphabets with macron and with circumflex
    is not supported.

(6) Selected kanji class:
     the minimum kanji class (Five kanji-like chars).
     the basic kanji class (Levels 1 and 2 kanji, JIS)..

AUTHOR

Tomoyuki SADAHIRO

 bqw10602@nifty.com
 http://homepage1.nifty.com/nomenclator/perl/

This program is free software; you can redistribute it and/or 
modify it under the same terms as Perl itself.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)