NAME

Search::Fulltext::Tokenizer::MeCab - Provides Japanese fulltext search for Search::Fulltext module

SYNOPSIS

use Search::Fulltext;
use Search::Fulltext::Tokenizer::MeCab;

my $query = '猫';
my @docs = (
    '我輩は猫である',
    '犬も歩けば棒に当る',
    '実家でてんちゃんって猫を飼ってまして，ものすっごい可愛いんですよほんと',
);

my $fts = Search::Fulltext->new({
    docs      => \@docs,
    tokenizer => "perl 'Search::Fulltext::Tokenizer::MeCab::tokenizer'",
});
my $results = $fts->search($query);
is_deeply($results, [0, 2]);        # 1st & 3rd include '猫'
my $results = $fts->search('猫 AND 可愛い');
is_deeply($results, [2]);

DESCRIPTION

Search::Fulltext::Tokenizer::MeCab is a Japanse tokenizer working with fulltext search module Search::Fulltext. Only you have to do is specify perl 'Search::Fulltext::Tokenizer::MeCab::tokenizer' as a tokenizer of Search::Fulltext.

my $fts = Search::Fulltext->new({
    docs      => \@docs,
    tokenizer => "perl 'Search::Fulltext::Tokenizer::MeCab::tokenizer'",
});

You are supposed to use UTF-8 strings for docs.

Although various queries are available like "QUERIES" in Search::Fulltext, wildcard query (e.g. '我*') and phrase query (e.g. '"我輩は猫である"') are not supported.

User dictionary can be used to change the tokenizing behavior of internally-used Text::MeCab. See ENVIRONMENTAL VARIABLES section for detailes.

ENVIRONMENTAL VARIABLES

Some environmental variables are provided to customize the behavior of Search::Fulltext::Tokenizer::MeCab.

Typical usage:

$ ENV1=foobar ENV2=buz perl /path/to/your_script_using_this_module ARGS

MECABDIC_USERDIC

Specify path(s) to MeCab's user dictionary.

See MeCab's manual to learn how to create user dictionary.

Examples:

MECABDIC_USERDIC="/path/to/yourdic1.dic"
MECABDIC_USERDIC="/path/to/yourdic1.dic, /path/to/yourdic2.dic"

MECABDIC_DEBUG

When set to not 0, debug strings appear on STDERR.

Especially, outputs below would help check how your docs are tokenized.

string to be parsed: 我輩は猫である (7)
token: 我輩 (2)
token: は (1)
token: 猫 (1)
token: で (1)
token: ある (2)
...
string to be parsed: 猫 AND 可愛い (9)
token: 猫 (1)
string to be parsed:  可愛い (4)
token: 可愛い (3)

Note that not only docs but also queries are also tokenized.

SUPPORTS

Bug reports and pull requests are welcome at https://github.com/laysakura/Search-Fulltext-Tokenizer-MeCab !

To read this manual via perldoc, use -t option for correctly displaying UTF-8 caracters.

$ perldoc -t Search::Fulltext::Tokenizer::MeCab

VERSION

Version 1.05

AUTHOR

Sho Nakatani <lay.sakura@gmail.com>, a.k.a. @laysakura