NAME

Lingua::EN::Segment - Split English-language domain names etc. into words

SYNOPSIS

my $segmenter = Lingua::EN::Segment->new;
for my $domain (<>) {
    chomp $domain;
    my @words = $segmenter->segment($domain);
    print "$domain: ", join(', ', @words), "\n";
}

DESCRIPTION

Sometimes you have a string that to a human eye is clearly made up of many words glommed together without spaces or hyphens. This module uses some mild cunning and a large list of known words from Google to try and work out how the string should be split into words.

new

Out: $segmenter

Returns a Lingua::EN::Segment object.

dist_dir

Out: $dist_dir

Returns the name of the directory where distribution-specific files are installed.

segment

In: $unsegmented_string
Out: @words

Supplied with an unsegmented string - e.g. a domain name - returns a list of words that are most statistically likely to be the words that make up this string.

unigrams

Out: \%unigrams

Returns a hashref of word => likelihood to appear in Google's huge list of words that they got off the Internet. The higher the likelihood, the more likely that this is a genuine regularly-used word, rather than an obscure word or a typo.

bigrams

Out: \%bigrams

As "unigrams", but returns a lookup table of "word1 word2" => likelihood for combinations of words.

ACKNOWLEDGEMENTS

This code is based on chapter 14 of Peter Norvig's book Beautiful Data.