NAME
Lingua::TokenParse - Parse a word into scored, fragment combinations
SYNOPSIS
use Lingua::TokenParse;
my $word = 'partition';
my %lexicon;
@lexicon{qw(ti art ion)} = qw(foo bar baz);
my $obj = Lingua::TokenParse->new(
word => $word,
lexicon => \%lexicon,
);
$obj->output_knowns;
# Okay. Now, let's parse a new word.
$obj->word('metaphysical');
$obj->lexicon({
'meta-' => 'more comprehensive',
'ta' => 'foo',
'phys' => 'natural science, singular',
'-ic' => 'being, containing',
'-al' => 'relating to, characterized by',
});
$obj->parse;
$obj->output_knowns;ABSTRACT
This class represents a Lingua::TokenParse object and contains methods to parse a given word into familiar combinations based on a lexicon of known word parts.
DESCRIPTION
A word like "partition" is actually composed of a few different word parts. Given a lexicon of known fragments, it is possible to partition this word into combinations of these (possibly overlapping) parts. Each of these combinations can be given a score, which represents a measure of familiarity.
Currently, this familiarity mesasure is a simple ratio of known to unknown parts.
* Please note that your lexicon absolutely must have definitions for each entry in order to have the current trim_knowns() method do the right thing. Yes, this will be fixed in a coming version... : )
* Please check out the sample code in the distribution's eg/ directory for exciting examples of how this module can be used.
METHODS
new()
$obj = Lingua::TokenParse->new(
word => $word,
lexicon => \%lexicon,
);Return a new Lingua::TokenParse object.
This method will automatically call the partition methods (detailed below) if a word and lexicon are provided.
_reset_parse()
$obj->_reset_parse();Reset the lists used to parse a word into fragment combinations.
This method is automatically called prior to reparsing a new word in the same session and should never need to be called explicitly.
parse()
$obj->parse();This is a convenience method that simply calls all the indiviual parsing methods that are detailed below.
Call this method after resetting the object with a new word and optionally, a new lexicon.
build_parts()
$obj->build_parts();Construct an array of the word partitions, accessed via the parts() method.
build_combinations()
$obj->build_combinations();Recursively compute the array of all possible word part combinations, accessed via the combinations() method.
build_knowns()
$obj->build_knowns();Compute the familiar word part combinations, accessed via the knowns() method.
This method handles word parts containing prefix and suffix hyphens, which are found in the dict.org sever (with web1913 database). These hyphens actually "encode" information about what is a syntactically legal word combination. Which can be used to score (or just throw out bogus combinations).
build_definitions()
$obj->build_definitions();Construct a hash of the definitions of the word parts in each combination in the keys of the knowns hash.
trim_knowns()
$obj->trim_knowns();Construct an array of the known combinations, with the adjacent unknown fragments concatinated.
output_knowns()
$obj->output_knowns();Convenience method to output the familiar word part combinations with their familiarity scores rounded to two decimals.
ACCESSORS
These accessors both get and set their respective values. Note that, if you set any of these after construction, you must manually run the partition methods. Also, note that it is pretty useless to set the parts, combinations and knowns lists, as they are computed by the partition methods.
word()
$word = $obj->word($word);The actual word to partition.
lexicon()
$lexicon = $obj->lexicon(\%lexicon);The hash reference of word parts (keys) with their (optional) definitions (values).
parts()
$parts = $obj->parts();The array reference of word partitions.
Note that this method is only useful for fetching, since the parts are computed by the build_parts() method.
combinations()
$combinations = $obj->combinations();The array reference of all possible word part combinations.
Note that this method is only useful for fetching, since the combinations are computed by the build_combinations() method.
knowns()
$knowns = $obj->knowns();The hash reference of known combinations (keys) with their familiarity scores (values). Note that only the non-zero scored combinations are kept.
Note that this method is only useful for fetching, since the knowns are computed by the build_knowns() method.
definitions()
$definitions = $obj->definitions();The hash reference of the definitions provided for each fragment of the combinations in the knowns hash. Note that the unknown fragments are defined as an empty string.
DEPENDENCIES
None
DISCLAIMER
This module uses some clunky, inefficient algorithms. For instance, a 50 letter word (like a medical term) just might take until the end of time to parse and possibly longer. Please write to me with improvements!
TO DO
Calculate familiarity with more granularity. Possibly with a multidimensional measure.
Output some type of concatinated, known combination definition.
Handle the build_combinations method and related globals correctly.
Compute the time required for a given parse.
Synthesize a term list based on word part (thesaurus) definitions. (That is, go in reverse! Non-trivial!)
DEDICATION
My Grandmother and English teacher - Frances Jones
AUTHOR
Gene Boggs <cpan@ology.net>
COPYRIGHT AND LICENSE
Copyright 2003 by Gene Boggs
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.