/ 
Search-Tools-0.27
River stage two • 17 direct dependents • 34 total dependents
/ Search::Tools::TokenListUtils

NAME

Search::Tools::TokenListUtils - mixin methods for TokenList and TokenListPP

SYNOPSIS

my $tokens = $tokenizer->tokenize( $string );
if ( $tokens->str eq $string) {
   print "string is same, before and after tokenize()\n";
}
else {
   warn "I'm filing a bug report against Search::Tools right away!\n";
}

my ($start_pos, $end_pos) = $tokens->get_window( 5, 20 );
# $start_pos probably == 0
# $end_pos probably   == 25

my $slice = $tokens->get_window_pos( 5, 20 );
for my $token (@$slice) {
   print "token = $token\n";
}

DESCRIPTION

Search::Tools::TokenListUtils contains pure-Perl methods inhertited by both Search::Tools::TokenList and Search::Tools::TokenListPP.

METHODS

str

Returns a serialized version of the TokenList. If you haven't altered the TokenList since you got it from tokenize(), then str() returns a scalar string identical to (but not the same) the string you passed to tokenize().

Both Search::Tools::TokenList and TokenListPP are overloaded to stringify to the str() value.

get_window( pos [, size] )

Returns array with two values: start and end positions for the array of length size on either side of pos. Like taking a slice of the TokenList.

Note that size is the number of tokens not matches. So if you're looking for the number of "words", think about size*2.

Note too that size is the number of tokens on one side of pos. So the entire window width (length of the returned slice) is size*2 +/-1. The window is guaranteed to be bounded by matches.

get_window_tokens( pos [, size] )

Like get_window() but returns an array ref of a slice of the TokenList containing Tokens.

AUTHOR

Peter Karman <karman@cpan.org>

BUGS

Please report any bugs or feature requests to bug-search-tools at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Search-Tools. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

You can find documentation for this module with the perldoc command.

perldoc Search::Tools

You can also look for information at:

COPYRIGHT

Copyright 2009 by Peter Karman.

This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.