NAME

Hailo::Role::Tokenizer - A role representing a Hailo tokenizer

METHODS

new

This is the constructor. It takes no arguments.

make_tokens

Takes a line of input and returns an array reference of tokens.

make_output

Takes an array reference of tokens and returns a line of output.

AUTHOR

Hinrik Örn Sigurðsson, hinrik.sig@gmail.com

LICENSE AND COPYRIGHT

Copyright 2010 Hinrik Örn Sigurðsson

This program is free software, you can redistribute it and/or modify it under the same terms as Perl itself.