/ 
Treex-Unilang-0.06903_1
River stage one • 3 direct dependents • 3 total dependents
/ Treex::Block::W2A::SegmentOnNewlines

NAME

Treex::Block::W2A::SegmentOnNewlines - segment text on new lines

VERSION

version 0.06903_1

DESCRIPTION

The source text is segmented into sentences which are stored in document bundles. If the document contained no bundles, the bundles are created. Otherwise, the document must contain the same number of bundles as the number of sentences (segmented by this blocks). This means that this block (or its derivatives) can be used in this way:

treex Read::Text language=en from=en.txt \
      Read::Text language=de from=de.txt \
      W2A::SegmentOnNewlines language=en \
      W2A::SegmentOnNewlines language=de \
      ...

This class detects sentences based on the newlines in the source text, but it can be used as an ancestor for more apropriate segmentations by overriding the method segment_text.

ATTRIBUTES

allow_empty_sentences

If set, empty sentences can be produced.

delete_empty_sentences

If set, empty sentences are automatically deleted.

If none of the previous attributes is set and empty sentence found, fatal is raised.

METHODS

get_segments()

This method produces segments from text given as parameter, can be overriden.

normalize_sentence()

This method does sentence normalization e.g. trims initial and terminal whitespaces.

AUTHOR

Martin Popel <popel@ufal.mff.cuni.cz>

COPYRIGHT AND LICENSE

Copyright © 2011 by Institute of Formal and Applied Linguistics, Charles University in Prague

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.