NAME

DiaColloDB::Corpus::Filters - collocation db, source corpus content filters

SYNOPSIS

##========================================================================
## PRELIMINARIES

use DiaColloDB::Corpus::Filters;

##========================================================================
## Methods

$filters = $CLASS_OR_OBJECT->new(%opts);
$filters = $CLASS_OR_OBJECT->null();
$filters = $filters->clear();
$bool = $filters1->equal($filters2);
\%name2obj = $filters->compile();
\%line2undef = $coldb->loadListFile($filename_or_undef);

DESCRIPTION

DiaColloDB::Corpus::Filters is a class representing corpus content filters (e.g. stopword lists and regular expressions) used by DiaColloDB::Corpus::Compiled and implicitly by the DiaColloDB::create()|DiaColloDB/create method as called by the top-level command-line utility dcdb-corpus-create.perl(1).

Administrivia

Variable: @ISA

DiaColloDB::Corpus::Filters inherits from DiaColloDB::Persistent. It also uses Exporter for compatibility with older versions of the DiaColloDB distribution in which the package-global default variables resided directly in the DiaColloDB package itself.

Defaults

(formerly defined in DiaColloDB.pm)

Don't use qr// for regex defaults, because Storable doesn't like pre-compiled Regexps.

Variable: $PGOOD_DEFAULT

Default positive PoS-regex for document parsing. Default = q/^(?:N|TRUNC|VV|ADJ)/.

Variable: $PBAD_DEFAULT

Default negative PoS-regex for document parsing. Default = undef (none).

Variable: $WGOOD_DEFAULT

Default positive word regex for document parsing. Default = q/[[:alpha:]]/

Variable: $WBAD_DEFAULT

Default negative word regex for document parsing. Default = q/[\.]/.

Variable: $LGOOD_DEFAULT

Default positive lemma regex for document parsing. Default = undef (none).

Variable: $LBAD_DEFAULT

Default negative lemma regex for document parsing. Default = undef (none).

Methods

new

$filters = $CLASS_OR_OBJECT->new(%opts);

Returns a new DiaColloDB::Corpus::Filters object, which is a simple HASH-ref wrapping %opts:

##-- part-of-speech filters
pgood     => $re,    ##-- PoS whitelist regex
pgoodfile => $file,  ##-- PoS whitelist filename
pbad      => $re,    ##-- PoS blacklist regex
pbadfile  => $file,  ##-- PoS blacklist filename

##-- word surface text filters
wgood     => $re,    ##-- word whitelist regex
wgoodfile => $file,  ##-- word whitelist filename
wbad      => $re,    ##-- word blacklist regex
wbadfile  => $file,  ##-- word blacklkist filename (= "stopword list")

##-- lemma filters
lgood     => $re,    ##-- lemma whitelist regex
lgoodfile => $file,  ##-- lemma whitelist filename
lbad      => $re,    ##-- lemma blacklist regex
lbadfile  => $file,  ##-- lemma blacklist filename

See "Defaults" for the default values.

null

$filters = $CLASS_OR_OBJECT->null();

Returns a new DiaColloDB::Corpus::Filters object representing a "null-filter", i.e. with all filter properties undefined.

clear

$filters = $filters->clear();

Deletes all filter properties (white- and blacklist regexes and filenames) from the $filters object.

isnull

$bool = $filters->isnull();

Returns true iff $filters does not define any supported filter properties at all (i.e. application of $filters would be a no-op).

equal

$bool = $filters1->equal($filters2);
$bool = $CLASS->equal($filters1,$filters2)

Returns true iff filter object operands define the all and only the same supported filter properties with identical values.

compile

\%name2obj = $filters->compile();
\%name2obj = $CLASS->compile(\%filters);

Returns a HASH-ref of compiled filter regexes and (stop|go)-hashes of the form

${NAME}     => $REGEXP,
${NAME}file => \%HASHREF,

loadListFile

\%line2undef = $coldb->loadListFile($filename_or_undef);

Low-level utility method used to load (stop|go)-list files.

AUTHOR

Bryan Jurish <moocow@cpan.org>

COPYRIGHT AND LICENSE

Copyright (C) 2015-2020 by Bryan Jurish

This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.14.2 or, at your option, any later version of Perl 5 you may have available.

SEE ALSO

dcdb-corpus-compile.per(1), DiaColloDB::Corpus::Compiled(3pm), DiaColloDB(3pm), perl(1), ...