/ 
Text-TFIDF-Ngram-0.0500
River stage zero No dependents
/ Text::TFIDF::Ngram

NAME

Text::TFIDF::Ngram - Compute the TF-IDF measure for ngram phrases

VERSION

version 0.0500

SYNOPSIS

use Text::TFIDF::Ngram;
my $obj = Text::TFIDF::Ngram->new(
  files => [qw( foo.txt bar.txt )],
  size  => 3,
);
my $w = $obj->tf( 'foo.txt', 'foo bar baz' );
my $x = $obj->idf('foo bar baz');
my $y = $obj->tfidf( 'foo.txt', 'foo bar baz' );
printf "TF: %.3f, IDF: %.3f, TFIDF: %.3f\n", $w, $x, $y;
my $z = $obj->tfidf_by_file;
print Dumper $z;

DESCRIPTION

This module computes the TF-IDF ("term frequency-inverse document frequency") measure for a corpus of text documents.

For a working example program, please see the eg/analyze file in this distribution.

ATTRIBUTES

files

ArrayRef of filenames to use in the ngram processing.

size

Integer ngram phrase size. Default is 1.

stopwords

Boolean indicating that phrases with stopwords will be ignored. Default is 1.

punctuation

Regular expression to be used to parse-out unwanted punctuation. Giving the constructor a value of '' or 0 will override this and not exclude any characters from the results.

Default: qr/[-!"#$%&()*+,.\/\\:;<=>?@\[\]^_`{|}~]/

Note that the default does not exclude the single quote.

lowercase

Boolean to render the ngrams in lowercase. Default is 0.

counts

HashRef of the ngram counts of each processed file. This is a computed attribute - providing it in the constructor will be ignored.

file_tfidf

HashRef of the TF-IDF values in each processed file. This is a computed attribute - providing it in the constructor will be ignored.

METHODS

new

$obj = Text::TFIDF::Ngram->new(
  files       => \@files,
  size        => $size,
  stopwords   => $stopwords,
  punctuation => $punctuation,
  lowercase   => $lowercase,
  digits      => $digits,
);

Create a new Text::TFIDF::Ngram object. If the files argument is passed in, the ngrams of each file are stored in the counts.

BUILD

Load the given file phrase counts.

tf

$tf = $obj->tf( $file, $phrase );

Returns the frequency of the given phrase in the document file. This is not the "raw count" of the phrase, but rather the percentage of times it is seen.

idf

$idf = $obj->idf($phrase);

Returns the inverse document frequency of a phrase.

tfidf

$tfidf = $obj->tfidf( $file, $phrase );

Computes the TF-IDF weight for the given file and phrase. If the phrase is not in the corpus, a warning is issued and undef is returned.

tfidf_by_file()

$tfidf = $obj->tfidf_by_file;

Construct a HashRef of all files with all phrases and their tfidf values.

SEE ALSO

The eg/analyze file in this distribution

https://en.wikipedia.org/wiki/Tf%E2%80%93idf

Lingua::EN::Ngram

Lingua::StopWords

List::Util

Moo

AUTHOR

Gene Boggs <gene@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2019 by Gene Boggs.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.