The Perl Toolchain Summit 2025 Needs You: You can help 🙏 Learn more

 / 
Text-TFIDF-Ngram-0.0509
River stage zero No dependents
/ Text::TFIDF::Ngram

NAME

Text::TFIDF::Ngram - Compute the TF-IDF measure for ngram phrases

VERSION

version 0.0509

SYNOPSIS


            

              
              
use Text::TFIDF::Ngram;
my $t = Text::TFIDF::Ngram->new(
  files => [qw( foo.txt bar.txt )],
  size  => 3,
);
my $w = $t->tf( 'foo.txt', 'foo bar baz' );
my $x = $t->idf('foo bar baz');
my $y = $t->tfidf( 'foo.txt', 'foo bar baz' );
printf "TF: %.3f, IDF: %.3f, TF-IDF: %.3f\n", $w, $x, $y;
my $z = $t->tfidf_by_file;
print Dumper $z;

DESCRIPTION

This module computes the TF-IDF ("term frequency - inverse document frequency") measure for a corpus of text documents.

This module will only work when given more than one document. Because the idf method is computed based on all documents, a single document in the given corpus will return 0.

ATTRIBUTES

files

ArrayRef of filenames.

size

Integer ngram phrase size.

Default: 1

stopwords

Boolean indicating that phrases with stopwords will be ignored.

Default: 1

punctuation

Regular expression to be used to parse-out unwanted punctuation. Giving the constructor a value of '' or 0 will override this and not exclude any characters from the results.

Default: qr/(?!')[[:punct:]]/

Note that the default does not exclude the single quote.

lowercase

Boolean to render the ngrams in lowercase.

Default: 0

counts

HashRef of the ngram counts of each processed file.

This is a computed attribute. Providing it to the constructor will be ignored.

file_tfidf

HashRef of the TF-IDF values in each processed file.

This is a computed attribute. Providing it to the constructor will be ignored.

METHODS

new


            

              
              
$t = Text::TFIDF::Ngram->new(
  files       => \@files,
  size        => $size,
  stopwords   => $stopwords,
  punctuation => $punctuation,
  lowercase   => $lowercase,
);

Create a new Text::TFIDF::Ngram object. If the files argument is passed in, the ngrams of each file are stored in counts.

tf


            

              
              
$tf = $t->tf( $file, $phrase );

Return the frequency of the given phrase in the document file. This is not the "raw count" of the phrase, but rather the percentage of times it is seen.

idf


            

              
              
$idf = $t->idf($phrase);

Return the inverse document frequency of a phrase across all corpus documents.

If the phrase is not in the corpus, a warning is issued and undef is returned.

tfidf


            

              
              
$tfidf = $t->tfidf( $file, $phrase );

Compute the TF-IDF weight for the given file and phrase.

tfidf_by_file


            

              
              
$tfidf = $t->tfidf_by_file;

Construct a HashRef of all files with all phrases and their tfidf values.

SEE ALSO

The eg/* and t/01-methods.t files in this distribution

https://en.wikipedia.org/wiki/Tf%E2%80%93idf

Lingua::EN::Ngram

Lingua::StopWords

List::Util

Moo

AUTHOR

Gene Boggs <gene@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2022 by Gene Boggs.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.