NAME
"Text::Corpus::Summaries::Wikipedia" - Creates corpora for summarization
testing.
SYNOPSIS
use Text::Corpus::Summaries::Wikipedia;
use Data::Dump qw(dump);
my $corpus = Text::Corpus::Summaries::Wikipedia->new;
$corpus->create;
dump $corpus->getListOfXmlFiles;
DESCRIPTION
"Text::Corpus::Summaries::Wikipedia" creates corpora for single document
summarization testing using the featured articles of various Wikipedias.
A criterion for an article in a Wikipedia to be *featured* is that it
have a well written lead section, or introduction. So the featured
articles of a Wikipedia can make an excellent corpus for testing single
document summarization methods. This module creates a corpus from the
featured articles of a Wikipedia by fetching and saving their content as
HTML, text, and XML, with the appropriate sections labeled as either
"summary" or "body".
CONSTRUCTOR
"new"
The constructor "new" creates an instance of the
"Text::Corpus::Summaries::Wikipedia" class with the following
parameters:
"languageCode"
languageCode => 'en'
"languageCode" is the language code of the Wikipedia from which the
corpus of featured articles are to be created. The supported
language codes are "af":Afrikaans, "ar":Arabic, "az":Azerbaijani,
"bg":Bulgarian, "bs":Bosnian, "ca":Catalan, "cs":Czech, "de":German,
"el":Greek, "en":English, "eo":Esperanto, "es":Spanish, "eu":Basque,
"fa":Persian, "fi":Finnish, "fr":French, "he":Hebrew, "hr":Croatian,
"hu":Hungarian, "id":Indonesian, "it":Italian, "ja":Japanese,
"jv":Javanese, "ka":Georgian, "kk":Kazakh, "km":Khmer, "ko":Korean,
"li":Limburgish, "lv":Latvian, "ml":Malayalam, "mr":Marathi,
"ms":Malay, "mzn":Mazandarani, "nl":Dutch, "nn":Norwegian (Nynorsk),
"no":Norwegian (Bokm?l), "pl":Polish, "pt":Portuguese,
"ro":Romanian, "ru":Russian, "sh":Serbo-Croatian, "simple":Simple
English, "sk":Slovak, "sl":Slovenian, "sr":Serbian, "sv":Swedish,
"sw":Swahili, "ta":Tamil, "th":Thai, "tl":Tagalog, "tr":Turkish,
"tt":Tatar, "uk":Ukrainian, "ur":Urdu, "vi":Vietnamese,
"vo":Volap?k, and "zh":Chinese. If the language code is "all", then
the corpus for each supported language is created (which takes a
long time). The default is "en".
"corpusDirectory"
corpusDirectory => 'cwd'
"corpusDirectory" is the corpus directory that the summaries and
articles will be stored in; the directory is created if it does not
exist. The default is the "cwd".
A language subdirectory is created at "corpusDirectory/languageCode"
that will contain the directories "log", "html", "unparsable",
"text", and "xml". The directory "log" will contain the file
"log.txt" that all errors, warnings, and informational messages are
logged to using Log::Log4perl. The directory "html" will contain
copies of the HTML versions of the featured article pages fetched
using LWP. The directory "text" will contain two files for each
article; one file will end with "_body.txt" and contain the body
text of the article, the other will end with "_summary.txt" and will
contain the summary. The directory "unparsable" will contain the
HTML files that could not be parsed into *body* and *summary*
sections. All files are UTF-8 encoded.
METHODS
"create"
The method "create" fetches the featured articles and creates the text
and XML versions of the files.
use Text::Corpus::Summaries::Wikipedia;
use Data::Dump qw(dump);
my $corpus = Text::Corpus::Summaries::Wikipedia->new;
$corpus->create;
dump $corpus->getListOfXmlFiles;
dump $corpus->getListOfTextFiles;
"maxProcesses"
maxProcesses => 1
"maxProcesses" is the maximum number of processes that can be
running simultaneously to parse the files. Parsing the files for the
summary and body sections may be computational intensive so the
module Forks::Super is used for parallelization. The default is one.
"test"
test => 0
If "test" is a positive integer than it will be treated as the
maximum number of pages that may be fetched and parsed. The default
is zero, meaning all possible pages are fetched and parsed.
"recreate"
The method "recreate" recreates the text and XML versions of the files
from the list of previously fetched HTML files in the "html" directory.
use Text::Corpus::Summaries::Wikipedia;
use Data::Dump qw(dump);
my $corpus = Text::Corpus::Summaries::Wikipedia->new;
$corpus->recreate;
dump $corpus->getListOfXmlFiles;
dump $corpus->getListOfTextFiles;
"maxProcesses"
maxProcesses => 1
"maxProcesses" is the maximum number of processes that can be
running simultaneously to parse the files. Parsing the files for the
summary and body sections may be computational intensive so the
module Forks::Super is used for parallelization. The default is one.
"test"
test => 0
If "test" is a positive integer than it will be treated as the
maximum number of pages that may be parsed. The default is zero,
meaning all possible pages are parsed.
"getListOfTextFiles"
The method "getListOfTextFiles" returns an array reference with each
item in the list having the form "{body => 'path_to_body_file', summary
=> 'path_to_summary_file'}".
use Text::Corpus::Summaries::Wikipedia;
use Data::Dump qw(dump);
my $corpus = Text::Corpus::Summaries::Wikipedia->new;
$corpus->create;
dump $corpus->getListOfTextFiles;
"getListOfXmlFiles"
The method "getListOfXmlFiles" returns an array reference containing the
path to each XML file.
use Text::Corpus::Summaries::Wikipedia;
use XML::Simple;
use Data::Dump qw(dump);
my $corpus = Text::Corpus::Summaries::Wikipedia->new;
foreach my $xmlFile (@{$corpus->getListOfXmlFiles})
{
my $article;
eval { $article = XMLin ($xmlFile) };
if ($@) { dump \$@; } else { dump $article; }
}
XML FORMAT
The XML files are created using XML::Code with the simple structure
outlined below:
<document>
<title>The Article Title</title>
<summary>
<p>This is the first paragraph of the summary.</p>
<p>This is the second paragraph of the summary.</p>
</summary>
<body>
<section>
<header>Head of first section</header>
<p>First paragraph of this section.</p>
<p>Second paragraph of this section.</p>
</section>
<section>
<p>First paragraph of this section.</p>
<p>Second paragraph of this section.</p>
<section>
<header>Head of sub-section</header>
<p>First paragraph of this sub-section.</p>
<p>Second paragraph of this sub-section.</p>
</section>
</section>
</body>
</document>
EXAMPLE
The following example computes and prints the median, mean, and standard
deviation of the fraction of words (ignoring repeats) in a summary that
also occur in the body of the text for all the articles in the corpora.
use Text::Corpus::Summaries::Wikipedia;
use Statistics::Descriptive;
use File::Slurp;
use Encode;
my $corpus = Text::Corpus::Summaries::Wikipedia->new;
my $statistics = Statistics::Descriptive::Full->new;
foreach my $textFilePair (@{$corpus->getListOfTextFiles})
{
my $summary = lc decode ('utf8', read_file ($textFilePair->{summary}, binmode => ':raw'));
my %summaryWords = map {($_, 1)} split (/\P{Letter}/, $summary);
my $totalUniqueSummaryWords = keys %summaryWords;
next unless $totalUniqueSummaryWords;
my $body = lc decode ('utf8', read_file ($textFilePair->{body}, binmode => ':raw'));
map {delete $summaryWords{$_}} split (/\P{Letter}/, $body);
my $totalUniqueSummaryWordsNotInBody = keys %summaryWords;
$statistics->add_data (1 - $totalUniqueSummaryWordsNotInBody / $totalUniqueSummaryWords);
}
print 'count: ', $statistics->count(), "\n";
print 'median: ', $statistics->median(), "\n";
print 'mean: ', $statistics->mean(), "\n";
print 'standard deviation: ', $statistics->standard_deviation(), "\n";
SCRIPTS
The script create_summary_corpus.pl makes a corpus for summarization
testing using this module.
INSTALLATION
Use CPAN to install the module and all its prerequisites:
perl -MCPAN -e shell
>install Text::Corpus::Summaries::Wikipedia
BUGS
This module creates corpora by parsing Wikipedia pages, the xpath
expressions used to extract links and text will become invalid as the
format of the various pages changes, causing some corpora not to be
created.
Please email bugs reports or feature requests to
"bug-text-corpus-summaries-wikipedia@rt.cpan.org", or through the web
interface at
kipedia>. The author will be notified and you can be automatically
notified of progress on the bug fix or feature request.
AUTHOR
Jeff Kubina<jeff.kubina@gmail.com>
COPYRIGHT
Copyright (c) 2010 Jeff Kubina. All rights reserved. This program is
free software; you can redistribute it and/or modify it under the same
terms as Perl itself.
The full text of the license can be found in the LICENSE file included
with this module.
KEYWORDS
corpus, information processing, summaries, summarization, wikipedia
SEE ALSO
create_summary_corpus.pl, Encode, Forks::Super,
HTML::TreeBuilder::XPath, Log::Log4perl, LWP::UserAgent, XML::Code