/ 
libwww-perl-5.21
/ LWP::RobotUA
Security Advisories (4)
CVE-2010-2253 (2010-07-06)

lwp-download in libwww-perl before 5.835 does not reject downloads to filenames that begin with a . (dot) character, which allows remote servers to create or overwrite files via (1) a 3xx redirect to a URL with a crafted filename or (2) a Content-Disposition header that suggests a crafted filename, and possibly execute arbitrary code as a consequence of writing to a dotfile in a home directory.

CPANSA-libwww-perl-2001-01 (2001-03-14)

If LWP::UserAgent::env_proxy is called in a CGI environment, the case-insensitivity when looking for "http_proxy" permits "HTTP_PROXY" to be found, but this can be trivially set by the web client using the "Proxy:" header.

CVE-2011-0633 (2011-01-20)

The Net::HTTPS module in libwww-perl (LWP) before 6.00, as used in WWW::Mechanize, LWP::UserAgent, and other products, when running in environments that do not set the If-SSL-Cert-Subject header, does not enable full validation of SSL certificates by default, which allows remote attackers to spoof servers via man-in-the-middle (MITM) attacks involving hostnames that are not properly validated.

CPANSA-libwww-perl-2017-01 (2017-11-06)

LWP::Protocol::file can open existent file from file:// scheme. However, current version of LWP uses open FILEHANDLE,EXPR and it has ability to execute arbitrary command

NAME

LWP::RobotUA - A class for Web Robots

SYNOPSIS

require LWP::RobotUA;
$ua = new LWP::RobotUA 'my-robot/0.1', 'me@foo.com';
$ua->delay(10);  # be very nice, go slowly
...
# just use it just like a normal LWP::UserAgent
$res = $ua->request($req);

DESCRIPTION

This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the robots.txt file to ensure that they are welcomed and they should not send too frequent requests.

But, before you consider writing a robot take a look at <URL:http://info.webcrawler.com/mak/projects/robots/robots.html>.

When you use a LWP::RobotUA as your user agent, then you do not really have to think about these things yourself. Just send requests as you do when you are using a normal LWP::UserAgent and this special agent will make sure you are nice.

METHODS

The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the same methods. In addition the following methods are provided:

$ua = LWP::RobotUA->new($agent_name, $from, [$rules])

Your robot's name and the mail address of the human responsible for the robot (i.e. you) is required by the constructor.

Optionally it allows you to specify the WWW::RobotRules object to use.

$ua->delay([$minutes])

Set the minimum delay between requests to the same server. The default is 1 minute.

$ua->use_sleep([$boolean])

Get/set a value indicating wether the UA should sleep() if request arrive to fast (before $ua->delay minutes has passed). The default is TRUE. If this value is FALSE then an internal SERVICE_UNAVAILABLE response will be generated. It will have an Retry-After header that indicate when it is OK to send another request to this server.

$ua->rules([$rules])

Set/get which WWW::RobotRules object to use.

$ua->no_visits($netloc)

Returns the number of documents fetched from this server host. Yes I know, this method should probably have been named num_visits() or something like that :-(

$ua->host_wait($netloc)

Returns the number of seconds (from now) you must wait before you can make a new request to this host.

$ua->as_string

Returns a text that describe the state of the UA. Mainly useful for debugging.

SEE ALSO

LWP::UserAgent, WWW::RobotRules

COPYRIGHT

Copyright 1996-1997 Gisle Aas.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.