NAME

App::WebPerimeterScan::HTTP - minimal HTTP/1.1 client for perimeter scanning

VERSION

Version 0.001.0

SYNOPSIS

Fetch a page and check for an error first:

use App::WebPerimeterScan::HTTP;

my $http = App::WebPerimeterScan::HTTP->new;
my $res = $http->fetch('https://example.com/');

if($res->{error}) {
	print "failed: $res->{error}\n";
} else {
	print "$res->{status} $res->{reason}\n";
	print 'Server: ', $http->header($res, 'Server') // '(none)', "\n";
}

Read every Set-Cookie header separately:

for my $cookie (@{ $res->{set_cookie} || [] }) {
	print "cookie: $cookie\n";
}

Send an OPTIONS request with an extra header:

my $options = $http->fetch('https://example.com/',
	method  => 'OPTIONS',
	headers => [ 'Origin: https://evil.example' ],
);
print $http->header($options, 'Allow') // '', "\n";

Follow a redirect yourself, only when the target is in scope:

my $first = $http->fetch('http://example.com/');
my $next  = $http->header($first, 'Location');
if(defined $next && $scope->allows(URI->new($next)->host)) {
	my $second = $http->fetch($next);
}

DESCRIPTION

This is a small HTTP client made for the scanner. It sends one request and returns the response as a hash reference. It is written for this module, instead of using LWP or HTTP::Tiny, for two reasons:

The scanner must see every Set-Cookie header separately. Those modules join repeated headers with commas, and a cookie's Expires date also contains a comma, so the join cannot be undone.
The scanner must see redirects, not follow them. The caller then decides if the next host is in scope.

The client waits between two requests to the same host (the delay), so that a scan does not overload the server.

The client does not check TLS certificates. This is on purpose: a site with an expired or wrong certificate can still be scanned. The certificate is checked, and reported, by App::WebPerimeterScan::TLS. Do not use this client to send anything secret.

Network problems are returned, not thrown: when a request fails, the result has an error field and no status. Only a programming mistake (an unknown argument, or a method that is not made only of upper-case letters A to Z) is fatal.

REQUIREMENTS

IO::Socket::INET 1.31 and IO::Select 1.17. HTTPS requests also need IO::Socket::SSL 2.000, whose installation needs the OpenSSL headers.

SUBROUTINES/METHODS

new

Purpose: create a client.

Args: named timeout (seconds per request), delay (seconds between requests to one host), max_size (bytes of body kept), agent (User-Agent). Defaults come from %App::WebPerimeterScan::HTTP::config.

Returns: the new object.

Side Effects: none; no connection is made until fetch.

Usage: my $http = App::WebPerimeterScan::HTTP->new(timeout => 5);

EXAMPLE

my $http = App::WebPerimeterScan::HTTP->new(delay => 0);	# no throttle, for tests

API SPECIFICATION

INPUT

{
	timeout  => { type => 'number', min => 0, optional => 1 },
	delay    => { type => 'number', min => 0, optional => 1 },
	max_size => { type => 'integer', min => 1, optional => 1 },
	agent    => { type => 'string', min => 1, optional => 1 },
}

OUTPUT

{ type => 'object', isa => 'App::WebPerimeterScan::HTTP' }

MESSAGES

Message                          Meaning / resolution
-------------------------------  -----------------------------------
Parameter 'X' ... (fatal)        A value is negative or not a number.
Unknown parameter 'X' (fatal)    A misspelt argument name.

requests

Purpose: the number of requests actually sent (connections that opened).

Args: none.

Returns: a non-negative integer.

Side Effects: none.

Usage: printf "Requests: %d\n", $http->requests;

EXAMPLE

$http->fetch($url);
print $http->requests, "\n";

API SPECIFICATION

INPUT

{}

OUTPUT

{ type => 'integer', min => 0 }

MESSAGES

None.

fetch

Purpose: send one request and read the response.

Args: $url, then named method (an upper-case token, default GET) and headers (an array reference of Name: value strings; any containing CR or LF are dropped so they cannot inject headers).

Returns: a hash reference. On failure it has url and error (a translated message), plus host and port when the URL was usable. On success: url, host, port, scheme, status, reason, http_version, headers (lower-case name => array of values, in order), set_cookie, body, truncated, content_type, raw_headers and elapsed.

Side Effects: network traffic; may sleep to honour delay; increments requests when a connection opens.

Usage: my $res = $http->fetch($url, method => 'OPTIONS');

EXAMPLE

my $res = $http->fetch('https://example.com/',
	headers => [ 'Origin: https://ws-probe.invalid' ]);
if($res->{error}) {
	print "failed: $res->{error}\n";
} else {
	print "$res->{status} ", scalar @{ $res->{set_cookie} }, " cookie(s)\n";
}

API SPECIFICATION

INPUT

{
	url     => { type => 'string', min => 1, position => 0 },
	method  => { type => 'string', matches => qr/\A[A-Z]+\z/, optional => 1 },
	headers => { type => 'arrayref', optional => 1 },
}

OUTPUT

{ type => 'hashref' }	# see Returns; 'error' present on failure

MESSAGES

These are returned in error, not thrown:

Message                       Meaning / resolution
----------------------------  ---------------------------------------
invalid URL                   The URL did not parse.
unsupported scheme 'S'        Only http and https are fetched.
URL has no host               The URL has no authority part.
connect failed: E             TCP or TLS connect failed; E says why.
request write failed          The connection closed or timed out
                              while sending.
no response headers           Nothing usable arrived before timeout.
malformed status line         The reply was not HTTP/1.x.

PSEUDOCODE

parse the URL; return an error for anything but http(s) with a host
wait until the per-host delay has passed
connect (TLS without verification for https); return an error on failure
write the request line, Host, User-Agent, fixed and extra headers
read until the blank line; parse the status line and headers
read the body by Content-Length, chunked encoding or to EOF
return the parsed response

Purpose: the last value of a response header, without dealing in array references.

Args: $response (from fetch) and $name (any case).

Returns: the value, or undef if the header is absent.

Side Effects: none.

Usage: my $server = $http->header($res, 'Server');

EXAMPLE

my $type = $http->header($res, 'Content-Type') // 'none';

API SPECIFICATION

INPUT

{
	response => { type => 'hashref', position => 0 },
	name     => { type => 'string', min => 1, position => 1 },
}

OUTPUT

{ type => 'string', optional => 1 }

MESSAGES

None.

COMMON PITFALLS

Check error before status. A failed request (no connection, a timeout, a reply that is not HTTP) returns a hash with error and no status. It does not die.
Redirects are not followed. A 301 or 302 is returned as it is. Read the Location header and decide yourself.
The body may be cut short. Only the first max_size bytes (2,000,000 by default) are kept. truncated is true when the rest was thrown away.
header returns only the last value. When a header appears more than once, header($res, $name) returns the last one. The full list is in $res->{headers}{$name}, where $name must be lower case. header itself ignores case.
header returns undef when the header is missing, and also for a failed request. Write $http->header($res, 'Server') // '' when you need a string.
The delay belongs to one object. Two HTTP objects do not wait for each other. Use one object for the whole scan.
A delay of 0 turns the wait off. Use it in tests only. Do not use it against a real site.
requests counts connections that opened. A request that failed before it connected is not counted.
The certificate is not checked. See "DESCRIPTION".

ENCODING

URLs must be ASCII. Percent-encode non-ASCII characters in the path and query (%C3%A9 for é in UTF-8), and write an international host name in its Punycode form. A URL that contains raw UTF-8 bytes is encoded a second time and asks for the wrong path.
The body is bytes. body is exactly what the server sent, after removing chunked transfer encoding. It is not decoded from its character set, and a compressed body is not uncompressed (the client does not ask for compression). To read the text, decode it yourself with the charset in content_type.
Header values are bytes. Most are ASCII. A server that puts UTF-8 in a header value gives you the UTF-8 bytes.
Extra request headers are sent as given. Pass them as ASCII byte strings. A header that contains a carriage return or a line feed is dropped, so it cannot add another header.
error is a message for people. It comes from the message catalogue and may contain the operating system's text in the language of the current locale. Do not compare it with a fixed string; test only whether it is true.

LIMITATIONS

HTTP/1.1 only, one request per connection, no redirects, no compression, no proxy support. An interim 100 Continue response is returned as if it were the final one.
The host name is resolved again for each connection; see "LIMITATIONS" in App::WebPerimeterScan::Scope for why that matters.
An IPv6 literal host is passed to the socket layer as given but the Host header does not bracket it.

FORMAL SPECIFICATION

This section states exactly what each method does, in the Z notation. You do not need it to use the module: the method descriptions above say the same thing in words. It is here for reviewers and for anyone who writes tests.

How to read a schema box:

The part above the middle line declares names. A name that ends in ? is an input. A name that ends in ! is an output.
The part below the middle line lists facts that are true after the call.
A name with a dash after it, such as items', is the value after the call. The same name without the dash is the value before the call.
Delta X (written with the Greek capital delta) means the call may change the state X. Xi X (the Greek capital xi) means the call does not change it.

The symbols used are:

Δ   may change            Ξ   does not change      •   such that
∀   for all               ∃   there is             ¬   not
∧   and                   ∨   or                   ⇒   implies
⇔   if and only if        ≠   not equal            ≤ ≥ at most, at least
∈   is in                 ¬in;   is not in            ⊆   is a subset of
∪   union                 ∩   intersection         ∅   the empty set
ℙ X sets of X             ⊥   undefined (undef)    …   and so on
⟨⟩  the empty sequence    ⁀   join sequences       ⇸   partial mapping
↦   maps to               ⊕   override a mapping   ↾   keep only these keys
∘   compose               →   function             × ÷ − times, divide, minus
ℕ   0, 1, 2, ...          ℕ₁  1, 2, 3, ...         ℝ   real numbers
𝔹   true or false         ⌊x⌋ x rounded down
dom M  the keys of M      ran M  the values of M   #S  the size of S

new

┌─ InitHTTP ──────────────────────────────────────────
│ HTTP'; timeout?, delay? : ℝ; max_size? : ℕ₁
├─────────────────────────────────────────────────────
│ timeout? ≥ 0 ∧ delay? ≥ 0
│ requests' = 0 ∧ last' = ∅
└─────────────────────────────────────────────────────

requests

┌─ Requests ──────────────────────────────────────────
│ ΞHTTP; result! : ℕ
├─────────────────────────────────────────────────────
│ result! = requests
└─────────────────────────────────────────────────────

fetch

┌─ Fetch ─────────────────────────────────────────────
│ ΔHTTP; url? : URL; method? : TOKEN; res! : RESPONSE
├─────────────────────────────────────────────────────
│ scheme(url?) ∈ {http, https} ∧ host(url?) ≠ ∅
│ now − last(host(url?)) ≥ delay        (after any sleep)
│ connected ⇒ requests' = requests + 1
│ ¬ connected ⇒ requests' = requests ∧ error ∈ dom res!
│ #res!.body ≤ max_size
└─────────────────────────────────────────────────────

header

┌─ Header ────────────────────────────────────────────
│ response? : RESPONSE; name? : STRING; result! : STRING ∪ {⊥}
├─────────────────────────────────────────────────────
│ n = lower(name?)
│ n ∈ dom response?.headers ⇒ result! = last(response?.headers(n))
│ n ¬in; dom response?.headers ⇒ result! = ⊥
└─────────────────────────────────────────────────────

STATE DIAGRAM

The object keeps two pieces of state: requests, the number of connections that opened, and the time of the last request to each host. Each call to fetch goes through the steps below. A step that fails ends the call with a result that has an error field.

  new(timeout, delay, max_size)
        |
        v
  +-------------+
  |    READY    |  requests = 0 at first; no host seen yet
  +-------------+
        |
        |  fetch(url, method, headers)
        v
  +-------------+
  |  VALIDATE   |--- bad method or unknown argument
  +-------------+       -> croak; back to READY, no change
        |
        |---------- URL not http or https, or it has no host
        |              -> result {error}; back to READY
        v
  +-------------+
  |    WAIT     |  sleep until 'delay' seconds have passed since
  +-------------+  the last request to this host; then record the
        |          time of this request for the host
        v
  +-------------+
  |   CONNECT   |--- refused, timeout, or TLS handshake failed
  +-------------+       -> result {error}; back to READY
        |
        |  connected: requests = requests + 1
        v
  +-------------+
  |    SEND     |--- write failed or timed out
  +-------------+       -> result {error}; back to READY
        |
        |  request line, Host, User-Agent and extra headers sent
        v
  +-------------+
  |  READ HEAD  |--- no blank line within 64 KiB or before the
  +-------------+    deadline, or a status line that is not HTTP
        |               -> result {error}; back to READY
        v
  +-------------+
  |  READ BODY  |  by Content-Length, by chunks, or to the end of
  +-------------+  the stream; stop at max_size (truncated = 1);
        |          no body for HEAD, 1xx, 204 or 304
        v
  result {status, headers, body, ...}; back to READY

The connection is closed before fetch returns, on every path after
CONNECT.

AUTHOR

Nigel Horne, <njh at nigelhorne.com>

LICENCE AND COPYRIGHT

Usage is subject to the GPL2 licence terms. If you use it, please let me know.

To install App::WebPerimeterScan, copy and paste the appropriate command in to your terminal.

cpanm

cpanm App::WebPerimeterScan

CPAN shell

perl -MCPAN -e shell
install App::WebPerimeterScan

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

VERSION

SYNOPSIS

DESCRIPTION

REQUIREMENTS

SUBROUTINES/METHODS

new

EXAMPLE

API SPECIFICATION

INPUT

OUTPUT

MESSAGES

requests

EXAMPLE

API SPECIFICATION

INPUT

OUTPUT

MESSAGES

fetch

EXAMPLE

API SPECIFICATION

INPUT

OUTPUT

MESSAGES

PSEUDOCODE

header

EXAMPLE

API SPECIFICATION

INPUT

OUTPUT

MESSAGES

COMMON PITFALLS

ENCODING

LIMITATIONS

FORMAL SPECIFICATION

new

requests

fetch

header

STATE DIAGRAM

AUTHOR

LICENCE AND COPYRIGHT

Module Install Instructions

Keyboard Shortcuts