/ 
App-WebPerimeterScan-0.001.0
/ App::WebPerimeterScan::Scope

NAME

App::WebPerimeterScan::Scope - authorisation boundary for an external scan

VERSION

Version 0.001.0

SYNOPSIS

Load a scope file and stop at once if a host may not be scanned:

use App::WebPerimeterScan::Scope;

my $scope = App::WebPerimeterScan::Scope->new;
$scope->load_file('scope.txt')->assert_target_allowed('www.example.com');

Ask before following a link, without stopping:

for my $link ('https://api.example.com/', 'https://other.example.net/') {
	my $host = URI->new($link)->host;
	next unless $scope->allows($host);
	print "may visit $link\n";
}

Scan a test server on your own machine. Loopback and private addresses are refused unless you say so:

my $lab = App::WebPerimeterScan::Scope->new(allow_private => 1);
$lab->load_file('lab-scope.txt');
print "allowed\n" if $lab->allows('127.0.0.1');

List the hosts that will be scanned:

for my $host (@{ $scope->hosts }) {
	print "$host\n";
}

DESCRIPTION

This module decides which hosts the scanner may contact. The scanner asks it before every network request.

A host may be scanned only when both of these are true:

1. The host is listed in the scope file (or matches a wildcard line in it).

The scope file

The scope file is plain text, one entry per line. Each line is one of:

example.com                 a host name
example.com:8443            a host name and a port
203.0.113.10                an IPv4 address
https://shop.example.com/   a URL: the scheme, host and any port are used
*.example.com               a wildcard: example.com and every name below it

Blank lines are ignored. A # starts a comment, which runs to the end of the line. Host names are compared without regard to case, and a final dot (example.com.) is removed. International names may be written as they are, in UTF-8; see "ENCODING". A line that cannot be understood is skipped with a warning. A file with no usable line at all is an error.

REQUIREMENTS

URI 1.70 and Data::Validate::IP 0.31, which classifies addresses against the IANA special-purpose registries for both IPv4 and IPv6. Addresses are resolved with Socket 2.020's getaddrinfo.

SUBROUTINES/METHODS

new

Purpose: create an empty scope.

Args: named allow_private (boolean, default 0) and source (string, the name to quote in refusals, default (none)).

Returns: the new object.

Side Effects: none.

Usage: my $scope = App::WebPerimeterScan::Scope->new(allow_private => 1);

EXAMPLE

my $scope = App::WebPerimeterScan::Scope->new;
$scope->load_file('scope.txt');

API SPECIFICATION

INPUT

{
	allow_private => { type => 'boolean', optional => 1, default => 0 },
	source        => { type => 'string', optional => 1 },
}

OUTPUT

{ type => 'object', isa => 'App::WebPerimeterScan::Scope' }

MESSAGES

Message                     Meaning / resolution
--------------------------  -------------------------------------
Unknown parameter 'X'       A misspelt argument name.

load_file

Purpose: read a scope file.

Args: $file, the path.

Returns: $self, for chaining.

Side Effects: appends the file's entries; records $file as the source quoted in refusals. Lines that cannot be parsed are skipped with a warning.

Usage: $scope->load_file('scope.txt')

Accepted lines, one per line, # starts a comment:

example.com
example.com:8443
https://example.com/
*.example.com            (the domain and any sub-domain)
192.0.2.10

EXAMPLE

my $scope = eval { App::WebPerimeterScan::Scope->new->load_file($path) }
	or die App::WebPerimeterScan::I18N::clean_error($@), "\n";

API SPECIFICATION

INPUT

{ file => { type => 'string', min => 1, position => 0 } }

OUTPUT

{ type => 'object', isa => 'App::WebPerimeterScan::Scope' }

MESSAGES

Message                                     Meaning / resolution
------------------------------------------  ------------------------------
Cannot read scope file 'F': E (fatal)       F is missing or unreadable; E
                                            is the operating system's text.
Ignoring unparsable scope line: L (warning) L is not a host, host:port,
                                            URL or *.domain; fix the line.
Scope file 'F' contains no usable hosts     Every line was blank, a
(fatal)                                     comment or unparsable.

PSEUDOCODE

open the file, or croak with the operating system's reason
for each line
	strip comments and surrounding white space; skip if empty
	parse it as a URL, a wildcard or host[:port]
	warn and skip if it does not parse
croak if no line produced an entry
remember the file name for refusal messages

entries

Purpose: every parsed entry, wildcards included.

Args: none.

Returns: an array reference of hash references with host and, where given, port, scheme and wildcard.

Side Effects: none.

Usage: for my $entry (@{ $scope->entries }) { ... }

EXAMPLE

my @wildcards = grep { $_->{wildcard} } @{ $scope->entries };

API SPECIFICATION

INPUT

{}

OUTPUT

{ type => 'arrayref', element_type => 'hashref' }

MESSAGES

None.

source

Purpose: where the scope came from, for messages and reports.

Args: none.

Returns: the file name given to load_file, or the source given to new, or (none).

Side Effects: none.

Usage: print 'Scope: ', $scope->source, "\n";

EXAMPLE

printf "Loaded %s\n", $scope->source;

API SPECIFICATION

INPUT

{}

OUTPUT

{ type => 'string' }

MESSAGES

None.

hosts

Purpose: the concrete hosts to scan.

Args: none.

Returns: an array reference of entries, wildcards removed and the first entry for each host kept. A wildcard still authorises its sub-domains, but sub-domain names are not guessed: only links the site serves are followed.

Side Effects: none.

Usage: for my $entry (@{ $scope->hosts }) { ... }

EXAMPLE

print "$_->{host}\n" for @{ $scope->hosts };

API SPECIFICATION

INPUT

{}

OUTPUT

{ type => 'arrayref', element_type => 'hashref' }

MESSAGES

None.

covers

Purpose: say whether a hostname is inside the authorised scope.

Args: $host (string). Case and a trailing dot are ignored.

Returns: 1 or 0.

Side Effects: none; this does not resolve the name.

Usage: next unless $scope->covers($uri->host);

EXAMPLE

# with "*.example.com" in the scope file
$scope->covers('api.example.com');	# 1
$scope->covers('notexample.com');	# 0, the match is on a label boundary

API SPECIFICATION

INPUT

{ host => { type => 'string', optional => 1, position => 0 } }

OUTPUT

{ type => 'boolean' }

MESSAGES

None.

is_private_target

Purpose: say whether a host resolves into address space that is not publicly routable.

Args: $host, a hostname or an IPv4 or IPv6 literal.

Returns: 1 if any address the name resolves to is loopback, private (RFC 1918, unique-local), link-local, shared (CGNAT), documentation, benchmarking, multicast, reserved or otherwise non-public; 0 if every address is public or the name does not resolve.

Side Effects: a DNS lookup.

Usage: warn "private\n" if $scope->is_private_target($host);

Checking every address matters: a name with one public and one internal A record would otherwise be judged on whichever the resolver listed first.

EXAMPLE

$scope->is_private_target('127.0.0.1');		# 1
$scope->is_private_target('::1');		# 1
$scope->is_private_target('192.30.252.1');	# 0

API SPECIFICATION

INPUT

{ host => { type => 'string', min => 1, position => 0 } }

OUTPUT

{ type => 'boolean' }

MESSAGES

None.

allows

Purpose: the full authorisation test for one host: in scope, and not in private address space unless that was permitted.

Args: $host (string).

Returns: 1 or 0.

Side Effects: a DNS lookup when the host is covered and allow_private is off.

Usage: next unless $scope->allows($uri->host);

This is what the crawler asks before following a link: a wildcard entry covers names nobody listed by hand, and one of them resolving to an internal address must be refused like a listed one would be.

EXAMPLE

my @fetchable = grep { $scope->allows(URI->new($_)->host) } @links;

API SPECIFICATION

INPUT

{ host => { type => 'string', optional => 1, position => 0 } }

OUTPUT

{ type => 'boolean' }

MESSAGES

None.

assert_target_allowed

Purpose: refuse, loudly, to scan a host that allows would reject.

Args: $host (string).

Returns: $self, for chaining.

Side Effects: croaks when the host is out of scope or private.

Usage: $scope->assert_target_allowed($host);

EXAMPLE

eval { $scope->assert_target_allowed('evil.example') };
print App::WebPerimeterScan::I18N::clean_error($@), "\n" if $@;

API SPECIFICATION

INPUT

{ host => { type => 'string', min => 1, position => 0 } }

OUTPUT

{ type => 'object', isa => 'App::WebPerimeterScan::Scope' }

MESSAGES

Message                                       Meaning / resolution
--------------------------------------------  ----------------------------
Refusing to scan 'H': not listed in F         Add H to the scope file F
(fatal)                                       if you are authorised to.
Refusing to scan 'H': it resolves into        H is loopback/private; pass
private address space (fatal)                 --allow-private if intended.

COMMON PITFALLS

  • A port in the scope file does not limit the ports. example.com:8443 tells the scanner where the web server is. It does not stop covers or allows from accepting example.com on any other port, and the port scan still checks every port in its list.

  • A wildcard is not scanned by itself. *.example.com lets the scanner follow links to shop.example.com, but hosts does not return it, so it is not a starting point. A file that has only wildcard lines loads without error, but there is nothing to scan, and App::WebPerimeterScan->run stops with an error.

  • A wildcard also covers the name itself, and every depth below it. *.example.com covers example.com, a.example.com and a.b.example.com. It does not cover notexample.com.

  • Names and addresses are separate entries. Listing example.com does not allow 203.0.113.10, even if that is its address, and listing the address does not allow the name.

  • load_file adds; it does not replace. Loading a second file keeps the entries of the first. source then names only the second file. Use a new object for a different scope.

  • undef and the empty string are never covered. covers(undef), allows(undef) and is_private_target(undef) return 0. For is_private_target, 0 means "no private address was found", so do not use it alone as a safety check: use allows, which also needs the host to be listed.

  • A name that does not resolve is not private. is_private_target returns 0 for it. The scan of that host then fails to connect and is reported as an error; nothing is sent.

  • The answer can change. A name is resolved every time it is checked, and its DNS answer can change between the check and the connection. See "LIMITATIONS".

ENCODING

  • Save the scope file as UTF-8. A byte-order mark at the start is not supported. Comments may contain any text.

  • International host names can be written as they are. A name such as bücher.example is stored in its ASCII (Punycode) form, xn--bcher-kva.example, which is the form DNS uses. This works in all three kinds of line: bücher.example, *.bücher.example and https://bücher.example/. Writing the Punycode form yourself gives the same entry.

    entries and hosts return the Punycode form, so that is what the reports show.

  • A file that is not UTF-8 is read as Latin-1 (ISO 8859-1), one character per byte. For Western European names this gives the right answer; for other encodings, save the file as UTF-8.

  • The checks accept any form of a name. covers, allows, is_private_target and assert_target_allowed accept a name in Punycode, as UTF-8 bytes or as a Perl character string, in any case.

  • The file name passed to load_file is used as Perl bytes, as the operating system gives them. A name with non-ASCII characters works when it is passed exactly as it appears on disk.

  • Error messages contain the operating system's text, in the language of the current locale, exactly as Perl reports it in $!.

LIMITATIONS

  • The private-address check and the connection resolve the name separately. A name whose DNS answer changes between the two (DNS rebinding) can pass the check and then connect to an internal address. Closing that needs every connection to be made to the address that was checked, with the name carried only in SNI and the Host header; the HTTP, TLS and port-scan layers do not do that yet.

  • Scope lines cannot name an IPv6 literal such as [2001:db8::1], and *.example.com covers example.com itself as well as its sub-domains.

  • The port scan resolves with inet_aton and so only ever connects over IPv4, even where the private-address check considered IPv6 addresses.

FORMAL SPECIFICATION

This section states exactly what each method does, in the Z notation. You do not need it to use the module: the method descriptions above say the same thing in words. It is here for reviewers and for anyone who writes tests.

How to read a schema box:

  • The part above the middle line declares names. A name that ends in ? is an input. A name that ends in ! is an output.

  • The part below the middle line lists facts that are true after the call.

  • A name with a dash after it, such as items', is the value after the call. The same name without the dash is the value before the call.

  • Delta X (written with the Greek capital delta) means the call may change the state X. Xi X (the Greek capital xi) means the call does not change it.

The symbols used are:

Δ   may change            Ξ   does not change      •   such that
∀   for all               ∃   there is             ¬   not
∧   and                   ∨   or                   ⇒   implies
⇔   if and only if        ≠   not equal            ≤ ≥ at most, at least
∈   is in                 ¬in;   is not in            ⊆   is a subset of
∪   union                 ∩   intersection         ∅   the empty set
ℙ X sets of X             ⊥   undefined (undef)    …   and so on
⟨⟩  the empty sequence    ⁀   join sequences       ⇸   partial mapping
↦   maps to               ⊕   override a mapping   ↾   keep only these keys
∘   compose               →   function             × ÷ − times, divide, minus
ℕ   0, 1, 2, ...          ℕ₁  1, 2, 3, ...         ℝ   real numbers
𝔹   true or false         ⌊x⌋ x rounded down
dom M  the keys of M      ran M  the values of M   #S  the size of S

new

┌─ InitScope ─────────────────────────────────────────
│ Scope'; allow_private? : 𝔹
├─────────────────────────────────────────────────────
│ entries' = ⟨⟩ ∧ allow_private' = allow_private?
└─────────────────────────────────────────────────────

load_file

┌─ LoadFile ──────────────────────────────────────────
│ ΔScope; file? : PATH
├─────────────────────────────────────────────────────
│ readable(file?)
│ parsed = ⟨ e : ENTRY | ∃ l ∈ lines(file?) • parse(l) = e ⟩
│ #parsed > 0
│ entries' = entries ⁀ parsed ∧ source' = file?
└─────────────────────────────────────────────────────

entries

┌─ Entries ───────────────────────────────────────────
│ ΞScope; result! : seq ENTRY
├─────────────────────────────────────────────────────
│ result! = entries
└─────────────────────────────────────────────────────

source

┌─ Source ────────────────────────────────────────────
│ ΞScope; result! : STRING
├─────────────────────────────────────────────────────
│ result! = source
└─────────────────────────────────────────────────────

hosts

┌─ Hosts ─────────────────────────────────────────────
│ ΞScope; result! : seq ENTRY
├─────────────────────────────────────────────────────
│ concrete = entries ↾ { e | ¬ e.wildcard }
│ result! = ⟨ e : concrete | e is the first in concrete with e.host ⟩
└─────────────────────────────────────────────────────

covers

┌─ Covers ────────────────────────────────────────────
│ ΞScope; host? : HOST; result! : 𝔹
├─────────────────────────────────────────────────────
│ h = lower(strip_dot(host?))
│ result! ⇔ ∃ e ∈ ran entries •
│     h = e.host ∨ (e.wildcard ∧ (∃ p • h = p ⁀ "." ⁀ e.host))
└─────────────────────────────────────────────────────

is_private_target

┌─ IsPrivateTarget ───────────────────────────────────
│ ΞScope; host? : HOST; result! : 𝔹
├─────────────────────────────────────────────────────
│ A = resolve(host?)
│ result! ⇔ ∃ a ∈ A • a ¬in; PUBLIC
└─────────────────────────────────────────────────────

allows

┌─ Allows ────────────────────────────────────────────
│ ΞScope; host? : HOST; result! : 𝔹
├─────────────────────────────────────────────────────
│ result! ⇔ Covers ∧ (allow_private ∨ ¬ IsPrivateTarget)
└─────────────────────────────────────────────────────

assert_target_allowed

┌─ AssertTargetAllowed ───────────────────────────────
│ ΞScope; host? : HOST
├─────────────────────────────────────────────────────
│ Allows ∧ result! = self
└─────────────────────────────────────────────────────
NotListed ≙ [ΞScope; host? : HOST | ¬ Covers] ⇒ croak
Private   ≙ [ΞScope; host? : HOST | Covers ∧ ¬ allow_private ∧ IsPrivateTarget] ⇒ croak

STATE DIAGRAM

A scope object is empty until a file is loaded. Loading only adds entries. The checks never change the object, but allows, is_private_target and assert_target_allowed make a DNS lookup.

          new(allow_private => 0 or 1)
                     |
                     v
            +------------------+
            |      EMPTY       |  entries = (), source = '(none)'
            +------------------+
              |      |      |
              |      |      |  covers / allows       -> 0
              |      |      |  hosts / entries       -> empty list
              |      |      |  assert_target_allowed -> croak (not listed)
              |      |      +------> (no change)
              |      |
              |      |  load_file(f): f cannot be read, or has no
              |      |  usable line
              |      +------> croak; (no change)
              |
              |  load_file(f): at least one usable line
              |  -> entries added; a warning per unusable line;
              |     source = f; returns the object
              v
            +------------------+
    +------>|      LOADED      |  entries >= 1
    |       +------------------+
    |         |       |       |
    |         |       |       +-- covers(h) / hosts / entries / source
    |         |       |           -> answer; (no change)
    |         |       |
    |         |       +---------- allows(h) / is_private_target(h)
    |         |                   -> DNS lookup; answer; (no change)
    |         |
    |         |  assert_target_allowed(h)
    |         |    h not listed          -> croak "not listed"
    |         |    h private, not allowed -> croak "private"
    |         |    otherwise             -> returns the object
    |         |  (no change in every case)
    |         |
    +---------+  load_file(f2): at least one usable line
                 -> more entries added; source = f2

load_file(f2) when f2 cannot be read or has no usable line: croak,
and the entries already loaded are kept.

AUTHOR

Nigel Horne, <njh at nigelhorne.com>

LICENCE AND COPYRIGHT

Copyright 2026 Nigel Horne.

Usage is subject to the GPL2 licence terms. If you use it, please let me know.