/ 
App-WebPerimeterScan-0.001.0
/ App::WebPerimeterScan::Source

NAME

App::WebPerimeterScan::Source - map scan results onto files and code

VERSION

Version 0.001.0

SYNOPSIS

Find the file on disk behind a URL:

use App::WebPerimeterScan::Source;

my $source = App::WebPerimeterScan::Source->new(
	docroots => ['/srv/www/example'],
);

my $hit = $source->file_for('/.git/config');
print "served from $hit->{path} ($hit->{size} bytes)\n" if $hit;

Find the line of code that defines an endpoint:

my $app = App::WebPerimeterScan::Source->new(
	source_dirs => ['/home/me/src/example/lib'],
);

my $route = $app->route_for('/view/dogs?page=2');
print "$route->{method} $route->{pattern} at $route->{file} line $route->{line}\n"
	if $route;

List every endpoint the source defines:

for my $route (@{ $app->routes }) {
	print "$route->{pattern}\n";
}

Look for secret files that exist on disk, even if the server did not serve them:

my $probes = App::WebPerimeterScan::Checks::sensitive_probes();
for my $hit (@{ $source->sensitive_on_disk($probes) }) {
	print "on disk: $hit->{relative}\n";
}

DESCRIPTION

A scan from outside can only say "this URL answered". This module adds two facts that tell you what to change:

  • The file. Given a document root (the directory the web server serves files from), it finds the file on disk that a URL path would be served from.

  • The route. Given your application's source code, it finds the line that declares the endpoint, for example $r->get('/view/:name').

The module only reads files. It never sends network traffic and never changes a file.

A path is only accepted if it is inside the document root after every symbolic link is followed. So .. in the path, a symbolic link that points out of the tree, and %2e%2e (an encoded ..) cannot reach a file outside it.

Route patterns

Routes are found in .pm, .pl, .psgi and .cgi files by looking for calls such as ->get('/path'), ->post('/path') and ->any('/path') whose pattern starts with /. In a pattern, :name matches one path segment, *name matches the rest of the path, and every other character must match exactly. A / at the end of the path is optional. The directories .git, .svn, blib and node_modules are not searched.

SUBROUTINES/METHODS

new

Purpose: describe where the site's files and code are.

Args: named docroots and source_dirs, each an array reference of directory names. Either may be empty.

Returns: the new object.

Side Effects: none; directories are read lazily.

Usage: my $source = App::WebPerimeterScan::Source->new(docroots => [$dir]);

EXAMPLE

my $source = App::WebPerimeterScan::Source->new(source_dirs => ['lib']);
printf "%d route(s)\n", scalar @{ $source->routes };

API SPECIFICATION

INPUT

{
	docroots    => { type => 'arrayref', optional => 1 },
	source_dirs => { type => 'arrayref', optional => 1 },
}

OUTPUT

{ type => 'object', isa => 'App::WebPerimeterScan::Source' }

MESSAGES

Message                       Meaning / resolution
----------------------------  ---------------------------------
Unknown parameter 'X'         A misspelt argument name.

docroots

Purpose: the document roots that exist, resolved once.

Args: none.

Returns: an array reference of real paths, without duplicates. A docroot that does not exist is dropped rather than fatal: the scan is still worth running without it.

Side Effects: caches the result.

Usage: for my $dir (@{ $source->docroots }) { ... }

EXAMPLE

print "serving from $_\n" for @{ $source->docroots };

API SPECIFICATION

INPUT

{}

OUTPUT

{ type => 'arrayref', element_type => 'string' }

MESSAGES

None.

file_for

Purpose: the file on disk a URL path would be served from.

Args: $url_path, the path part of a URL (a query or fragment is ignored; percent-encoding is decoded first, as a server would).

Returns: a hash reference of docroot, path (the real path), relative and size; or undef when the file is not under any docroot.

Side Effects: file system reads.

Usage: my $hit = $source->file_for('/.env');

Decoding first is deliberate: %2e%2e%2f becomes ../ and is then refused, rather than slipping through as an opaque name.

EXAMPLE

if(my $hit = $source->file_for('/.git/config')) {
	print "served from $hit->{path} ($hit->{size} bytes)\n";
}

API SPECIFICATION

INPUT

{ url_path => { type => 'string', optional => 1, position => 0 } }

OUTPUT

{
	type => 'hashref', optional => 1,
	schema => {
		docroot  => { type => 'string' },
		path     => { type => 'string' },
		relative => { type => 'string' },
		size     => { type => 'integer', min => 0 },
	},
}

MESSAGES

None.

sensitive_on_disk

Purpose: which of the sensitive paths the scan probes actually exist under a docroot. The caller passes the probe table so that what is looked for on disk is exactly what was requested over the wire.

Args: $probes, an array reference of hash references each with a path.

Returns: an array reference of file_for results, each with a probe key holding the probe it matched.

Side Effects: file system reads.

Usage: my $hits = $source->sensitive_on_disk(App::WebPerimeterScan::Checks::sensitive_probes());

EXAMPLE

for my $hit (@{ $source->sensitive_on_disk([ { id => 'env-exposed', path => '/.env' } ]) }) {
	print "$hit->{probe}{id}: $hit->{path}\n";
}

API SPECIFICATION

INPUT

{ probes => { type => 'arrayref', element_type => 'hashref', position => 0 } }

OUTPUT

{ type => 'arrayref', element_type => 'hashref' }

MESSAGES

None.

routes

Purpose: every route declaration under the source directories.

Args: none.

Returns: an array reference of hash references: method, pattern, file, line and re (a compiled regexp matching the paths the route serves).

Side Effects: walks the source directories the first time and caches the result. An unreadable file is skipped with a warning.

Usage: printf "%d route(s)\n", scalar @{ $source->routes };

EXAMPLE

for my $route (@{ $source->routes }) {
	print "$route->{method} $route->{pattern} at $route->{file}:$route->{line}\n";
}

API SPECIFICATION

INPUT

{}

OUTPUT

{ type => 'arrayref', element_type => 'hashref' }

MESSAGES

Message                                Meaning / resolution
-------------------------------------  ---------------------------------
Skipping source file 'F': E (warning)  F could not be opened; check its
                                       permissions.

PSEUDOCODE

if not yet indexed
	walk each source directory, pruning .git, .svn, blib, node_modules
	for each .pm, .pl, .psgi or .cgi file
		record every ->get('/...')-style declaration with its line
return a copy of the index

route_for

Purpose: the route that would serve a path, in declaration order.

Args: $url_path (a query or fragment is ignored).

Returns: a route hash reference as in routes, or undef.

Side Effects: indexes the source on first use.

Usage: my $route = $source->route_for('/view/dogs');

EXAMPLE

if(my $route = $source->route_for('/api/dirs')) {
	print "$route->{method} $route->{pattern} ($route->{file}:$route->{line})\n";
}

API SPECIFICATION

INPUT

{ url_path => { type => 'string', optional => 1, position => 0 } }

OUTPUT

{ type => 'hashref', optional => 1 }

MESSAGES

As routes.

COMMON PITFALLS

  • A document root that does not exist is ignored without a warning. docroots returns only the directories that exist. Check it after new if a typo would matter: warn "no docroot\n" unless @{ $source->docroots }.

  • The route list is read once. routes reads the source tree the first time it is called, and then always returns that list. Create a new object after the source code changes.

  • file_for and route_for return undef when nothing is found. Test the result before you use it. file_for also returns undef for a directory, for a path that contains .., and for undef or an empty string.

  • file_for wants the path of a URL, not the whole URL. Pass /a/b.txt, not https://example.com/a/b.txt. A query (?x=1) or a fragment (#top) is ignored.

  • The first route that matches wins. Routes are tried in the order they were found, which is the order of the files on disk. When /items/new and /items/:id both match, the one found first is returned.

  • Paths are real paths. path has every symbolic link resolved. On macOS, for example, a file under /var is returned as /private/var/....

ENCODING

  • file_for decodes the URL path once. %20 becomes a space and %C3%A9 becomes the two UTF-8 bytes of é, as a web server would do. The result is matched against file names as bytes, so a file whose name is UTF-8 on disk is found. A name stored in another encoding is not. route_for does not decode: pass the path as it appears in the URL.

  • File and directory names are bytes. The paths returned are the bytes the operating system gave. Do not decode them before using them as file names.

  • Source files are read as bytes. Route patterns must be ASCII to be useful, because URLs are ASCII. UTF-8 elsewhere in a source file does no harm.

LIMITATIONS

  • Only Mojolicious-style ->get('/path') declarations are recognised. Dancer's get '/path' => sub, Catalyst actions, Plack mounts and routes built at run time are not, and a regexp route is skipped rather than approximated.

  • A route pattern can only be matched against a path, so two routes that differ only by HTTP method are both reported for the same path.

  • The check that a file is inside its document root compares paths that use /, which is what realpath returns on Unix. Windows paths are not tested.

FORMAL SPECIFICATION

This section states exactly what each method does, in the Z notation. You do not need it to use the module: the method descriptions above say the same thing in words. It is here for reviewers and for anyone who writes tests.

How to read a schema box:

  • The part above the middle line declares names. A name that ends in ? is an input. A name that ends in ! is an output.

  • The part below the middle line lists facts that are true after the call.

  • A name with a dash after it, such as items', is the value after the call. The same name without the dash is the value before the call.

  • Delta X (written with the Greek capital delta) means the call may change the state X. Xi X (the Greek capital xi) means the call does not change it.

The symbols used are:

Δ   may change            Ξ   does not change      •   such that
∀   for all               ∃   there is             ¬   not
∧   and                   ∨   or                   ⇒   implies
⇔   if and only if        ≠   not equal            ≤ ≥ at most, at least
∈   is in                 ¬in;   is not in            ⊆   is a subset of
∪   union                 ∩   intersection         ∅   the empty set
ℙ X sets of X             ⊥   undefined (undef)    …   and so on
⟨⟩  the empty sequence    ⁀   join sequences       ⇸   partial mapping
↦   maps to               ⊕   override a mapping   ↾   keep only these keys
∘   compose               →   function             × ÷ − times, divide, minus
ℕ   0, 1, 2, ...          ℕ₁  1, 2, 3, ...         ℝ   real numbers
𝔹   true or false         ⌊x⌋ x rounded down
dom M  the keys of M      ran M  the values of M   #S  the size of S

new

┌─ InitSource ────────────────────────────────────────
│ Source'; docroots?, source_dirs? : seq PATH
├─────────────────────────────────────────────────────
│ docroots' = docroots? ↾ DEFINED ∧ source_dirs' = source_dirs? ↾ DEFINED
│ routes' = ⊥
└─────────────────────────────────────────────────────

docroots

┌─ Docroots ──────────────────────────────────────────
│ ΞSource; result! : seq PATH
├─────────────────────────────────────────────────────
│ result! = squash(⟨ realpath(d) | d ∈ docroots ∧ is_dir(d) ⟩)
└─────────────────────────────────────────────────────

file_for

┌─ FileFor ───────────────────────────────────────────
│ ΞSource; url_path? : STRING; result! : HIT ∪ {⊥}
├─────────────────────────────────────────────────────
│ r = unescape(strip_query(url_path?)) without leading "/"
│ ".." ∈ segments(r) ⇒ result! = ⊥
│ result! ≠ ⊥ ⇒ is_file(result!.path)
│              ∧ result!.path = realpath(result!.docroot / r)
│              ∧ result!.docroot prefix_of result!.path
└─────────────────────────────────────────────────────

sensitive_on_disk

┌─ SensitiveOnDisk ───────────────────────────────────
│ ΞSource; probes? : seq PROBE; result! : seq HIT
├─────────────────────────────────────────────────────
│ result! = ⟨ FileFor(p.path) ⊕ {probe ↦ p}
│             | p ∈ probes? ∧ FileFor(p.path) ≠ ⊥ ⟩
└─────────────────────────────────────────────────────

routes

┌─ Routes ────────────────────────────────────────────
│ ΔSource; result! : seq ROUTE
├─────────────────────────────────────────────────────
│ routes = ⊥ ⇒ routes' = ⁀/ ⟨ declared(f) | f ∈ files(source_dirs) ⟩
│ routes ≠ ⊥ ⇒ routes' = routes
│ result! = routes'
└─────────────────────────────────────────────────────

route_for

┌─ RouteFor ──────────────────────────────────────────
│ ΞSource; url_path? : STRING; result! : ROUTE ∪ {⊥}
├─────────────────────────────────────────────────────
│ p = "/" ⁀ strip_query(url_path?) without leading "/"
│ M = ⟨ r ∈ routes | p matches r.re ⟩
│ M ≠ ⟨⟩ ⇒ result! = head M
│ M = ⟨⟩ ⇒ result! = ⊥
└─────────────────────────────────────────────────────

STATE DIAGRAM

An object has two caches, each filled the first time it is needed and never changed after that: the list of document roots that exist, and the list of routes.

      new(docroots, source_dirs)
                 |
                 v
      +----------------------+
      |        FRESH         |  nothing read from disk yet
      +----------------------+
         |                |
         | docroots,      | routes, route_for
         | file_for,      |
         | sensitive_on_  |
         | disk           |
         v                v
+-----------------+  +-----------------+
| DOCROOTS KNOWN  |  |  ROUTES KNOWN   |
| each docroot    |  | every source    |
| checked once    |  | file read once; |
| with realpath;  |  | a warning for   |
| missing ones    |  | each file that  |
| dropped         |  | cannot be read  |
+-----------------+  +-----------------+
         |                |
         +-------+--------+
                 |  the other group of methods is called
                 v
      +----------------------+
      |     BOTH KNOWN       |
      +----------------------+
                 |
                 |  any method
                 v
        answer from the caches; file_for and
        sensitive_on_disk still check each file
        on disk; (no change to the object)

AUTHOR

Nigel Horne, <njh at nigelhorne.com>

LICENCE AND COPYRIGHT

Copyright 2026 Nigel Horne.

Usage is subject to the GPL2 licence terms. If you use it, please let me know.