NAME
Punk::Plugin::Sitemap - sitemap.xml and robots.txt from the route table
SYNOPSIS
package MyApp;
use Punk;
use Punk::Plugin::Sitemap; # for the `sitemap` keyword
plugin 'Sitemap' => { base => 'https://example.com' };
get '/' => sub { ... }; # in the sitemap
get '/about' => sub { ... }; # in the sitemap
get '/users/:id' => sub { ... }; # out: a capture is not a URL
post '/orders' => sub { ... }; # out: not a GET
get '/admin' => sub { ... }, { sitemap => 0 }; # out: opted out
under '/account' => 'auth'; # out: guarded
sitemap users => sub { # the ids the route table cannot know
map { { loc => "/users/$_->{id}", lastmod => $_->{updated} } } @rows;
};DESCRIPTION
Punk compiles every route at boot, so an application already holds a complete list of what it serves. A sitemap is that list with a filter over it, and this is the filter.
Which means it cannot list a page the application does not serve, and it cannot drift from the routes the way a hand-written file does.
What it refuses to list, which is the part that matters
A route is listed only when every one of these holds:
its method is
GET(or aroutethat includes it)its path contains no capture - no
:name, no*splatit carries no guard
it did not say
sitemap => 0
Everything else is left out, and out is the safe direction. A page missing from a sitemap is still crawled if anything links to it; a page wrongly present is a crawler fetching a 404 or a login redirect on a schedule, for as long as the site exists.
A route with a capture is not a URL. /users/:id names a shape, and the ids are not something the route table knows. Those are supplied by the application.
A guarded route is excluded without anyone maintaining a list. under copies its guard chain into every route declared inside it, so the plugin can see what somebody writing a sitemap by hand has to remember. Every hand-written sitemap eventually lists a page behind a login; this one cannot.
base is configuration, and the application's host is its default
host 'https://example.com';
plugin 'Sitemap'; # base = the host
plugin 'Sitemap' => { base => 'https://example.com' }; # explicit winsThe sitemap protocol wants absolute URLs. The obvious place to get the scheme and host is the request that asked for the sitemap, and that is a host-header injection delivered to search engines: a request carrying Host: evil.example would produce a file naming that host for every page on the site, and the owner could not see it, because their own request produces a correct file.
So the request is never consulted. The base comes from configuration - an explicit base, or the application's declared "host" in Punk, which is the same fact stated once for every plugin that needs it. An application with neither croaks, for the same reason session refuses to start without a secret; it croaks at to_app rather than at the plugin line, so host may be declared on either side of it.
Several hosts
host 'https://example.com', allow => [ '*.example.com' ];
plugin 'Sitemap';An application with a host allowlist serves a sitemap per tenant: a request from an allowlisted host is answered with a document naming that host, and robots.txt advertises the same. The route table and the dynamic sections are shared - they are computed once per ttl however many hosts ask - and only the origin differs. A host on neither the canonical origin nor the allowlist is handed the canonical document, because the alternative is the host-header injection above.
What it serves
GET /sitemap.xml the document, or an index over the parts
GET /sitemap/1.xml a part, when there is more than one
GET /robots.txt the disallow list, plus a C<Sitemap:> lineBoth from the site root regardless of any mount prefix, because /sitemap.xml is a fixed location and a crawler will not look anywhere else.
The document is rendered once, at to_app, and served from memory - the routes cannot change while the process runs, so a request is a write of bytes that already exist.
The protocol caps a sitemap at 50,000 URLs and 50MB, and past either one crawlers reject the whole file rather than the excess. Both are counted as the document is written and whichever binds first starts a new part; with more than one part /sitemap.xml becomes an index naming them. With one part there is no index, because an index wrapping a single sitemap is a second fetch for nothing.
A part number nobody generated is a 404, so a crawler following a stale index is told the part is gone rather than handed an empty document.
Encoding, then escaping
Two different jobs, in that order. A path holding a space has to be percent-encoded to be a URL at all, and escaping alone would produce a well-formed document containing an invalid one. Encoding first also makes the whole document ASCII, so a route path that is not valid UTF-8 cannot make it unparseable.
Then XML escaping over every URL without exception. One bare & makes the document not well-formed, and a crawler rejects all of it rather than the offending entry - which makes this a correctness problem before it is a security one.
There is no <lastmod> on routes. "When did this static page last change" is not something the route table knows, and stamping boot time would tell crawlers every page changed on every deploy, which is the fastest way to have the file ignored.
THE DYNAMIC HALF
package MyApp;
use Punk;
use Punk::Plugin::Sitemap; # for the keyword, see below
plugin 'Sitemap' => { base => 'https://example.com', ttl => 3600 };
get '/users/:id' => sub { ... }; # a shape, so it contributes nothing
sitemap users => sub { # the ids, which only the app knows
my @u = MyApp->model('User')->all;
return map { { loc => "/users/$_->{id}", lastmod => $_->{updated} } } @u;
};A section returns locations: a rooted path, or a hashref with loc and an optional lastmod. Sections are emitted in declaration order after the route half, each sorted, and the whole document is de-duplicated - so a page that is both a static route and a section row appears once.
Declaring the same name twice replaces rather than appends, keeping its original position, so a base class can declare a section and a subclass override it without the document reshuffling.
The keyword needs the `use`
plugin 'Sitemap' runs at runtime of the package body, long after sitemap users => sub {...} has been compiled - so a keyword installed only there is one perl has already refused to parse. use installs it at compile time, which is why both lines are in the synopsis.
Without the use, the parenthesised form still works, because perl resolves a call with parentheses at runtime:
sitemap('users' => sub { ... });ttl, and the staleness it buys
A section reads a database, so its answer changes while the process runs. Running it per request would be a query nobody is watching, on a schedule somebody else chooses; running it once at boot would produce a sitemap correct on the day of the deploy and progressively wrong afterwards.
So it is rebuilt when ttl seconds have passed, an hour by default, and the document is stale by up to that long. That is fine and worth saying rather than engineering away: a crawler learning about a page an hour late is a crawler behaving normally, and no search engine promises to fetch a sitemap promptly.
There is no stampede to guard against. Hyperman is single-threaded and a worker serves its requests one after another, so the second of two simultaneous crawler requests finds what the first built. The cost across a pool is one rebuild per worker per ttl.
An application with no sections is never rebuilt at all: the routes cannot change, so there is nothing to refresh.
What a section may not return
A section returns data that goes straight into a structured document, so every field is checked. The route half needs none of this, because its paths are declarations; a section's are database rows.
A loc is dropped, with a warning, unless it is a rooted, same-origin, concrete path:
it must start with
/- the base supplies the originit must not start with
//, which is protocol-relative and names another host.//evil.example/xin your sitemap is somebody else's pages published under your name.it must hold no
:or*, which would mean the section returned the route pattern by mistake and a literal:idwould go in the fileno control bytes and no backslash
$c->safe_path encodes the same rules for redirects, for the same reason: bytes the application did not write, reaching somewhere structured.
An unparseable lastmod is dropped while the URL still goes in. One bad date invalidates the whole document, and losing the page would cost more than losing the date.
A section that dies warns and contributes nothing, rather than taking the request with it - a failed section degrades the sitemap, not the site. One returning more than 50,000 locations is truncated there, because an unbounded SELECT on a table that grew is how this becomes an out-of-memory rather than a large file.
ROBOTS.TXT
GET /robots.txtIt is not access control. It is a request to well-behaved crawlers, ignored by everything else, and a Disallow line is a signpost to anything hostile - a published list of the paths worth attacking. The guarding is auth_guard's job; this only tells Google not to bother.
It is generated here rather than by hand, and from the same pass over the same records as the sitemap, because they are two spellings of one decision. Written apart they drift within a release, and the drift is silent both ways: a path disallowed but listed is a crawler told two things, and a path excluded but not disallowed is crawled anyway - which is how a staging environment ends up indexed.
User-agent: *
Disallow: /account/orders/
Disallow: /account/settings
Disallow: /admin
Sitemap: https://example.com/sitemap.xmlThree sources: guarded routes, routes that said sitemap => 0, and whatever disallow added. Plus the Sitemap: line, which is how a crawler finds the sitemap without being told and is the line most hand-written robots.txt files are missing.
A guarded route with a capture is disallowed as a prefix. Nobody benefits from Disallow: /account/orders/:id, so the path is truncated at the first capture and the prefix stands for everything under it - which is exactly the set the route matches.
Nothing the sitemap lists can be disallowed. Every candidate is checked against the listed URLs first, so a Disallow can never shadow a page the same file is advertising. That is enforced rather than tested for afterwards.
The plugin's own routes are kept out of both. They are absent from the sitemap because a sitemap listing itself is noise, and absent from Disallow because forbidding /sitemap.xml while the Sitemap: line advertises it is the exact contradiction this section exists to prevent.
OPTIONS
disallow
An arrayref of extra prefixes for robots.txt, for anything the route table cannot see - a mounted PSGI app, a static directory.
disallow_all
plugin 'Sitemap' => { base => ..., disallow_all => 1 }; # staging
User-agent: *
Disallow: /A staging environment being indexed is routine, embarrassing and slow to undo, and it is usually caused by a robots.txt copied from production. This is one flag, driven from the config that already differs between environments.
No Sitemap: line is emitted with it, because advertising a sitemap while disallowing everything says two opposite things.
ttl
Seconds before a dynamic section is run again. An hour by default, and irrelevant to an application with no sections.
base
The scheme and host every path is joined onto. Defaults to the application's "host" in Punk; with neither declared, to_app croaks. A trailing slash is trimmed, since every path already starts with one and https://x//about is a different URL to a crawler.
ROUTE OPTIONS
sitemap => 0
Keeps a route out.
sitemap => 1
Puts a route in despite a guard.
This is not the opt-in half of a pair - inclusion is already the default for a route that says nothing, and an opt-in default would mean an empty sitemap until somebody remembered, which looks like an answer and is not.
It exists because a scope guard may be an authentication check or may be an ordinary filter, and Punk cannot read its intent. The plugin assumes the first, because that is the safe direction, and this is how an application says a page is public anyway.
The method and the shape are not overridable. sitemap => 1 on a POST or on /users/:id does nothing, because a POST is not a page and a pattern is not a URL - those are mistakes rather than instructions.
SEE ALSO
AUTHOR
LNATION <email@lnation.org>
LICENSE AND COPYRIGHT
This software is Copyright (c) 2026 by LNATION <email@lnation.org>.
This is free software, licensed under:
The Artistic License 2.0 (GPL Compatible)