NAME

Punk::Plugin::Observe - mount Punk::Observe in a Punk application

SYNOPSIS

plugin 'Observe' => {
    prefix => '/observe',
    guard  => 'Web::Auth#observe_admin',       # required
    store  => '/var/lib/punk-observe',
    ingest => { prefix => '/v1', keys => '/etc/punk-observe/keys' },
    limits => { series => 1_000_000, rate_records => 50_000 },
};

DESCRIPTION

Mounts the observability UI under a guarded scope and, optionally, the OTLP ingest endpoint beside it.

The guard is not optional

Registration croaks without one.

An unguarded mount is every log line the application has ever written, served to anybody who finds the prefix. A loud failure at boot beats a silent hole that nobody notices, so the failure is at registration rather than at the first request.

Setting PUNK_OBSERVE_INSECURE is the deliberate escape, named for this distribution so that meaning it is possible and doing it by accident is not.

The UI is registered as an under scope rather than a mount. Under a scope the guard covers every path beneath it including ones added later; a mount with a guard on each route is the same thing right up until somebody adds a route and forgets, and the failure mode of forgetting is an unauthenticated page.

Ingest is a separate scope, in both directions

The ingest prefix sits outside the UI scope on purpose.

It is authenticated by key and not by the UI guard, because an exporter has no session. It is CSRF-exempt, because an exporter has no form token.

Both directions are bugs. A UI route that became CSRF-exempt by sharing this scope would be a security hole; an ingest route that required a form token would return 403 to every exporter in the world with a message about forms.

An ingest key cannot read the UI. There is no option to widen it: a key that could do both leaks a whole installation the first time it is baked into a container image, which is where ingest keys go.

Options

prefix

Where the UI mounts. Defaults to /observe.

guard

Required. A coderef, or 'Controller#action'.

store

The store root. Every path this distribution builds is rooted here.

tenant

A constant tenant id, or a coderef resolving one. Defaults to a constant, so the self-hosted shape is the whole shape and the hosted one is a callback through the same code. Whatever a resolver returns is validated before a byte of it reaches a path - a resolver is host code, and host code that returns ../other is a bug to catch rather than a value to trust.

A tenant id is never taken from anything a client sends.

ingest

{ prefix => '/v1', keys => $path }. Omit it and no ingest endpoint is registered at all.

limits

rate_records and rate_bytes per second, series for cardinality, storage in bytes, and attributes for the indexed-attribute allowlist.

The rate limit is off unless configured: a default rate limit on a self-hosted box is a surprise in the wrong direction. The cardinality limit has a default, because a store with no cardinality limit is a store waiting for one bad deploy.

rate_records and rate_bytes truncate the batch before it reaches the log and answer with an OTLP partial success naming what was dropped. That ordering is the contract: a partial success means "I did not keep these", so an exporter resending what it was told was rejected must not find those records already stored.

The counters behind all of this live in a small shared page mapped at registration, which is before the fork. One mapped afterwards would be private per worker, and the symptom is not a crash - it is a rate limit N times what was configured and a status page showing whichever shard of the traffic answered the request. Where the platform cannot share one, the status page says so rather than presenting a fraction as a total.

limits and what the status page can show

Two record counts appear on the status page and they answer different questions. accepted is what arrived, counted at ingest; records is what the store still holds. Retention makes the second smaller over time and a limit makes it smaller immediately, so a gap between them is not an error.

The store cannot report the first. Refused data never reaches it, so a receiver throwing every batch away and a receiver being sent nothing have identical stored totals.

retain

retain => { keep => '7d' }              # hourly, at :17
retain => { keep => '30d', at => '40 2 * * *' }
retain => { keep => '48h', bytes => '2G' }
retain => { keep => '7y' }              # what a production store keeps

Schedules retention on the host's queue. There is no default window: absent this option nothing is ever deleted, because a retention job with a silently defaulted window is a deletion job. A keep that does not parse is a boot failure - the alternative is an operator who believes deletion is running when nothing is.

keep takes any duration the query language takes, up to and including years: y is exactly 365 days. A window past the last representable instant - beyond roughly 584 years - is refused rather than wrapped, because a wrapped window is a cutoff of now and deletes the store.

bytes is an optional budget over and above the window ('500M', '2G', or a plain byte count): a store still past it after the time sweep loses its oldest segments even inside the window, because a full disk loses everything rather than the oldest hour. A size that does not parse is a boot failure, for the same reason keep is.

What it does when it runs: deletes, never archives. Whole sealed segments whose newest record is older than now - keep are unlinked with their index sidecars; the live log and any segment still holding one record inside the window survive untouched, so keep is a floor rather than an exact edge. Nothing is copied anywhere first. A reader already holding a deleted segment keeps reading it to the end - the primitive is unlink, never truncate, and Punk::Observe::Retain carries the reasons.

health_allow

Hosts the outbound-URL policy should admit when a health target is saved and when it is polled, as an arrayref. The policy refuses loopback, link-local and private ranges by default - correct for a URL a stranger typed, and exactly wrong for a self-hosted install watching services on its own network, which is what the allowlist is for.

on_alert

on_alert => sub {
    my ($event) = @_;
    # app           the application class
    # tenant
    # rule          { id, name, query, op, threshold, for_ns, every_ns }
    # kind          firing | resolved | vanished | error
    # series        [ { series, kind, value, at, fired_at }, ... ]
    #               one entry, or many when a group delivered together
    # count
    # at            when this delivery became due (ns string)
    # path          "/observe/alerts/<id>" - the host owns its origin
    # delivery_key  idempotency token, per group per send
},

The delivery seam, and the host's whole contribution to alerting: rules are edited on the screen, evaluated by the cron this plugin registers on the host's queue, and every notification arrives here. Webhook, email, pager - delivery is the one thing the core cannot know, so the core does no outbound HTTP at all; Punk::Observe::Target is available for hosts that send to URLs.

Values are raw - nanoseconds where the query measured time - because formatting belongs to the host. A die is retried by the queue, five attempts with backoff, and the terminal failure is a dead letter on the group, visible, never dropped. Delivery is at least once; delivery_key is the token for a host that wants exactly-once on its own side.

Absent, notifications are still recorded and marked; there is simply nobody to tell.

The ordering rule: alerting runs on the host's queue, so use Punk::Plugin::Queue and plugin 'Queue' must come before this plugin - the same rule Punk-Mailer enforces, with the same loud boot failure when it is missing. The registered cron is observe-evaluate; pausing it through the queue's controls survives deploys, because cron reconcile never resets enabled.

alerts

Policy for the built-in alerting path, all optional:

alerts => {
    every           => '@every 30s',  # the cron cadence - the resolution
    group_wait      => '30s',  # a group holds this long before its first
                               # send: one bad deploy, one message
    repeat_interval => '15m',  # re-page a STILL-firing group; off absent
};

Or the escape: a reader seam - a coderef or hashref, as below - for a host that keeps rules somewhere of its own. A host that supplies one keeps its own evaluator too: no cron is registered for it, and on_alert does not apply. Rules are configuration with an owner, a review and a history, so they live in the configuration database rather than in a telemetry store that retention deletes from; sqitch/ is the schema.

alerts => sub {
    # ONE argument, and it is the request - not ($id, $req). The
    # dashboards seam below takes its slug first; this one does not, and
    # reading it that way binds the request hashref to $id.
    my ($req) = @_;             # $req->{id} on /observe/alerts/:id
    return {
        rules    => [ { id, name, series, state, value, held } ],
        silences => [ { pattern, until, by, reason } ],
        events   => [ { series, to, at } ],     # optional
        can_edit => 1,
        to       => $now_ns,                    # optional
    };
};

state is one of ok, pending, firing, stale or error, per series rather than per rule - one state for a whole rule is the bug that makes an alert resolve because a different service recovered.

held and value are numbers, and the screen formats them. held is how long the series has been in its current state, in nanoseconds, rendered as a duration; value is the number the rule last compared, rendered with %.4g and no unit. A pre-formatted string does not survive either: "2m30s" is read as the number 2 and drawn as two nanoseconds. A latency is therefore worth converting to milliseconds before it is handed over, since nanoseconds under %.4g come out as 2.911e+09.

events is the state history, and the screen draws a timeline from it. Each entry is a transition: the series, the state it moved to, and the instant at which it did, as nanoseconds. That is a row of alert_events in the shipped schema. Supply it and the timeline appears; omit it and the screen is the table alone.

The timeline is drawn only from recorded transitions. One inferred from current state would be a straight line claiming the present has always been the case - which is exactly the question a reader opens it to answer, so answering it wrongly is worse than not answering it.

to is where the timeline's right edge sits, defaulting to now. The last band runs to it, because a state nobody has left is still in force.

dashboards

The same shape for dashboards: a hashref or a coderef taking a slug, returning title, cols, panels and a list of the others. A panel is an OQL string with a title, validated at save time by the parser that will run it.

Panel bodies are deferred. The page ships each panel's title and a placeholder; the browser fetches every body - chart or table - from its own fragment route, in parallel, and refreshes it every thirty seconds. ?full=1 renders everything with the page instead, which is also the no-JavaScript path. A panel is addressed by its id, or by its position for a reader whose panels carry none - give panels ids if reordering while a page is open matters to you. The editor runs no panel queries at all: it renders forms.

The argument order differs from alerts, and always has. A dashboard reader is called as ($slug, $req) - the slug first, because a dashboard page is a request for one particular dashboard - while an alerts reader takes only ($req). Getting them the same way round would mean either passing a slug nothing uses or digging one out of the request, and both are worse than the asymmetry.

Both seams are now overrides. Absent, they read from the configuration store this distribution ships - see db - rather than meaning there are no dashboards. To write as well as read, give a hashref instead of a coderef:

dashboards => { read => \&r, write => \&w, delete => \&d }

A bare coderef is still a reader and still means what it did.

db

Where the configuration lives: dashboards, panels, alert rules, silences. A dsn, a hashref of Punk::Observe::Backend options, an object answering to dbh and migrate, or 0 to have none.

Absent, it is SQLite in a file called config.db beside store - so a self-hosted installation has dashboards without deciding anything. With no store either there is nowhere obvious to put it and there is no default, because guessing a path in the working directory is worse than having none.

The schema is migrated at registration, in every worker. That is idempotent and locked, so it is a version check that occasionally does work.

identity

A coderef given the request, returning a stable opaque string for whoever is looking, or undef. Optional, and there is no user model here - the guard decides who may look and this only says who they are, for the state that is per viewer rather than per installation.

Whatever it returns is used as a key and never rendered, so an application keying on an email address does not put one on a page. Absent, per-viewer state lives in the browser and is per browser.

editing

Not an option. The editor is at <prefix>/dashboards/:slug/edit and <prefix>/dashboards/new, and both are the ordinary dashboard page's data rendered through a different template - so a panel shown on one is the same panel the other saved, and there is no second reader for them to disagree through.

The forms POST to six routes, all registered only when writable:

POST <prefix>/dashboards                       create
POST <prefix>/dashboards/:slug                 update
POST <prefix>/dashboards/:slug/delete
POST <prefix>/dashboards/:slug/panels          add a panel
POST <prefix>/dashboards/:slug/panels/save     save the whole table
POST <prefix>/dashboards/:slug/panels/:id      update one
POST <prefix>/dashboards/:slug/panels/:id/delete

A successful write answers 303 to the reader's own page, so a reload cannot re-submit and the editor shows what happened rather than what it hoped would. A refused one comes back as the form, 400, with the reason against the field it is about.

writable

0 to mount read-only. Editing is otherwise on when there is somewhere to write and the application has turned on Punk::CSRF - a write route without a token is forgeable from another origin, and this plugin cannot enable CSRF for the host because the token needs a session only the application can hold. When it is off the write routes are not registered at all, and the screens say why rather than showing a button that does not work.

stats

Discouraged. It predates the built-in configuration store and the health targets, which between them cover what it was for - a number of your own on the status page is better expressed as a health check or a dashboard panel, both of which have history and neither of which can overwrite error. It keeps working; a callback that dies now warns instead of vanishing.

A coderef called as ($c, $page_name, $store) on every screen, whose returned hashref is merged over the template variables. It is a blunt instrument: the merge is flat and last-writer-wins, so it can overwrite error or rows as easily as add a number, and it runs after the decision about whether to load the charting library, so a *_plot key it returns will not draw.

root

The template and asset directory, if you are shipping your own.

seal_bytes / max_rows

Passed to Punk::Observe::Store as given.

on_records

A coderef called as ($tenant, $signal, $records) after a batch is persisted. Materialising the records costs something, so this is off unless asked for.

What each limit does at its cap

They fail three different ways, and treating them as one limit is the mistake:

Ingest rate

Returns an OTLP partial success naming the rejected count, never a bare 429. A 429 makes the exporter re-send the whole batch, forever, at the moment the server is already under pressure - the limit becomes an amplifier.

The limiter covers the ingest prefix and nothing else. Rate-limiting a health endpoint takes the box out of a load balancer under exactly the load the limiter exists for.

Cardinality

The new series is dropped, counted, and surfaced. An existing series is never evicted to admit a new one: that converts a cardinality problem into data loss on the exact series somebody has open in a dashboard.

The counter lives in an arena mapped before the fork, so the limit is per pool. One mapped afterwards is private per worker, and the symptom is a limit silently N times what was configured.

Storage bytes

The retention job shortens retention. Writes are never refused. A store over its byte budget should lose old data; it must not lose the incident happening now.

The allowlist matters more than any of the numbers

Logs and spans carry unbounded attributes, and only the configured set becomes an index dimension. The rest stay in the record and are reachable by a residual filter, so nothing is lost - it is just slower to find.

Without it, one service putting a request id in a resource attribute takes the store down. The overflow counter names the attribute, because the person who hits this first is a self-hoster with no support contract and no dashboard telling them which one did it.

SEE ALSO

Punk::Observe, Punk::Observe::Tenant, Punk::Observe::Key, Punk::Observe::Limit