NAME

Fetch - HTTP/2 Future-based user agent

VERSION

Version 0.16

SYNOPSIS

use Fetch;

# works out of the box - no event loop to set up
my $res = Fetch->new->get('https://example.com')->get;
print $res->content if $res->is_success;

# every call returns a Fetch::Future; ->get awaits it
my $ua  = Fetch->new(timeout => 10);
my $f   = $ua->post('https://api/things',
                    headers => { 'Content-Type' => 'application/json' },
                    body    => '{"name":"x"}');
my $res = $f->get;

# many requests concurrently on one loop
my @futs = map { $ua->get($_) } @urls;
Fetch::Future->needs_all(@futs)->get;
my @bodies = map { $_->get->content } @futs;

# a live WebSocket
my $ws = $ua->websocket('wss://host/socket')->get;
$ws->send('hello');
my $reply = $ws->next_message->get;

DESCRIPTION

Fetch is an HTTP user agent whose socket, TLS, HTTP/2 framing and HTTP/1.1 parsing hot path lives in vendored C, and whose asynchronous results are Fetch::Future objects that compose with the Hyperman event loop and other CPAN loops (IO::Async, AnyEvent) - or with nothing at all, since Fetch ships its own event loop (Fetch::Loop::Standalone) and uses it automatically.

HTTP/1.1 and HTTP/2 (ALPN-negotiated over TLS), over cleartext and TLS, with keep-alive connection pooling, redirect following, per-request timeouts, streaming response bodies, a cookie jar, JSON request/response helpers, and native WebSockets. Every request method returns a Fetch::Future; ->get on one awaits it, pumping whichever event loop is active (its own if none), so the same code serves both a simple synchronous call and thousands of requests multiplexed on one loop.

CONSTRUCTOR

new(%args)

my $ua = Fetch->new(
    timeout    => 10,
    tls_verify => 1,
    headers    => { 'Accept' => 'application/json' },
    cookie_jar => 1,
);

Create a user agent. All arguments are optional:

loop

The event loop to run on. Omit it and Fetch uses its own Fetch::Loop::Standalone (so ->get just works with no framework). Pass a raw IO::Async::Loop or Hyperman::Loop and it is wrapped automatically; pass the string 'AnyEvent' to drive AnyEvent; or pass a ready-made Fetch::Loop adapter. See "EVENT LOOPS".

Omitting it gives every such agent the same loop - one implicit Fetch::Loop::Standalone per process, rebuilt automatically in a child after a fork. Agents therefore multiplex: awaiting a request on one drives whatever the others have in flight rather than stalling them. Pass an explicit loop to opt out and get an isolated one.

headers

Default headers sent on every request, as a hashref, an arrayref of name => value pairs (duplicates preserved), or a Fetch::Headers. Per-request headers are merged on top (see "REQUEST OPTIONS").

agent

The User-Agent string. Defaults to "Fetch/$VERSION".

tls_verify

Whether to verify the peer certificate and hostname for https. Default true. Overridable per request.

timeout

Default per-request deadline in seconds (fractional allowed). 0 (the default) means no timeout. Overridable per request.

max_redirects

How many redirects to follow. Default 5; 0 disables following. Overridable per request.

keep_alive

Reuse connections via a keep-alive pool (default true). Set false to close every connection after one request.

pool_size

Maximum idle connections the keep-alive pool parks (default 32).

cookie_jar

A Fetch::CookieJar to store and send cookies (applied across redirects), or a true scalar to create a fresh one. Default: no jar.

simple_response

Resolve requests to a plain unblessed hashref { status => ..., headers => [k, v, ...], content => ... } instead of a blessed Fetch::Response. Read fields directly ($res->{status}, $res->{content}) rather than via methods. This skips the response object's method dispatch on the hot path; the saving is small (a couple of percent) and only shows when you actually read the response, so reach for it when you are consuming millions of responses and want the leanest possible per-response cost. Default false.

REQUEST METHODS

Each returns a Fetch::Future that resolves to a Fetch::Response (or fails with an error string). They never block; call ->get on the future to await the result.

get / head / delete

my $f = $ua->get($url, %opt);

post / put

my $f = $ua->post($url, body => $bytes, %opt);

request($method, $url, %opt)

my $f = $ua->request('PATCH', $url, body => $bytes, %opt);

The general form the verb helpers dispatch to; use it for any method.

REQUEST OPTIONS

Passed as a trailing key => value list to any request method:

headers

Extra headers for this request - a hashref, an arrayref of pairs (keeping duplicate names, e.g. multiple X-* values), or a Fetch::Headers. Each named field overrides the agent default of the same name.

body

The request body (bytes). Content-Length is added automatically unless you set it yourself.

json

A Perl data structure to send as a JSON body: it is encoded and Content-Type: application/json is set (unless you gave your own). Takes precedence over body. Pair it with "json" in Fetch::Response to decode the reply. Encoding uses Cpanel::JSON::XS when installed, else core JSON::PP.

my $res  = $ua->post($url, json => { name => 'x', ok => \1 })->get;
my $data = $res->json;

timeout

Override the agent timeout for this request (seconds; 0 disables it).

tls_verify

Override certificate/hostname verification for this https request.

max_redirects

Override how many redirects this request follows.

on_body

A coderef called with each body chunk as it arrives, instead of buffering. Suits large downloads and server-sent events: the buffer is compacted so an endless stream does not grow memory, and the resolved response body is empty.

$ua->get($url, on_body => sub { my ($chunk) = @_; print $chunk })->get;

on_headers

A coderef called once, as soon as the response status line and headers have been parsed - before any on_body chunk - with the status code and the header list:

$ua->get($url,
    on_headers => sub { my ($status, $headers) = @_; ... },  # $headers: [k,v,...]
    on_body    => sub { my ($chunk) = @_; ... },
)->get;

This lets a streaming consumer act on the status and headers up front (for example, to open a downstream writer) rather than waiting for the whole response. Not fired for a WebSocket upgrade.

WEBSOCKETS

websocket($url, %opt)

my $ws = $ua->websocket('ws://host/echo')->get;   # or wss://

Open a WebSocket (RFC 6455). Returns a Fetch::Future that resolves, after the 101 handshake, to a Fetch::WebSocket for sending and receiving messages. Accepts ws:///wss:// (and http/https); tls_verify and timeout options apply to the handshake.

clone(%overrides)

my $shared  = Fetch->new;                        # built once
my $scoped  = $shared->clone(cookie_jar => 1);   # its own jar, same pool

Another agent over this one's connection pool and event loop, with the given options replaced. Everything not named is inherited.

The case it exists for is a per-request cookie jar. A jar belongs to the agent, so a jar on a long-lived agent is shared by every request that agent serves: fine when the cookies authenticate the application itself, a cross-request leak the moment they identify an end user. Building a whole fresh agent per request avoids the leak but throws away the keep-alive pool and re-resolves the loop adapter, which is most of what new costs. A clone gives you the isolation without the bill.

cookie_jar, headers, agent, timeout, tls_verify, max_redirects, keep_alive and simple_response can be overridden; cookie_jar => undef drops an inherited jar. Headers are copied rather than shared, so a clone cannot write into the parent's defaults.

loop and pool_size cannot be overridden and croak if given: they are what the clone shares, and an agent on a different loop driving the parent's parked connections would be reaching into the wrong one.

A clone holds references to the shared pool and loop, so they live until the last agent using them goes. Keep the parent alive for as long as its clones, as you would anyway.

ACCESSORS

loop

The event-loop adapter this agent runs on.

cookie_jar

The Fetch::CookieJar in use, or undef.

OBSERVING OUTBOUND REQUESTS

Fetch->on_request(\&start, \&done)

Watch every request this process makes, including ones it did not write.

Fetch->on_request(
    sub {
        my ($method, $url, $headers) = @_;
        push @$headers, traceparent => current_span_id();
        return { at => time, url => $url };        # the token
    },
    sub {
        my ($token, $res, $err) = @_;
        record($token->{url}, time - $token->{at},
               $err ? "failed: $err" : $res->status);
    },
);

start fires just before a request goes out, with the merged header list still mutable: push a name and a value onto $headers and the request carries them. That is the point of the hook - "REQUEST OPTIONS"' own headers is set by whoever made the call, which is no use to a tracing layer that has to annotate a request the application wrote.

Whatever start returns is the token, handed back to done when that same request settles. It is an ordinary Perl scalar - a hashref of what you want to remember is the usual thing - so the two halves correlate without a lookup table keyed on something that might repeat.

done fires exactly once for every start, with exactly one of $res (a Fetch::Response) and $err (the failure message). That includes a timeout, a refused connection and a DNS failure, which are the endings an instrumented client most wants and the easiest to leak. done is optional.

Three things to know, all of which are the contract rather than an oversight:

Per hop, not per call. A redirect chain is several requests, each of which went somewhere, and each is observed. One ->get that follows two redirects fires start three times.
Registration is process-global and permanent. Not per agent: an agent is a per-worker object and a process may hold several, while an observer is a property of the process. There is no deregistration, so register at boot - and note that the callbacks (and everything they close over) live as long as the process does.
A death is warned, not propagated. An observer is a bystander, and a broken bystander must not be able to fail the request it was watching. If either callback dies, the death becomes a warning naming which half it came from, and the request carries on - without whatever the callback had not yet done, such as the header it died before pushing.

Returns 1, or 0 when the observer table is full (FETCH_ABI_MAX_OBSERVERS, 8). Croaks if the callbacks are not code references, at registration, where the mistake is.

This costs one Perl call per hop, paid only by a process that asked for it: a request with no observer registered pays a single branch. Where that is too much - a proxy or a server instrumenting every request it forwards - the same registry takes C function pointers through the C ABI's "The table" on_request, which is the same contract with no Perl on the path.

THE RESULT

Awaiting a request future yields a Fetch::Response with status, headers (a Fetch::Headers), content, header($name), is_success and is_redirect. A failed request (connection error, timeout, bad TLS, a rejected WebSocket upgrade) fails the future with a message; ->get rethrows it, or inspect $f->failure / $f->is_failed.

EVENT LOOPS

$future->get awaits by pumping the active loop, so a bare synchronous call needs no setup. Hand Fetch a loop to cooperate with an existing async program - requests then fly concurrently on that one loop without blocking it:

use IO::Async::Loop;
my $loop = IO::Async::Loop->new;
my $ua   = Fetch->new(loop => $loop);       # shares this loop
my $f    = $ua->get('https://example.com/');
$loop->loop_once until $f->is_ready;
my $res  = $f->get;

Supported loops: the built-in Fetch::Loop::Standalone, plus Fetch::Loop::IOAsync, Fetch::Loop::AnyEvent and Fetch::Loop::Hyperman.

A request future remembers the loop it was issued on, and every future derived from it by then/followed_by/transform inherits that. So ->get always pumps the loop that owns the socket, however many agents and loops the program has, and in whatever order they were built. Awaiting a future on a loop that was never going to resolve it - one with no watchers and no timers - dies with a diagnostic rather than blocking in the kernel forever.

C ABI

Fetch exposes a small C ABI so another XS module can drive it from C, with no per-request Perl round trip on the hot path. It is how Reverse::Proxy builds its upstream requests entirely in C. This is not part of the Perl API - if you are writing Perl, use the request methods above; the ABI is only for XS consumers.

The ABI is a versioned function pointer table. A consumer vendors a copy of the header fetch_abi.h (shipped in this distribution under include/fetch/) at a pinned FETCH_ABI_VERSION, then at boot resolves the table and checks the version:

#include "fetch_abi.h"   /* vendored, after the perl.h includes */

/* in BOOT: */
IV p = 0;
if (call_pv("Fetch::_abi_ptr", G_SCALAR) > 0) { SPAGAIN; p = POPi; PUTBACK; }
const fetch_abi *FETCH = NULL;
if (p) {
    const fetch_abi *a = INT2PTR(const fetch_abi *, p);
    if (a && a->abi_version >= FETCH_ABI_VERSION) FETCH = a;
}
/* FETCH == NULL means the running Fetch is too old; the consumer decides
 * whether that is a hard error or a Perl fallback. The check is >=, never
 * ==: the table only grows at the end, so a newer Fetch still holds every
 * entry the consumer was written against. */

Fetch::_abi_ptr

Returns the address of Fetch's fetch_abi table as an IV. Call it once, at boot, and INT2PTR the result to a const fetch_abi *. A version mismatch must never be treated as a crash - compare abi_version first and fall back.

The table

fetch_abi (see fetch_abi.h for the exact signatures and ownership rules) holds, after abi_version:

ua_new(kv, nkv)

Construct a Fetch user agent from nkv flat key/value SVs (the same options new takes: loop, pool_size, tls_verify, timeout, agent, headers, cookie_jar, keep_alive, max_redirects, simple_response). Returns the blessed Fetch UA SV (+1 owned). A consumer may instead call Fetch->new from Perl once and cache the object; ua_new just removes that last Perl call.

request(ua_sv, method, url, hdrs, nhdrs, body, blen, timeout, max_redirects, map, ud)

Issue one HTTP request on ua_sv, building it entirely from C. The headers are a flat fetch_hdr array; max_redirects below zero means "use the UA default". Returns a Fetch::Future SV (+1 owned) - hand it to an awaiting server or call ->get on it. When the request settles, your map callback shapes the value the future resolves to (for a proxy, a PSGI [ status, \@headers, \@body ]).

res_parts(res, status, headers, body)

Pull the status, the flat header AV and the content SV out of an already-resolved response (a Fetch::Response or a simple_response hash) with no method dispatch. Any out-pointer may be NULL; the returned headers and body are borrowed. For the blocking (awaited) path.

request_stream(ua_sv, method, url, hdrs, nhdrs, body, blen, timeout, max_redirects, on_headers, on_body, on_done, ud)

Like request, but streams the response instead of buffering it: on_headers fires once up front with the status and header AV, on_body once per body chunk, and on_done at completion (with success/failure). Returns the request future SV (+1 owned) - keep it alive until on_done fires. Lets a consumer forward a large download or an endless SSE stream with flat memory.

tunnel_connect(host, port, tls, verify) and friends

A raw blocking upstream TCP connection Fetch owns, for a proxy's Upgrade/WebSocket tunnel where the consumer splices bytes both ways itself. When tls is true the connection reuses Fetch's own client SSL_CTX (SNI, and hostname/certificate verification when verify is true), so the consumer tunnels to a wss/https upstream without linking OpenSSL. This is what lets Reverse::Proxy tunnel to a TLS upstream.

Unlike the rest of the table, these six entries are pure C - they take no pTHX and touch no SV, so they can be called directly from inside a select() splice loop:

tunnel_connect(host, port, tls, verify) - open the connection; returns an opaque handle, or NULL on failure (DNS, connect, or TLS handshake).
tunnel_fd(conn) - the underlying socket fd, to hand to select()/poll().
tunnel_read(conn, buf, len) - read bytes; returns the count (>0), 0 at EOF, or -1 on error.
tunnel_write_all(conn, buf, len) - write the whole buffer; returns 0 when all bytes are written, -1 on error.
tunnel_pending(conn) - bytes already buffered inside the TLS layer; drain these before trusting select() readiness (always 0 for a plain connection).
tunnel_close(conn) - shut the connection down and free the handle.

on_request(start, done, ud) (v2)

Observe outbound requests.

static void *start(pTHX_ const char *method, STRLEN mlen,
                   const char *url, STRLEN ulen, AV *headers, void *ud);
static void  done(pTHX_ void *token, SV *res, SV *err, void *ud);

start fires once per hop with the merged header list still mutable - push a name and a value onto headers and the request carries them. That is what the hook is for: "REQUEST OPTIONS"' headers is set by the caller, which is no use to anything that wants to annotate a request the application wrote. It returns an opaque token, handed back to done when that same hop settles, so the two correlate with no lookup table on the hot path.

done fires exactly once for every start, with exactly one of res and err - including a timeout, a refused connection and a DNS failure. Both are borrowed.

Per hop, not per call: a redirect chain is several requests, each of which went somewhere, and anything measuring them should see all of them.

Registration is process-global, not per agent, and there is no deregistration; register at boot. Neither callback may croak. Returns 1, or 0 when the table is full (FETCH_ABI_MAX_OBSERVERS). A request with no observer registered pays one branch and allocates nothing.

"Fetch->on_request(\&start, \&done)" is the same registry reached from Perl, for a consumer that is not an XS module: it registers a pair of shims holding coderefs. Same contract, one Perl call per hop instead of none, and the one rule C could state and Perl cannot enforce - "neither may croak" - becomes "a death is warned and the request carries on".

tunnel_starttls(conn, host, verify) (v3)

Upgrade a tunnel opened with tls false to TLS, after the application protocol has negotiated it - SMTP's STARTTLS, where the connection starts in plaintext, the server says 220, and only then does the handshake begin. The handshake is the one tunnel_connect performs when tls is true: the same client SSL_CTX, SNI, and hostname and certificate verification when verify is true.

Returns 0 on success, after which tunnel_read, tunnel_write_all and tunnel_pending speak TLS. Returns -1 when conn is NULL or already TLS, when this build of Fetch has no TLS, or when the handshake fails; after a failed handshake the socket is in an undefined state and the caller should tunnel_close it. Pure C, no pTHX, like the rest of the tunnel.

The tunnel carries no timeout of its own. A consumer that cannot wait forever on a peer sets SO_RCVTIMEO and SO_SNDTIMEO on tunnel_fd(conn); the handshake and every later read honour them.

AUTHOR

LNATION <email@lnation.org>

LICENSE AND COPYRIGHT

This is free software, licensed under the Artistic License 2.0.

To install Fetch, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Fetch

CPAN shell

perl -MCPAN -e shell
install Fetch

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)