NAME

CGI::Buffer - Verify, Cache and Optimise CGI Output

VERSION

Version 0.89

SYNOPSIS

CGI::Buffer verifies the HTML that you produce by passing it through HTML::Lint.

CGI::Buffer optimises CGI programs by reducing, filtering and compressing output to speed up the transmission and by nearly seamlessly making use of client and server caches.

To make use of client caches, that is to say to reduce needless calls to your server asking for the same data, all you need to do is to include the package, and it does the rest.

use CGI::Buffer;
# ...

To also make use of server caches, that is to say to save regenerating output when different clients ask you for the same data, you will need to create a cache. But that's simple:

    use CHI;
    use CGI::Buffer;

    # Put this at the top before you output anything
    if(CGI::Buffer::init(cache => CHI->new(driver => 'File'))->is_cached()) {
	# Nothing has changed - use the version in the cache
	exit;
    }

    # ...

To temporarily prevent the use of server-side caches, for example whilst debugging before publishing a code change, set the NO_CACHE environment variable to any non-zero value. If you get errors about Wide characters in print it means that you've forgotten to emit pure HTML on non-ASCII characters. See HTML::Entities. As a hack work around you could also remove accents and the like by using Text::Unidecode, which works well but isn't really what you want.

HOW THE OUTPUT IS SENT

When the script ends, CGI::Buffer:

  • optimises (optimise_content) and checks (lint_content) HTML;

  • answers conditional requests (If-None-Match, If-Modified-Since) with 304, or 412 for methods other than GET and HEAD;

  • answers a single byte Range with 206, or 416 if it can't be met, when there's no server-side cache;

  • compresses with the first encoding in Accept-Encoding that it can produce (gzip, br or zstd);

  • stores the output in, or sends it from, the server-side cache;

  • sends one Status header, and a Content-Length worked out from what is actually sent.

If any of that fails, a warning is given and the output is sent as the script printed it.

SUBROUTINES/METHODS

init

Set various options and override default values.

Purpose

Configures CGI::Buffer. Options which affect the server-side cache must be given before anything is printed; the others can be given at any time before the script ends.

Arguments

A hash or a hash reference of options, any of which may be left out:

  • generate_etag - send an ETag, and answer If-None-Match with 304. By default, on if the protocol is HTTP/1.1 or later.

  • generate_last_modified - send a Last-Modified header, default on. It needs a cache or an info object.

  • generate_304 - answer conditional requests with 304, default on.

  • compress_content - compress the output if the client accepts it, default on.

  • optimise_content - 0 (default) leaves HTML alone, 1 optimises the HTML and CSS, 2 also does aggressive JavaScript optimisations which may fail.

  • lint_content - check the HTML with HTML::Lint, default off. Errors are sent to the client as a 500, so only use this when developing.

  • cache - a server-side cache: an object that understands get_object(), set(), get() and remove(), whose objects understand value() and created_at(), such as a CHI object. Undef turns the cache off.

  • cache_key - the key for this page in the cache. If not given, one is generated which may not be unique. It should be a unique value dependent upon the values set by the browser.

  • cache_duration - how long to keep pages in the cache, default '10 minutes'. cache_age is an old name for this.

  • logger - an object that understands trace(), debug(), info(), warn() and error(), such as a Log::Log4perl logger.

  • lingua - a CGI::Lingua object; the language becomes part of the generated cache key.

  • info - a CGI::Info object. One is created when needed if it isn't given. It's only taken from the first call.

  • messages - a hash of message keys to replacement text, e.g. to translate the messages, see "MESSAGES".

Items stay in the server-side cache by default for 10 minutes. This can be overridden by the max-age directive of the Cache-Control HTTP header in the request, if it's a number no larger than 600 seconds, and the default can be changed by the cache_duration argument to init().

Responses aren't stored in the server-side cache if their headers include Set-Cookie, a Cache-Control of no-store, no-cache or private, or a Vary on anything other than Accept-Encoding (or Accept-Language when lingua is given), since they would be given to clients that they weren't meant for. Neither are HTML pages with no </html>, which suggests that the script died. A page that includes the value of the client's csrf_token cookie is never stored. The csrf_token and Google Analytics cookies aren't part of the generated cache key, other cookies and PATH_INFO are.

Returns

The string 'CGI::Buffer', so that calls can be chained, e.g. CGI::Buffer::init(...)->is_cached().

Side Effects

Sets the options for the rest of the request. Gives a warning for each unknown or invalid option, which is ignored, and if the cache is given after output has started.

EXAMPLE

    # Put this toward the top of your program before you do anything
    use CHI;
    use CGI::Lingua;
    use CGI::Buffer;

    CGI::Buffer::init(
	generate_etag => 1,	# make good use of client's cache
	generate_last_modified => 1,	# more use of client's cache
	compress_content => 1,	# gzip the output
	optimise_content => 0,	# optimise your program's HTML, CSS and JavaScript
	cache => CHI->new(driver => 'File'),	# cache requests
	cache_key => 'string',	# key for the cache
	cache_duration => '10 minutes',	# how long to store responses in the cache
	logger => $logger,
	lint_content => 0,	# Pass through HTML::Lint
	generate_304 => 1,	# Generate 304: Not modified
	lingua => CGI::Lingua->new(supported => ['en']),
    );

Init allows a reference of the options to be passed. So both of these work:

CGI::Buffer::init(generate_etag => 1);
CGI::Buffer::init({ generate_etag => 1, info => CGI::Info->new() });

Generally speaking, passing by reference is better since it copies less on to the stack.

Alternatively you can give the options when loading the package:

use CGI::Buffer { optimise_content => 1 };

API SPECIFICATION

INPUT

{
	generate_etag => { type => 'boolean', optional => 1 },
	generate_last_modified => { type => 'boolean', optional => 1 },
	generate_304 => { type => 'boolean', optional => 1 },
	compress_content => { type => 'boolean', optional => 1 },
	lint_content => { type => 'boolean', optional => 1 },
	optimise_content => { type => 'integer', min => 0, optional => 1 },
	cache => { type => 'object', can => ['get_object', 'set'], optional => 1 },
	cache_key => { type => 'string', optional => 1 },
	cache_duration => { type => 'string', optional => 1 },
	cache_age => { type => 'string', optional => 1 },
	logger => { type => 'object', can => ['trace', 'debug', 'info', 'warn', 'error'], optional => 1 },
	lingua => { type => 'object', can => 'language', optional => 1 },
	info => { type => 'object', can => ['status', 'host_name'], optional => 1 },
	messages => { type => 'hashref', optional => 1 },
}

OUTPUT

{ type => 'string', value => 'CGI::Buffer' }

MESSAGES

Key             Meaning                                   Resolution
--------------  ----------------------------------------  ---------------------------------
unknown_option  An option that init() doesn't know about  Check the spelling, see above
bad_option      An option of the wrong type               Pass the type listed above
logger_method   The logger lacks a method, so is ignored  Use a logger with all five methods
too_late        The cache was set up after printing       Call init() before printing

FORMAL SPECIFICATION

┌─ Init ──────────────────────────────────────────────┐
│ ΔOptions                                            │
│ params? : OPTION ⇸ VALUE                            │
│ printed : ℕ                                         │
│ result! : STRING                                    │
├─────────────────────────────────────────────────────┤
│ valid == { o : dom params? | o ∈ dom INIT_OPTIONS ∧ │
│            params?(o) ∈ INIT_OPTIONS(o) }           │
│ ∀ o : valid \ {cache} • options'(o) = params?(o)    │
│ cache ∈ valid ∧ params?(cache) = ⊥ ⇒                │
│     options'(cache) = ⊥                             │
│ cache ∈ valid ∧ params?(cache) ≠ ⊥ ∧ can_cache ⇒    │
│     options'(cache) = params?(cache)                │
│ result! = "CGI::Buffer"                             │
└─────────────────────────────────────────────────────┘

PSEUDOCODE

get the options from a hash or a hash reference
for each option
    warn about, and drop, those that are unknown or of the wrong type
replace any messages first, so that the warnings below use them
set each option that can be changed at any time
if anything has been printed, warn that it's too late
if the cache option was given
    if it's undef, turn the cache off
    else if this request may use the cache
        work out how long to keep pages for
        remember the cache and its key
return 'CGI::Buffer'

set_options

Synonym for init, kept for historical reasons.

Purpose, Arguments, Returns and Side Effects

As for "init".

EXAMPLE

CGI::Buffer::set_options(optimise_content => 1);

API SPECIFICATION

INPUT

As for "init".

OUTPUT

{ type => 'string', value => 'CGI::Buffer' }

MESSAGES

As for "init".

FORMAL SPECIFICATION

SetOptions ≙ Init

can_cache

Returns true if the server is allowed to store the results locally.

Purpose

Says whether this request may use the server-side cache. The cache isn't used if the NO_CACHE or NO_STORE environment variables are set, if the request method isn't GET or HEAD, if the client has logged in (the Authorization header or REMOTE_USER is set), or if the client's Cache-Control header includes any of no-store, no-cache, private or max-age=0.

Arguments

None.

Returns

1 if the cache may be used, otherwise 0.

Side Effects

The answer is decided on the first call, and doesn't change for the rest of the request.

EXAMPLE

    if(CGI::Buffer::can_cache()) {
	CGI::Buffer::init(cache => CHI->new(driver => 'File'));
    }

API SPECIFICATION

INPUT

None.

OUTPUT

{ type => 'boolean' }

MESSAGES

None.

FORMAL SPECIFICATION

┌─ CanCache ──────────────────────────────────────────┐
│ ΞEnvironment                                        │
│ result! : 𝔹                                         │
├─────────────────────────────────────────────────────┤
│ result! = true ⇔                                    │
│   {NO_CACHE, NO_STORE, HTTP_AUTHORIZATION,          │
│    REMOTE_USER} ∩ dom env = ∅ ∧                     │
│   (REQUEST_METHOD ¬in; dom env ∨                       │
│    env(REQUEST_METHOD) ∈ {GET, HEAD}) ∧             │
│   directives(env(HTTP_CACHE_CONTROL)) ∩             │
│    {no-store, no-cache, private, max-age=0} = ∅     │
└─────────────────────────────────────────────────────┘

is_cached

Returns true if the output is cached. If it is then it means that all of the expensive routines in the CGI script can be by-passed because we already have the result stored in the cache.

Purpose

Says whether the page is in the server-side cache, in which case the script can exit without printing anything, and the cached copy is sent.

Arguments

None. Any that are given, e.g. the class name when chained after init(), are ignored.

Returns

1 if the page is in the cache, otherwise 0.

An entry that has expired isn't cached, and neither is any entry if the script's file has changed since, or if the age of the script can't be determined. If the cache fails, a warning is given and the output isn't cached.

Side Effects

Remembers the cache entry for when the output is sent. An entry made before the script was last changed is removed from the cache.

EXAMPLE

    # Put this toward the top of your program before you do anything

    # Example key generation - use whatever you want as something
    # unique for this call, so that subsequent calls with the same
    # values match something in the cache
    use CGI::Info;
    use CGI::Lingua;
    use CGI::Buffer;

    my $i = CGI::Info->new();
    my $l = CGI::Lingua->new(supported => ['en']);

    # To use server side caching you must give the cache argument, however
    # the cache_key argument is optional - if you don't give one then one will
    # be generated for you
    if(CGI::Buffer::can_cache()) {
        CGI::Buffer::init(
	    cache => CHI->new(driver => 'File'),
	    cache_key => $i->domain_name() . '/' . $i->script_name() . '/' . $i->as_string() . '/' . $l->language()
        );
        if(CGI::Buffer::is_cached()) {
	    # Output will be retrieved from the cache and sent automatically
	    exit;
        }
    }
    # Not in the cache, so now do our expensive computing to generate the
    # results
    print "Content-type: text/html\n";
    # ...

API SPECIFICATION

INPUT

None.

OUTPUT

{ type => 'boolean' }

MESSAGES

Key             Meaning                                   Resolution
--------------  ----------------------------------------  ---------------------------------
failed          The cache couldn't be read                Check the cache's configuration
no_script_path  CGI::Info can't find the script (fatal)   Check SCRIPT_FILENAME/SCRIPT_NAME
no_script_age   The script's file can't be read (logged)  Check the script's permissions

FORMAL SPECIFICATION

┌─ IsCached ──────────────────────────────────────────┐
│ ΞCache                                              │
│ key : KEY                                           │
│ script_mtime : TIME                                 │
│ now : TIME                                          │
│ result! : 𝔹                                         │
├─────────────────────────────────────────────────────┤
│ result! = true ⇔                                    │
│   key ∈ dom cache ∧                                 │
│   cache(key).value ≠ ⊥ ∧                            │
│   cache(key).expires_at > now ∧                     │
│   script_mtime ≤ cache(key).created_at              │
└─────────────────────────────────────────────────────┘

PSEUDOCODE

if there's no cache, the page isn't cached
look up the key, ignoring expired entries and a failing cache
if there's no entry, or it has no data, the page isn't cached
if the script's age can't be found, the page isn't cached
if the script has changed since the entry was made
    remove the entry, the page isn't cached
otherwise the page is cached

MESSAGES

Every message that CGI::Buffer gives starts with "CGI::Buffer: ". Each can be replaced, for example to translate it, by giving init() a messages hash. Placeholders are written %{name} followed by a sprintf conversion, e.g. %{key}s. A message may instead be a hash of plural forms, one and other, chosen by its count.

    CGI::Buffer::init(messages => {
	unknown_option => 'init: option inconnue %{option}s ignoree',
	too_late => {
	    one => 'Trop tard, %{count}d caractere a ete imprime',
	    other => 'Trop tard, %{count}d caracteres ont ete imprimes',
	},
    });

    Key             Placeholders      Meaning
    --------------  ----------------  ------------------------------------------
    too_late        count             The cache was set up after printing
    unknown_option  option            init() was given an option it doesn't know
    bad_option      option, reason    init() was given an option of the wrong type
    failed          what, error       Something non-essential, e.g. the cache, failed
    no_cache_data   key               The page was cached, but now can't be read
    no_cache_body   key, fields       A cache entry has no body, so is removed
    logger_method   method            The logger lacks a method, so is ignored
    no_script_path  -                 CGI::Info can't find the script (fatal)
    no_script_age   path, error       The script's file can't be read (logged)
    no_info         -                 A CGI::Info object can't be created (fatal)
    send_failed     error             Processing the output failed; it's sent as is
    lint            where, text       An error found by HTML::Lint

None of the messages refers to a person, so there are no gendered forms.

LIMITATIONS

  • All of the state is in package variables, and the work is done in an END block. That fits CGI, where each request is a new process, but it means CGI::Buffer can't be used for more than one request, can't be reset, and can only be tested by running a script in another process. FCGI::Buffer is the object-oriented version for persistent processes.

  • HTML is optimised with regular expressions, not an HTML parser. They are written to run in linear time and to leave <pre> and <textarea> alone, but HTML that's unusual enough can still be changed in ways that are visible, e.g. whitespace in an element styled with white-space: pre. HTML::Packer can take quadratic time on many unterminated <script> tags.

  • Links to this site are shortened to absolute paths (/foo/bar.htm), not relative ones (../bar.htm). Relative links would save a few more bytes, but break pages that use <base href>, and pages reached through more than one path.

  • Only one byte range is supported, and Range is ignored when there's a server-side cache.

  • The server-side cache key can't include headers named by Vary, since it's needed before the script prints them; such pages aren't stored.

  • Only HTML is checked for having been cut short by the script dying.

  • lint_content sends HTML::Lint's errors to the client, and the X-CGI-Buffer header gives the version of CGI::Buffer. Neither should be relied upon in production.

  • init()'s options are checked by hand, rather than with Params::Validate::Strict, and private routines are private by convention rather than enforced with Sub::Private. Both add about 50ms to every request, since a CGI program is compiled afresh each time. For the same reason CGI::Info is only loaded when it's needed, so a script that uses CGI::Info must load it itself.

  • The server-side cache's duration isn't taken from the Expires or Cache-Control: max-age headers that the script gives.

AUTHOR

Nigel Horne, <njh at bandsman.co.uk>

BUGS

CGI::Buffer should be safe even in scripts which produce lots of different output, e.g. e-commerce situations. On such pages, however, I strongly urge to setting generate_304 to 0 and sending the HTTP header "Cache-Control: no-cache".

When using Template, ensure that you don't use it to output to STDOUT, instead you will need to capture into a variable and print that. For example:

my $output;
$template->process($input, $vars, \$output) || ($output = $template->error());
print $output;

Can produce buggy JavaScript if you use the <!-- HIDING technique. This is a bug in JavaScript::Packer, not CGI::Buffer. See https://github.com/nevesenin/javascript-packer-perl/issues/1#issuecomment-4356790

Mod_deflate can confuse this when compressing output. Ensure that deflation is off for .pl files:

SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|pl)$ no-gzip dont-vary

If-None-Match uses the weak comparison of RFC 9110, so a client that has the compressed version of a page is sent a 304 when it asks for the uncompressed one, and vice versa.

I advise adding CGI::Buffer as the last use statement so that it is cleared up first. In particular it should be loaded after Log::Log4perl, if you're using that, so that any messages it produces are printed after the HTTP headers have been sent by CGI::Buffer;

CGI::Buffer is not compatible with FastCGI, use FCGI::Buffer instead.

Please report any bugs or feature requests to bug-cgi-buffer at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=CGI-Buffer. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SEE ALSO

HTML::Packer, HTML::Lint, FCGI::Buffer

SUPPORT

You can find documentation for this module with the perldoc command.

perldoc CGI::Buffer

You can also look for information at:

ACKNOWLEDGEMENTS

The inspiration and code for some of this is cgi_buffer by Mark Nottingham: https://www.mnot.net/blog/2003/04/24/etags.

LICENSE AND COPYRIGHT

The licence for cgi_buffer is:

"(c) 2000 Copyright Mark Nottingham <mnot@pobox.com>

This software may be freely distributed, modified and used,
provided that this copyright notice remain intact.

This software is provided 'as is' without warranty of any kind."

The rest of the program is Copyright 2011-2026 Nigel Horne, and is released under the following licence: GPL2