NAME

PAR::Tutorial - Cross-Platform Packaging and Deployment with PAR

SYNOPSIS

This is a tutorial on PAR, first appeared at the 7th Perl Conference. The HTML version of this tutorial is available online as http://aut.dyndns.org/par-tutorial/.

DESCRIPTION

On Deploying Perl Applications

% sshnuke.pl 10.2.2.2 -rootpw="Z1ON0101"
Perl v5.6.1 required--this is only v5.6.0, stopped at sshnuke.pl line 1.
BEGIN failed--compilation aborted at sshnuke.pl line 1.

Q: "Help! I can't run your program!"
A1: Install Perl & perl -MCPAN -e'install(...)'
- How do we know which modules are needed?
- New versions of CPAN modules may break sshnuke.pl
A2: Install Perl & tar zxf my_perllib.tgz
- Possibly overwriting existing modules; not cross-platform at all
A3: Use the executable generated by perlcc sshnuke.pl
- Impossible to debug; perlcc usually does not work anyway

PAR, the Perl Archive Toolkit

Do what JAR (Java Archive) does for Perl
- Aggregates modules, scripts and other files into a Zip file
- Easy to generate, update and extract
- Version consistency: solves forward-compatibility problems
- Developed by community: par@perl.org
PAR files can be packed into self-contained scripts
- Automatically scans perl script for dependencies
- Bundles all necessary 3rd-party modules with it
- Requires only core Perl to run on the target machine
- PAR also comes with pp, the Perl Packager:
```
% pp -o sshnuke.exe sshnuke.pl	# stand-alone executable!
```

Simple Packaging

PAR files are just Zip files with modules in it

Any Zip tools can generate them:

% zip foo.par Hello.pm World.pm	# pack two modules
% zip -r bar.par lib/		# grab all modules in lib/

To load modules from PAR files:

use PAR;
use lib "foo.par";		# the .par part is optional
use Hello;

This also works:

use PAR "/home/mylibs/*.par";	# put all of them into @INC
use Hello;

PAR Loaders

Use par.pl to run files inside a PAR archive:

% par.pl foo.par		# looks for 'main.pl' by default
% par.pl foo.par test.pl	# runs script/test.pl in foo.par

Same thing, with the stand-alone parl or parl.exe:

% parl foo.par			# no perl or PAR.pm needed!
% parl foo.par test.pl		# ditto

The PAR loader can prepend itself to a PAR file:
- -b bundles non-core modules needed by PAR.pm:
```
% par.pl -b -O./foo.pl foo.par	# self-contained script
```
- -B bundles core modules in addition to -b:
```
% parl -B -O./foo.exe foo.par	# self-contained binary
```

Dependency Scanning

Recursively scan dependencies with scandeps.pl:

% scandeps.pl sshnuke.pl
# Legend: [C]ore [X]ternal [S]ubmodule [?]NotOnCPAN
'Crypt::SSLeay'       => '0', #  X   #
'Net::HTTP'           => '0', #      #
'Crypt::SSLeay::X509' => '0', # S    # Crypt::SSLeay
'Net::HTTP::Methods'  => '0', # S    # Net::HTTP
'Compress::Zlib'      => '0', #  X   # Net::HTTP::Methods

Scan an one-liner, list all involved files:

% scandeps.pl -V -e "use Dynaloader;"
...
# auto/DynaLoader/dl_findfile.al [autoload]
# auto/DynaLoader/extralibs.ld [autoload]
# auto/File/Glob/Glob.bs [data]
# auto/File/Glob/Glob.so [shared]
...

Perl Packager: `pp`

Combines scanning, zipping and loader-embedding:

% pp -o out.exe src.pl		# self-contained .exe
% out.exe			# runs anywhere on the same OS

Bundle additional modules:

% pp -o out.exe -M CGI src.pl	# pack CGI + its dependencies, too

Pack one-liners:

% pp -o out.exe -e 'print "Hi!"'   # turns one-liner into executable

Generate PAR files instead of executables:

% pp -p src.pl			# makes 'source.par'
% pp -B -p src.pl		# include core modules

How it works

Command-line options are almost identical to perlcc's
- Also supports gcc-style long options:
```
% pp --gui --verbose --output=out.exe src.pl
```
Small initial overhead; no runtime overhead
Dependencies are POD-stripped before packing
Loads modules directly into memory on demand
Shared libraries (DLLs) are extracted with File::Temp
Works on Perl 5.6.0 or above
Tested on Win32 (VC++ and MinGW), FreeBSD, NetBSD, Linux, MacOSX, Cygwin, AIX, Solaris, HP-UX, Tru64...

Aggregating multiple programs

A common question:

> I have used pp to make several standalone applications which work
> great, the only problem is that for each executable that I make, I am
> assuming the parl.exe is somehow bundled into the resulting exe.

The obvious workaround:

You can ship parl.exe by itself, along with .par files built
by "pp -p", and run those PAR files by associating them to parl.exe.

On platforms that have ln, there is a better solution:

% pp --output=a.out a.pl b.pl	# two scripts in one!
% ln a.out b.out		# symlink also works
% ./a.out			# runs a.pl
% ./b.out			# runs b.pl

Cross-platform Packages

Of course, there is no cross-platform binary format
Pure-perl PAR packages are cross-platform by default
- However, XS modules are specific to Perl version and platform
- Multiple versions of a XS module can co-exist in a PAR file

Suppose we need out.par on both Win32 and Finix:

C:\> pp --multiarch --output=out.par src.pl
...copy src.pl and out.par to a Finix machine...
% pp --multiarch --output=out.par src.pl

Now it works on both platforms:

% parl out.par			# runs src.pl
% perl -MPAR=out.par -e '...'	# uses modules inside out.par

The Anatomy of a PAR file

Modules can reside in several directories:

/			# casual packaging only
/lib/			# standard location
/arch/			# for creating from blib/ 
/i386-freebsd/		# i.e. $Config{archname}
/5.8.0/		# i.e. Perl version number
/5.8.0/i386-freebsd/	# combination of the two above

Scripts are stored in one of the two locations:

/			# casual packaging only
/script/		# standard location

Shared libraries may be architecture- or perl-version-specific:
```
/shlib/(5.8.0/)?(i386-freebsd/)?
```
PAR files may recursively contain other PAR files:
```
/par/(5.8.0/)?(i386-freebsd/)?
```

Special files

MANIFEST
- Index of all files inside PAR
- Can be parsed with ExtUtils::Manifest
META.yml
- Dependency, license, runtime options
- Can be parsed with YAML
SIGNATURE
- OpenPGP-signed digital signature
- Can be parsed and verified with Module::Signature

Advantages over perlcc, PerlApp and Perl2exe

This is not meant to be a flame
- All three maintainers have contributed to PAR directly; I'm grateful
perlcc
- "The code generated in this way is not guaranteed to work... Use for production purposes is strongly discouraged." (from perldoc perlcc)
- Guaranteed to not work is more like it
PerlApp / Perl2exe
- Expensive: Need to pay for each upgrade
- Non-portable: Only available for limited platforms
- Proprietary: Cannot extend its features or fix bugs
- Obfuscated: Vendor and black-hats can see your code, but you can't
- Inflexible: Does not work with existing Perl installations

MANIFEST: Best viewed with Mozilla

The URL of MANIFEST inside /home/autrijus/foo.par:
```
jar:file:///home/autrijus/foo.par!/MANIFEST
```
Open it in a Gecko browser (e.g. Netscape 6+) with Javascript enabled:
No needed to unzip anything; just click on files to view them

META.yml: Metadata galore

Static, machine-readable distribution metadata
- Supported by Module::Build, ExtUtils::MakeMaker, Module::Install

A typical pp-generated META.yml looks like this:

build_requires: {}
conflicts: {}
dist_name: out.par
distribution_type: par
dynamic_config: 0
generated_by: 'Perl Packager version 0.03'
license: unknown
par:
  clean: 0
  signature: ''
  verbatim: 0
  version: 0.68

The par: settings controls its runtime behavior

SIGNATURE: Signing and verifying packages

OpenPGP clear-signed manifest with SHA1 digests
- Supported by Module::Signature, CPANPLUS and Module::Build

A typical SIGNATURE looks like this:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

SHA1 8a014cd6d0f6775552a01d1e6354a69eb6826046 AUTHORS
...
-----BEGIN PGP SIGNATURE-----
...
-----END PGP SIGNATURE-----

Use pp and cpansign to work with signatures:

% pp -s -o foo.par bar.pl	# make and sign foo.par from bar.pl
% cpansign -s foo.par	# sign this PAR file
% cpansign -v foo.par	# verify this PAR file

Perl Servlets with Apache::PAR

Framework for self-contained Web applications
- Similar to Java's "Web Application Archive" (WAR) files
- Works with mod_perl 1.x or 2.x
A complete web application inside a .par file
- Apache configuration, static files, Perl modules...
- Supports Static, Registry and PerlRun handlers
- Can also load all PARs under a directory

One additional special file: web.conf

Alias /myapp/cgi-perl/ ##PARFILE##/
<Location /myapp/cgi-perl>
    Options +ExecCGI
    SetHandler perl-script
    PerlHandler Apache::PAR::Registry
</Location>

Hon Dah, A-par-che!

First, make a hondah.par from an one-liner:

# use the "web.conf" from the previous slide
% pp -p -o hondah.par -e 'print "Hon Dah!\n"' \
     --add web.conf
% chmod a+x hondah.par

Add this to httpd.conf, then restart apache:

<IfDefine MODPERL2>
PerlModule Apache2
</IfDefine>
PerlAddVar PARInclude /home/autrijus/hondah.par
PerlModule Apache::PAR

Test it out:

% GET http://localhost/myapp/cgi-perl/main.pl
Hon Dah!

Instant one-liner web application that works!

On-demand library fetching

With LWP installed, your can use remote PAR files:

use PAR;
use lib 'http://aut.dyndns.org/par/DBI-latest.par';
use DBI;    # always up to date!

Modules are now cached under $ENV{PAR_GLOBAL_TEMP}
Auto-updates with LWP::Simple::mirror
- Download only if modified
- Safe for offline use after the first time
- May use SIGNATURE to prevent DNS-spoofing
Makes large-scale deployment a breeze
- Upgrades from a central location
- No installers needed

Code Obfuscation

Also known as source-hiding techniques
- It is not encryption
- Offered by PerlApp, Perl2Exe, Stunnix...
Usually easy to defeat
- Take optree dump from memory, feed to B::Deparse
- If you just want to stop a casual grep, "deflate" already works
PAR now supports pluggable input filters with pp -f
- Bundled examples: Bleach, PodStrip and PatchContent
- True encryption using Crypt::*
- Or even _product activation_ over the internet
Alternatively, just keep core logic in your server and use RPC

Accessing packed files

To get the host archive from a packed program:

my $zip = PAR::par_handle($0);	# an Archive::Zip object
my $content = $zip->contents('MANIFEST');

Same thing, but with read_file():

my $content = PAR::read_file('MANIFEST');

Loaded PAR files are stored in %PAR::LibCache:

use PAR '/home/mylibs/*.par';
while (my ($filename, $zip) = each %PAR::LibCache) {
    print "[$filename - MANIFEST]\n";
    print $zip->contents('MANIFEST');
}

Packing GUI applications

GUI toolkits often need to link with shared libraries:

# search for libncurses under library paths and pack it
% pp -l ncurses curses_app.pl	# same for Tk, Wx, Gtk, Qt...

Use pp --gui on Win32 to eliminate the console window:

# pack 'src.pl' into a console-less 'out.exe' (Win32 only)
% pp --gui -o out.exe src.pl

"Can't locate Foo/Widget/Bar.pm in @INC"?
- Some toolkits (notably Tk) autoloads modules without use or require
- Hence pp and Module::ScanDeps may fail to detect them
- Tk problems mostly fixed by now, but other toolkits may still break
- You can work around it with pp -M or an explicit require
- Or better, send a short test-case to par@perl.org so we can fix it

Precompiled CPAN distributions

Installing XS extensions from CPAN was difficult
- Some platforms do not come with a compiler (Win32, MacOSX...)
- Some headers or libraries may be missing
- PAR.pm itself used to suffer from both problems

...but not anymore -- Module::Install to the rescue!

# same old Makefile.PL, with a few changes
use inc::Module::Install;	# was "use ExtUtils::MakeMaker;"
WriteMakefile( ... );		# same as the original
check_nmake();			# make sure the user have nmake
par_base('AUTRIJUS');		# your CPAN ID or a URL
fetch_par() unless can_cc();	# use precompiled PAR only if necessary

Users will not notice anything, except now it works
- Of course, you still need to type make par and upload the precompiled package
- PAR users can also install it directly with parl -i

Platform-specific Tips

Win32 and other icon-savvy platforms
- Needs 3rd-party tools to add icons to pp-generated executables
- PE Header manipulation in Perl -- volunteers wanted!
Linux and other libc-based platforms
- Try to avoid running pp on a bleeding-edge version of the OS
- Older versions with an earlier libc won't work with new ones
Solaris and other zlib-lacking platforms (but not Win32)
- You need a static-linked Compress::Zlib before installing PAR
- In the future, PAR may depend on Compress::Zlib::Static instead
Any platform with limited bandwidth or disk space
- Use UPX to minimize the executable size

Thank you!

Additional resources
- Mailing list: par@perl.org
- Subscribe: Send a blank email to par-subscribe@perl.org
- List archive: http://nntp.x.perl.org/group/perl.par
- PAR::Intro: http://search.cpan.org/dist/PAR/lib/PAR/Intro.pod
- Apache::PAR: http://search.cpan.org/dist/Apache-PAR/
- Module::Install: http://search.cpan.org/dist/Module-Install/
Any questions?

Bonus Slides: PAR Internals

Overview of PAR.pm's Implementation

Here begins the scary part
- Grues, Dragons and Jabberwocks abound...
- You are going to learn weird things about Perl internals
PAR invokes four areas of Perl arcana:
- @INC code references
- On-the-fly source filtering
- Overriding DynaLoader::bootstrap() to handle XS modules
- Making self-bootstrapping binary executables
The first two only works on 5.6 or later
- DynaLoader and %INC are there since Perl 5 was born
- PAR currently needs 5.6, but a 5.005 port is possible

Code References in @INC

On 1999-07-19, Ken Fox submitted a patch to P5P
- To _enable using remote modules_ by putting hooks in @INC
- It's accepted to come in Perl 5.6, but undocumented until 5.8
- Type perldoc -f require to read the nitty-gritty details

Coderefs in @INC may return a fh, or undef to 'pass':

push @INC, sub {
    my ($coderef, $filename) = @_;  # $coderef is \&my_sub
    open my $fh, "wget ftp://example.com/$filename |";
    return $fh;	# using remote modules, indeed!
};

Perl 5.8 let you open a file handle to a string, so we just use that:

open my $fh, '<', \($zip->memberNamed($filename)->contents);
return $fh;

But Perl 5.6 does not have that, and I don't want to use temp files...

Source Filtering without Filter::* Modules

... Undocumented features to the rescue!
- It turns out that @INC hooks can return two values
- The first is still the file handle
- The second is a code reference for line-by-line source filtering!

This is how Acme::use::strict::with::pride works:

# Force all modules used to use strict and warnings
open my $fh, "<", $filename or return;
my @lines = ("use strict; use warnings;\n", "#line 1 \"$full\"\n");
return ($fh, sub {
    return 0 unless @lines;	
    push @lines, $_; $_ = shift @lines; return length $_;
});

Source Filtering without Filter::* Modules (cont.)

But we don't really have a filehandle for anything
Another undocumented feature saves the day!

We can actually omit the first return value altogether:

# Return all contents line-by-line from the file inside PAR
my @lines = split(
    /(?<=\n)/,
    $zip->memberNamed($filename)->contents
);
return (sub {
    $_ = shift(@lines);
    return length $_;
});

Overriding DynaLoader::bootstrap

XS modules have dynamically loaded libraries
- They cannot be loaded as part of a zip file, so we extract them out
- Must intercept DynaLoader's library-finding process
Module names are passed to bootstrap for XS loading
- During the process, it calls dl_findfile to locate the file
- So we install pre-hooks around both functions
Our _bootstrap just checks if the library is in PARs
- If yes, extract it to a File::Temp temp file
  - The file will be automatically cleaned up when the program ends
- It then pass the arguments to the original bootstrap
- Finally, our dl_findfile intercepts known filenames and return it

Anatomy of a Self-Contained PAR executable

The par script ($0) itself
- May be in plain-text or native executable format
Any number of embedded files
- Typically used to bootstrap PAR's various dependencies
- Each section begins with the magic string "FILE"
- Length of filename in pack('N') format and the filename (auto/.../)
- File length in pack('N') and the file's content (not compressed)
One PAR file
- Just a regular zip file with the magic string "PK\003\004"
Ending section
- A pack('N') number of the total length of FILE and PAR sections
- Finally, there must be a 8-bytes magic string: "\012PAR.pm\012"

Self-Bootstrapping Tricks

All we can expect is a working perl interpreter
- The self-contained script *must not* use any modules at all
- But to process PAR files, we need XS modules like Compress::Zlib
Answer: bundle all modules + libraries used by PAR.pm
- That's what the FILE section in the previous slide is for
- Load modules to memory, and write object files to disk
- Then use a local @INC hook to load them on demand
Minimizing the amount of temporary files
- First, try to load PerlIO::scalar and File::Temp
- Set up an END hook to unlink all temp files up to this point
- Load other bundled files, and look in the compressed PAR section
- This can be much easier with a pure-perl inflate(); patches welcome!

Thank you (again)!

Any questions, please?

AUTHORS

Audrey Tang <autrijus@autrijus.org>

http://par.perl.org/ is the official PAR website. You can write to the mailing list at <par@perl.org>, or send an empty mail to <par-subscribe@perl.org> to participate in the discussion.

Please submit bug reports to <bug-par@rt.cpan.org>.

COPYRIGHT

This document is free documentation; you can redistribute it and/or modify it under the same terms as Perl itself.

See http://www.perl.com/perl/misc/Artistic.html

To install PAR, copy and paste the appropriate command in to your terminal.

cpanm

cpanm PAR

CPAN shell

perl -MCPAN -e shell
install PAR

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)