NAME

Test::Against::Dev - Test CPAN modules against Perl dev releases

SYNOPSIS

my $self = Test::Against::Dev->new( {
    application_dir => '/path/to/application',
} );

my ($tarball_path, $work_dir) = $self->perform_tarball_download( {
    host                => 'ftp.funet.fi',
    hostdir             => /pub/languages/perl/CPAN/src/5.0,
    perl_version        => 'perl-5.27.6',
    compression         => 'gz',
    work_dir            => "~/tmp/Downloads",
    verbose             => 1,
    mock                => 0,
} );

my $this_perl = $self->configure_build_install_perl({
    verbose => 1,
});

my $this_cpanm = $self->fetch_cpanm( { verbose => 1 } );

my $gzipped_build_log = $self->run_cpanm( {
    module_file => '/path/to/cpan-river-file.txt',
    title       => 'cpan-river-1000',
    verbose     => 1,
} );

DESCRIPTION

Who Should Use This Library?

This library should be used by anyone who wishes to assess the impact of month-to-month changes in the Perl 5 core distribution on the installability of libraries found on the Comprehensive Perl Archive Network (CPAN).

The Problem to Be Addressed

This problem is typically referred to as Blead Breaks CPAN (or BBC for short). Perl 5 undergoes an annual development cycle characterized by monthly releases whose version numbers follow the convention of 5.27.0, 5.27.1, etc., where the middle digits are always odd numbers. (Annual production releases and subsequent maintenance releases have even-numbered middle digits, e.g., 5.26.0, 5.26.1, etc.) A monthly development release is essentially a roll-up of a month's worth of commits to the master branch known as blead (pronounced "bleed"). Changes in the Perl 5 code base have the potential to adversely impact the installability of existing CPAN libraries. Hence, various individuals have, over the years, developed ways of testing those libraries against blead and reporting problems to those people actively involved in the ongoing development of the Perl 5 core distribution -- people typically referred to as the Perl 5 Porters.

This library is intended as a contribution to those efforts. It is intended to provide a monthly snapshot of the impact of Perl 5 core development on important CPAN libraries.

The Approach Test-Against-Dev Currently Takes and How It May Change in the Future

Unlike other efforts, Test-Against-Dev does not depend on test reports sent to CPANtesters.org. Hence, it should be unaffected by any technical problems which that site may face. As a consequence, however, a user of this library must be willing to maintain more of her own local infrastructure than a typical CPANtester would maintain.

While this library could, in principle, be used to test the entirety of CPAN, it is probably better suited for testing selected subsets of CPAN libraries which the user deems important to her individual or organizational needs.

This library is currently focused on monthly development releases of Perl 5. It does not directly provide a basis for identifying individual commits to blead which adversely impacted particular CPAN libraries. It "tests against dev" more than it "tests against blead" -- hence, the name of the library. However, once it has gotten some production experience, it may be extended to, say, measure the effect of individual commits to blead on CPAN libraries using the previous monthly development release as a baseline.

This library is currently focused on Perl 5 libraries publicly available on CPAN. In the future, it may be extended to be able to include an organization's private libraries as well.

This library is currently focused on blead, the master branch of the Perl 5 core distribution. However, it could, in principle, be extended to assess the impact on CPAN libraries of code in non-blead ("smoke-me") branches as well.

What Is the Result Produced by This Library?

Currently, if you run code built with this library on a monthly basis, you will produce an updated version of a pipe-separated-values (PSV) plain-text file suitable for opening in a spreadsheet. The columns in that PSV file will be these:

dist
perl-5.27.0.author
perl-5.27.0.distname
perl-5.27.0.distversion
perl-5.27.0.grade
perl-5.27.1.author
perl-5.27.1.distname
perl-5.27.1.distversion
perl-5.27.1.grade
...

So the output for particular CPAN libraries will look like this:

dist|perl-5.27.0.author|perl-5.27.0.distname|perl-5.27.0.distversion|perl-5.27.0.grade|perl-5.27.1.author|perl-5.27.1.distname|perl-5.27.1.distversion|perl-5.27.1.grade|...
Acme-CPANAuthors|ISHIGAKI|Acme-CPANAuthors-0.26|0.26|PASS|ISHIGAKI|Acme-CPANAuthors-0.26|0.26|PASS|...
Algorithm-C3|HAARG|Algorithm-C3-0.11|0.11|PASS|HAARG|Algorithm-C3-0.11|0.11|PASS|...

If a particular CPAN library receives a grade of PASS one month and a grade of FAIL month, it ought to be inspected for the cause of that breakage. Sometimes the change in Perl 5 is wrong and needs to be reverted. Sometimes the change in Perl 5 is correct (or, at least, plausible) but exposes sub-optimal code in the CPAN module. Sometimes the failure is due to external conditions, such as a change in a C library on the testing platform. There's no way to write code to figure out which situation -- or mix of situations -- we are in. The human user must intervene at this point.

What Preparations Are Needed to Use This Library?

Platform

The user should select a machine/platform which is likely to be reasonably stable over one Perl 5 annual development cycle. We presume that the platform's system administrator will be updating system libraries for security and other reasons over time. But it would be a hassle to run this software on a machine scheduled for a complete major version update of its operating system.
Perl 5 Configuration

The user must decide on a Perl 5 configuration before using Test-Against-Dev on a regular basis and then must refrain from changing that configuration over the course of the testing period. Otherwise, the results may reflect changes in that configuration rather than changes in Perl 5 core distribution code or changes in the targeted CPAN libraries.

By "Perl 5 configuration" we mean the way one calls Configure when building Perl 5 from source, <e.g.>:
```
sh ./Configure -des -Dusedevel \
    -Duseithreads \
    -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"
```
So, you should not configure without threads one month but with threads another month. You should not switch to debugging builds half-way through the testing period.
Selection of CPAN Libraries for Testing

This is the most important step in preparation to use this library.

When you use this library, you are in effect saying: Here is a list of CPAN modules important enough to me that I don't want to see them start breaking in the course of Perl's annual development cycle. (If they do break, then the Perl 5 Porters and the modules' authors/maintainers must address how to handle the breakage.) To keep track of the problem, I'm going to build perl from each monthly release and attempt to install this entire list against that perl.

Hence, once you decide to track a certain CPAN library, you should continue to include it in your list of modules to be tracked for the balance of the development cycle. You can, it is true, add additional modules to your list part way through the development cycle. You simply won't have the same baseline data that you have for the modules you selected at the very beginning.

Here are some approaches that come to mind:
- CPAN river
  
  The CPAN river is a concept developed by Neil Bowers and other participants in the Perl Toolchain Gang and Perl QA Hackathons and Summits. The concept starts from the premise that CPAN libraries upon which many other CPAN libraries depend are more important than those upon which few other libraries depend. That's a useful definition of importance even if it is not strictly true. Modules "way upstream" feed modules and real-world code "farther downstream". Hence, if Perl 5's development branch changes in a way such that "upstream" modules start to fail to configure, build, test and install correctly, then we have a potentially serious problem. The author of this library has primarily developed it with the idea that it would be run monthly to see what happens with the 1000 "farthest upstream" modules -- the so-called "CPAN River Top 1000".
- Organizational dependencies
  
  Many organizations use technologies such as Carton and cpanfile to keep track of their dependencies on CPAN libraries. The lists compiled by such applications could very easily be translated into a list of modules tested once a month against a Perl development release.
- What repeatedly breaks
  
  Certain CPAN libraries get broken relatively frequently. While this can happen because of sub-standard coding practices in those libraries, it more often happens because these libraries, in order to do what they want to do, reach down deep into the Perl 5 guts and use undocumented or not publicly supported features of Perl.

Notice of Breaking Change in Version 0.06 (March 20 2018)

If you are first using Test-Against-Dev in version 0.06 released on the date above, you may skip this section.

If you used Test-Against-Dev in an ongoing way prior to that version, please be advised that the library now creates a slightly different directory structure beneath the directory specified by the value of application_dir passed to the constructor. Up through version 0.05, that structure looked like this:

$> find . -maxdepth 4 -type d
./results
./results/perl-5.27.6
./results/perl-5.27.6/storage
./results/perl-5.27.6/analysis
./results/perl-5.27.6/analysis/01
./results/perl-5.27.6/buildlogs
./testing
./testing/perl-5.27.6
./testing/perl-5.27.6/.cpanreporter
./testing/perl-5.27.6/.cpanm
./testing/perl-5.27.6/.cpanm/work
./testing/perl-5.27.6/lib
./testing/perl-5.27.6/lib/site_perl
./testing/perl-5.27.6/lib/5.27.6
./testing/perl-5.27.6/bin

The results/<perl-version>/analysis/01 directory would hold .log.json files like these:

$> ls -l ./results/perl-5.27.6/analysis/01 | head -5
total 5824
-rw-r--r-- 1 jkeenan jkeenan    757 Dec 16 13:58 ABH.Mozilla-CA-20160104.log.json
-rw-r--r-- 1 jkeenan jkeenan   6504 Dec 16 13:58 ABIGAIL.Regexp-Common-2017060201.log.json
-rw-r--r-- 1 jkeenan jkeenan  11639 Dec 16 13:58 ABW.Template-Toolkit-2.27.log.json
-rw-r--r-- 1 jkeenan jkeenan    645 Dec 16 13:58 ABW.XML-Namespace-0.02.log.json

In versions 0.06 and later, the .log.json files are placed one directory higher, i.e., in results/<perl-version>/analysis. An analysis/01 directory is no longer created, as it was deemed unnecessary. Please upgrade to a newer version at the completion of your tracking of the Perl 5.27 development cycle, i.e., once Perl 5.28.0 has been released.

As a consequence of this change, the analyze_json_logs() method no longer needs a key-value pair like run => 1 in the hash reference passed to the method as argument.

METHODS

`new()`

Purpose

Test::Against::Dev constructor. Guarantees that the top-level directory for the application (application_dir) already exists, then creates two directories thereunder: testing/ and results/.
Arguments
```
my $self = Test::Against::Dev->new( {
    application_dir => '/path/to/application',
} );
```
Takes a hash reference with the following elements:
- application_dir
  
  String holding path to the directory which will serve as the top level for your application.
Return Value

Test::Against::Dev object.
Comment

This class has two possible constructors: this method and new_from_existing_perl_cpanm() (see below). Use new() when you need to do a fresh install of a perl by compiling it from a downloaded tarball. Use new_from_existing_perl_cpanm() when you have already installed such a perl on disk and have installed a cpanm against that perl.

The method will guarantee that underneath the application directory there are two directories: testing and results.

`perform_tarball_download()`

Purpose
Arguments
```
($tarball_path, $work_dir) = $self->perform_tarball_download( {
    host                => 'ftp.funet.fi',
    hostdir             => /pub/languages/perl/CPAN/src/5.0,
    perl_version        => 'perl-5.27.6',
    compression         => 'gz',
    work_dir            => "~/tmp/Downloads",
    verbose             => 1,
    mock                => 0,
} );
```
Hash reference with the following elements:
- host
  
  String. The FTP mirror from which you wish to download a tarball of a Perl release. Required.
- hostdir
  
  String. The directory on the FTP mirror specified by host in which the tarball is located. Required.
- perl_version
  
  String denoting a Perl release. The string must start with perl-, followed by the major version, minor version and patch version delimited by periods. The major version is always 5. Required.
- compression
  
  String denoting the compression format of the tarball you wish to download. Eligible compression formats are gz, bz2 and bz2. Required.
  
  Note that not all compression formats are available for all tarballs on our FTP mirrors and that the compression formats offered may change over time.
  
  Note further that gz is currently the recommended format, as the other methods have not been thorougly tested.
- work_dir
  
  String holding absolute path to the directory in which the work of configuring and building the new perl will be performed. Optional; if not provided a temporary directory created via File::Temp::tempdir() will be used.
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
- mock
  
  Display the expected results of the download on STDOUT, but don't actually do it. Optional; defaults to being off; provide a Perl-true value to turn it on. Any program using this option will terminate with a non-zero status once the results have been displayed.
Return Value

Returns a list of two elements:
- Tarball path
  
  String holding absolute path to the tarball once downloaded.
- Work directory
  
  String holding path of directory in which work of configuring and building perl will be performed. (This is probably only useful if you want to see the path to the temporary directory. It will be uninitialized if mock is turned on.)
Comment

The method guarantees the existence of a directory whose name will be the value of the perl_version argument and which will be found underneath the testing directory (discussed in new() above). This "release directory" -- accessible by calling $self-get_release_dir()> -- will be the directory below which a new perl will be installed.

`configure_build_install_perl()`

Purpose

Configures, builds and installs perl from the downloaded tarball.
Arguments
```
my $this_perl = $self->configure_build_install_perl({
    verbose => 1,
});
```
Hash reference with the following elements:
- configure_command
  
  String holding a shell command to call Perl's Configure program with command-line options. Optional; will default to:
```
my $release_dir = $self->get_release_dir();

sh ./Configure -des -Dusedevel -Uversiononly -Dprefix=$release_dir \
    -Dman1dir=none -Dman3dir=none
```
  The spelling of the command is subsequently accessible by calling $self-access_configure_command()>.
- make_install_command
  
  String holding a shell command to build and install perl underneath the release directory. Optional; will default to:
```
make install
```
  The spelling of the command is subsequently accessible by calling $self-access_make_install_command()>.
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off. Set to 1 (recommended) for moderate verbosity. Set to 2 for extra verbosity (full output of decompression commands, Configure and make). Scope is limited to this method.
Return Value

String holding absolute path to the new perl executable. This location can subsequently be accessed by calling $self-get_this_perl()>.
Comment

The new perl executable will sit two levels underneath the release directory in a directory named bin/. That directory will sit next to a directory named lib/ under which libraries will be installed. Those locations can subsequently be accessed by calling $self-get_bin_dir()> and $self-get_lib_dir()>, respectively.

`fetch_cpanm()`

Purpose

Fetch the fatpacked cpanm executable and install it against the newly installed perl.
Arguments
```
my $this_cpanm = $self->fetch_cpanm( { verbose => 1 } );
```
Hash reference with these elements:
- uri
  
  String holding URI from which cpanm will be downloaded. Optional; defaults to http://cpansearch.perl.org/src/MIYAGAWA/App-cpanminus-1.7043/bin/cpanm.
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
Return Value

String holding the absolute path to the newly installed cpanm executable.
Comment

The executable's location can subsequently be accessed by calling $self-get_this_cpanm()>. The method also guarantees the existence of a .cpanm directory underneath the release directory. This directory can subsequently be accessed by calling $self-get_cpanm_dir()>.

`run_cpanm()`

Purpose

Use cpanm to install selected Perl modules against the perl built for testing purposes.
Arguments

Two mutually exclusive interfaces:
- Modules provided in a list
```
$gzipped_build_log = $self->run_cpanm( {
    module_list => [ 'DateTime', 'AnyEvent' ],
    title       => 'two-important-libraries',
    verbose     => 1,
} );
```
- Modules listed in a file
```
$gzipped_build_log = $self->run_cpanm( {
    module_file => '/path/to/cpan-river-file.txt',
    title       => 'cpan-river-1000',
    verbose     => 1,
} );
```
Each interface takes a hash reference with the following elements:
- module_list OR module_file
  
  Mutually exclusive; must use one or the other but not both.
  
  The value of module_list must be an array reference holding a list of modules for which you wish to track the impact of changes in the Perl 5 core distribution over time. In either case the module names are spelled in Some::Module format -- i.e., double-colons -- rather than in Some-Module format (hyphens).
- title
  
  String which will be used to compose the name of project-specific output files. Required.
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
Return Value

String holding the absolute path of a gzipped copy of the build.log generated by the cpanm run which this method conducts. The basename of this file, using the arguments supplied, would be:
```
cpan-river-1000.perl-5.27.6.01.build.log.gz
```

Comment

The method guarantees the existence of several directories underneath the "results" directory discussed above. These are illustrated as follows:

/path/to/application/results/
                    /results/perl-5.27.6/
                    /results/perl-5.27.6/analysis/
                    /results/perl-5.27.6/buildlogs/
                    /results/perl-5.27.6/storage/

`analyze_cpanm_build_logs()`

Purpose

Parse the build.log created by running run_cpanm(), creating JSON files which log the results of attempting to install each module in the list or file.
Arguments
```
$ranalysis_dir = $self->analyze_cpanm_build_logs( { verbose => 1 } );
```
Hash reference which, at the present time, can only take one element: verbose. Optional.
Return Value

String holding absolute path to the directory holding .log.json files for a particular run of run_cpanm().
Comment

`analyze_json_logs()`

Purpose

Create a character-delimited-values file summarizing the results of a given run. The delimiter defaults to a pipe (|), thereby creating a pipe-separated values file (.psv), but you may select a comma (,), generating a comma-separated-values file (.csv) as well.
Arguments
```
my $fcdvfile = $self->analyze_json_logs( { verbose => 1, sep_char => '|' } );
```
Hash reference with these elements:
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
- sep_char
  
  Delimiter character. Optional; defaults to pipe (|), but comma (,) may also be chosen.
Return Value

String holding absolute path to the .psv or .csv file created.
Comment

As a precaution, the function creates a tarball to archive the .log.json files for a given run.

`new_from_existing_perl_cpanm()`

Purpose

Alternate constructor to be used when you have already built a perl executable to be used in tracking Perl development and have installed a cpanm against that perl.
Arguments
```
$self = Test::Against::Dev->new_from_existing_perl_cpanm( {
    path_to_perl    => '/path/to/perl-5.27.0/bin/perl',
    application_dir => '/path/to/application',
    perl_version    => 'perl-5.27.0',
} );
```
Takes a hash reference with the following elements:
- path_to_perl
  
  String holding path to an installed perl executable. Required.
- application_dir
  
  String holding path to the directory which will serve as the top level for your application. (Same meaning as in new().) Required.
- perl_version
  
  String denoting a Perl release. The string must start with perl-, followed by the major version, minor version and patch version delimited by periods. The major version is always 5. (Same meaning as in perform_tarball_download().) Required.
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
Return Value

Test::Against::Dev object.
Comment

As was the case with new(), this method guarantees the existence of the application directory and the testing and results directories thereunder. It also performs sanity checks for the paths to installed perl and cpanm.

If you already have a perl installed which suffices for a monthly development release, then you can start with this method, omit calls to perform_tarball_download(), configure_build_install_perl() and fetch_cpanm() and go directly to run_cpanm().

LIMITATIONS

This library has a fair number of direct and indirect dependencies on other CPAN libraries. Consequently, the library may experience problems if there are major changes in those libraries. In particular, the code is indirectly dependent upon App::cpanminus::reporter, which in turn is dependent upon cpanm. (Nonetheless, this software could never have been written without those two libraries by Breno G. de Oliveira and Tatsuhiko Miyagawa, respectively.)

This library has been developed in a Unix programming environment and is unlikely to work in its current form on Windows, Cygwin or VMS.

AUTHOR

James E Keenan
CPAN ID: JKEENAN
jkeenan@cpan.org
http://thenceforward.net/perl

SUPPORT

Please report any bugs by mail to bug-Test-Against-Dev@rt.cpan.org or through the web interface at http://rt.cpan.org.

COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

The full text of the license can be found in the LICENSE file included with this module.

ACKNOWLEDGEMENTS

This library emerged in the wake of the author's participation in the Perl 5 Core Hackathon held in Amsterdam, Netherlands, in October 2017. The author thanks the lead organizers of that event, Sawyer X and Todd Rinaldo, for the invitation to the hackathon. The event could not have happened without the generous contributions from the following companies:

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

SYNOPSIS

DESCRIPTION

Who Should Use This Library?

The Problem to Be Addressed

The Approach Test-Against-Dev Currently Takes and How It May Change in the Future

What Is the Result Produced by This Library?

What Preparations Are Needed to Use This Library?

Notice of Breaking Change in Version 0.06 (March 20 2018)

METHODS

new()

perform_tarball_download()

configure_build_install_perl()

fetch_cpanm()

run_cpanm()

analyze_cpanm_build_logs()

analyze_json_logs()

new_from_existing_perl_cpanm()