NAME

Test::Against::Commit - Test CPAN modules against Perl dev releases

SYNOPSIS

my $self = Test::Against::Commit->new( {
    application_dir => '/path/to/application',
    commit          => <commit_ID_tag_or_branch>,
} );

my $this_cpanm = $self->fetch_cpanm( { verbose => 1 } );

my $gzipped_build_log = $self->run_cpanm( {
    module_file => '/path/to/cpan-river-file.txt',
    title       => 'cpan-river-1000',
    verbose     => 1,
} );

DESCRIPTION

Who Should Use This Library?

This library should be used by anyone who wishes to assess the impact of day-to-day changes in the Perl 5 core distribution on the installability of libraries found on the Comprehensive Perl Archive Network (CPAN). This library supersedes the existing CPAN library Test-Against-Dev.

The Problem Addressed by This Library

In the development of Perl as a language we face a problem typically referred to as Blead Breaks CPAN (or BBC for short). Perl 5 undergoes an annual development cycle characterized by:

Commits on a near daily basis to a GitHub (GH) repository.
Monthly development releases (tarballs) whose version numbers follow the convention of 5.43.0, 5.43.1, etc., where the middle digits are always odd numbers.
Annual production releases and subsequent maintenance releases whose version numbers have even-numbered middle digits, e.g., 5.44.0, 5.44.1, etc.

A monthly development release is essentially a roll-up of a month's worth of commits to the master repository branch known as blead (pronounced "bleed"). Changes in the Perl 5 code base have the potential to adversely impact the installability of existing CPAN libraries. Hence, various individuals have, over the years, developed ways of testing those libraries against blead and reporting problems to those people actively involved in the ongoing development of the Perl 5 core distribution. The latter are typically referred to as "core developers" or as the "Perl 5 Porters."

This library is intended as a contribution to those efforts by enabling the Perl 5 Porters to assess the impact of changes in the Perl 5 core distribution on important provide a monthly snapshot of the impact of core development on CPAN libraries well in advance of production and maintenance releases.

The Approach Test-Against-Commit Takes

Unlike other efforts, Test-Against-Commit does not depend on test reports sent to CPANtesters.org. Hence, it should be unaffected by any technical problems which that site may face. As a consequence, however, a user of this library must be willing to maintain more of her own local infrastructure than a typical CPANtester would maintain.

While this library could, in principle, be used to test the entirety of CPAN, it is probably better suited for testing selected subsets of CPAN libraries which the user deems important to her individual or organizational needs.

Unlike its ancestor Test-Against-Dev, this library is designed to test CPAN libraries against either Perl 5 monthly development releases or against individual commits to any branch of any GH repository holding the Perl 5 core distribution. This library presumes that the user knows how to configure and build a perl executable and how to run the core distribution's test suite. This library leaves the configuration, build and installation of a perl executable to the user. The scope of this library's activity begins at the point that a perl has been installed on disk, continues through installation of libraries needed for testing CPAN libraries against that executable to analysis of the results of that testing and presentation of those results in a usable form.

While this library is currently focused on Perl 5 libraries publicly available on CPAN, it could in principle be extended to test an organization's private libraries as well. This functionality, however, has not yet been implemented or tested.

What Is the Result Produced by This Library?

We will use the term run to describe an instance of (i) testing one or more CPAN libraries against a given installed perl and (ii) the recording of data from that instance of testing. Our objective is to be able to compare the results of different runs against different perl executables.

For example, suppose a person working on the Perl core distribution wants to assess the impact of certain changes being proposed in a pull request on set of fifty specific CPAN libraries. The user will first create a benchmark perl probably built from a monthly development release, the GH tag associated with that release, or the GH commit from which the pull request was generated. She will use this library to conduct a run against that executable. In the run, each CPAN library will be graded PASS, FAIL (or NA for "not applicable").

At a certain point in the course of the pull request's development, the user will build a new perl executable and conduct a run against that perl. If a particular CPAN library receives a grade of PASS during the first run and a grade of FAIL during the next, the Perl 5 Porters will be asked to determine the cause of that breakage.

Sometimes the change in Perl 5 is wrong and needs to be reverted.
Sometimes the change in Perl 5 is correct (or, at least, plausible) but exposes sub-optimal code in the CPAN module.
Sometimes the failure is due to external conditions, such as a change in a C library on the testing platform.

There's no way to write code to figure out which situation -- or mix of situations -- we are in. The human user must intervene at this point.

What Preparations Are Needed to Use This Library?

Platform

The user should select a machine/platform which is likely to be reasonably stable over one Perl 5 annual development cycle. We presume that the platform's system administrator will be updating system libraries for security and other reasons over time. But it would be a hassle to run this software on a machine scheduled for a complete major version update of its operating system.
Perl 5 Configuration

The user must decide on a Perl 5 configuration before using Test-Against-Commit on a regular basis and then must refrain from changing that configuration over the course of the testing period. Otherwise, the results may reflect changes in that configuration rather than changes in Perl 5 core distribution code or changes in the targeted CPAN libraries.

By "Perl 5 configuration" we mean the way one calls Configure when building Perl 5 from source, <e.g.>:
```
sh ./Configure -des -Dusedevel
```
or:
```
sh ./Configure -des -Dusedevel \
    -Duseithreads \
    -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"
```
For instance, you should not configure without threads in one run but with threads in the next. Nor should you switch from regular to debugging builds between threads.
perl Executable Installation Location

As noted above, this library leaves to the user the choice of a way to get the Perl source code and the decision of how to configure an individual perl executable. It also leaves to the user, with one caveat, the decision of where to install that executable on disk. That caveat is that the perl installation should reside in a directory named testing which in turn sits underneath a directory which we'll refer to as the application directory. The user will have to manually create the application directory as well as the testing directory underneath.

Example: Suppose you want all your Test-Against-Commit data to sit in the directory tree /path/to/application:
```
/path/to/application
```
You will need a subdirectory called testing underneath that:
```
/path/to/application
/path/to/application/testing
```
You can create these directories with this command:
```
$ mkdir -p /path/to/application/testing
```
Now suppose that for one project you want to use as your baseline a perl built from a git checkout at commit 03f24b8a08, and for a different project you want to use as your baseline a perl built from a maintenance release tarball of perl-5.40.2. That means you will ultimately want a directory structure like this:
```
/path/to/application/
/path/to/application/testing/
/path/to/application/testing/03f24b8a08/
/path/to/application/testing/perl-5.40.2/
```
Using the second of the Configure examples above, for the first project you would configure with:
```
$ sh ./Configure -des -Dusedevel \
    -Duseithreads \
    -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"
    -Dprefix=/path/to/application/testing/03f24b8a08
```
For the second project you would configure with:
```
$ sh ./Configure -des -Dusedevel \
    -Duseithreads \
    -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"
    -Dprefix=/path/to/application/testing/perl-5.40.2
```
After running sh ./Configure, you would then call in each project:
```
$ make install
```
You would end up with a directory structure the top of which would look like this:
```
/path/to/application/
/path/to/application/testing/
/path/to/application/testing/03f24b8a08/
/path/to/application/testing/03f24b8a08/bin
/path/to/application/testing/03f24b8a08/lib
/path/to/application/testing/perl-5.40.2/
/path/to/application/testing/perl-5.40.2/bin
/path/to/application/testing/perl-5.40.2/lib
```
Selection of CPAN Libraries for Testing

This is the most important step in preparation to use this library.

When you use this library, you are in effect saying: Here is a list of CPAN modules important enough to me that I don't want to see them begin to break in the course of Perl's annual development cycle. (If they do break, then the Perl 5 Porters and the modules' authors/maintainers must address how to handle the breakage.) To keep track of the problem, I'm going to build perl from a starting point where those modules are working proprerly and assess their installability at later points.

Hence, once you decide to track a certain CPAN library, you should continue to include it in your list of modules to be tracked for the balance of that year's development cycle. You can, it is true, add additional modules to your list part way through the development cycle. You simply won't have the same baseline data that you have for the modules you selected at the very beginning.

Here are some approaches that come to mind:
- CPAN river
  
  The CPAN river is a concept developed by Neil Bowers and other participants in the Perl Toolchain Gang and Perl QA Hackathons and Summits. The concept starts from the premise that CPAN libraries upon which many other CPAN libraries depend are more important than those upon which few other libraries depend. That's a useful definition of importance even if it is far from strictly true. Modules "way upstream" feed modules and real-world code "farther downstream." Hence, if Perl 5's development branch changes in a way such that "upstream" modules start to fail to configure, build, test and install correctly, then we have a potentially serious problem.
- Organizational dependencies
  
  Many organizations use technologies such as Carton and cpanfile to keep track of their dependencies on CPAN libraries. The lists compiled by such applications could very easily be translated into a list of modules tested once a month against a Perl development release.
- What repeatedly breaks
  
  Certain CPAN libraries get broken relatively frequently. While this can happen because of sub-standard coding practices in those libraries, it more often happens because these libraries, in order to do what they want to do, reach down deep into the Perl 5 guts and use features of the Perl-to-C API.

METHODS

`new()`

Purpose

Test::Against::Commit constructor. Guarantees that the top-level directory for the application (application_dir) already exists, then creates two directories thereunder: testing/ and results/.
Arguments
```
my $self = Test::Against::Commit->new( {
    application_dir => '/path/to/application',
    commit => 'blead',
} );
```
Takes a hash reference with the following elements:
- application_dir
  
  String holding path to the directory which will serve as the top level for your application.
- commit
  
  String holding a name for the specific perl executable against which you will be attempting to install CPAN modules. If you are building perl from a git checkout, this should be the git commit ID (SHA), git tag or git branch name from which you are starting. If you are building perl from a release tarball, consider using a string such as perl-5.42.0 from the tarball's basename.
Return Value

Test::Against::Commit object.
Comment

The constructor merely verifies the existence of certain directories on your machine. It does not install a perl executable. That is the user's responsibility. The user will subsequently have to call the prepare_testing_directory() and perhaps fetch_cpanm() to be fully ready to test.

`get_application_dir() get_testing_dir() get_results_dir()`

Purpose

Three methods which simply return the path to relevant directories (along with short-hand versions of their name):
- application directory (application_dir)
  
  The top-level directory for all code and data implemented by Test-Against-Commit. It will typically hold 2 subdirectories: testing and results, described below.
- testing directory (testing_dir)
  
  A directory which holds one or more subdirectories, each of which contains an installation of a perl executable. That installation will start off with bin/ and lib/ subdirectories and ./bin/perl -Ilib -v will be called to demonstrate the presence of a viable perl.
- results directory (results_dir)
  
  The directory under which all data created by runs of programs using Test::Against::Commit will be placed. This will include data in JSON and pipe-separated-value (PSV) formats.

Arguments

$application_dir = $self->get_application_dir();

$testing_dir = $self->get_testing_dir();

$results_dir = $self->get_results_dir();

Return Value

String holding a path to the named directory.
Comment

These methods become available once a perl executable has been installed and new() has been run.

`get_commit()`

Purpose

Each perl installed underneath testing_dir needs a unique name. If we build this perl from a git checkout, this should be one of the commit ID (SHA), tag or branch name of the checkout.
Arguments
```
my $commit = $self->get_commit();
```
Return Value

String holding a git commit ID, tag or branch name.
Comment

Since commit is one of the key-value pairs we are handing to new(), this method essentially just gives us back what we already told it. However, we will use it internally later to derive the path to the installed perl against which we are trying to install modules.

TK: What about when we're building from a tarball?

`prepare_testing_directory`

Purpose

Determines whether the perl executable has been installed -- if not, it's the user's responsibility to install it -- and whether this application has the correct directory structure.
Arguments
```
$self->prepare_testing_directory()
```
Return Value

Returns the Test::Against::Commit object, which now holds additional data.
Comment

TK

`get_commit_dir() <get_bin_dir() get_lib_dir()`

Purpose

Once prepare_testing_directory() has been run, three additional methods become available to help the code determine where it is.
- commit directory (commit_dir)
  
  A directory underneath testing_dir holding perl installation. This directory will start off life with two subdirectories, bin and lib.
- bin directory (bin_dir)
  
  The directory underneath an individual commit_dir directory holding installed executables such as perl, cpan and cpanm.
- lib directory (lib_dir)
  
  The directory underneath an individual commit_dir directory holding the libraries supporting the installed executables found in the bin_dir.

Arguments

$commit_dir = $self->get_commit_dir();

$bin_dir = $self->get_bin_dir();

$lib_dir = $self->get_lib_dir();

Return Value

String holding a path to the named directory.
Comment

If the perl executable has not yet been installed, these methods will throw exceptions.

`get_this_perl()`

Purpose

Identify the location of the perl executable file being tested.
Arguments
```
$this_perl = $self->get_this_perl()
```
Return Value

String holding the path to the perl executable being tested.
Comment

Will throw an exception if such a perl executable has not yet been installed.

`fetch_cpanm() get_this_cpanm() get_cpanm_dir()`

Purpose

Determine whether cpanm has been installed. If it has not, fetch the fatpacked cpanm executable and install it against the newly installed perl.
Arguments
```
my $rv = $self->fetch_cpanm( { verbose => 1 } );
```
Hash reference with these elements:
- uri
  
  String holding URI from which cpanm will be downloaded. Optional; defaults to https://fastapi.metacpan.org/source/MIYAGAWA/App-cpanminus-1.7048/bin/cpanm.
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
Return Value

Returns the Test::Against::Commit object, which now holds additional data.
Comment

The cpanm executable's location can subsequently be accessed by calling $self-get_this_cpanm()>. The method also guarantees the existence of a .cpanm directory underneath the commit directory, i.e., side-by-side with bin and lib. This directory can subsequently be accessed by calling $self-get_cpanm_dir()>.

`run_cpanm()`

Purpose

Use cpanm to install selected Perl modules against the perl built for testing purposes.
Arguments

Two mutually exclusive interfaces:
- Modules provided in a list
```
$gzipped_build_log = $self->run_cpanm( {
    module_list => [ 'DateTime', 'AnyEvent' ],
    title       => 'two-important-libraries',
    verbose     => 1,
} );
```
- Modules listed in a file
```
$gzipped_build_log = $self->run_cpanm( {
    module_file => '/path/to/cpan-river-file.txt',
    title       => 'cpan-river-1000',
    verbose     => 1,
} );
```
Each interface takes a hash reference with the following elements:
- module_list OR module_file
  
  Mutually exclusive; you may use one or the other but not both.
  
  The value of module_list must be an array reference holding a list of modules for which you wish to assess the impact of changes in the Perl 5 core distribution. In either case the module names are spelled in Some::Module format -- i.e., double-colons -- rather than in Some-Distribution format (hyphens).
- title
  
  String which will be used to compose the name of project-specific output files. Required.
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
Return Value

String holding the absolute path of a gzipped copy of the build.log generated by the cpanm run which this method conducts. The basename of this file, using the arguments supplied, would be:
```
cpan-river-1000.perl-5.43.6.01.build.log.gz
```

Comment

The method confirms the existence of several directories underneath the results_dir directory discussed above. These are illustrated as follows:

/path/to/application/results/
                    /results/perl-5.43.6/
                    /results/perl-5.43.6/analysis/
                    /results/perl-5.43.6/buildlogs/
                    /results/perl-5.43.6/storage/

`analyze_cpanm_build_logs()`

Purpose

Parse the build.log created by running run_cpanm(), creating JSON files which log the results of attempting to install each module in the list or file.
Arguments
```
$ranalysis_dir = $self->analyze_cpanm_build_logs( { verbose => 1 } );
```
Hash reference which, at the present time, can only take one element: verbose. Optional.
Return Value

String holding absolute path to the directory holding .log.json files for a particular run of run_cpanm().
Comment

`analyze_json_logs()`

Purpose

Create a character-delimited-values file summarizing the results of a given run. The delimiter defaults to a pipe (|), thereby creating a pipe-separated values file (.psv), but you may select a comma (,), generating a comma-separated-values file (.csv) as well.
Arguments
```
my $fcdvfile = $self->analyze_json_logs( { verbose => 1, sep_char => '|' } );
```
Hash reference with these elements:
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
- sep_char
  
  Delimiter character. Optional; defaults to pipe (|), but comma (,) may also be chosen.
Return Value

String holding absolute path to the .psv or .csv file created.
Comment

As a precaution, the function creates a tarball to archive the .log.json files for a given run.

LIMITATIONS

This library has a fair number of direct and indirect dependencies on other CPAN libraries. Consequently, the library may experience problems if there are major changes in those libraries. In particular, the code is indirectly dependent upon App::cpanminus::reporter, which in turn is dependent upon cpanm. (Nonetheless, this software could never have been written without those two libraries by Breno G. de Oliveira and Tatsuhiko Miyagawa, respectively.)

This library has been developed in a Unix programming environment and is unlikely to work in its current form on Windows, Cygwin or VMS.

AUTHOR

James E Keenan
CPAN ID: JKEENAN
jkeenan@cpan.org
http://thenceforward.net/perl

SUPPORT

Please report any bugs in our GitHub Issues queue at https://github.com/jkeenan/perl5-test-cpan-against-commit/issues.

COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

The full text of the license can be found in the LICENSE file included with this module.

ACKNOWLEDGEMENTS

This library's ancestor, Test-Against-Dev, emerged in the wake of the author's participation in the Perl 5 Core Hackathon held in Amsterdam, Netherlands, in October 2017. The author thanks the lead organizers of that event, Sawyer X and Todd Rinaldo, for the invitation to the hackathon. The event could not have happened without the generous contributions from the following companies:

Additional Contributors

Mohammad S Anwar

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

SYNOPSIS

DESCRIPTION

Who Should Use This Library?

The Problem Addressed by This Library

The Approach Test-Against-Commit Takes

What Is the Result Produced by This Library?

What Preparations Are Needed to Use This Library?

METHODS

new()

get_application_dir() get_testing_dir() get_results_dir()

get_commit()

prepare_testing_directory

get_commit_dir() <get_bin_dir() get_lib_dir()

get_this_perl()

fetch_cpanm() get_this_cpanm() get_cpanm_dir()

run_cpanm()

analyze_cpanm_build_logs()

analyze_json_logs()