NAME

Test::Against::Commit - Test CPAN modules against Perl dev releases

SYNOPSIS

my $self = Test::Against::Commit->new( {
    application_dir => '/path/to/application',
    project         => 'business_project',
    install          => <commit_ID_tag_or_branch>,
} );

$self->prepare_testing_directories();

my $this_cpanm = $self->fetch_cpanm( { verbose => 1 } );

my $modules_ref = $self->process_modules( {
    module_file => '/path/to/cpan-river-file.txt',
    title       => 'cpan-river-1000',
    verbose     => 1,
} );

DESCRIPTION

Who Should Use This Library?

This library should be used by anyone who wishes to assess the impact of day-to-day changes in the Perl 5 core distribution on the installability of libraries found on the Comprehensive Perl Archive Network (CPAN). This library supersedes the existing CPAN library Test-Against-Dev.

The Problem Addressed by This Library

In the development of Perl as a language we face a problem typically referred to as Blead Breaks CPAN (or BBC for short). Perl 5 undergoes an annual development cycle characterized by:

Commits on a near daily basis to a GitHub (GH) repository.
Monthly development releases (tarballs) whose version numbers follow the convention of 5.43.0, 5.43.1, etc., where the middle digits are always odd numbers.
Annual production releases and subsequent maintenance releases whose version numbers have even-numbered middle digits, e.g., 5.44.0, 5.44.1, etc.

A monthly development release is essentially a roll-up of a month's worth of commits to the master repository branch known as blead (pronounced "bleed"). Changes in the Perl 5 code base have the potential to adversely impact the installability of existing CPAN libraries. Hence, various individuals have, over the years, developed ways of testing those libraries against blead and reporting problems to those people actively involved in the ongoing development of the Perl 5 core distribution. The latter are typically referred to as "core developers" or as the "Perl 5 Porters."

This library is intended as a contribution to those efforts by enabling the Perl 5 Porters to assess the impact of changes in the Perl 5 core distribution CPAN libraries well in advance of production and maintenance releases.

The Approach Test-Against-Commit Takes

Unlike other efforts, Test-Against-Commit does not depend on test reports sent to CPANtesters.org. Hence, it should be unaffected by any technical problems which that site may face. As a consequence, however, a user of this library must be willing to maintain more of her own local infrastructure than a typical CPANtester would maintain.

While this library could, in principle, be used to test the entirety of CPAN, it is probably better suited for testing selected subsets of CPAN libraries which the user deems important to her individual or organizational needs.

Unlike its ancestor Test-Against-Dev, this library is designed to test CPAN libraries against either Perl 5 monthly development releases or against individual commits to any branch of any GH repository holding the Perl 5 core distribution. This library presumes that the user knows how to configure and build a perl executable and how to run the core distribution's test suite. This library leaves the configuration, build and installation of a perl executable to the user. The scope of this library's activity begins at the point that a perl has been installed on disk, continues through installation of libraries needed for testing CPAN libraries against that executable to analysis of the results of that testing and presentation of those results in a usable form.

While this library is currently focused on Perl 5 libraries publicly available on CPAN, it could in principle be extended to test an organization's private libraries as well. This functionality, however, has not yet been implemented or tested.

Terminology

Here are some terms which we use in a specific way in this library:

application directory

A directory to which the user has write-privileges and which holds the input and output for one or more projects. Example: /path-to-application.
project

A short-hand description of the focus of a particular investigation using Test-Against-Commit. Example: goto-fatal could describe an investigation of the impact of fatalization of certain uses of the Perl goto function on CPAN libraries.
project directory

A subdirectory of the application directory which holds the input and output data for one project. Example: /path-to-application/goto-fatal.
installation

An installation of one perl executable, the libraries that get installed with the core distribution, CPAN libraries whose installability we tested against that perl and data gathered to analyze that installability and answer questions for the business purpose of the project.

An installation will be built either from a particular checkout, tag or branch from a git repository of the Perl 5 source code (23ae7f95ea, v5.43.3, blead) or from a Perl 5 development, production or maintenance release in tarball form (perl-5.44.0).

A project will consist of at least one installation but will probably hold two or three installations: the first will be used to determine a baseline state, the second will be used to assess the impact of a proposed change in the Perl 5 core distribution on installability of CPAN libraries.

installation directory

A subdirectory of the project directory holding one installation. Example:

/path-to-application/                         # <-- application directory
/path-to-application/goto-fatal/              # <-- project directory
/path-to-application/goto-fatal/23ae7f95ea/   # <-- installation directory
/path-to-application/goto-fatal/v5.43.3/      # <-- another installation directory

Each installation directory will have exactly two subdirectories: testing and results. (See next two items.)

testing directory

A subdirectory of an installation directory which in turn initially holds two subdirectories, bin/ and lib/.

The bin/ directory holds the perl, perldoc, cpan and other executable when a particular perl is built for the project. The lib/ directory holds all modules installed either initially with the executable or subsequently, including those whose functionality we are assessing as part of the project's business purpose. Test-Against-Commit methods will create other subdirectories next to bin/ and lib/, some of which are hidden, as part of the testing progress.
```
/path-to-application/goto-fatal/23ae7f95ea/               # <-- installation directory
/path-to-application/goto-fatal/23ae7f95ea/testing/       # <-- testing directory
/path-to-application/goto-fatal/23ae7f95ea/testing/bin/   # <-- bin directory
/path-to-application/goto-fatal/23ae7f95ea/testing/lib/   # <-- lib directory
```
The data in the testing directory can be thought of as the project's input data.

results directory

A subdirectory of an installation directory which holds the data created by running a program using Test::Against::Commit methods. This directory will in turn hold two subdirectories: analysis/ and storage/.

/path-to-application/goto-fatal/23ae7f95ea/           # <-- installation directory
/path-to-application/goto-fatal/23ae7f95ea/testing/   # <-- testing directory
...
/path-to-application/goto-fatal/23ae7f95ea/results/   # <-- results directory
/path-to-application/goto-fatal/23ae7f95ea/results/analysis/
/path-to-application/goto-fatal/23ae7f95ea/results/storage/

The data in the results directory can be thought of as the project's output data.

run

A run is an instance of (i) testing a set of one or more CPAN libraries against a given installation and (ii) the recording of data from that instance of testing.
Perl 5 configuration

The way one calls Configure when building Perl 5 from source, <e.g.>:
```
sh ./Configure -des -Dusedevel
```
or:
```
sh ./Configure -des -Dusedevel \
    -Duseithreads \
    -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"
```
Once you begin to use Test-Against-Commit for a particular project, you should use the same configuration for each installation over the life of that project. For instance, you should not configure without threads in one run but with threads in the next. Nor should you switch from regular to debugging builds between runs. Otherwise, the results may reflect changes in that configuration rather than changes in Perl 5 core distribution code or changes in the targeted CPAN libraries.

What Is the Result Produced by This Library?

Our objective is to be able to compare output data recorded in one run for a given project with data recorded in a different run for a different (presumably subsequent) installation within the same project. To return to the example of the goto-fatal project, let's assume we have two different installations, the first of which sets our baseline (which CPAN libraries currently PASS and which currently FAIL) and a second which determines the impact of applying a pull request.

/path-to-application/goto-fatal/              # <-- project directory
/path-to-application/goto-fatal/23ae7f95ea/   # <-- first installation directory
/path-to-application/goto-fatal/v5.43.3/      # <-- second installation directory

We will end up comparing data stored in these installations' respective results/analysis/ subdirectories.

/path-to-application/goto-fatal/23ae7f95ea/results/analysis/
/path-to-application/goto-fatal/v5.43.3/results/analysis/

What Preparations Are Needed to Use This Library?

Platform

The user should select a machine/platform which is likely to be reasonably stable over one Perl 5 annual development cycle. We presume that the platform's system administrator will be updating system libraries for security and other reasons over time. But it would be a hassle to run this software on a machine scheduled for a complete major version update of its operating system.
Perl 5 Configuration

The user must decide on a Perl 5 configuration for a given project and then must refrain from changing configurations over the course of the project's existence. See item under Terminology above.
perl Executable Installation Location

As noted above, this library leaves to the user the choice of a way to get the Perl source code and the decision of how to configure an individual perl executable. It also leaves to the user, with one caveat, the decision of where to install that executable on disk. That caveat is that the installation should reside in a directory named testing which in turn sits underneath a directory which we'll refer to as the application directory. The user will have to manually create the application directory, the project directory, the installation directory and the testing directory and then use the testing directory as the value for the -Dprefix option in the invocation of Configure.

In terms of the directory structure discussed above, that the user would create a directory structure something like this:
```
$ cd ~/tmp
$ export TESTINGDIR=`pwd`/all-tad-projects/goto-fatal/23ae7f95ea/testing
$ echo $TESTINGDIR
.../tmp/all-tad-projects/goto-fatal/23ae7f95ea/testing
$ mkdir -p $TESTINGDIR
$ ls -l $TESTINGDIR
total 0

$ cd <git checkout of perl branch or decompressed release tarball>
```
The user would then invoke Configure in a way something like this:
```
$ sh ./Configure -des -Dusedevel -Dprefix=$TESTINGDIR \
    -Uversiononly -Dman1dir=none -Dman3dir=none
$ make install
```
The user could then confirm installation with this:
```
$ $TESTINGDIR/bin/perl -v | head -2 | tail -1
This is perl 5, version 43, subversion 3 (v5.43.3 (v5.43.2-343-g5fdb3e501b)) built for x86_64-linux
```
Note that at this point we have not yet created the results directory ...
```
$ cd ~/tmp
$ ls -l ./all-tad-projects/goto-fatal/23ae7f95ea/results
... No such file or directory
```
... but no worries; Test-Against-Commit methods will handle that.
Selection of CPAN Libraries for Testing

This is the most important step in preparation to use this library.

When you use this library, you are in effect saying: Here is a list of CPAN modules important enough to me that I don't want to see them begin to break in the course of Perl's annual development cycle. (If they do break, then the Perl 5 Porters and the modules' authors/maintainers must address how to handle the breakage.) To keep track of the problem, I'm going to build perl from a starting point where those modules are working proprerly and assess their installability at later points.

Hence, once you decide to track a certain CPAN library, you should continue to include it in your list of modules to be tracked for the balance of that year's development cycle. You can, it is true, add additional modules to your list part way through the development cycle. You simply won't have the same baseline data that you have for the modules you selected at the very beginning.

Here are some approaches that come to mind:
- CPAN river
  
  The CPAN river is a concept developed by Neil Bowers and other participants in the Perl Toolchain Gang and Perl QA Hackathons and Summits. The concept starts from the premise that CPAN libraries upon which many other CPAN libraries depend are more important than those upon which few other libraries depend. That's a useful definition of importance even if it is far from strictly true. Modules "way upstream" feed modules and real-world code "farther downstream." Hence, if Perl 5's development branch changes in a way such that "upstream" modules start to fail to configure, build, test and install correctly, then we have a potentially serious problem.
- Organizational dependencies
  
  Many organizations use technologies such as Carton and cpanfile to keep track of their dependencies on CPAN libraries. The lists compiled by such applications could very easily be translated into a list of modules tested once a month against a Perl development release.
- What repeatedly breaks
  
  Certain CPAN libraries get broken relatively frequently. While this can happen because of sub-standard coding practices in those libraries, it more often happens because these libraries, in order to do what they want to do, reach down deep into the Perl 5 guts and use features of the Perl-to-C API.

METHODS

`new()`

Purpose

Test::Against::Commit constructor. Guarantees that the top-level directory for the application (application_dir) already exists, then creates two directories thereunder: testing/ and results/.
Arguments
```
my $self = Test::Against::Commit->new( {
    application_dir => '/path/to/application',
    project => 'goto-fatal'
    install => '23ae7f95ea',
} );
```
Takes a hash reference with the following elements:
- application_dir
  
  String holding path to the directory which will serve as the top level for all projects using Test-Against-Commit technology.
- project
  
  String holding a short name for your current business project.
- install
  
  String holding a name for the specific installation of perl against which you will be attempting to install CPAN modules. If you have built perl from a git checkout, this should be the git commit ID (SHA), git tag or git branch name from which you are starting. If you are building perl from a release tarball, consider using a string such as perl-5.42.0 from the tarball's basename.
Return Value

Test::Against::Commit object.
Comment

The constructor merely verifies the existence of certain directories on your machine. It does not install a perl executable. That is the user's responsibility. The user will subsequently have to call the prepare_testing_directory() and perhaps fetch_cpanm() to be fully ready to test.

`get_application_dir() get_project_dir() get_install_dir() get_testing_dir() get_results_dir()`

Purpose

Methods which simply return the path to relevant directories (along with short-hand versions of their name):
- application directory (application_dir)
  
  The top-level directory for all code and data implemented by Test-Against-Commit. It will typically hold 1 subdirectory for each business project using Test-Against-Commit technology.
- project directory (project_dir)
  
  TK
- install directory (install_dir)
  
  TK
- testing directory (testing_dir)
  
  A directory which holds one or more subdirectories, each of which contains an installation of a perl executable. That installation will start off with bin/ and lib/ subdirectories and ./bin/perl -Ilib -v will be called to demonstrate the presence of a viable perl.
- results directory (results_dir)
  
  The directory under which all data created by runs of programs using Test::Against::Commit will be placed. This will include data in JSON and pipe-separated-value (PSV) formats.

Arguments

$application_dir = $self->get_application_dir();

$project_dir = $self->get_project_dir();

$install_dir = $self->get_install_dir();

$testing_dir = $self->get_testing_dir();

$results_dir = $self->get_results_dir();

Return Value

String holding a path to the named directory.
Comment

These methods become available once a perl executable has been installed and new() has been run.

`get_install()`

Purpose

Each perl installed underneath testing_dir needs a unique name. If we build this perl from a git checkout, this should be one of the commit ID (SHA), tag or branch name of the checkout.
Arguments
```
my $install = $self->get_install();
```
Return Value

String holding a git commit ID, tag or branch name.
Comment

Since install is one of the key-value pairs we are handing to new(), this method essentially just gives us back what we already told it. However, we will use it internally later to derive the path to the installed perl against which we are trying to install modules.

TK: What about when we're building from a tarball?

`prepare_testing_directory`

Purpose

Determines whether the perl executable has been installed -- if not, it's the user's responsibility to install it -- and whether this application has the correct directory structure.
Arguments
```
$self->prepare_testing_directory()
```
Return Value

Returns the Test::Against::Commit object, which now holds additional data.
Comment

TK

`get_bin_dir() get_lib_dir()`

Purpose

Once prepare_testing_directory() has been run, two additional methods become available to help the code determine where it is.
- bin directory (bin_dir)
  
  The directory underneath an individual install_dir directory holding installed executables such as perl, cpan and cpanm.
- lib directory (lib_dir)
  
  The directory underneath an individual install_dir directory holding the libraries supporting the installed executables found in the bin_dir.

Arguments

$bin_dir = $self->get_bin_dir();

$lib_dir = $self->get_lib_dir();

Return Value

String holding a path to the named directory.
Comment

If the perl executable has not yet been installed, these methods will throw exceptions.

`get_this_perl()`

Purpose

Identify the location of the perl executable file being tested.
Arguments
```
$this_perl = $self->get_this_perl()
```
Return Value

String holding the path to the perl executable being tested.
Comment

Will throw an exception if such a perl executable has not yet been installed.

`get_this_cpan()`

Purpose

Identify the location of the cpan executable file.
Arguments
```
$this_cpan = $self->get_this_cpan()
```
Return Value

String holding the path to the cpan executable being tested.
Comment

Will throw an exception if such a cpan executable has not yet been installed. We will use cpan to subsequently install App::cpanminus.

`fetch_cpanm() get_this_cpanm() get_cpanm_dir()`

Purpose

Determine whether cpanm has been installed. If it has not, install App::cpanminus and the cpanm executable against the installed perl.
Arguments
```
my $rv = $self->fetch_cpanm();
```
None. All information is already inside the object. No verbose output.
Return Value

Returns the Test::Against::Commit object, which now holds additional data.
Comment

The cpanm executable's location can subsequently be accessed by calling $self-get_this_cpanm()>. The method also guarantees the existence of a .cpanm directory underneath the install directory, i.e., side-by-side with bin and lib. This directory can subsequently be accessed by calling $self-get_cpanm_dir()>.

`process_modules()`

Purpose

Use cpanm to install selected Perl modules against the perl built for testing purposes.
Arguments

Two mutually exclusive interfaces:
- Modules provided in a list
```
my $modules_ref = $self->process_modules( {
    module_list => [ 'DateTime', 'AnyEvent' ],
    title       => 'two-important-libraries',
    verbose     => 1,
} );
```
- Modules listed in a file
```
my $modules_ref = $self->process_modules( {
    module_file => '/path/to/cpan-river-file.txt',
    title       => 'cpan-river-1000',
    verbose     => 1,
} );
```
Each interface takes a hash reference with the following elements:
- module_list OR module_file
  
  Mutually exclusive; you may use one or the other but not both.
  
  The value of module_list must be an array reference holding a list of modules for which you wish to assess the impact of changes in the Perl 5 core distribution.
  
  The value of module_file must be an absolute path to a file which holds a list of modules, one module per line. Lines in such a file that start with a '#' (hash-mark or sharp) be treated as a comment and no module listed on that line will be processed.
  
  In either case the module names are spelled in Some::Module format -- i.e., double-colons -- rather than in Some-Distribution format (hyphens).
- title
  
  String which will be used to compose the name of project-specific output files. Required.
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
- dryrun
  
  Optional; defaults to being off. Program runs only as far as determining modules which the user will attempt to load, prints number of modules being attempted, then process_modules() returns an undefined value (which would prevent subsequent methods from running correctly).
Return Value

Default: Single array reference holding a list of all modules that process_modules() at least attempted to process.

With true-value for dryrun: Undefined value.

Comment

The method creates or confirms the existence of several directories underneath the results_dir directory discussed above. These are illustrated as follows:

/path-to-application/                                 # <-- application directory
/path-to-application/goto-fatal/                      # <-- project directory
/path-to-application/goto-fatal/23ae7f95ea/           # <-- installation directory
/path-to-application/goto-fatal/23ae7f95ea/testing/   # <-- testing directory
/path-to-application/goto-fatal/23ae7f95ea/results/   # <-- results directory
/path-to-application/goto-fatal/23ae7f95ea/results/analysis/   # <-- analysis directory
/path-to-application/goto-fatal/23ae7f95ea/results/storage/   # <-- storage directory

`get_analysis_dir() get_storage_dir()`

Purpose

Once process_modules() has been run, two additional methods become available to help code determine where output data are located.
- analysis directory (analysis_dir)
  
  The directory underneath the results directory holding files representing the parsed content of the build log of the most recent run. These files are in .json format.
- storage directory (storage_dir)
  
  The directory underneath the results directory holding final output results.

Arguments

$storage_dir = $self->get_storage_dir();

Return Value

String holding a path to the named directory.
Comment

These directories are only confirmed to exist once internal method setup_results_directories() has been executed. (That method is called within process_modules().) Otherwise, these methods will throw exceptions.

`analyze_json_logs()`

Purpose

Create a character-delimited-values file summarizing the results of a given run. The delimiter defaults to a pipe (|), thereby creating a pipe-separated values file (.psv), but you may select a comma (,), generating a comma-separated-values file (.csv) as well.
Arguments
```
my $fcdvfile = $self->analyze_json_logs( { verbose => 1, sep_char => '|' } );
```
Hash reference with these elements:
- verbose
  
  Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
- sep_char
  
  Delimiter character. Optional; defaults to pipe (|), but comma (,) may also be chosen.
Return Value

String holding absolute path to the .psv or .csv file created.
Comment

As a precaution, the function creates a tarball to archive the .log.json files for a given run.

LIMITATIONS

This library has a fair number of direct and indirect dependencies on other CPAN libraries. Consequently, the library may experience problems if there are major changes in those libraries. In particular, the code is indirectly dependent upon App::cpanminus::reporter, which in turn is dependent upon cpanm. (Nonetheless, this software could never have been written without those two libraries by Breno G. de Oliveira and Tatsuhiko Miyagawa, respectively.)

This library has been developed in a Unix programming environment and is unlikely to work in its current form on Windows, Cygwin or VMS.

AUTHOR

James E Keenan
CPAN ID: JKEENAN
jkeenan@cpan.org
http://thenceforward.net/perl

SUPPORT

Please report any bugs in our GitHub Issues queue at https://github.com/jkeenan/perl5-test-cpan-against-commit/issues.

COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

The full text of the license can be found in the LICENSE file included with this module.

ACKNOWLEDGEMENTS

This library's ancestor, Test-Against-Dev, emerged in the wake of the author's participation in the Perl 5 Core Hackathon held in Amsterdam, Netherlands, in October 2017. The author thanks the lead organizers of that event, Sawyer X and Todd Rinaldo, for the invitation to the hackathon. The event could not have happened without the generous contributions from the following companies:

Additional Contributors

Mohammad S Anwar

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

SYNOPSIS

DESCRIPTION

Who Should Use This Library?

The Problem Addressed by This Library

The Approach Test-Against-Commit Takes

Terminology

What Is the Result Produced by This Library?

What Preparations Are Needed to Use This Library?

METHODS

new()

get_application_dir() get_project_dir() get_install_dir() get_testing_dir() get_results_dir()

get_install()

prepare_testing_directory

get_bin_dir() get_lib_dir()

get_this_perl()

get_this_cpan()

fetch_cpanm() get_this_cpanm() get_cpanm_dir()

process_modules()

get_analysis_dir() get_storage_dir()

analyze_json_logs()

LIMITATIONS

AUTHOR

SUPPORT

COPYRIGHT

ACKNOWLEDGEMENTS

Additional Contributors

SEE ALSO

Module Install Instructions

`new()`

`get_application_dir() get_project_dir() get_install_dir() get_testing_dir() get_results_dir()`

`get_install()`

`prepare_testing_directory`

`get_bin_dir() get_lib_dir()`

`get_this_perl()`

`get_this_cpan()`

`fetch_cpanm() get_this_cpanm() get_cpanm_dir()`

`process_modules()`

`get_analysis_dir() get_storage_dir()`

`analyze_json_logs()`