/ 
Test-Against-Dev-0.03
River stage zero No dependents
/ Test::Against::Dev

NAME

Test::Against::Dev - Test CPAN modules against Perl dev releases

SYNOPSIS

my $self = Test::Against::Dev->new( {
    application_dir => '/path/to/application',
} );

my ($tarball_path, $work_dir) = $self->perform_tarball_download( {
    host                => 'ftp.funet.fi',
    hostdir             => /pub/languages/perl/CPAN/src/5.0,
    perl_version        => 'perl-5.27.6',
    compression         => 'gz',
    work_dir            => "~/tmp/Downloads",
    verbose             => 1,
    mock                => 0,
} );

my $this_perl = $self->configure_build_install_perl({
    verbose => 1,
});

my $this_cpanm = $self->fetch_cpanm( { verbose => 1 } );

my $gzipped_build_log = $self->run_cpanm( {
    module_file => '/path/to/cpan-river-file.txt',
    title       => 'cpan-river-1000',
    verbose     => 1,
} );

DESCRIPTION

Who Should Use This Library?

This library should be used by anyone who wishes to assess the impact of month-to-month changes in the Perl 5 core distribution on the installability of libraries found on the Comprehensive Perl Archive Network (CPAN).

The Problem to Be Addressed

This problem is typically referred to as Blead Breaks CPAN (or BBC for short). Perl 5 undergoes an annual development cycle characterized by monthly releases whose version numbers follow the convention of 5.27.0, 5.27.1, etc., where the middle digits are always odd numbers. (Annual production releases and subsequent maintenance releases have even-numbered middle digits, e.g., 5.26.0, 5.26.1, etc.) A monthly development release is essentially a roll-up of a month's worth of commits to the master branch known as blead (pronounced "bleed"). Changes in the Perl 5 code base have the potential to adversely impact the installability of existing CPAN libraries. Hence, various individuals have, over the years, developed ways of testing those libraries against blead and reporting problems to those people actively involved in the ongoing development of the Perl 5 core distribution -- people typically referred to as the Perl 5 Porters.

This library is intended as a contribution to those efforts. It is intended to provide a monthly snapshot of the impact of Perl 5 core development on important CPAN libraries.

The Approach Test-Against-Dev Currently Takes and How It May Change in the Future

Unlike other efforts, Test-Against-Dev does not depend on test reports sent to CPANtesters.org. Hence, it should be unaffected by any technical problems which that site may face. As a consequence, however, a user of this library must be willing to maintain more of her own local infrastructure than a typical CPANtester would maintain.

While this library could, in principle, be used to test the entirety of CPAN, it is probably better suited for testing selected subsets of CPAN libraries which the user deems important to her individual or organizational needs.

This library is currently focused on monthly development releases of Perl 5. It does not directly provide a basis for identifying individual commits to blead which adversely impacted particular CPAN libraries. It "tests against dev" more than it "tests against blead" -- hence, the name of the library. However, once it has gotten some production experience, it may be extended to, say, measure the effect of individual commits to blead on CPAN libraries using the previous monthly development release as a baseline.

This library is currently focused on Perl 5 libraries publicly available on CPAN. In the future, it may be extended to be able to include an organization's private libraries as well.

This library is currently focused on blead, the master branch of the Perl 5 core distribution. However, it could, in principle, be extended to assess the impact on CPAN libraries of code in non-blead ("smoke-me") branches as well.

What Is the Result Produced by This Library?

Currently, if you run code built with this library on a monthly basis, you will produce an updated version of a pipe-separated-values (PSV) plain-text file suitable for opening in a spreadsheet. The columns in that PSV file will be these:

dist
perl-5.27.0.author
perl-5.27.0.distname
perl-5.27.0.distversion
perl-5.27.0.grade
perl-5.27.1.author
perl-5.27.1.distname
perl-5.27.1.distversion
perl-5.27.1.grade
...

So the output for particular CPAN libraries will look like this:

dist|perl-5.27.0.author|perl-5.27.0.distname|perl-5.27.0.distversion|perl-5.27.0.grade|perl-5.27.1.author|perl-5.27.1.distname|perl-5.27.1.distversion|perl-5.27.1.grade|...
Acme-CPANAuthors|ISHIGAKI|Acme-CPANAuthors-0.26|0.26|PASS|ISHIGAKI|Acme-CPANAuthors-0.26|0.26|PASS|...
Algorithm-C3|HAARG|Algorithm-C3-0.10|0.10|PASS|HAARG|Algorithm-C3-0.10|0.10|PASS|...

If a particular CPAN library receives a grade of PASS one month and a grade of <FAIL> month, it ought to be inspected for the cause of that breakage. Sometimes the change in Perl 5 is wrong and needs to be reverted. Sometimes the change in Perl 5 is correct (or, at least, plausible) but exposes sub-optimal code in the CPAN module. Sometimes the failure is due to external conditions, such as a change in a C library on the testing platform. There's no way to write code to figure out which situation -- or mix of situations -- we are in. The human user must intervene at this point.

What Preparations Are Needed to Use This Library?

  • Platform

    The user should select a machine/platform which is likely to be reasonably stable over one Perl 5 annual development cycle. We understand that the platform's system administrator will be updating system libraries for security and other reasons over time. But it would be a hassle to run this software on a machine scheduled for a complete major version update of its operating system.

  • Perl 5 Configuration

    The user must decide on a Perl 5 configuration before using Test-Against-Dev on a regular basis and not change that over the course of the testing period. Otherwise, the results may reflect changes in that configuration rather than changes in Perl 5 core distribution code or changes in the targeted CPAN libraries.

    "Perl 5 configuration" means the way one calls Configure when building Perl 5 from source, <e.g.>:

    sh ./Configure -des -Dusedevel \
    -Duseithreads \
    -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"

    So, you should not configure without threads one month but with threads another month. You should not switch to debugging builds half-way through the testing period.

  • Selection of CPAN Libraries for Testing

    This is the most important step in preparation to use this library.

    When you use this library, you are in effect saying: Here is a list of CPAN modules important enough to me that I don't want to see them start breaking in the course of Perl's annual development cycle. (If they do break, then the Perl 5 Porters and the modules' authors/maintainers must address how to handle the breakage.) To keep track of the problem, I'm going to build perl from each monthly release and attempt to install this entire list against that perl.

    Hence, once you decide to track a certain CPAN library, you should continue to include it in your list of modules to be tracked for the balance of the development cycle. You can, it is true, add additional modules to your list part way through the development cycle. You simply won't have the same baseline data that you have for the modules you selected at the very beginning.

    Here are some approaches that come to mind:

    • CPAN river

      The CPAN river is a concept developed by Neil Bowers and other participants in the Perl Toolchain Gang and Perl QA Hackathons and Summits. The concept starts from the premise that CPAN libraries upon which many other CPAN libraries depend are more important than those upon which few other libraries depend. That's a useful definition of importance even if it is not strictly true. Modules "way upstream" feed modules and real-world code "farther downstream". Hence, if Perl 5's development branch changes in a way such that "upstream" modules start to fail to configure, build, test and install correctly, then we have a potentially serious problem. The author of this library has primarily developed it with the idea that it would be run monthly to see what happens with the 1000 "farthest upstream" modules -- the so-called "CPAN River Top 1000".

    • Organizational dependencies

      Many organizations use technologies such as Carton and cpanfile to keep track of their dependencies on CPAN libraries. The lists compile by such applications could very easily be translated into a list of modules tested once a month against a Perl development release.

    • What repeatedly breaks

      Certain CPAN libraries get broken relatively frequently. While this can happen because of sub-standard coding practices in those libraries, it more often happens because these libraries, in order to do what they want to do, reach down deep into the Perl 5 guts and use undocumented or not publicly supported features of Perl.

METHODS

new()

  • Purpose

    Test::Against::Dev constructor. Guarantees that the top-level directory for the application exists, then creates two directories thereunder: testing/ and results/.

  • Arguments

    my $self = Test::Against::Dev->new( {
    application_dir => '/path/to/application',
} );

    Takes a hash reference with the following elements:

    • application_dir

      String holding path to the directory which will serve as the top level for your application.

  • Return Value

    Test::Against::Dev object.

  • Comment

    This class has two possible constructors: this method and new_from_existing_perl_cpanm(). Use this one when you need to do a fresh install of a perl by compiling it from a downloaded tarball. Use the other one when you have already installed such a perl on disk and have installed a cpanm against that perl.

    The method will guarantee that underneath the application directory there are two directories: testing and results.

perform_tarball_download()

  • Purpose

  • Arguments

    ($tarball_path, $work_dir) = $self->perform_tarball_download( {
    host                => 'ftp.funet.fi',
    hostdir             => /pub/languages/perl/CPAN/src/5.0,
    perl_version        => 'perl-5.27.6',
    compression         => 'gz',
    work_dir            => "~/tmp/Downloads",
    verbose             => 1,
    mock                => 0,
} );

    Hash reference with the following elements:

    • host

      String. The FTP mirror from which you wish to download a tarball of a Perl release. Required.

    • hostdir

      String. The directory on the FTP mirror specified by host in which the tarball is located. Required.

    • perl_version

      String denoting a Perl release. The string must start with perl-, followed by the major version, minor version and patch version delimited by periods. The major version is always 5. Required.

    • compression

      String denoting the compression format of the tarball you wish to download. Eligible compression formats are gz, bz2 and bz2. Required.

      Note that not all compression formats are available for all tarballs on our FTP mirrors and that the compression formats offered may change over time.

      Note further that gz is currently the recommended format, as the other methods have not been thorougly tested.

    • work_dir

      String holding absolute path to the directory in which the work of configuring and building the new perl will be performed. Optional; if not provided a temporary directory created via File::Temp::tempdir() will be used.

    • verbose

      Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.

    • mock

      Display the expected results of the download on STDOUT, but don't actually do it. Optional; defaults to being off; provide a Perl-true value to turn it on. Any program using this option will terminate with a non-zero status once the results have been displayed.

  • Return Value

    Returns a list of two elements:

    • Tarball path

      String holding absolute path to the tarball once downloaded.

    • Work directory

      String holding path of directory in which work of configuring and building perl will be performed. (This is probably only useful if you want to see the path to the temporary directory. It will be uninitialized if mock is turned on.)

  • Comment

    The method guarantees the existence of a directory whose name will be the value of the perl_version argument and which will be found underneath the testing directory (discussed in new() above). This "release directory" -- accessible by calling $self-get_release_dir()> -- will be the directory below which a new perl will be installed.

configure_build_install_perl()

  • Purpose

    Configures, builds and installs perl from the downloaded tarball.

  • Arguments

    my $this_perl = $self->configure_build_install_perl({
    verbose => 1,
});

    Hash reference with the following elements:

    • configure_command

      String holding a shell command to call Perl's Configure program with command-line options. Optional; will default to:

      my $release_dir = $self->get_release_dir();

sh ./Configure -des -Dusedevel -Uversiononly -Dprefix=$release_dir \
    -Dman1dir=none -Dman3dir=none

      The spelling of the command is subsequently accessible by calling $self-access_configure_command()>.

    • make_install_command

      String holding a shell command to build and install perl underneath the release directory. Optional; will default to:

      make install

      The spelling of the command is subsequently accessible by calling $self-access_make_install_command()>.

    • verbose

      Extra information provided on STDOUT. Optional; defaults to being off. Set to 1 (recommended) for moderate verbosity. Set to 2 for extra verbosity (full output of decompression commands, Configure and make). Scope is limited to this method.

  • Return Value

    String holding absolute path to the new perl executable. This location can subsequently be accessed by calling $self-get_this_perl()>.

  • Comment

    The new perl executable will sit two levels underneath the release directory in a directory named bin/. That directory will sit next to a directory named lib/ under which libraries will be installed. Those locations can subsequently be accessed by calling $self-get_bin_dir()> and $self-get_lib_dir()>, respectively.

fetch_cpanm()

  • Purpose

    Fetch the fatpacked cpanm executable and install it against the newly installed perl.

  • Arguments

    my $this_cpanm = $self->fetch_cpanm( { verbose => 1 } );

    Hash reference with these elements:

  • Return Value

    String holding the absolute path to the newly installed cpanm executable.

  • Comment

    The executable's location can subsequently be accessed by calling $self-get_this_cpanm()>. The method also guarantees the existence of a .cpanm directory underneath the release directory. This directory can subsequently be accessed by calling $self-get_cpanm_dir()>.

run_cpanm()

  • Purpose

    Use cpanm to install selected Perl modules against the perl built for testing purposes.

  • Arguments

    Two mutually exclusive interfaces:

    • Modules provided in a list

      $gzipped_build_log = $self->run_cpanm( {
    module_list => [ 'DateTime', 'AnyEvent' ],
    title       => 'two-important-libraries',
    verbose     => 1,
} );

    • Modules listed in a file

      $gzipped_build_log = $self->run_cpanm( {
    module_file => '/path/to/cpan-river-file.txt',
    title       => 'cpan-river-1000',
    verbose     => 1,
} );

    Each interface takes a hash reference with the following elements:

    • module_list OR module_file

      Mutually exclusive; must use one or the other but not both.

      The value of module_list must be an array reference holding a list of modules for which you wish to track the impact of changes in the Perl 5 core distribution over time. In either case the module names are spelled in Some::Module format -- i.e., double-colons -- rather than in Some-Module format (hyphens).

    • title

      String which will be used to compose the name of project-specific output files. Required.

    • verbose

      Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.

  • Return Value

    String holding the absolute path of a gzipped copy of the build.log generated by the cpanm run which this method conducts. The basename of this file, using the arguments supplied, would be:

    cpan-river-1000.perl-5.27.6.01.build.log.gz

  • Comment

    The method guarantees the existence of several directories underneath the "results" directory discussed above. These are illustrated as follows:

    /path/to/application/results/
                    /results/perl-5.27.6/
                    /results/perl-5.27.6/analysis/
                    /results/perl-5.27.6/buildlogs/
                    /results/perl-5.27.6/storage/

analyze_cpanm_build_logs()

  • Purpose

    Parse the build.log created by running run_cpanm(), creating JSON files which log the results of attempting to install each module in the list or file.

  • Arguments

    $ranalysis_dir = $self->analyze_cpanm_build_logs( { verbose => 1 } );

    Hash reference which, at the present time, can only take one element: verbose. Optional.

  • Return Value

    String holding absolute path to the directory holding .log.json files for a particular run of run_cpanm().

  • Comment

analyze_json_logs()

  • Purpose

  • Arguments

    my $fpsvfile = $self->analyze_json_logs( { run => 1, verbose => 1 } );

    Hash reference with these elements:

    • run

      A positive integer.

    • verbose

      Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.

  • Return Value

    String holding absolute path

  • Comment

new_from_existing_perl_cpanm()

  • Purpose

    Alternate constructor to be used when you have already built a perl executable to be used in tracking Perl development and installed a cpanm against that perl.

  • Arguments

    $self = Test::Against::Dev->new_from_existing_perl_cpanm( {
    path_to_perl    => '/path/to/perl-5.27.0/bin/perl',
    application_dir => '/path/to/application',
    perl_version    => 'perl-5.27.0',
} );

    Takes a hash reference with the following elements:

    • path_to_perl

      String holding path to an installed perl executable. Required.

    • application_dir

      String holding path to the directory which will serve as the top level for your application. (Same meaning as in new().) Required.

    • perl_version

      String denoting a Perl release. The string must start with perl-, followed by the major version, minor version and patch version delimited by periods. The major version is always 5. (Same meaning as in perform_tarball_download().) Required.

    • verbose

      Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.

  • Return Value

    Test::Against::Dev object.

  • Comment

    As was the case with new(), this method guarantees the existence of the application directory and the testing and results directories thereunder. It also performs sanity checks for the paths to installed perl and cpanm.

    If you already have a perl installed which suffices for a monthly development release, then you can start with this method, omit calls to perform_tarball_download(), configure_build_install_perl() and fetch_cpanm() and go directly to run_cpanm().

LIMITATIONS

This library has a fair number of direct and indirect dependencies on other CPAN libraries. Consequently, the library may experience problems if there are major changes in those libraries. In particular, the code is indirectly dependent upon App::cpanminus::reporter, which in turn is dependent upon cpanm. (Nonetheless, this software could never have been written without those two libraries by Breno G. de Oliveira and Tatsuhiko Miyagawa, respectively.)

AUTHOR

James E Keenan
CPAN ID: JKEENAN
jkeenan@cpan.org
http://thenceforward.net/perl

SUPPORT

Please report any bugs by mail to bug-Test-Against-Dev@rt.cpan.org or through the web interface at http://rt.cpan.org.

COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

The full text of the license can be found in the LICENSE file included with this module.

Copyright James E Keenan 2017. All rights reserved.

ACKNOWLEDGEMENTS

This library emerged in the wake of the author's participation in the Perl 5 Core Hackathon held in Amsterdam, Netherlands, in October 2017. The author thanks the lead organizers of that event, Sawyer X and Todd Rinaldo, for the invitation to the hackathon. The event could not have happened without the generous contributions from the following companies:

SEE ALSO

perl(1). CPAN::cpanminus::reporter::RetainReports(3). Perl::Download::FTP(3). App::cpanminus::reporter(3). cpanm(3).

2017 Perl 5 Core Hackathon Discussion on Testing.

perl.cpan.testers.discuss Thread on Testing.