NAME
30shell - The (old) main test script for CPAN.pm (superceded mostly by 31sections.t)
SYNOPSIS
make test # standard
make test TEST_FILES=t/30shell.t # traditional on file invocation
make testshell-with-protocol # collects output in ../protocolsDESCRIPTION
Coverage
30shell.coverage collects results from Devel::Cover
How this script works
The heart of the testing mechanism is t/30shell.t which is based on Expect and as such is able to test a shell session. The following provides an overview about how this testscript works.
After the __END__ token you find small groups of lines like the following:
########
#P:make CPAN::Test::Dummy::Perl5::BuildOrMake
#E:(?s:Running Build.*?Creating new.*?Build\s+-- OK)
#R:Module::Build
#T:15
#C:comment
#N:internal note
########P stands for program or print
E for expect
T for timeout
R for requires or relies on
C for comment (should be visible during testing)
S for status
N for notes (just for the test writer, otherwise invisible)
The script starts a CPAN shell and feeds it with chunks. The P line is injected and the output of the shell is parsed and compared to the expression in the E line. With T the timeout can be changed (the default is rather low, maybe 30 seconds, see the code for details). The expression in R is used to filter tests. The keyword in S may be one of run to run this test, skip to skip it, and quit to stop testing immediately when this test is reached.
To get reliable and debuggable results, Expect.pm should be installed. Without Expect.pm, a fallback mode is started that should in principle also succeed but is pretty hard to debug because there is no mechanism to sync state between reader and writer.
How to add new pseudo distributions
To make reproducable tests we need a shell session that is based on a clone of a miniaturized CPAN site. This site lives under t/CPAN/{authors,modules}.
The first distribution in the fake CPAN site was CPAN-Test-Dummy-Perl5-Make-1.01.tar.gz in the ./CPAN/authors/id/A/AN/ANDK/ directory which was a clone of PITA::Test::Dummy::Perl5::Make.
We need distros based on the following (and more) criteria:
Testing: success, failure
Installer: EU:MM, M:B, M:I
YAML: with YAML, with YAML::Syck, without
SIGNATURE: with, without
Zipping: tar.gz, tar.bz2, zip
Requires: requires, build_requiresAny new distro must be separately available on CPAN so that our CHECKSUMS files can be the real (signed) ones and we need not introduce a backdoor into the shell to ignore signatures.
To add a new distro, the following steps must be taken:
(1) Collect the source
- mkdir the author's directory if it doesn't exist yet; e.g.
mkdir -p t/CPAN/authors/id/A/AN/ANDK
cd t/CPAN/authors/id/A/AN/ANDK- introduce the whole source code under the author's homedir, often just a copy; e.g.
rsync -va CPAN-Test-Dummy-Perl5-Make-CircDepeOne CPAN-Test-Dummy-Perl5-Make-Expect- add the source code directory with a trailing slash to ../MANIFEST.SKIP
- finish now the distro until it does what you intended
(2) Create the distro tarball
- add a stanza to CPAN.pm's ../Makefile.PL that produces the distro with the whole dependency on all files within the distro and moves it up into the author's homedir. Run this with 'make testdistros'.
- *git add* the new testdistro (first we did that, then we stopped doing it for "it makes no sense"; then I realized we need to do it because with a newer MakeMaker or Module::Build we cannot regenerate them byte-by-byte and lose the signature war)
- add it to the ../MANIFEST
(3) Upload and embed into our "./CPAN" minicpan
- verify that 'make dist' on CPAN.pm still works
- if you want more distros, repeat (1) and (2) now
- upload the distro(s) to the CPAN and wait until the indexer has produced a CHECKSUMS file
- git add/commit the relevant CHECKSUMS files
- add the CHECKSUMS files to the MANIFEST
(4) Work with the results
- verify that 'make dist' on CPAN.pm still works
- add the distro(s) to CPAN/modules/02packages.details.txt: this step needs special care: pay attention to both the module version and the distro name; if there is more than one module or bundle inside the distro, write two lines; watch the line count;
- if this distro replaces another, git-rm the other one
- if this distro replaces another, fix the tests that rely on the other one
- add the test to 30shell.t that triggered the demand for a new distro
Problems
With SVN we had the problem that when you set up a new working copy of the SVN repository, you first had to run 'make testdistros' to get the pseudo distros that were not in the repository. This made too many testdistros, so you had to run 'svk st' and see which were marked with 'M'. Then you had to revert those and then the 30shell test should succeed. This has now been corrected for git repos.