/ 
Test-Smoke-1.36
River stage zero No dependents
/ FAQ

NAME

FAQ - Test::Smoke frequently asked questions

DESCRIPTION

What is Test::Smoke?

Test::Smoke is the symbolic name for set of scripts and modules that try to run the perl core tests on as many configurations as possible and combine the results into an easy to read report.

The basic cycle looks like:

fetch_sourcetree
foreach $config ( @configurations ) {
    make distclean
    ./Configure $config
    make
    make test (and log failures)
}
create_report()
mail_report()

Why is Test::Smoke?

The reports from Test::Smoke give a good picture of the current state of Perl development. We try to get as many of the supported platforms as possible.

Where is Test::Smoke?

On CPAN and http://search.cpan.org/dist/Test-Smoke

What are these configuration files about?

Test::Smoke uses two types of configuration files:

smoke-configuration-file

This is a file that uses a _config suffix and contains settings for the smokeperl.pl program and is created by configsmoke.pl.

This is basically a Data::Dumper::Dump() of a hash.

build-configurations-file

This file contains the information needed to assemble the list of perl configurations that will be used to build and test perl.

The file is devided into sections separated by a line that starts with an equals-sign (=). Each line in a section is combined with all lines from all the other sections. The default build-configurations file perlcurrent.cfg constist of four (4) sections:

section 1 one (1) (blank) line
section 2 two (2) lines
section 3 seven (7) lines
section 4 two (2) lines (and a policy indicator line)

This file wil yield 1 * 2 * 7 * 2 = 28 configurations. Test::Smoke will build perl in all of these 28 configurations and run the core test suite one (1), two (2) or three (3) times for each of those perls:

  • with $ENV{PERLIO}=stdio (allways)

  • with $ENV{PERLIO}=perlio (unless -Uuseperlio)

  • with $ENV{PERL_UNICODE}="", $ENV{LC_ALL}=xx_YY.utf8 (if specified and !-Uuseperlio)

If a section consists of only one single line, like section 1 in the example, the options on this line are used in every configuration. This can be useful to set an alternate compiler (-Dcc=gcc) or to disable features that would fail anyway (-Uusefaststdio). [Thank you Merijn for contributing that insight!]

How do I create different smoke-configurations?

By specifying a diffent prefix to configsmoke.pl:

$ perl configsmoke.pl -p mysmoke

This will create the smokeconfiguration mysmoke_config and a small shellscript mysmoke.sh (or mysmoke.cmd if you are on Windows).

There are three (3) special prefixes: smoke55x, smoke562 and smoke58x. These will set the default answers for configsmoke.pl to be appropriate for smoking the maintenance branches for these versions of perl.

How do I include copy-on-write testing in my smokes?

TIMTOWTDO :-)

Most of this works for flags that need to be added to ccflags in general.

-Accflags='-DPERLL_COPY_ON_WRITE'

The fastest way to do it, is to add a new section to your buildconfiguration file, like this:

=
# Add a new two line section for -DPERL_COPY_ON_WRITE

-Accflags='-DPERL_COPY_ON_WRITE'
=

That will always work, but IMO looks a bit "loud" in the reports.

NOTE: This is the only way to smoke COW on MSWin32!

Default Policy.sh

Since maintenence version 1.18.10, the default Policy.sh is generated from the policy-target sections in the buildconfigurations file. It is now possible to just add this three line section to your buildconfiguration file:

=
/-DPERL_COPY_ON_WRITE/

-DPERL_COPY_ON_WRITE
=
Your own Policy.sh

Some people use the posibility to handle site specific build environments from <builddir>/../Policy.sh. This excludes the use of the default Policy.sh. You will need to add the -DPERL_COPY_ON_WRITE flag to the ccflags='' line in your Policy.sh. After that you must add the policy-target section, as above, to your buildconfiguration file.

What are all the scripts in the smoke suite for?

smokeperl.pl

This is a wrapper script that goes through all the steps in the smoke process.

synctree.pl

Small wrapper around Test::Smoke::Syncer that can use the settings from the smoke-configuration file. It will bring the source-tree in sync.

patchtree.pl

Small wrapper around Test::Smoke::Patcher that can use the settings from the smoke-configuration file. It will apply the patches that are specified in the patches-to-be-applied file.

runsmoke.pl

Small wrapper around Test::Smoke::run_smoke() that can use the settings from the smoke-configuration file. It runs the smoke cycle for all configurations and is a replacement for the now removed mktest.pl.

mailrpt.pl

Small wrapper around Test::Smoke::Mailer that can use the settings from the smoke-configuration file. It will check if the report is already generated and do so if not. This is the replacement for the now removed mkovz.pl.

archiverpt.pl

This script copies the report and the logfile to a configurable place.

smokestatus.pl

This script prints a little status report for a smoke confifuration:

$ ./smokestatus.pl -c smoke58x
Checking status for configuration 'smoke58x' (5.8.x)
  Change number 21537 started on Sun Oct 26 06:30:39 2003.
    4 out of 10 configurations finished in 3 hours 6 minutes.
    1 configuration showed failures (F).
    0 failures in the running configuration.
    6 configurations to finish, estimated completion in 4 hours 40 minutes
    Average smoke duration: 46 minutes 42 seconds.
W32Configure.bat

Small wrapper around Test::Smoke::Util::Configure_win32() that can use some of the settings from the smoke-configuration file.

This helps users on Windows to create the <builddir>/win32/smoke.mk makefile for a specific configuration in a unixy way.

How can I skip a step in smokeperl.pl?

smokeperl.pl takes arguments to skip some of the steps it takes:

--nofetch Do not sync the source-tree before smoking
--nopatch Do not patch the source-tree (if applicable)
--nomail Do not send the report out by mail
--noarchive Do not archive the report and logfile

smokeperl.pl will always do the equivalent of runsmoke.pl and generate the report.

After running configsmoke.pl you should have a small wrapper-script, that sets up the environment and calls smokeperl.pl. All arguments passed to this script will be passed to smokeperl.pl! Please see smokeperl.pl for details.

How do I smoke my patch?

When preparing a patch for admission to perl5-porters it is always a good idea to test it. Test::Smoke has a mechanism to smoke-test the effects of that patch. Here is a way to set it up:

Set up Test::Smoke

Install Test::Smoke and create a separate directory to keep all your configurations in.

Create a build-configurations file

In this configs directory you then create a build-configurations file. This is usually done from one of default *.cfg files in the Test::Smoke distribution.

Configure Test::Smoke with configsmoke.pl

Now you need to configure Test::Smoke by running configsmoke.pl in your configs directory (optionally specifying a prefix).

$ ../smoke/configsmoke.pl -p testpatch

You should be careful with the e-mail options and make sure you do not send the results to either <smokers-reports@perl.org> or <perl5-porters@perl.org>.

Create a patches-to-apply-file

The patches-to-apply-file has a simple format. It has each patch to apply on a single line and each line is semicolon separated with the full path to the patchfile in the first column and optionally the options to pass to patch(1) in the second column. By default -p1 is passed if no -pN option was specified.

$ cat mypatch.inc
~/p5p/patches/mypatch.diff;-p0
Start the smoke with the --pfile option
   $ ./testpatch.sh --pfile mypatch.inc

You can also configure Test::Smoke to always use a patches-to-apply-file during configsmoke.pl.

Can I interrupt a smoke run?

Yes, you can kill all the smoke related processes. Each stage is directly logged in <builddir>/mktest.out.

As of version 1.17.02 you can specify a timeout (if you have alarm())

Can I still generate a report after an interrupted smoke?

Yes, but some information might be missing (the compiler info).

$ perl mailrpt.pl -c [configname]

How can I run continues smokes?

On a unixy system you can try this shellscript if you adjust the basedir variable (thank you Johan Vromans for helping with this).

#! /bin/sh

basedir=$HOME/Test-Smoke/smoke
cd "$basedir"
pidfile=`pwd`/smokeloop.pid
stpfile=`pwd`/.stopsmoking
versions="58x current"

if test -f "$pidfile" ; then
    echo "We seem to be running as: `cat $pidfile`"
    exit
fi

trap 'trap "" 0 ; rm -f "$pidfile" ; echo "exiting now" ; exit' 0 1 2 3 15

checkstop() {
    if test -f "$stpfile" ; then
        echo "`date`: Stopping on user request"
        rm -f "$stpfile"
        exit
    fi
}

echo $$ > "$pidfile"
while test 1 ; do
    checkstop
    touch "$pidfile"

    for smokever in $versions ; do
        nice $basedir/smoke$smokever.sh
        checkstop
    done

    perl -e 'sleep 3600 if (1+2*rand)*(time-(stat(shift))[9])<600' \
        "$pidfile"
done

This script assumes it is located in the directory where the Test::Smoke suite is installed. You can stop execution between smokes for different versions by putting the file .stopsmoking in that same directory.

Where do the reports go?

To the address configured in configsmoke.pl. The default is <smokers-reports@perl.org>.

There are several ways to monitor the reports send in:

Subscribe to the mailinglist

You can subscribe to that list by sending an empty message to <daily-build-reports-subscribe@perl.org>. You do not have to be subscribed to send reports!

The usenet mirror

The mailinglist also has a mirror on usenet: <news:nntp.perl.org/perl.daily-build.reports>

Posts to the newsgroup will also be mirrored on the mailinglist.

Google groups

The newsgroup is mirrored on Google groups: http://groups.google.com/group/perl.daily-build.reports

The usenet WWW archive

The usenet mirror also has an archive on the web: http://www.nntp.perl.org/group/perl.daily-build.reports

How do I investigate failures?

When investigating a testfailure you need to make sure you have Configured and build perl with the same options as the one with the failures.

$ cd <builddir>
$ make distclean
$ sh Configure -des -Dusedevel [other build options]
$ make
$ make test-prep
$ cd t

Now you can start with a manual run of the failing test:

$ ./perl harness -v <pathto/failingtest>
Note for Win32

As Windows lacks a Configure to create the Makefile, Test::Smoke supplies the helper script W32Configure.bat that will create a new Makefile with the right options: <builddir>\win32\smoke.mk.

After that you can:

S:\perl-current\win32>nmake -f smoke.mk
S:\perl-current\win32>nmake -f smoke.mk test-prep
S:\perl-current\win32>cd ..\t

For -Duseperlio builds (per default) failures can occur in different environments, and you will also need to set these up:

PERLIO=stdio
$ PERLIO=stdio ./perl harness -v <pathto/failingtest>

Note that Windows builds (MSVC, Borland or MinGW) also need the "crlf" layer:

S:>(set PERLIO=stdio :crlf) && .\perl harness -v <pathto\failingtest>

Be careful with the environment variables on Windows, they are sticky.

PERLIO=perlio
$ PERLIO=perlio ./perl harness -v <pathto/failingtest>

Note that Windows builds (MSVC, Borland or MinGW) also need the "crlf" layer:

S:>(set PERLIO=perlio :crlf) && .\perl harness -v <pathto\failingtest>

Be careful with the environment variables on Windows, they are sticky.

UTF-8 locale

If the failure occurs in the UTF-8 locale test-run the environment setup is a bit different (assuming "en_US.utf8"):

$ PERL_UNICODE= LC_ALL=en_US.utf8 ./perl harness -v <pathto/failingtest>

What's with the "daily-build" and "smokers" names?

There has been enough discussion on the subject, so lets agree to disagree (it was all before my time).

The case is now that the official name for the mailinglist is daily-build-reports and there is an alias to smokers-reports.

REVISION

$Id: FAQ 909 2005-09-10 11:59:47Z abeltje $

COPYRIGHT

Copyright 2002-2003, Abe Timmerman <abeltje@cpan.org> All rights reserved.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

See:

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.