Revision history for Test-Harness
3.02 2007-11-15
- Process I/O now unbuffered, uses sysread, plays better with
select. Fixes #30740.
- Made Test::Harness @INC propagation more compatible with 2.64.
Was breaking Parrot's test suite.
- Added HARNESS_OPTIONS (#30676)
3.01 2007-11-12
- Fix for RHEL incpush.patch related failure.
- Output real time of test completion with --timer
- prove -b adds blib/auto to @INC
- made SKIP plan parsing even more liberal for pre-v13 TAP
3.00 2007-11-06
- Non-dev release. No changes since 2.99_09.
2.99_09 2007-11-05
- Implemented TODO-in-PLAN syntax for TAP version 12 and earlier.
2.99_08 2007-11-04
- Tiny changes. New version pushed to get some smoke coverage.
2.99_07 2007-11-01
- Fix for #21938: Unable to handle circular links
- Fix for #24926: prove -b and -l should use absolute paths
- Fixed prove switches. Big oops. How the hell did we miss that?
- Consolidated quiet, really_quiet, verbose into verbosity.
- Various VMS related fixes to tests
2.99_06 2007-10-30
- Added skip_all method to TAP::Parser.
- Display reason for skipped tests.
- make test now self tests.
2.99_05 2007-10-30
- Fix for occasional rogue -1 exit code on Windows.
- Fix for @INC handling under CPANPLUS.
- Added real time to prove --timer output
- Improved prove error message in case where 't' not found and
no tests named.
2.99_04 2007-10-11
- Fixed bug where 'All tests successful' would not be printed if bonus
tests are seen.
- Fixed bug where 'Result: FAIL' would be printed at the end of a test
run if there were unexpectedly succeeding tests.
- Added -M, -P switches to allow arbitrary modules to be loaded
by prove. We haven't yet defined what they'll do once they
load but it's a start...
- Added testing under simulated non-forking platforms.
2.99_03 2007-10-06
- Refactored all display specific code out of TAP::Harness.
- Relaxed strict parsing of skip plan for pre v13 TAP.
- Elapsed hi-res time is now displayed in integer milliseconds
instead of fractional seconds.
- prove stops running if any command-line switches are invalid.
- prove -v would try to print an undef.
- Added support for multiplexed and forked parallel tests. Use
prove -j 9 to run tests in parallel and prove -j 9 --fork to
fork. These features are experimental and currently
unavailable on Windows.
- Rationalized the management of the environment that we give to
test scripts (PERL5LIB, PERL5OPT, switches).
- Fixed handling of STDIN (we no longer close it) for test
scripts.
- Performance enhancements. Parser is now 30% - 40% faster.
2.99_02 2007-09-07
- Ensure prove (and App::Prove) sort any recursively
discovered tests
- It is now possible to register multiple callback handlers for
a particular event.
- Added before_runtests, after_runtests callbacks to
TAP::Harness.
- Moved logic of prove program into App::Prove.
- Added simple machine readable summary.
- Performance improvement: The processing pipeline within
TAP::Parser is now a closure which speeds up access to the
various attribtes it needs.
- Performance improvement: Test count spinner now updates
exponentially less frequently as the count increases which
saves a lot of I/O on big tests.
- More improvements in test coverage from Leif.
- Fixes to TAP spooling - now captures YAML blocks correctly.
- Fix YAMLish handling of empty arrays, hashes.
- Renamed TAP::Harness::Compatible to Test::Harness,
runtests to prove.
- Fixes to @INC handling. We didn't always pass the correct path
to subprocesses.
- We now observe any switches in HARNESS_PERL_SWITCHES.
- Changes to output formatting for greater compatibility with
Test::Harness 2.64.
- Added unicode test coverage and fixed a couple of
unicode issues.
- Additions to documentation.
- Added support for non-forking Perls. If forking isn't
available we fall back to open and disable stream merging.
- Added support for simulating non-forking Perls to improve our
test coverage.
========================================================================
Version numbers below this point relate to TAP::Parser - which was the
name of this version of Test::Harness during its development.
========================================================================
0.54
- Optimized I/O for common case of 'runtests -l'
- Croak if supplied an empty (0 lines) Perl script.
- Made T::P::Result::YAML return literal input YAML correctly.
- Merged speed-ups from speedy branch.
0.53 18 August 2007
- Fixed a few docs nits.
- Added -V (--version) switch to runtests. Suggested by markjugg on
Perlmonks.
- Fixed failing t/030-grammer.t under 5.9.5. Exact cause still
unknown; something to do with localisation of $1 et all I think.
- Fixed use of three arg open in t/compat/test-harness-compat; was
failing on 5.6.2.
- Fixed runtests --exec option. T::H wasn't passing the exec option
to T::P.
- Merged Leif Eriksen's coverage enhancing changes to
t/080-aggregator.t, t/030-grammar.t
- Made various changes so that we test cleanly on 5.0.5.
- Many more coverage enhancements by Leif.
- Applied Michael Peters' patch to add an EOF callback to
TAP::Parser.
- Added --reverse option to runtests to run tests in reverse order.
- Made runtests exit with non-zero status if the test run had
problems.
- Stopped TAP::Parser::Iterator::Process from trampling on STDIN.
0.52 14 July 2007
- Incorporate Schwern's investigations into TAP versions.
Unversioned TAP is now TAP v12. The lowest explicit version number
that can be specified is 13.
- Renumbered tests to eliminate gaps.
- Killed execrc. The '--exec' switch to runtests handles all of this for
us.
- Refactored T::P::Iterator into
T::P::Iterator::(Array|Process|Stream) so that we have a
process specific iterator with which to experiment with
STDOUT/STDERR merging.
- Removed vestigial exit status handling from T::P::I::Stream.
- Removed unused pid interface from T::P::I::Process.
- Fixed infinite recursion in T::P::I::Stream and added regression
coverage for same.
- Added tests for T::P::I::Process.
- TAP::Harness now displays the first five TAP syntax errors and
explains how to pass the -p flag to runtests to see them all.
- Added merge option to TAP::Parser::Iterator::Process,
TAP::Parser::Source, TAP::Parser and TAP::Harness.
- Added --merge option to runtests to enable STDOUT/STDERR merging.
This behaviour used to be the default.
- Made T::P::I::Process use open3 for both merged and non-merged
streams so that it works on Windows.
- Implemented Eric Wilhelm's IO::Select based multiple stream
handler so that STDERR is piped to us even if stream merging is
turned off. This tends to reduce the temporal skew between the
two streams so that error messages appear closer to their
correct location.
- Altered the T::P::Grammar interface so that it gets a stream
rather than the next line from the stream in preparation for
making it handle YAML diagnostics.
- Implemented YAML syntax. Currently YAML may only follow a
test result. The first line of YAML is '---' and the last
line is '...'.
- Made grammar version-aware. Different grammars may now be selected
depending on the TAP version being parsed.
- Added formatter delegate mechanism for test results.
- Added prototype stream based YAML(ish) parser.
- Added more tests for T::P::YAMLish
- Altered T::P::Grammar to use T::P::YAMLish
- Removed T::P::YAML
- Added raw source capture to T::P::YAMLish
- Added support for double quoted hash keys
- Added TAP::Parser::YAMLish::Writer and renamed T::P::YAMLish as
T::P::YAMLish::Reader.
- Added extra TAP::Parser::YAMLish::Writer output options
- Inline YAML documents must now be indented by at least one space
- Fixed broken dependencies in bin/prove
- Make library paths absolute before running tests in case tests
chdir before loading modules.
- Added libs and switches handling to T::H::Compatible. This and the
previous change fix [24926]
- Added PERLLIB to libraries stripped in _default_inc [12030]
- Our version of prove now handles directories containing circular
links correctly [21938]
- Set TAP_VERSION env var in Parser [11595]
- Added setup, teardown hooks to T::P::I::Process to facilitate the
setup and cleanup of the test script's environment
- Any additional libs added to the command line are also added to
PERL5LIB for the duration of a test run so that any Perl children
of the test script inherit the same library paths.
- Fixed handling of single quoted hash keys in T::P::Y::Reader
- Made runtests return the TAP::Parser::Aggregator
- Fixed t/120-harness.t has failures if TAP::Harness::Color cannot
load optional modules [27125] - thanks DROLSKY
- Fixed parsing of \# in test description
0.51 12 March 2007
- 'execrc' file now allows 'regex' matches for tests.
- rename 'TAPx' --> 'TAP'
- Reimplemented the parse logic of TAP::Parser as a state machine.
- Removed various ad-hoc state variables from TAP::Parser and moved
their logic into the state machine.
- Removed now-unused is_first / is_last methods from Iterator and
simplified remaining logic to suit.
- Removed now-redundant t/140-varsource.t.
- Implemented TAP version syntax.
- Tidied TAP::Harness::Compatible documentation
- Removed redundant modules below TAP::Harness::Compatible
- Removed unused compatibility tests
0.50_07 5 March 2007
- Fixed bug where we erroneously checked the test number instead of number
of tests run to determine if we've run more tests than we planned.
- Add a --directives switch to 'runtests' which only shows test results
with directives (such as 'TODO' or 'SKIP').
- Removed some dead code from TAPx::Parser.
- Added color support for Windows using Win32::Console.
- Made Color::failure_output reset colors before printing
the trailing newline.
- Corrected some issues with the 'runtests' docs and removed some
performance notes which no longer seem accurate.
- Fixed bug whereby if tests without file extensions were included then
the spacing of the result leaders would be off.
- execrc file is now a YAML file.
- Removed white background on the test failures. It was too garish for
me. Just more proof that we need better ways of overriding color
support.
- Started work on TAPx::Harness::Compatible. Right now it's mainly just
a direct lift of Test::Harness to make sure the tests work.
- Commented out use Data::Dumper::Simple in T::Harness.pm - it's not
a core module.
- Added next_raw to TAPx::Parser::Iterator which skips any fixes for
quirky TAP that are implemented by next. Used to support
TAPx::Harness::Compatible::Iterator
- Applied our version number to all T::H::Compatible modules
- Removed T::H::C::Assert. It's documented as being private to
Test::Harness and we're not going to need it.
- Refactored runtests to call aggregate_tests to expose the
interface we need for the compatibility layer.
- Make it possible to pass an end time to summary so that it needn't
be called immediately after the tests complete.
- Moved callback handling into TAPx::Base and altered TAPx::Parser
to use it.
- Made TAPx::Harness into a subclass of TAPx::Base and implemented
made_parser callback.
- Moved the dispatch of callbacks out of run and into next so that
they're called when TAPx::Harness iterates through the results.
- Implemented PERL_TEST_HARNESS_DUMP_TAP which names a directory
into which the raw TAP of any tests run via TAPx::Harness will
be written.
- Rewrote the TAPx::Grammar->tokenize method to return a
TAPx::Parser::Result object. Code is much cleaner now.
- Moved the official grammar from TAPx::Parser to TAPx::Parser::Grammar,
provided a link and updated the grammar.
- Fixed bug where a properly escaped '# TODO' line in a test description
would still be reported as a TODO test.
- Added patches/ExtUtils-MakeMaker-6.31.patch - a patch against EUMM
that makes test_harness use TAPx::Harness instead of Test::Harness
if PERL_EUMM_USE_TAPX is true and TAPx::Harness is installed. In
other words cause 'make test' for EUMM based models to use
TAPx::Harness.
- Added support for timer option to TAPx::Harness which causes the
elapsed time for each test to be displayed.
- Setup tapx-dev@hexten.net mailing list.
- Fixed accumulating @$exec bug in TAPx::Harness.
- Made runtests pass '--exec' option as an array.
- (#24679) TAPx::Harness now reports failure for tests that die
after completing all subtests.
- Added in_todo attribute on TAPx::Parser which is true while the
most recently seen test was a TODO.
- (#24728) TAPx::Harness now supresses diagnostics from failed
TODOs. Not sure if the semantics of this are correct yet.
0.50_06 18 January 2007
- Fixed doc typo in examples/README [rt.cpan.org #24409]
- Colored test output is now the default for 'runtests' unless
you're running under windows or -t STDOUT is false.
[rt.cpan.org #24310]
- Removed the .t extension from t/source_tests/*.t since those are
'test tests' which caused false negatives when running recursive
tests. [Adrian Howard]
- Somewhere along the way, the exit status started working again.
Go figure.
- Factored color output so that disabling it under Windows is
cleaner.
- Added explicit switch to :crlf layer after open3 under Windows.
open3 defaults to raw mode resulting in spurious \r characters input
parsed input.
- Made Iterator do an explicit wait for subprocess termination.
Needed to get process status correctly on Windows.
- Fixed bug which didn't allow t/010-regression.t to be run directly
via Perl unless you specified Perl's full path.
- Removed SIG{CHLD} handler (which we shouldn't need I think because
we explicitly waitpid) and made binmode ':crlf' conditional on
IS_WIN32. On Mac OS these two things combined to expose a problem
which meant that output from test scripts was sometimes lost.
- Made t/110-source.t use File::Spec->catfile to build path to
test script.
- Made Iterator::FH init is_first, is_last to 0 rather than undef
for consistency with array iterator.
- Added t/120-varsource.t to test is_first and is_last semantics
over files with small numbers of lines.
- Added check for valid callback keys.
- Added t/130-results.t for Result classes.
0.50_05 15 January 2007
- Removed debugging code accidentally left in bin/runtests.
- Removed 'local $/ = ...' from the iterator. Hopefully that will fix the
line ending bug, but I don't know about the wstat problem.
0.50_04 14 January 2007
- BACKWARDS IMCOMPATIBLE: Renamed all '::Results' classes to '::Result'
because they represent a single result.
- Fixed bug where piping would break verbose output.
- IPC::Open3::open3 now takes a @command list rather than a $command
string. This should make it work under Windows.
- Added 'stdout_sterr' sample test back to regression tests. IPC::Open3
appears to make it work.
- Bug fix: don't print 'All tests successful' if no tests are run.
- Refactored 'runtests' to make it a bit easier to follow.
- Bug fix: Junk and comments now allowed before a leading plan.
- HARNESS_ACTIVE and HARNESS_VERSION environment variables now set.
- Renamed 'problems' in TAPx::Parser and TAPx::Aggregator to
'has_problems'.
0.50_03 08 January 2007
- Fixed bug where '-q' or '-Q' with colored tests weren't suppressing all
information.
- Fixed an annoying MANIFEST nit.
- Made '-h' for runtests now report help. Using a new harness requires
the full --harness switch.
- Added 'problems' method to TAPx::Parser and TAPx::Parser::Aggregator.
- Deprecatd 'todo_failed' in favor of 'todo_passed'
- Add -I switch to runtests.
- Fixed runtests doc nit (smylers)
- Removed TAPx::Parser::Builder.
- A few more POD nits taken care of.
- Completely removed all traces of C<--merge> as IPC::Open3 seems to be
working.
- Moved the tprove* examples to examples/bin in hopes of them no longer
showing up in CPAN's docs.
- Made the 'unexpectedly succeeded' message clearer (Adam Kennedy)
0.50_02 06 January 2007
- Added some files I left out of the manifest (reported by Florian
Ragwitz).
- Added strict to Makefile.PL and changed @PROGRAM to @program (reported
Florian Ragwitz).
0.50_01 06 January 2007
- Added a new example which shows to how test Perl, Ruby, and URLs all at
the same time using 'execrc' files.
- Fixed the diagnostic format mangling bug.
- We no longer override Test::Builder to merge streams. Instead, we go
ahead and use IPC::Open3. It remains to be seen whether or not this is
a good idea.
- Fixed vms nit: for failing tests, vms often has the 'not' on a line by
itself.
- Fixed bugs where unplanned tests were not reporting as a failure (test
number greater than tests planned).
- TAPx::Parser constructor can now take an 'exec' option to tell it what
to execute to create the stream (huge performance boost).
- Added TAPx::Parser::Source. This allows us to run tests in just about
any programming language.
- Renamed the filename() method to source() in TAPx::Parser::Source::Perl.
- We now cache the @INC values found for TAPx::Parser::Source::Perl.
- Added two test harnesses, TAPx::Harness and TAPx::Harness::Color.
- Removed references to manual stream construction from TAPx::Parser
documentation. Users should not (usually) need to worry about streams.
- Added bin/runtests utility. This is very similar to 'prove'.
- Renumbered tests to make it easier to add new ones.
- Corrected some minor documentation nits.
- Makefile.PL is no longer auto-generated (it's built by hand).
- Fixed regression test bug where driving tests through the harness I'm
testing caused things to break.
- BUG: exit() values are now broken. I don't know how to capture them
with IPC::Open3. However, since no one appears to be using them, this
might not be an issue.
0.41 12 December 2006
- Fixed (?) 10-regression.t test which failed on Windows. Removed the
segfault test as it has no meaning on Windows. Reported by PSINNOTT
<link@redbrick.dcu.ie> and fix recommended by Schwern based on his
Test::Harness experience.
http://rt.cpan.org/Ticket/Display.html?id=21624
0.40 05 December 2006
- Removed TAPx::Parser::Streamed and folded its functionality into
TAPx::Parser.
- Fixed bug where sometimes is_good_plan() would return a false positive
(exposed by refactoring).
- A number of tiny performance enhancements.
0.33 22 September 2006
- OK, I'm getting ticked off by some of the comments on Perl-QA so I
rushed this out the door and broke it :( I'm backing out one test and
slowing down a bit.
0.32 22 September 2006
- Applied patch from Schwern which fixed the Builder package name (TAPx::
instead of TAPX:: -- stupid case-insensitive package names!).
[rt.cpan.org #21605]
0.31 21 September 2006
- Fixed bug where Carp::croak without parens could cause Perl to fail to
compile on some platforms. [Andreas J. Koenig]
- Eliminated the non-portable redirect of STDERR to STDOUT (2>&1) and
fixed the synchronization issue. This involves overridding
Test::Builder::failure_output() in a very sneaky way. I may have to
back this out.
- Renamed boolean methods to begin with 'is_'. The methods they replace
are documented, deprecated, and will not be removed prior to version
1.00.
0.30 17 September 2006
- Fixed bug where no output would still claim to have a good plan.
- Fixed bug where no output would cause parser to die.
- Fixed bug where failing to specify a plan would be two parse errors
instead of one.
- Fixed bug where a correct plan count in an incorrect place would still
report as a 'good_plan'.
- Fixed bug where comments could accidently be misparsed as directives.
- Eliminated testing of internal structure of result objects. The other
tests cover this.
- Allow hash marks in descriptions. This was causing a problem because
many test suites (Regexp::Common and Perl core) allowed them to exist.
- Added support for SKIP directives in plans.
- Did some work simplifying &TAPx::Parser::_initialize. It's not great,
but it's better than it was.
- TODO tests now always pass, regardless of actual_passed status.
- Removed 'use warnings' and now use -w
- 'switches' may now be passed to the TAPx::Parser constructor.
- Added 'exit' status.
- Added 'wait' status.
- Eliminated 'use base'. This is part of the plan to make TAPx::Parser
compatible with older versions of Perl.
- Added 'source' key to the TAPx::Parser constructor. Making new parsers
is now much easier.
- Renamed iterator first() and last() methods to is_first() and is_last().
Credit: Aristotle.
- Planned tests != tests run is now a parse error. It was really stupid
of me not to do that in the first place.
- Added massive regression test suite in t/100-regression.t
- Updated the grammar to show that comments are allowed.
- Comments are now permitted after an ending plan.
0.22 13 September 2006
- Removed buggy support for multi-line chunks from streams. If your
streams or iterators return anything but single lines, this is a bug.
- Fixed bug whereby blank lines in TAP would confuse the parser. Reported
by Torsten Schoenfeld.
- Added first() and last() methods to the iterator.
- TAPx::Parser::Source::Perl now has a 'switches' method which allows
switches to be passed to the perl executable running the test file.
This allows tprove to accept a '-l' argument to force lib/ to be
included in Perl's @INC.
0.21 8 September 2006
- Included experimental GTK interface written by Torsten Schoenfeld.
- Fixed bad docs in examples/tprove_color
- Applied patch from Shlomi Fish fixing bug where runs from one stream
could leak into another when bailing out. [rt.cpan.org #21379]
- Fixed some typos in the POD.
- Corrected the grammar to allow for a plan of "1..0" (infinite stream).
- Started to add proper acknowledgements.
0.20 2 September 2006
- Fixed bug reported by GEOFFR. When no tap output was found, an
"Unitialized value" warning occurred. [rt.cpan.org #21205]
- Updated tprove to now report a test failure when no tap output found.
- Removed examples/tprove_color2 as tprove_color now works.
- Vastly improved callback system and updated the docs for how to use
them.
- Changed TAPx::Parser::Source::Perl to use Symbol::gensym() instead of a
hard-to-guess filehandle name.
0.12 30 July 2006
- Added a test colorization script
- Callback support added.
- Added TAPx::Parser::Source::Perl.
- Added TAPx::Parser::Aggregator.
- Added version numbers to all classes.
- Added 'todo_failed' test result and parser.
- 00-load.t now loads all classes instead of having individual tests load
their supporting classes.
- Changed $parser->results to $parser->next
0.11 25 July, 2006
- Renamed is_skip and is_todo to has_skip and has_todo. Much less
confusing since a result responding true to those also responded true to
is_test.
- Added simplistic bin/tprove to run tests. Much harder than I thought
and much code stolen from Test::Harness.
- Modified stolen iterator to fix a bug with stream handling when extra
newlines were encountered.
- Added TAPx::Parser::Iterator (stolen from Test::Harness::Iterator)
- Normalized internal structure of result objects.
- All tokens now have a 'type' key. This greatly simplifies internals.
- Copied much result POD info into the main docs.
- Corrected the bug report URLs.
- Minor updates to the grammar listed in the POD.
0.10 23 July, 2006
- Oh my Larry, we gots docs!
- _parse and _tap are now private methods.
- Stream support has been added.
- Moved the grammar into its own class.
- Pulled remaining parser functionality out of lexer.
- Added type() method to Results().
- Parse errors no longer croak(). Instead, they are available through the
parse_errors() method.
- Added good_plan() method.
- tests_planned != tests_run is no longer a parse error.
- Renamed test_count() to tests_run().
- Renamed num_tests() to tests_planned().
0.03 17 July, 2006
- 'Bail out!' is now handled.
- The parser is now data driven, thus skipping a huge if/else chain
- We now track all TODOs, SKIPs, passes and fails by test number.
- Removed all non-core modules.
- Store original line for each TAP line. Available through
$result->raw().
- Renamed test is_ok() to passed() and added actual_passed(). The former
method takes into account TODO tests and the latter returns the actual
pass/fail status.
- Fixed a bug where SKIP tests would not be identified correctly.
0.02 8 July, 2006
- Moved some lexer responsibility to the parser. This will allow us to
eventually parse streams.
- Properly track passed/failed tests, even accounting for TODO.
- Added support for comments and unknown lines.
- Allow explicit and inferred test numbers to be mixed.
- Allow escaped hashes in the test description.
- Renamed to TAPx::Parser. Will probably rename it again.
0.01 Date/time
- First version, unreleased on an unsuspecting world.
- No, you'll never know when ...