1.059 2021-07-31 Released-By: PERLANCAR; Urgency: medium
- [bugfix] Fix goto bug that caused empty item names.
- [action list-items] Show _succinct_name for debugging.
1.058 2021-07-31 Released-By: PERLANCAR; Urgency: medium
- [bugfix] When rendering result as Benchmark.pm result, fix
calculating rate from time (time is in ms not s).
- [bugfix] Avoid mangling non-function participant names when
creating succinct names (e.g. 'perl -e (baseline)'.
1.057 2021-07-23 Released-By: PERLANCAR; Urgency: high
- Doh, typo.
1.056 2021-07-23 Released-By: PERLANCAR; Urgency: medium
- chart_result(): Change to use items directly, and use
_succinct_name.
- Tweak/fix logging statement.
1.055 2021-07-23 Released-By: PERLANCAR; Urgency: medium
- Add new renderer formatter: RenderAsBenchmarkPm.
1.054 2021-07-23 Released-By: PERLANCAR; Urgency: medium
- Split Sah schemas to its own dist Sah-Schemas-Bencher.
1.053 2021-04-10 Released-By: PERLANCAR; Urgency: medium
- Update dependency's new name {,Data::}TableData::Object.
1.052 2020-09-21 Released-By: PERLANCAR; Urgency: medium
- Improve multimodver/multiperl mode: Do not dump _code (coderef),
but reconstruct from _code_str instead.
- Allow participant that does not mention module to be permuted in
multimodver mode.
1.051 2020-08-16 Released-By: PERLANCAR; Urgency: high
- Merge two releases 1.050 and 1.050.1.
1.050.1 2020-08-15 Released-By: PERLANCAR; Urgency: medium; Status: Cancelled
- Add bencher options: tidy, keep_tempdir.
- [Rinci metadata] Tweak argument categories.
- UPDATE 2020-08-16: rename release from 1.050 to 1.050.1 and cancel
(duplicate).
1.050 2020-06-18 Released-By: PERLANCAR; Urgency: medium
- Observe module_startup_precision property.
1.049 2020-05-10 Released-By: PERLANCAR; Urgency: medium
- When profiling, observe --precision to set a repetition loop.
1.048 2020-04-29 Released-By: PERLANCAR; Urgency: low
- Tweak Rinci metadata: replace deprecated
x.schema.{,element_}entity with more specific schema.
- Avoid perl undef warning.
1.047 2020-01-12 Released-By: PERLANCAR; Urgency: medium
- Allow specifying field format in add_field().
- [experimental] Replace formatter AddVsSlowest with AddComparisonFields
(which adds pct_faster_vs_slowest, pct_slower_vs_fastest) to make it
easy to say "xx times/percent slower than ..." as well as "xx
times/percent faster than ...".
1.046 2019-11-29 Released-By: PERLANCAR; Urgency: high
- Merge releases 1.045 & 1.045.1.
1.045.1 2019-11-29 Released-By: PERLANCAR; Urgency: medium
- Update coerce rule names (Data::Sah::Coerce 0.034+). UPDATE
2019-12-02: rename release from 1.045 to 1.045.1, cancel release
(duplicate version).
1.045 2019-10-15 Released-By: PERLANCAR; Urgency: medium
- Update renamed prereq Data::Clean::JSON -> Data::Clean::ForJSON.
1.044 2019-02-24 Released-By: PERLANCAR
- Support loading scenario from an Acme::CPANModules module (add
--cpanmodules-module option).
1.043 2019-02-24 Released-By: PERLANCAR
- Use Exporter to check imported args.
1.042 2019-02-15 Released-By: PERLANCAR
- [ux] Allow specifying scenario module with Bencher::Scenario::
prefix.
- Stringify arg_XXX result column.
1.041 2018-04-03 Released-By: PERLANCAR
- Implement import_args & import_args_array.
- [ux] In multimodver mode, allow if there is only one path/version
of the module.
- Do not turn on --with-process-size on --module-startup or
--code-startup because they are meaningless, the participant is
of type=command.
1.040 2017-07-13 Released-By: PERLANCAR
- Add code_startup mode.
1.039 2017-07-02 Released-By: PERLANCAR
- [Bugfix] Incomplete conversion to Log::ger.
1.038 2017-07-02 Released-By: PERLANCAR
- [ux] If there is only one item and we are in interactive mode, open
the profile result in browser.
1.037 2017-07-02 Released-By: PERLANCAR
- Add actions: dump-parsed-scenario, dump-items, profile (to profile
items with Devel::NYTProf).
- Replace Log::Any with Log::ger.
1.036 2017-02-20 Released-By: PERLANCAR
- [Bugfix] Fix completion of participant & dataset tags.
1.035 2017-02-19 Released-By: PERLANCAR
- Show participant tags (p_tags) & dataset tags (ds_tags) columns
in result.
1.034 2017-01-25 Released-By: PERLANCAR
- Introduce participant property: helper_modules (allow specifying
modules that are require for running code but are not subject of
benchmark).
- Return Bencher::Backend version in result meta.
- [ux] Show elapsed time when output format is text.
1.033 2017-01-04 Released-By: PERLANCAR
- [ux] Format numbers etc when output format is /html/, show
platform information when output format is text.
1.032 2016-12-10 Released-By: PERLANCAR
- Turn on with_process_size by default when in module_startup mode
on linux.
1.031 2016-11-23 Released-By: PERLANCAR
- Record scenario file's mtime and checksums (MD5, SHA1, SHA256) in
result metadata.
1.030 2016-10-28 Released-By: PERLANCAR
- [ux] Provide completion for --sort/--sorts.
1.029 2016-10-27 Released-By: PERLANCAR
- Add argument: scenario (to pass scenario data structure).
1.028 2016-10-20 Released-By: PERLANCAR
- Do not require Sys::Load on Windows because it doesn't install there.
Bencher now works on Windows.
1.027 2016-10-14 Released-By: PERLANCAR
- Add options: --exclude-xs-modules (--noxs) --exclude-pp-modules
(--nopp).
1.026 2016-10-13 Released-By: PERLANCAR
- Add hooks before_parse_{datasets,participants} and deprecated
before_list_{datasets,participants}. To generate
datasets/participants dynamically, use the before_parse_* hooks
because the datasets/participants can later still be filtered.
- Add hooks 'before_test_item' & 'after_test_item'.
1.025 2016-10-09 Released-By: PERLANCAR
- [Bugfix] Several bug fixes.
- Add property/option 'test' to skip testing participant code prior to
benchmarking. Useful if code runs for at least a few seconds and we
don't want to wait for too long.
1.024 2016-10-09 Released-By: PERLANCAR
- [Incompatible change] Remove action 'bench-with-benchmark-pm' (see
below to run benchmark with Benchmark.pm).
- Add property/option 'runner' which can be the default
'Benchmark::Dumb', or 'Benchmark', or the new
'Benchmark::Dumb::SimpleTime' which can be used if you want to
benchmark code that runs for at least a few seconds with very few
iterations (like 1 or 2) without "Number of runs too small" warning
from Benchmark::Dumb.
1.023 2016-09-09 Released-By: PERLANCAR
- Aside from observing per-dataset 'result' property, also observe
scenario's 'result' property for when we don't have any datasets.
1.022 2016-09-09 Released-By: PERLANCAR
- [workaround] Some completions were broken because we use
'str_comma_sep' coercion rules for --include-path, this is
because include_path is now a string instead or array[str] before
validation+coercion, and completion currently does not use
Data::Sah, so now we disable this coercion rule for now.
1.021 2016-08-26 Released-By: PERLANCAR
- Introduce --with-args-sizes.
1.020 2016-08-24 Released-By: PERLANCAR
- [Incompatible change] Rename options for consistency
--include-process-size to --with-process-size,
--include-result-size to --with-result-size (--include-* are for
filter options).
- [ux] Add completion for comma-separated filter options, e.g.
--include-participants, --exclude-dataset-seqs, etc.
1.019 2016-08-24 Released-By: PERLANCAR
- [ux] Allow specifying options like --include-datasets with
comma-separated value, so --include-datasets 1,2,3 instead of
--include-dataset 1 --include-dataset 2 --include-dataset 3 or
--include-datasets-json '[1,2,3]'.
- [Bugfix][Regression] Accidentally removed a code to permute
dataset, breaking v1.018.
1.018 2016-08-20 Released-By: PERLANCAR
- UPDATE: this release is broken, don't use.
[NEW FEATURES]
- Allow items to be benchmarked using Benchmark.pm instead of
Benchmark::Dumb. To do this, specify bencher option
--bench-with-benchmark-pm. Currently if you do this, the raw result
from Benchmark.pm will be used and no result table will be
returned/saved. Also don't forget to set --benchmark-pm-count when
appropriate (the default is -0.5).
- Introduce scenario property 'env_hashes' (can also be set from the CLI
using --env-hash). This allows permuting multiple environment variable
sets.
1.017 2016-07-13 Released-By: PERLANCAR
- Add chart_result().
1.016 2016-07-13 Released-By: PERLANCAR
- Include permutation in result metadata (func.permute).
1.015 2016-07-12 Released-By: PERLANCAR
- Handle negative time in results.
- [Bugfix] Still allow stringy matching of --include with name even
though value is a number.
1.014 2016-06-30 Released-By: PERLANCAR
- Add function: split_result().
1.013 2016-06-29 Released-By: PERLANCAR
- [Bugfix] Determine number of significant digits from 'time', not from
'rate' because 'rate' is removed in module startup result.
1.012 2016-06-28 Released-By: PERLANCAR
- format_result(): Add a convenient way to skip using the
RenderAsTextTable formatter by passing option render_as_text_table=>0.
- Avoid some perl undef warnings, e.g. for older bencher results where
'perl' field is not defined.
1.011 2016-06-23 Released-By: PERLANCAR
- More tweaks/fixes for alignments.
1.010 2016-06-17 Released-By: PERLANCAR
- [Bugfix] DeleteConstantFields formatter didn't use
delete_fields() yet.
1.009 2016-06-17 Released-By: PERLANCAR
- [ux] Add column alignment to result.
- [ux] Use decimal notation by default, except for 'errors' field
or when option --scientific-notation is given.
1.008 2016-05-20 Released-By: PERLANCAR
- Follow Sah::Schemas specification, rename
Sah::Schema::Bencher::Scenario to Sah::Schema::bencher::scenario.
- Avoid perl undef warning.
1.007 2016-04-19 Released-By: PERLANCAR
- [Bugfix] save_result was turned *on* all the time, it should be
only when --save-result or (--result-dir or BENCHER_RESULT_DIR
was set).
1.006 2016-04-13 Released-By: PERLANCAR
- Add --{in,ex}clude-{participant,dataset,item}-{name,seq} to
unambiguously include/exclude by name or sequence.
- [ux] Warn when there is a participant/dataset/item which has name that
is also a sequence number, which can be confusing when
including/excluding.
- [Bugfix] Defeat the annoying compile errors when using Capture::Tiny
due to mismatching prototypes.
1.005 2016-04-08 Released-By: PERLANCAR
[ENHANCEMENTS]
- Add --include-process-size to show process memory stats. UPDATE: this
option has been renamed to --with-process-size in 1.020.
- Add output capturing (--show-items-outputs, --capture-stdout,
--capture-stderr).
[BUG FIXES]
- Regression: --show-items-results and --show-items-results-sizes
returned JSON by default when it should return a formatted text.
- When trying to come up with a name for a command participant, also
take a look at cmdline_template/perl_cmdline/perl_cmdline_template in
addition to just cmdline.
1.004 2016-04-08 Released-By: PERLANCAR
- Eval code template in 'main' package.
- Allow <VARNAME:raw> in template to avoid escaping.
1.003 2016-04-03 Released-By: PERLANCAR
- [Bugfix] completer for participant/dataset name didn't include
generated names.
- [Bugfix] save_result was not turned on automatically when user
sets result_dir.
1.002 2016-04-01 Released-By: PERLANCAR
- [Bug] Logic bug: --return-meta accidentally defaulted to off when
--save-result is not enabled.
1.001 2016-04-01 Released-By: PERLANCAR
- Turn off include_result_size when in module startup mode.
- Add feature to save result to a JSON file (--save-result,
--result-dir, --result-filename).
- Grok scenario property 'extra_modules' to allow extra modules to
be loaded and their versions recorded in benchmark result
metadata.
1.000 2016-03-31 Released-By: PERLANCAR
- No functional changes.
- Bump version number to 1.000.
0.46 2016-03-31 Released-By: PERLANCAR
- No functional changes.
- [doc] Add description, fixes.
0.45 2016-03-30 Released-By: PERLANCAR
- [ux] Check that what's specified in
--{in,ex}clude-{dataset,participant,item} are actually known, to
avoid silent surprise.
- [Bugfix] Fix typo that caused sometimes specifying
--include-participant by name doesn't work.
- [Bugfix] RoundNumbers formatter: no need to round result_size
except when it's not a round number.
0.44 2016-03-26 Released-By: PERLANCAR
- Show result size (as measured by Devel::Size), add
--include-result-size (& include_result_size scenario property), add
action show-items-results-sizes. UPDATE: the option has been renamed
to --with-result-size in 1.020.
- [Bugfix] Fix completion of -m when there is -I specified.
0.43 2016-03-24 Released-By: PERLANCAR
- [ux] Directory specified in -I (--include-path) is now also used
for searching and loading scenario module, this is so we can say
something like 'bencher -Ilib -m Foo' instead of 'PERL5OPT=-Ilib
bencher -m Foo'.
- [ux] When a dataset has no name, try to assign a name from
argument value.
0.42 2016-03-22 Released-By: PERLANCAR
- Allow specifying 'modules' in participant to benchmark startup
overhead of several modules at once.
0.41 2016-03-18 Released-By: PERLANCAR
- No functional changes.
- [Bugfix] Include modver & perl in result items even when in not
multiperl/multimodver mode (ModuleStartup formatter module uses
the perl key).
- Add workaround for RT#113117 (List::MoreUtils sometimes return
undef max).
0.40 2016-03-16 Released-By: PERLANCAR
- Add action: list-perls; by default only use perls that have
Bencher::Backend installed.
- [Bugfix] Avoid benchmarking duplicate items, e.g. 'perl -e1
(baseline)' when in module_startup mode + multimodver mode.
- [Bugfix] fix calculating mod_overhead_time when in multiperl mode.
0.39 2016-03-15 Released-By: PERLANCAR
[ENHANCEMENTS]
- Support benchmarking against multiple perls (introduce options
`--multiperl`, `--include-perl`, `--exclude-perl`). Limitation:
Currently this is implemented by dumping (using Data::Dmp, which means
using B::Deparse) the items' codes into a temporary scenario file,
then executing the scenario file using each perl executable. This
means codes that contains closures Released-By: references to outside variable
won't work correctly.
- Support benchmarking multiple versions of a module (`--multimodver`).
To do this, first add more module search paths using `--include-path`
(`-I`).
- In participant, introduce 'perl_cmdline' and 'perl_cmdline_template'
in addition to 'cmdline' and 'cmdline_template'. They are the same as
'cmdline*' except there is implicit first argument which is the perl
executable. They are for executing commands using perl, without having
to hardcode/include the perl executable location.
0.38 2016-03-15 Released-By: PERLANCAR
- First release, split from Bencher distribution to reduce dependencies
when installing Bencher to a perl installation (this is part of
developing support for benchmarking against multiple perls/module
versions).