==================================================
Changes from 2021-07-20 00:00:00 +0000 to present.
==================================================
----------------------------------------
version 8.6 at 2023-07-20 20:26:02 +0000
----------------------------------------
Change: 69b286e396cc816a60053c94df43713ea390d24a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2023-07-20 08:53:17 +0000
Preparing 8.6 Release
* Add `App::ElasticSearch::Utilities::Metrics`
* Convert es-graphite-dynamic.pl to use
App::ElasticSearch::Utilities::Metrics * Updates for
`App::ElasticSearch::Utilities::Aggregations`:
* Adding tests for the `es_flatten_aggs()` function
* For aggs with `key_as_string`, don't lose data by adding `$field.raw` as
the `key` value * New Utilities:
* `es-index-fields.pl` shows storage details about the fields in indexes
* `es-index-scan.pl` scans indexes for potential issues * Updates for
`App::ElasticSearch::Utilities`
* `es_index_strip_date()` - Add a fall back parser for date patterns in
index names
* **(new)** `es_human_count()` to transform 10_000 into "10 thousand"
* **(new)** `es_human_size()` to transform 10_000 into "10 Kb" * Require
Perl 5.16+, and test on 5.38
----------------------------------------
version 8.5 at 2023-04-12 23:51:37 +0000
----------------------------------------
Change: 52766654c246913695280adfc9fd74247543d4f4
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2023-04-12 16:51:37 +0000
Prepping 8.5 release
Change: b3de99668fa520fc98432f4a1cbaddc40b6d76bd
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2023-04-12 16:47:40 +0000
Minor Improvements
* Remove old script `es-nagios-check.pl` * Update documentation * Disable
imports from `YAML::XS`
Change: 159f1dad0a7f81d8b436ef00ab1d9086a04d1c0a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2023-04-12 16:33:58 +0000
Fix bug in date parsing logic
Change: 853c42e60acef360f1f2e91d6cbc82ce77afe74e
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2023-04-12 16:33:58 +0000
es-cluster-settings.pl to manage cluster settings
Added `es-cluster-settings.pl` to the distribution to manage cluster
settings more easily.
Change: 250fe6f9738808b493c518fe58c98a1f750d58c9
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-09-08 12:06:35 +0000
Fix warning for unitialized values
``` Use of uninitialized value $out_of in numeric gt (>) at
~/perl5/perlbrew/perls/perl-5.32.1/bin/es-search.pl line 624. ```
----------------------------------------
version 8.4 at 2022-09-02 19:51:04 +0000
----------------------------------------
Change: b6112b07c154fe669e7ab9407b3b20f6f845e7bc
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-09-02 12:51:04 +0000
Convert to YAML::XS
* `YAML::XS` is more compatible with other YAML parsers * Enable boolean
detection for `es-search.pl`
Change: 400b50196d211697fe4d40cca9b587b72d981e48
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-06-01 17:59:00 +0000
Fix display of statistics so they all follow --precision
Change: c6c0e96db37f637bc62492acda8072700244d342
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-06-01 04:20:05 +0000
Add client certificate authentication to the tooling
Change: ee969f539f64bd15bd3c3e30cd85cf273b728a1c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-06-01 03:56:41 +0000
Correct version check for TLS and Auth
* Use `App::ElasticSearch::Utilities::HTTPRequest` for checking the
version string. * Add a check for `PASSEXEC` to `_get_es_version()` so we
can use the
provided credentials * Correct `::Connection->request()` to set
`authorization_basic()` when
the `->password` has a length greater than 0.
Change: e978a9b33eecd59191cfbf88ffc4f7b54d698c74
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-05-31 19:02:24 +0000
More fixes for HTTP Authentication
* Remove `--http-password` and parsing `~/.es-utils.yaml` for the
password. This is not a secure default. * Do not allow HTTP Basic Auth over
HTTP as it leaks credentials * Check for `value_as_string` in
`es_flatten_aggregations`
Change: e49f4394b7a35b3f0539debec53f693f36d43da7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-05-31 03:41:01 +0000
Support HTTPS and Basic Authorization
* Adds support for HTTP Basic Authorization username/password to the
libraries * Add TLS support to `App::ElasticSearch::Utilities::Connection`
and
`es_connect()`
----------------------------------------
version 8.3 at 2022-03-23 00:15:12 +0000
----------------------------------------
Change: 823b184da9c4238d91fc2455922510aa937491c4
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-03-22 17:15:12 +0000
Prepping 8.3 release
Change: b065c26c080ccc10d9f1860bcb033657fe3e4097
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-02-01 10:38:12 +0000
Allow wildcard and regexp matches from files
Adds the `wildcard` and `regexp` queries into the
`::QueryString::FileExpansion` plugin.
``` $ cat wildcards.dat *gmail.com *yahoo.com
$ es-search.pl email:*wildcard.dat
$ cat regexps.dat .*\.gmail\.com$ .*\.yahoo\.com$
$ es-search.pl email:~regexp.dat ```
Change: f7ea1b13fe33f941d66af30dc8b2d999d10e70b1
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2022-01-31 17:51:35 +0000
Options to disable mucking with @ARGV at import
Support `delay_argv` to prevent messing with `@ARGV` at import time.
Added import options:
* `delay_argv` - Don't muck with `@ARGV` at import time * `preprocess_argv`
- Current default, messes with `@ARGV` at import
time. * `copy_argv` - Don't mess with `@ARGV` at all, just make a copy
Disable `auto_abbrev` from `Getopt::Long`.
Change: b9da116757d0a543560797d06983972008df1f3f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-10-01 12:53:30 +0000
Use `key_as_string` instead of `key` when provided
This is most noticeable when using `date_histogram` aggregations.
Change: 04890b682bcf156edba62e231798e9705d62d9ae
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-09-24 15:54:14 +0000
Fix merging of query stash and bool sections
This is related to the earlier bug I discovered in handling array
references with ES 7.x and later.
----------------------------------------
version 8.2 at 2021-09-24 21:30:36 +0000
----------------------------------------
Change: 569436f7847833b3b3b20d15d14f753ab07c61aa
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-09-24 14:30:36 +0000
Prepping the 8.2 Release
Change: ff914b2e69d3bf4c1862ad4970e119ab9a150e12
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-09-16 15:57:37 +0000
ES 7.x fixes + bonus
* Fix `::Query->add_bool()` for stricter parser in ES 7.x * Add
`value_count` as a valid aggregation for sorting
Change: 963498bd7d45ba7480a8ef2ba82306bd98f9175c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-31 20:25:48 +0000
Add --with-missing to es-search.pl
If specified, all terms aggregations will calculate the missing element as
'MISSING'.
----------------------------------------
version 8.1 at 2021-09-01 00:43:28 +0000
----------------------------------------
Change: dc6735a8f9fcb22c3bad773b759c47d4519dc2c1
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-31 17:43:28 +0000
Setting up the 8.1 release
Change: cc2d07e3e2515d02d983dd8dd35fe3a061047a66
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-31 17:41:12 +0000
Add test for query and minimum_should_match for consistency
Change: 5257a66d11394ea8ec140c9e5de0c3e7a8a1d50c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-31 17:24:45 +0000
Fix tabs which errantly made it into the file
Change: 94b47afe216d3361d821ef6585888fed4a6c958b
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-31 17:20:53 +0000
Fix minimum should match handling in the ::Query
Change: 365a0a129d40b505bcb8ef3d72809f6e4c75ccfe
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-31 17:18:31 +0000
Update Aggregations documentation
* Adds documentation to `::Aggregations` * Adds more docs to
`es-aggregate.pl`
Change: ce7787fe56b295dc83476f58f9082ade7f0e3661
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-16 21:02:18 +0000
Starting point for aggregation documentation
----------------------------------------
version 8.0 at 2021-08-14 22:15:28 +0000
----------------------------------------
Change: 40dbb86b7b8d9b98f4572a9f362fdd94f4d57b40
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-14 15:15:28 +0000
Support OpenSearch 1.0.0
Add a check to the version lookup for opensearch. If opensearch is
discovered, set version to 7.10.
------------------------------------------
version 7.9.2 at 2021-08-14 02:17:04 +0000
------------------------------------------
Change: 8adedd9d89bdcaac26844547d36813bfc75f511f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-13 19:17:04 +0000
Fix a warning in the 7.9.1 Release
I was unable to `Dist::Zilla` to let me tag a `7.10` release, so I went
with `7.9.1`. This caused a warning in `::Connection` for formatting the
version string.
Change: 84dd4b08f113cadc0d8f71ac7bd5448536910f3f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-13 19:07:49 +0000
Version number fixed in Markdown
------------------------------------------
version 7.9.1 at 2021-08-14 01:48:01 +0000
------------------------------------------
Change: 3eef2f64709f5b9643c91544b839a1727b7f32c0
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-13 18:48:01 +0000
Fix POD for xtests
Change: bc51180e9f87730ee0d220e5c1c8b22e7e740253
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-13 18:38:33 +0000
Aggregations Updated
* Recursive aggregation parsing to flatten records into a row/column format
* Documentation updates for the
`App::ElasticSearch::Utilities::Aggregations` * Make `es-aggregate.pl` more
useful generally and move to `scripts/` * Fix alias handling for simple
terms aggs
Change: d8c8be912a60156290cb68c046e3b3c8fafbc2ae
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-04 17:26:42 +0000
Tests started to the aggregations parser
Change: 25e3fc64e388b3a84002d594e68516602970eb89
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-08-04 17:13:23 +0000
Suss out the es_expand_aggregate syntax
Now looks like:
``` [ALIAS=]AGG:FIELD[:PARAMS]
PARAMS := [k=v,k1=v1,v1a] ```
Add a few additional aggregations to the supported list.
----------------------------------------
version 7.9 at 2021-07-30 18:19:09 +0000
----------------------------------------
Change: f386e7003a442a0419fc62aa30d576312afa9efa
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-30 11:19:09 +0000
Release 7.9
* Fix timeout handling * Allow for connection preferences into the `meta`
settings section * Allow `_id` to be shown by `es-search.pl` * Look in
`~/.config` for `es-utils.yaml` * Started some work on library for handling
aggregation queries
Change: ba56db091ef15fd365100e034b17a8de530333e6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-14 15:07:21 +0000
WIP: Substantial progress on the multi level aggregation script
Change: 5c2517a1264b4a381b11387125e63dada6fcef2a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 14:50:45 +0000
Adding es-aggregate.pl script
`es-search.pl` has a lot of cruft. Creating `es-aggregate.pl` as a
proof-of-concept starting from a clean slate to see if I can make the code
more readable and more reusable. The idea being to push this code back into
the library to make working with aggregations easier and cleaner.
Change: cfd98ad2e84ee1ea8791debe7f6e1809bd6a9af6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 07:34:21 +0000
Move the aggregation parsing out to it's own sub
This is a long time coming, the handling of aggregations needs a major
cleanup. First step is to break things up into more parseable blocks.
Change: ea453e9190173af63adce15b684dcad13e3b13c5
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 07:15:14 +0000
Pass timeout to the object creation
When converting to `App::ElasticSearch::Utilities::Connection`, I
mistakenly removed the pass through of the `--timeout` parameter to the
object creation. This restores the timeout capability.
Change: c8f664cfb09b928831945f82607eab5830faa066
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 07:14:50 +0000
Allow HTTP username/password in meta
The previous commit allowed overriding the destination for the connection.
This allows users to override the HTTP basic auth username/password.
```yaml host: es-main.int.example.com meta:
access:
timestamp: t
security:
timestamp: @timestamp
host: es-security.int.example.com
proto: https
http-username: bob
password-exec: ~/bin/get-es-security-password.sh ```
Change: 7c56813974cc069a6214daef8667551a8d31de65
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-07-08 06:55:14 +0000
Added connection details to the meta section
This update allows the metadata in the `~/.es-utils.yaml` file to include
the `host`, `port`, and `proto`.
Example:
```yaml host: es-main.int.example.com port: 80 meta:
security:
host: es-security.int.example.com
port: 443
proto: https ```
Change: 797199f7af436597ffa67c486d5ea275b825e471
Author: Matthew Feinberg <mattf@intex.com>
Date : 2021-06-24 14:03:07 +0000
For es-search.pl allow _id to be shown as a column when using --show and
display _id when showing full result records.
Change: 6547b9893ad05e7cef565f4a2caf51b321091de0
Author: Andrei Grechkin <andrei.grechkin@booking.com>
Date : 2021-06-11 10:11:34 +0000
Add support for the XDG Base Directory Specification
Support configuration file placed in
`$XDG_CONFIG_HOME/es-utils/config.yaml`
Change: 0f799636f22bf8bff51c25a890f12341f6f7b9d7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-06-05 14:57:35 +0000
Fix warning in es_request()
When the index is undefined, the code throws a warning.
Change: fa084386a44bbda1a75ee961f8ea6228ab62fe0c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2021-03-20 17:17:13 +0000
The GitHub::Meta plugin stopped working and needed configuration
Change: c118b27b05d5e5e75b444b9537362d78e099176f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2020-10-08 13:42:10 +0000
Fix warnings when we encounter an unallocated shard
=================================================
Plus 85 releases after 2021-07-20 00:00:00 +0000.
=================================================