NAME

App::ElasticSearch::Utilities - Utilities for Monitoring ElasticSearch

VERSION

version 7.6

SYNOPSIS

This library contains utilities for unified interfaces in the scripts.

This a set of utilities to make monitoring ElasticSearch clusters much simpler.

Included are:

SEARCHING:

scripts/es-search.pl - Utility to interact with LogStash style indices from the CLI

MONITORING:

scripts/es-nagios-check.pl - Monitor ES remotely or via NRPE with this script
scripts/es-graphite-dynamic.pl - Perform index maintenance on daily indexes
scripts/es-status.pl - Command line utility for ES Metrics
scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster
scripts/es-nodes.pl - View node information

MAINTENANCE:

scripts/es-daily-index-maintenance.pl - Perform index maintenance on daily indexes
scripts/es-alias-manager.pl - Manage index aliases automatically
scripts/es-open.pl - Open any closed indices matching a index parameters

MANAGEMENT:

scripts/es-copy-index.pl - Copy an index from one cluster to another
scripts/es-apply-settings.pl - Apply settings to all indexes matching a pattern
scripts/es-storage-overview.pl - View how shards/data is aligned on your cluster

DEPRECATED:

scripts/es-graphite-static.pl - Send ES Metrics to Graphite or Cacti

The App::ElasticSearch::Utilities module simply serves as a wrapper around the scripts for packaging and distribution.

FUNCTIONS

es_globals($key)

Grab the value of the global value from the es-utils.yaml files.

es_basic_auth($host)

Get the user/password combination for this host. This is called from LWP::UserAgent if it recieves a 401, so the auth condition must be satisfied.

Returns the username and password as a list.

es_pass_exec(host, username)

Called from es_basic_auth to exec a program, capture the password and return it to the caller. This allows the use of password vaults and keychains.

es_pattern

Returns a hashref of the pattern filter used to get the indexes { string => '*', re => '.*', }

es_connect

Without options, this connects to the server defined in the args. If passed an array ref, it will use that as the connection definition.

es_master([$handle])

Returns true (1) if the handle is to the the cluster master, or false (0) otherwise.

es_request([$handle],$command,{ method => 'GET', uri_param => { a => 1 } }, {})

Retrieve URL from ElasticSearch, returns a hash reference

First hash ref contains options, including:

uri_param           Query String Parameters
index               Index name
type                Index type
method              Default is GET

If the request is not successful, this function will throw a fatal exception. If you'd like to proceed you need to catch that error.

es_nodes

Returns the hash of index meta data.

es_indices_meta

Returns the hash of index meta data.

es_indices

Returns a list of active indexes matching the filter criteria specified on the command line. Can handle indices named:

logstash-YYYY.MM.DD
dcid-logstash-YYYY.MM.DD
logstash-dcid-YYYY.MM.DD
logstash-YYYY.MM.DD-dcid

Makes use of --datesep to determine where the date is.

Options include:

state: Default is 'open', can be used to find 'closed' indexes as well.
check_state: Default is 1, set to 0 to disable state checks. The combination of the default with this option and the default for state means only open indices are returned.
check_dates: Default is 1, set to 0 to disable checking index age.

es_index_strip_date( 'index-name' )

Returns the index name with the date removed.

es_index_bases( 'index-name' )

Returns an array of the possible index base names for this index

es_index_days_old( 'index-name' )

Return the number of days old this index is.

es_index_shards( 'index-name' )

Returns the number of replicas for a given index.

es_index_valid( 'index-name' )

Checks if the specified index is valid

es_index_fields('index-name')

Returns a hash reference with the following data:

key_name:
  type: field_data_type
  # If the field is nested
  nested_path: nested_path
  nested_key: nested_key

es_close_index('index-name')

Closes an index

es_open_index('index-name')

Open an index

es_delete_index('index-name')

Deletes an index

es_optimize_index('index-name')

Optimize an index to a single segment per shard

es_apply_index_settings('index-name', { settings })

Apply a HASH of settings to an index.

es_index_segments( 'index-name' )

Exposes GET /$index/_segments

Returns the segment data from the index in hashref:

es_segment_stats($index)

Return the number of shards and segments in an index as a hashref

es_index_stats( 'index-name' )

Exposes GET /$index/_stats

Returns a hashref

es_settings()

Exposes GET /_settings

Returns a hashref

es_node_stats()

Exposes GET /_nodes/stats

Returns a hashref

es_flatten_hash

Performs flattening that's compatible with Elasticsearch's flattening.

def('key')

Exposes Definitions grabbed by options parsing

es_local_index_meta(key => 'base' || 'index')

Fetch meta-data from the local config file, i.e. ~/.es-utils.yaml.

Format is:

---
meta:
  index_name:
    key: value
  index_basename:
    key: value

The most specific version is searched first, followed by the index stripped of it's date, and then on through all the bases discovered with es_index_bases().

This is used by the es-search.pl utility to do lookups of the timestamp field it needs to sort documents, i.e.:

---
meta:
  logstash:
    timestamp: '@timestamp'
  bro:
    timestamp: 'timestamp'

ARGS

From App::ElasticSearch::Utilities:

--local         Use localhost as the elasticsearch host
--host          ElasticSearch host to connect to
--port          HTTP port for your cluster
--proto         Defaults to 'http', can also be 'https'
--http-username HTTP Basic Auth username
--http-password HTTP Basic Auth password (if not specified, and --http-user is, you will be prompted)
--password-exec Script to run to get the users password
--noop          Any operations other than GET are disabled, can be negated with --no-noop
--timeout       Timeout to ElasticSearch, default 30
--keep-proxy    Do not remove any proxy settings from %ENV
--index         Index to run commands against
--base          For daily indexes, reference only those starting with "logstash"
                 (same as --pattern logstash-* or logstash-DATE)
--datesep       Date separator, default '.' also (--date-separator)
--pattern       Use a pattern to operate on the indexes
--days          If using a pattern or base, how many days back to go, default: 1

See also the "CONNECTION ARGUMENTS" and "INDEX SELECTION ARGUMENTS" sections from App::ElasticSearch::Utilities.

ARGUMENT GLOBALS

Some options may be specified in the /etc/es-utils.yaml or $HOME/.es-utils.yaml file:

---
base: logstash
days: 7
host: esproxy.example.com
port: 80
timeout: 10
proto: https
http-username: bob
password-exec: /home/bob/bin/get-es-passwd.sh

CONNECTION ARGUMENTS

Arguments for establishing a connection with the cluster. Unless specified otherwise, these options can all be set in the globals file.

local

Assume ElasticSearch is running locally, connect to localhost.

host

Use a different hostname or IP address to connect.

port

Defaults to 9200.

proto

Defaults to 'http', can also be 'https'.

http-username

If HTTP Basic Authentication is required, use this username.

See also the "HTTP Basic Authentication" section for more details

http-password

If HTTP Basic Authentication is required, use this password, **INSECURE**, set in globals, netrc, or use the password-exec option below.

password-exec

If HTTP Basic Authentication is required, run this command, passing the arguments:

<command_to_run> <es_host> <es_username>

The script expects the last line to contain the password in plaintext.

noop

Prevents any communication to the cluster from making changes to the settings or data contained therein. In short, it prevents anything but HEAD and GET requests, except POST requests to the _search endpoint.

timeout

Timeout for connections and requests, defaults to 10.

keep-proxy

By default, HTTP proxy environment variables are stripped. Use this option to keep your proxy environment variables in tact.

INDEX SELECTION ARGUMENTS

base

In an environment using monthly, weekly, daily, or hourly indexes. The base index name is everything without the date. Parsing for bases, also provides splitting and matching on segments of the index name delineated by the '-' character. If we have the following indexes:

web-dc1-YYYY.MM.DD
web-dc2-YYYY.MM.DD
logstash-dc1-YYYY.MM.DD
logstash-dc2-YYYY.MM.DD

Valid bases would be:

web
web-dc1
web-dc2
logstash
logstash-dc1
logstash-dc2
dc1
dc2

Combining that with the days option can provide a way to select many indexes at once.

days

How many days backwards you want your operation to be relevant.

datesep

Default is '.' Can be set to an empty string for no separator.

pattern

A pattern to match the indexes. Can expand the following key words and characters:

'*'    expanded to '.*'
'ANY'  expanded to '.*'
'DATE' expanded to a pattern to match a date,

The indexes are compared against this pattern.

HTTP Basic Authentication

The implementation for HTTP Basic Authentication leverages the LWP::UserAgent's underlying HTTP 401 detection and is automatic. There is no way to force basic authentication, it has to be requested by the server. If the server does request it, here's what you need to know about how usernames and passwords are resolved.

The username is selected by going through these mechanisms until one is found:

--http-username
'http-username' in /etc/es-utils.yml or ~/.es-utils.yml
Netrc element matching the hostname of the request
CLI::Helpers prompt()

Once the username has been resolved, the following mechanisms are tried in order:

--http-password
'http-password' in /etc/es-utils.yml or ~/.es-utils.yml
Netrc element matching the hostname of the request
Password executable defined by --password-exec
'password-exec' in /etc/es-utils.yml, ~/.es-utils.yml
CLI::Helpers prompt()

Password Exec

It is BAD practice to specify passwords as a command line argument, or store it in a plaintext file. There are cases where this may be necessary, but it is not recommended. The best method for securing your password is to use the password-exec option.

This option must point to an executable script. That script will be passed two arguments, the hostname and the username for the request. It expects the password printed to STDOUT as the last line of output. Here's an example password-exec setup using Apple Keychain:

#!/bin/sh

HOSTNAME=$1;
USERNAME=$2;

/usr/bin/security find-generic-password -w -a "$USERNAME" -s "$HOSTNAME"

If we save this to "$HOME/bin/get-passwd.sh" we can execute a script like this:

$ es-search.pl --http-username bob --password-exec $HOME/bin/get-passwd.sh \
                --base secure-data --fields

Though it's probably best to set this in your ~/.es-utils.yml file:

---
host: secured-cluster.example.org
port: 443
proto: https
http-username: bob
password-exec: /home/bob/bin/get-passwd.sh

CLI::Helpers and Password Prompting

If all the fails to yield a password, the last resort is to use CLI::Helpers::prompt() to ask the user for their password. If the user is using version 1.1 or higher of CLI::Helpers, this call will turn off echo and readline magic for the password prompt.

INSTALL

This library attempts to provide scripts compatible with version 0.19 through 1.1 of ElasticSearch.

Recommended install with CPAN Minus:

cpanm App::ElasticSearch::Utilities

You can also use CPAN:

cpan App::ElasticSearch::Utilities

Or if you'd prefer to manually install:

export RELEASE=<CurrentRelease>

wget "https://github.com/reyjrar/es-utils/blob/master/releases/App-ElasticSearch-Utilities-$RELEASE.tar.gz?raw=true" -O es-utils.tgz

tar -zxvf es-utils.tgz

cd App-ElasticSearch-Utilities-$RELEASE

perl Makefile.PL

make

make install

This will take care of ensuring all the dependencies are satisfied and will install the scripts into the same directory as your Perl executable.

USAGE

The tools are all wrapped in their own documentation, please see:

$UTILITY --help
$UTILITY --manual

For individual options and capabilities

PATTERNS

Patterns are used to match an index to the aliases it should have. A few symbols are expanded into regular expressions. Those patterns are:

*       expands to match any number of any characters.
DATE    expands to match YYYY.MM.DD, YYYY-MM-DD, or YYYYMMDD
ANY     expands to match any number of any characters.

AUTHOR

Brad Lhotsky <brad@divisionbyzero.net>

CONTRIBUTORS

Alexey Shatlovsky <alexey.shatlovsky@booking.com>
Vitaly Shupak <vitaly.shupak@deshaw.com>
Alexey Surikov <ksurent@gmail.com>
Daniel Ostermeier <daniel.ostermeier@gmail.com>
Jason Rojas <jason.rojas@mgo.com>
Kang-min Liu <gugod@gugod.org>
Lisa Hare <lhare@inview.co.uk>
Markus Linnala <Markus.Linnala@cybercom.com>
Mohammad S Anwar <mohammad.anwar@yahoo.com>
Samit Badle <Samit.Badle@gmail.com>
Takumi Sakamoto <takumi.saka@gmail.com>

SUPPORT

Websites

The following websites have more information about this module, and may be of help to you. As always, in addition to those websites please use your favorite search engine to discover more resources.

MetaCPAN

A modern, open-source CPAN search engine, useful to view POD in HTML format.

https://metacpan.org/release/App-ElasticSearch-Utilities
CPAN Testers

The CPAN Testers is a network of smoke testers who run automated tests on uploaded CPAN distributions.

http://www.cpantesters.org/distro/A/App-ElasticSearch-Utilities
CPAN Testers Matrix

The CPAN Testers Matrix is a website that provides a visual overview of the test results for a distribution on various Perls/platforms.

http://matrix.cpantesters.org/?dist=App-ElasticSearch-Utilities

Bugs / Feature Requests

This module uses the GitHub Issue Tracker: https://github.com/reyjrar/es-utils/issues

Source Code

This module's source code is available by visiting: https://github.com/reyjrar/es-utils

COPYRIGHT AND LICENSE

This is free software, licensed under:

The (three-clause) BSD License

To install App::ElasticSearch::Utilities, copy and paste the appropriate command in to your terminal.

cpanm

cpanm App::ElasticSearch::Utilities

CPAN shell

perl -MCPAN -e shell
install App::ElasticSearch::Utilities

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

VERSION

SYNOPSIS

FUNCTIONS

es_globals($key)

es_basic_auth($host)

es_pass_exec(host, username)

es_pattern

es_connect

es_master([$handle])

es_request([$handle],$command,{ method => 'GET', uri_param => { a => 1 } }, {})

es_nodes

es_indices_meta

es_indices

es_index_strip_date( 'index-name' )

es_index_bases( 'index-name' )

es_index_days_old( 'index-name' )

es_index_shards( 'index-name' )

es_index_valid( 'index-name' )

es_index_fields('index-name')

es_close_index('index-name')

es_open_index('index-name')

es_delete_index('index-name')

es_optimize_index('index-name')

es_apply_index_settings('index-name', { settings })

es_index_segments( 'index-name' )

es_segment_stats($index)

es_index_stats( 'index-name' )

es_settings()

es_node_stats()

es_flatten_hash

def('key')

es_local_index_meta(key => 'base' || 'index')

ARGS

ARGUMENT GLOBALS

CONNECTION ARGUMENTS

INDEX SELECTION ARGUMENTS

HTTP Basic Authentication

Password Exec

CLI::Helpers and Password Prompting

INSTALL

USAGE

PATTERNS

AUTHOR

CONTRIBUTORS

SUPPORT

Websites

Bugs / Feature Requests

Source Code

COPYRIGHT AND LICENSE

Module Install Instructions