/ 
App-ElasticSearch-Utilities-8.7
River stage zero No dependents
/ es-alias-manager.pl

NAME

es-alias-manager.pl - Allow easy alias management for daily indexes

VERSION

version 8.7

SYNOPSIS

es-alias-manager.pl --local --config /etc/elasticsearch/aliases.yml

Options:

--help              print help
--manual            print full manual
--config            Location of Config File, default /etc/elasticsearch/aliases.yml
--skip              Action name to be skipped, 'add' or 'remove', default none

From App::ElasticSearch::Utilities:

--local         Use localhost as the elasticsearch host
--host          ElasticSearch host to connect to
--port          HTTP port for your cluster
--proto         Defaults to 'http', can also be 'https'
--http-username HTTP Basic Auth username
--password-exec Script to run to get the users password
--insecure      Don't verify TLS certificates
--cacert        Specify the TLS CA file
--capath        Specify the directory with TLS CAs
--cert          Specify the path to the client certificate
--key           Specify the path to the client private key file
--noop          Any operations other than GET are disabled, can be negated with --no-noop
--timeout       Timeout to ElasticSearch, default 10
--keep-proxy    Do not remove any proxy settings from %ENV
--index         Index to run commands against
--base          For daily indexes, reference only those starting with "logstash"
                 (same as --pattern logstash-* or logstash-DATE)
--pattern       Use a pattern to operate on the indexes
--days          If using a pattern or base, how many days back to go, default: 1

See also the "CONNECTION ARGUMENTS" and "INDEX SELECTION ARGUMENTS" sections from App::ElasticSearch::Utilities.

From CLI::Helpers:

--data-file         Path to a file to write lines tagged with 'data => 1'
--tags              A comma separated list of tags to display
--color             Boolean, enable/disable color, default use git settings
--verbose           Incremental, increase verbosity (Alias is -v)
--debug             Show developer output
--debug-class       Show debug messages originating from a specific package, default: main
--quiet             Show no output (for cron)
--syslog            Generate messages to syslog as well
--syslog-facility   Default "local0"
--syslog-tag        The program name, default is the script name
--syslog-debug      Enable debug messages to syslog if in use, default false
--nopaste           Use App::Nopaste to paste output to configured paste service
--nopaste-public    Defaults to false, specify to use public paste services
--nopaste-service   Comma-separated App::Nopaste service, defaults to Shadowcat

DESCRIPTION

This script assists in maintaining the aliases for relative or daily indexes across multiple datacenters.

Use with cron:

22 4 * * * es-alias-manager.pl --local --config /etc/elasticsearch/aliases.yml

This will allow you to split your cluster between datacenters (or whatever) and alias the split clusters to a homogenous index that standard LogStash/Kibana interfaces will understand.

If I create the following in /etc/elasticsearch/aliases.yml

---
pickle: ~
logstash:
  pattern: \*-logstash-{{DATE}}
  daily: logstash-{{DATE}}
  relative:
    alias: logstash-{{PERIOD}}
    periods:
      today:
        from:
          days: 0
        to:
          days: 0
      lastweek:
        from:
          days: 14
        to:
          days: 7

The pickle alias is flagged as an alias to ignore in the addition/removal process. This script will automatically ignore any alias that begins with a '.'.

Assuming today is the 2013.07.18 and I have 3 datacenters (IAD, NYC, AMS) with the following indices:

iad-logstash-2013.07.17
iad-logstash-2013.07.18
nyc-logstash-2013.07.17
nyc-logstash-2013.07.18
ams-logstash-2013.07.17
ams-logstash-2013.07.18

The following aliases would be created

logstash-2013.07.17
    |- iad-logstash-2013.07.17
    |- nyc-logstash-2013.07.17
    `- ams-logstash-2013.07.17

logstash-2013.07.18
    |- iad-logstash-2013.07.18
    |- nyc-logstash-2013.07.18
    `- ams-logstash-2013.07.18

logstash-today
    |- iad-logstash-2013.07.18
    |- nyc-logstash-2013.07.18
    `- ams-logstash-2013.07.18

This lets you use index templates and the index.routing.allocation to isolate data by datacenter or another parameter to certain nodes while allowing all the nodes to work together as cleanly as possible. This also facilitates the default expectations of Kibana to have a single index per day when you may need more.

PATTERN VARIABLES

Patterns are used to match an index to the aliases it should have. A few symbols are expanded into regular expressions. Those patterns are:

The '*' expands to match any number of any characters.
The '?' expands to match any single character.
{{DATE}} expands to match YYYY.MM.DD, YYYY-MM-DD, or YYYYMMDD

ALIAS VARIABLES

For daily indices, the following variables are available:

{{DATE}} - Expands to YYYY.MM.DD for the current day of the current index

For relative period indices, the following variable is required.

{{PERIOD}} - Name of the period

OPTIONS

config

Location of the config file, default is /etc/elasticsearch/aliases.yml

skip

Optionally skip a phase of alias management, valid phases are: add, remove

AUTHOR

Brad Lhotsky <brad@divisionbyzero.net>

COPYRIGHT AND LICENSE

This software is Copyright (c) 2023 by Brad Lhotsky.

This is free software, licensed under:

The (three-clause) BSD License