/ 
App-ElasticSearch-Utilities-1.2
River stage zero No dependents
/ es-alias-manager.pl

NAME

es-alias-manager.pl - Allow easy alias management for daily indexes

VERSION

version 1.2

SYNOPSIS

es-alias-manager.pl --local --config /etc/elasticsearch/aliases.yml

Options:

--help              print help
--manual            print full manual
--config            Location of Config File, default /etc/elasticsearch/aliases.yml
--local             Poll localhost and use name reported by ES
--host|-H           Host to poll for statistics
--local             Assume localhost as the host
--quiet             Ideal for running on cron, only outputs errors
--verbose           Send additional messages to STDERR

DESCRIPTION

This script assists in maintaining the aliases for relative or daily indexes across multiple datacenters.

Use with cron:

22 4 * * * es-alias-manager.pl --local --config /etc/elasticsearch/aliases.yml

This will allow you to split your cluster between datacenters (or whatever) and alias the split clusters to a homogenous index that standard LogStash/Kibana interfaces will understand.

If I create the following in /etc/elasticsearch/aliases.yml

---
logstash:
  pattern: \*-logstash-{{DATE}}
  daily: logstash-{{DATE}}
  relative:
    alias: logstash-{{PERIOD}}
    periods:
      today:
        from:
          days: 0
        to:
          days: 0
      lastweek:
        from:
          days: 14
        to:
          days: 7

Assuming today is the 2013.07.18 and I have 3 datacenters (IAD, NYC, AMS) with the following indices:

IAD-logstash-2013.07.17
IAD-logstash-2013.07.18
NYC-logstash-2013.07.17
NYC-logstash-2013.07.18
AMS-logstash-2013.07.17
AMS-logstash-2013.07.18

The following aliases would be created

logstash-2013.07.17
    |- IAD-logstash-2013.07.17
    |- NYC-logstash-2013.07.17
    `- AMS-logstash-2013.07.17

logstash-2013.07.18
    |- IAD-logstash-2013.07.18
    |- NYC-logstash-2013.07.18
    `- AMS-logstash-2013.07.18

logstash-today
    |- IAD-logstash-2013.07.18
    |- NYC-logstash-2013.07.18
    `- AMS-logstash-2013.07.18

This lets you use index templates and the index.routing.allocation to isolate data by datacenter or another parameter to certain nodes while allowing all the nodes to work together as cleanly as possible. This also facilitates the default expectations of Kibana to have a single index per day when you may need more.

PATTERN VARIABLES

Patterns are used to match an index to the aliases it should have. A few symbols are expanded into regular expressions. Those patterns are:

The '*' expands to match any number of any characters.
The '?' expands to match any single character.
{{DATE}} expands to match YYYY.MM.DD, YYYY-MM-DD, or YYYYMMDD

ALIAS VARIABLES

For daily indices, the following variables are available:

{{DATE}} - Expands to YYYY.MM.DD for the current day of the current index

For relative period indices, the following variable is required.

{{PERIOD}} - Name of the period

OPTIONS

help

Print this message and exit

manual

Print this message and exit

config

Location of the config file, default is /etc/elasticsearch/aliases.yml

local

Optional, operate on localhost (if not specified, --host required)

host

Optional, the host to maintain (if not specified --local required)

verbose

Verbose stats to STDERR

AUTHOR

Brad Lhotsky <brad@divisionbyzero.net>

COPYRIGHT AND LICENSE

This software is Copyright (c) 2012 by Brad Lhotsky.

This is free software, licensed under:

The (three-clause) BSD License