==================================================
Changes from 2015-12-05 00:00:00 +0000 to present.
==================================================
------------------------------------------
version 0.006 at 2017-12-04 00:18:54 +0000
------------------------------------------
Change: b1231e5a3f970d07f07ec4e00cc4746634293c09
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-12-04 01:18:50 +0000
Release 0.006 with the indexers fixed and the iptables context.
Change: a121f4f64caa7b60bbbe9da6b422fb7e33991799
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-12-04 01:14:28 +0000
Fix the eris-es-indexer.pl
The config was pointing to the wrong depth in the hash. Ensure when the
config is passed from the commandline, eris::schemas are instantiated
correctly. Fix the mapping for the geo_point field in the geoip mapping.
Add the eris::log::context::iptables to parse iptables logs into the
indexes.
------------------------------------------
version 0.005 at 2017-12-03 23:10:58 +0000
------------------------------------------
Change: a4aa7aa00b3bb87e3a11f1e40fc1bd2d1d949688
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-12-04 00:10:58 +0000
Release version 0.005
Update documentaiton to demonstrate enabling the debug dictionary in the
syslog schema.
Change: 537771e0cbd49dea95f9f2d8358cfa7275d5a260
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-11-24 11:15:46 +0000
Correct the shebang line for install
Change: b2090fc15bdee9533c4732afbbc73c74114051cf
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-11-24 09:34:31 +0000
Use auto-detection of MinimumPerl
I incorrectly set the minimum Perl version. Removing this hard-coded config
detects the correct minimum Perl version.
------------------------------------------
version 0.004 at 2017-11-16 17:43:11 +0000
------------------------------------------
Change: b7737fd29826f94cfab9f91d02bfb142ff7a9dac
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-11-16 18:43:11 +0000
Remove App::ElasticSearch::Utilites from the prereqs and fix some bugs with
the indexer.
Change: ed5eb795a6a7beb21e6ce36c66b527e5b727fb40
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2017-11-16 09:56:57 +0000
Remove the eris::dictionary global singleton
It makes more sense to allow the schema to define it's own dictionary.
Dictionaries can now be configured per-schema allowing them to be as
configurable as necessary. Allow hash flattening of the documents and
enable that option in eris-context.pl.
Change: eea981c6695f603eccde4865581b84a689a877c5
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-11-05 10:33:28 +0000
Regenerated README
Change: 6c9167d2ed6e6f2b6b655b9a2137482791c418b2
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-11-04 16:20:31 +0000
Catch documentation up to the current state of affairs.
Change: 0d15fa7fcf4b0fad7d8f83904ce5c1edcc47dc9a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-11-04 15:53:56 +0000
Removing the type library since I'm not using it anymore
Change: 773f3a247bb14dde25e4f393e5ac6dfbf7c132e6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-11-04 15:53:22 +0000
Added all POD required for author tests to pass
Add version tags in the modules where they were missing Add abstracts
everywhere they were missing Ensure all the final POD elements were closed
Change: 55293219e715ac9668c09c25afe80fa901ae917c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-11-04 13:04:27 +0000
Fix Perl::Critic and POD syntax errors.
Change: 129c0539a53e7f28f44e506c412c231a053fb76c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-29 08:09:15 +0000
Fix parse and pod errors.
Change: ddfe01b50fc4e2893924feb3d036ec7559059b9c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-29 07:46:18 +0000
Started documenting the overall project goals and design
Change: 128d6c3fc0ec36504055fcbfaa379012e0e018d4
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-28 11:54:01 +0000
Documentation added to more classes.
Change: 570dfb821e79bbffd202520ead08731837e28daf
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-28 10:47:53 +0000
Documentation added to roles
* Technical debt being collected * Migrate the '_build_name' method all the
way back to
eris::role::plugin.
Push down smarter logic for automatically determining the name of a
plugin. Require a namespace parameter for eris::role::plugin that's
automatically passed from the eris::role::pluggable consumer from it's
required parameter. This makes naming the consumers easier and smarter.
Change: 21799c506625b11dcd27696297fda40de866d865
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-27 17:36:48 +0000
Allow schemas to choose not to be final.
This means a single log entry can be interpretted by more than one schema.
This might be useful for storing events in a large short term index, but
particular events in a longer term index.
Change: fdf10a5c1ad228c6555e57d858e97089baecc45c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-26 23:04:28 +0000
Working out the kinks in cleaning up the separation of eris::log and the
underlying elasticsearch schemas.
Change: e24a07509590e27cdfa8e228bdd3a48b2e0f284a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-26 19:14:51 +0000
Separation of the schema and parsing done!
Change: 27201c0df79a737549e701596e710d7797521c1e
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-20 21:14:58 +0000
Reworking the system
* Contextualizing and Storage separated so you can apply different
storage rules to the same message * Working out how to mimic the ES
mappings
Change: 06deb277d97b92779b1539ec7a851242eedbdd73
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-08-15 20:19:02 +0000
Store the raw message
* Add the raw context and update the elasticsearch mappings to store the
raw data without indexing it. * Anchor dhcpd parser * Fix protocol
extraction in pfsense::filterlog
Change: bc4da89bcb71189294d75cf95cfedc1ea0ec2eb8
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-07-14 23:32:17 +0000
Add static context to add k/v pairs to every log event.
Add a special "double star" matcher to match every event. Add a
"SuppressWarnings" variable to the contexts. If a context wishes to be
silently ignored, it can set "our $SuppressWarnings".
Use both of these features in the static context. The advantage is one less
subroutine dispatch if the static context isn't configured.
Change: 867feb6940ef4d11b275bcd6e56acf63296d3558
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-07-14 18:22:46 +0000
Overhaul of the reference implementations.
* eris-eris-client.pl - Added options to control the flow of information.
Added graphite output for statistics reporting. Currently only
"dispatched" statistics work. Reads the config file for a client
section, which is then passed to the POE::Component::Client::eris
constructor, e.g.:
client:
Subscribe: [ "sshd", "sudo", "kernel" ]
Match: [ "error", "failed", "failure" ]
* eris-es-indexer.pl - Added support for ES versioning via the
--es-version option. Defaults to '5'. Supports the following options
via the config file:
es_addr: a host in the cluster to index to
es_default_type: Type to index message
es_default_index: 'syslog' Index to write the message to
TODO: eris-es-indexer.pl should autodetect the version of the cluster and
apply the appropriate mapping.
es_addr should accept an array
Change: 385513cd4c1e713dd2565d693832960110f4ada2
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-07-06 07:38:26 +0000
More cleanup, record pid and program sub in a CEE Compatible way.
Change: 93fa0a2b5926c7d1c697f90cdf0149e26c7dff9c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-07-05 07:37:51 +0000
Fix up tags and streamline context calls.
Change: abb30c813e3589349dc66f8d89058fb08ab50471
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-07-05 06:54:45 +0000
Fix protocol handling
Normalize protocols to lowercase. Use 'proto_app' instead of 'proto' per
CEE. Add 'service' of 'firewall' to the logs.
Change: fe5849d08fd174325074fb84bfd0db4791da3ba6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-07-05 06:30:33 +0000
Added parsing for pfSense's CSV filterlog.
Changes to the eris-context.pl tool to accomodate mixed case names.
pfSense::filterlog can parse out IPv(4|6) and TCP/UDP meta-data
Change: 9d845334e6c321fad188936fa1ba8228e8b8da49
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-07-05 05:28:58 +0000
Added dhcpd parser.
Fixed up older contexts to take advantage of better logic. Added help
option to the eris-context script
------------------------------------------
version 0.003 at 2017-03-01 06:24:50 +0000
------------------------------------------
Change: acee6050a6cae7fee3db0b8f962404302c6a8c1a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-03-01 07:24:50 +0000
Add ability to swap index basenames and types via contexts.
This patch allows a context to set a different index base name or type.
This will be sent along to the cluster in the bulk api end point.
Change: 63c8d264be2e86678ecbecf1262cfa53c7afde6f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-02-02 20:54:29 +0000
Update location of GeoIP2 mmdb, rename 'desc' to 'name' for clarity.
Change: 2579cad2f0a63c7a6ba4578b6844769085d822ef
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-02-02 09:26:43 +0000
eris::log->add_tags() created to allow contexts to tag messages, total_time
added to allow easy searching.
Change: 1307cf25da94f1b47ce7afac0f548a78f0a9de7d
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-02-02 08:40:51 +0000
Added suricata/snort log decoding to the library.
Change: e18c8d66f30ec0cf1bbbdfe708ffa87a58757fb6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-12-16 07:47:54 +0000
Bump release in the docs.
------------------------------------------
version 0.002 at 2016-12-16 06:45:24 +0000
------------------------------------------
Change: 503862bb34803e756ed43f25d4b887a7fcc9d039
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-12-16 07:45:24 +0000
Fix race condition in the scheduling that was preventing bulk writes from
occurring.
Change: 05590cf74a82b9d5ae36af97e7a676c9cea7c60c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-12-16 07:01:14 +0000
Postfix Context: Make sure the K/V pairs extract before trying to use them
in the log context
Change: d4fb6699edf3ad40b36411413bcf4ec6547a8ee3
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-29 01:36:59 +0000
Update ignore list.
Change: 47ab113720483941f3e2c617e00617a1fe98a8d8
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-28 20:39:17 +0000
Cron support on FreeBSD, Invalid user tracking on sshd.
Change: be5ead9c3f26270d43d3313370678a6e1dc9532b
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-26 03:19:14 +0000
CROND and postfix parsers added, dictionary fixes.
Change: 936fd1c3ae01e64843c5ff1b615c8fcfea171a9f
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-23 21:38:10 +0000
Conversion to Moo and Type::Tiny
I'm not utilizing the full Moosey stack, so I converted to Moo. If I need
to use Moose for something later, the Type::Tiny conversion will make that
trivial.
Restructured the contexts, decoders, and dictionaries to use the
eris::role::plugin to remove duplicate code.
---------------------------------------------
version 0.001_01 at 2016-11-20 04:14:44 +0000
---------------------------------------------
Change: 4101a001d306c9780e0f9884c126472fc0c5268a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-20 05:14:44 +0000
Ignore the dzil distribution builds.
Change: 30f6b501ec302af320b1b65744421ed465c98807
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-20 05:11:19 +0000
Added rudimentary README to the dist.
Change: 9169cc3302c39d40b86aadaf46cd443225283e58
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-20 05:10:35 +0000
Append newlines to the STDERR log messages.
Change: 7f4526adf6caea02a9535d97a094fd065bf387b1
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-20 05:00:21 +0000
Automatically manage index templates
This patch adds a simple template checking in to the worker startup.
Change: a07d8b9fdfe20e4af22803a25f48315a0f051b99
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-20 03:38:55 +0000
Prerelease alpha
The eris-eris-client.pl can work with a POE::Component::Server::eris server
to connect and stream the logs to an ElasticSearch cluster. Stats are
bubbled up from the workers to the server, and documents land in the index!
Change: b4f4bc012cf0d2cb71376fce5aacf60a1a0fd5f0
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-19 19:09:58 +0000
Output basic stats for testing purposes.
Change: b4ceb83d54897b2ca949a2b13f1106ddf05958fc
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-19 19:06:06 +0000
Allow worker pools to be a configurable size.
Change: a7f8a9663379711fa7b2813335e700ac982c6760
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-19 18:59:56 +0000
Added eris-eris-client.pl
Uses POE::Component::Client::eris to connect to a local syslog server as
it's source.
Change: ef7b6799ce619d9f490c0fc329144e30b17c9b14
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-19 18:24:43 +0000
Prerelease Candidate 1
dzil configs updated to reflect my standard dzil layout. Renaming the
scripts in the bin directory to have clearer, more appropriate names and
avoid collisions. Restructure the timing data so it can be easily indexed
in ElasticSearch. Convert the config file to an optional parameter so the
utilities run cleanly from the the dist.
Change: b0554bc33417834348579abedecb7ea202a63722
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-11-19 17:29:36 +0000
Remove static license file.
Change: 725576bc10475a79cf33017a197c0a56c60b5fcb
Author: Brad Lhotsky <brad.lhotsky@gmail.com>
Date : 2016-09-21 22:28:38 +0000
Delete postfix.pm
This was just a placeholder, removing to remove confusion.
Change: ce5b8590f3e645374af835df96a86b437d9898f7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Catch errors and remove the input streams from the system. Bubble stats up
to the main process for handling.
Change: ecf1e1deb89f853e8cb92d355b14975da6b785e4
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Getting close to a releasable package.
Change: 3190609932c28ae1a828a9519000b5e2443adafa
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
more stuff
Change: bd85f8eff29ff4969875268e738816e6f3fa7c91
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Allow context/complete objects to be duplicated as in the case of the GeoIP
context.
Change: acb23a094a573a1d22b8697a4b6d571e135dbc07
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
GeoIP Context Established
Change: 815920085af300f40e6406620e759587eb5604e3
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Pass plugin configs down to the context objects
Change: b6d6c0831413c34a51b856db09923586980fd767
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Add timing information to contextualize script.
Change: 0cf3b9ba37b8e59faaedae37a73b2aa7939cd4ce
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Added sudo and yum log parsers.
Change: 58c41cc831df9eea401fdf2b35473f054670e08b
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Added sudoers parsing and dictionary entries for src/dst usernames.
Change: 4788e0dacf903066deb68f018e3fe90d5a0c1eb2
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
pause.
Change: 70f28672cc14b2fcdafb2e57ca344e0ba85a66d8
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Added an empty postfix context and fixed output from the library functions.
Change: c08994a3cffeda013ede875892358418c0952ee7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Adding timing element to log object.
Change: 25c4a3f9d29775cba801c82d71ecc4dc1521efaa
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Added basic context establishment Added dictionary lookups for fields Fixes
to the decoders to behave
Change: 338bea1dad1d28ef2e7d66ec3d242beff1e9d2f7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Simplify the contexts by providing a SubType to coerce things into the
eris::log::context->target() code ref. It will return true if the context
matches the field / target pair, or false otherwise.
Change: 89279c0b767432a17b1b0dd5a9c53f43eb62d41b
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Typos, syntax, and logic fix ups. Included a demo script to use to test the
whole eris:* name space. Decoders and object creation are working.
Change: dac379507a39fc7e6f4e530913da04a6a0916f82
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
Ignore the main config file.
Change: c307f13774d6829af2e51ce221f07e28db49d7b6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2016-09-21 22:20:22 +0000
initial revision
Change: 8a3d2b7f64ba059fcb3cd2cd12f12bad620c61b7
Author: Brad Lhotsky <brad.lhotsky@gmail.com>
Date : 2016-09-21 22:19:43 +0000
Initial commit
================
End of releases.
================