A typical evolutionary path for event and log management in an organization runs like this: paleolithic admin uses just eyes and brain to review logs, looking for evidence of misbehavior, misconfiguration, and mischance; crafty neolithic admin cleverly adds scripts to the mix and automates as much of the review as possible; later, the tools come from others rather than being made by his or her own hand, but the basic premise is the same - automate the scanning that was once was manual. Functional refinement adds flourishes: the crude, slightly sharpened flint scraper (home-brewed Perl scripts) becomes a razor sharp, polished obsidian axe (Snort) becomes a manufactured steel axe (insert your favorite full blown SIEM system with alarms and canned reports). Life is good, and whenever the system catches too little, we just pour more data into it, add some rules, and see if it finds more; when it catches too much, and alarms us with unnecessary alarms, well, we try to sharpen up the axe a bit by refining the thresholds, and hope for the best. We are in the industrial age of commoditized inputs (logs) and mass-production (rules-based scans).

But still we are vexed by attacks, surprised by inconsistent and incomplete configurations. Where's an up and coming post-modern post-industrial admin to go next?

Back to the paleolithic, of course! We solve the problem in part by reintroducing a greater degree of personal involvement in the process, but without giving up the advances of modernity: using the full arsenal of aggregation, indexing, and search capabilities to empower the admin in new ways. By not just tuning them back into using their paleolithic hunting and tracking skills but by also giving them power tools with which to equip their hunting, we can give admins greater hope of getting the most leverage out of their automation by allowing it to do what it does best and helping them do better what they can do best. Instead of just throwing more log data at systems when they miss an attack or audits turn up a misconfiguration or user reports indicate some kind of unexpected system interaction, admins can dive in and search through the data streams already provided and see if there is a pattern there to discover. When hunting for a needle in a haystack, after all, making the haystack larger is not an obviously productive course; getting a tool that can assist in the hunt - a magnet, or a metal detector - makes more sense!