It's not all about security, it's not all about events, it's not all about compliance. All those things are critically important to IT, of course, but even more fundamental is the task of keeping things running. All those other things depend on this one. System logs reveal a wealth of information about normal (and aberrant) operations, but they don't cover everything. Worse, even without logging absolutely everything that can be logged, many data centers are drowning in operational metadata and having to develop new strategies for managing this introspective data explosion as well as the explosion of actual enterprise data. For many, logging everything is just not practical - the power to log is the power to destroy.

So, can admins add more data to the mix without creating more data? Is there hay already in the stack that they are not sifting through in their search for needles?

There is another excellent source of information about our systems that is already in our systems: the configuration files, inventory databases, and administration scripts that we use to create and manage systems in the first place. If our administrative power tools can extend both our canned (rules-based) and ad hoc (first-time or one-time) searches into that territory, then we can discover things like mistaken configuration commands without having to poll devices to see what they are set to do; and to poll them and compare intention to reality to see if something has changed that should not have. We can find out if two applications are, mistakenly, writing output to the same file, and so inadvertently interfering with each others efforts. We can see whether the inventory database says a system is retired and off the network but the NAC system log says it is still up and running. In fact, IT can use the data both proactively and reactively: to find problems before systems go into production rather than only inferring them from logged events after the fact, to confirm compliance pre-deployment as well as audit for it afterwards.

By extending the scope, the breadth of field, of search-based function for the administrator, whether canned or ad hoc, we can get the maximum amount of leverage possible out of the data we already have, rather than having to generate new streams that replicate these existing sources in log form in order to search them alongside logs. Search, unfettered and unbounded, can save the admin and the enterprise from drowning in hay.