Re: analysis of java application logs
Ulrich Scholz wrote:
I'm looking for an approach to the problem of analyzing application
log files.
I need to analyse Java log files from applications (i.e., not logs of
web servers). These logs contain Java exceptions, thread dumps, and
free-form log4j messages issued by log statements inserted by
programmers during development. Right now, these man-made log entries
do not have any specific format.
What I'm looking for is a tool and/or strategy that supports in lexing/
parsing, tagging, and analysing the log entries. Because there is only
little defined syntax and grammar - and because you might not know
what you are looking for - the task requires the quick issuing of
queries against the log data base. Some sort of visualization would be
nice, too.
Pointers to existing tools and approaches as well as appropriate tools/
algorithms to develop the required system would be welcome.
It helps if you have a logging strategy that mandates a consistent logging
format, specific information in particular positions or marked by particular
markup, logging levels and other such so that your analysis tool isn't faced
with a completely open-ended input. What you describe requires a general
text-analysis approach, as you indicate that you can make no guarantees about
the format. Based on that, your best tool is "less" or equivalent text-file
reader.
What is a tool supposed to do, read your mind?
It's really hard to extract information from a garbage can where people just
randomly dumped whatever they individually felt like dumping without regard
for operational needs. You can't build a skyscraper on a bad foundation, and
you can't build a good log analysis off a crappy log.
Fix the logging system, then the analysis problem will be tractable.
--
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedia/commons/c/cf/Friz.jpg