There are numerous Log Viewer / Log Analyzer tools on the market, reflecting the fact that the standard log analysis functionality provided by most operating systems is insufficient to adequately address the investigation of complicated software problems.
Therefore, the command line oriented Linux / Unix operating system provides powerful standard log analysis tools such as grep and regular expressions. These are quite sufficient for single log line data analysis, for example to calculate transaction rate statistics. However, they fall short if we need, for example, to analyze the correlation between the log lines, that is, the recognition of logically linked lines. So for the example above, if we were to calculate the system response time statistics, we would first have to find the start and end line of each transaction. Even on Linux, this task requires advanced scripting, and eventually the task may be too slow to complete in a reasonable time.
Windows uses an essentially graphical user interface, but there is no standard graphical log analyzer, and standard text editors are designed for office use and only offer a simple string search. The same applies to Mac OS, although (since it is based on Linux) you also have the option of using a command line interface. The Linux environment can be simulated on Windows with the help of Cygwin software.
In our opinion, to select the most suitable log viewer for your tasks, you should consider 5 main questions.
Question 1. What is the size of the log file you need to analyze? Although in many cases software systems have some method of log rotation, log files can sometimes grow very large. Software professionals sometimes have to analyze files larger than 1GB. Especially in Windows, handling large files is not an easy task, so you have to evaluate the performance of a tool for a log file of its size.
Question 2. Do you need to analyze binary files? Most tools only support records in text format.
Question 3. Do you anticipate the need to merge multiple log files for analysis at longer time intervals? For example, to obtain more reliable statistics. Not all tools support this.
Question # 4. Would you prefer a graphical analysis tool? It is common knowledge that a picture can be worth a thousand words, but is this also true in log analysis? The answer is yes. Each log line is overloaded with information and, as we saw earlier, the number of lines in the log is often too large to easily navigate between them visually; you can’t just keep that much information in your memory when scrolling the screen. Charts add a new dimension to analysis, making it easier to see trends or just faster to find what you’re looking for.
Adding graphical functionality to a log viewer is not an easy task. A log file does not have a regular structure: generally, the log is the place where many independent processes write their messages, each one may be using its own format. Compared to such a well structured file such as a spreadsheet, we cannot just plot one column against another.
One way to solve this problem is to add the registration pattern recognition and analysis functionality, which leads us to
Question # 5. Would you need to look for record patterns and analyze them? The record pattern is a repeatable combination of keywords that can have different associated values.
Let’s take an example. Any web server log contains a series of repeated HTTP GET requests and 200 OK responses (or other code). Each request has a URI that points to a certain page on the server. These 3 objects (GET, URI and 200 OK) form a natural pattern, and their occurrence can be traced as a function of time, where the request URI and / or response code can be represented as a title for the point of the graphic.
This example shows how powerful log analysis can be when patterned log data processing is combined with a graphical presentation.