Collecting Application Logs and Configurations from Customer

The Classical Problem

In most of the enterprise applications, collecting the logs from the customer systems is a nightmare. If the customer experiences and reports a bad day scenario, it is very essential to troubleshoot what went wrong. Along with the log files, it is also important to collect the application configuration files. These configuration files will differ from applications to applications.  It will help a lot to re-create the error/failure scenario in internal test systems. In a standard enterprise application, there would be many types of logs stored in different places and in different formats. 

3 Types Log Files

The commonly used logs are

• User generated events logs
• Application logs
• Server life cycle logs

Typically the size of the overall log files are very huge. It ranges from few MBs to many GBs depends on the log levels set on the application. 
 

Collecting Logs & Configuration Files

Every applications will have a different way of bundling the logs. If the logs are scattered multiple places, it will be very difficult for the customer to collect all the required logs. If the customer is not sure about what to collect, the customer support executive needs to spend considerable amount of time to explain the same. Collecting the logs from the customer systems involves three steps.

• Collection Identify the locations where the log files & configuration files are stored
• Compression (To save the bandwidth)
• Transmission

In few application domains it is not allowed to collect the customer configurations. But there would be a program to generate the dummy configuration by masking the confidential details.

Challenges in Collecting Logs & Configuration Files

Few applications will use the third party tools like 7-Zip, WinZip/WinRAR to collect and compress these files. It adds the following limitations

• Licensing Overhead
• OS dependent
• No way to control from the application
• No way to monitor these scripts

It is always better to develop the in-house log collection tools and ship with the application itself. So that we can have more control over it. But sometimes, few configuration/logs files are not accessible until the sever is stopped and this might introduce the application downtime. 

4 Ways to Send back to Home

Once the files are compressed, one of following options are mostly used.

• FTP/SFTP
• E-Mail
• Company’s Upload Portal
• Attachment in the Ticketing Tool

The first two types are the old/classical solutions and still used by start-ups & mid size organizations. The last two options are very much popular in big enterprise. Whatever approach we use, it should be easy and hassle free solution for the customer. Otherwise user & usability experience of the application will take a huge hit when a customer runs into a problem.