Proof of concept to analyse huge application log files using Hadoop cluster on IBM Cloud Platform

Analysing the application log files generated on production environment are very challenging. Data in the log files are in unstructured format and hence to leverage the query functionality, they can’t be stored in RDBMS/traditional database systems without conversion to structured format. Hence if an application behaves abruptly for very short duration, troubleshooting the application based on the information recorded in a large log file, probably of size hundreds of terabytes, is nearly impossible.As part of our POC development, we found that from an E-Commerce application running on Oracle Web Commerce platform (ATG), sometimes for order fulfilment asynchronous communication was not established to a third party vendor. JMS messaging protocol was responsible to delivered the order submission message from ATG third party vendor and vice versa, but periodically it was unable to do that. Using Hadoop cluster with customized Map-Reduce programming model, we extracted the exact recorded warnings and errors from log files produced from out of box ATG component. After performing the intricate analysis within the framework component, based on the analysed reports produced by Hadoop framework, we concluded that the issue was lying within the ATG framework itself. The same was communicated to the software vendor and subsequently received the patch from them.