To thrive in a digital age, businesses must look beyond log data

With the amount of data in the world predicted to increase at least 50 fold between 2010 and 2020, how we store that data has come into sharp focus. Collecting large volumes of raw log data from multiple applications and infrastructure components and sending it to a central location for storage and processing, for example, increases the size and cost of storage. And as the volume of data grows and storage and processing costs increase dramatically, businesses risk undermining the advantages big data brings. Furthermore, the surging demand for data has environmental implications; by 2020, 12 percent of the world’s energy consumption will be taken by our digital ecosystem, and this is expected to grow annually at approximately 7 percent until 2030. 

Data storage hindering business growth

Since the costs associated with exporting large volumes of data from the cloud to an on-premise data center are frequently prohibitive, businesses are opting to store log collected data locally in the cloud, which requires a considerable amount of space. In an attempt to reduce some of the large volumes of data held, administrators may be forced to decide which logs to erase and which to keep. However, while this approach can help to reduce storage space and costs, it is inefficient, time-consuming and prone to human error, meaning valuable and irreplaceable information can be lost from the log data set.

In addition, as log data is collected from a wide range of systems and variety of vendors, such as load balancers, other network appliances, servers, databases and service enablers, it lacks a common schema and structure, and can differ from system to system. This is further compounded by the fact that the developers of applications running on these systems decide which events to log, creating huge inconsistencies, and potentially abstracting the “bigger picture.” It is also impossible to access information in real time due to the time it takes to collect data, and with 99 percent of IT and business decision makers noticing an increasing pace of change in today’s connected world, being unable to act in real time presents a major obstacle to success.

Getting smart with data

To address these challenges, a fundamental change in approach is required.

Leave a Reply

Your email address will not be published. Required fields are marked *