With the amount of data in the world predicted to increase at least 50 fold between 2010 and 2020, how we store that data has come into sharp focus. Collecting large volumes of raw log data from multiple applications and infrastructure components and sending it to a central location for storage and processing, for example, increases the size and cost of storage. And as the volume of data grows and storage and processing costs increase dramatically, businesses risk undermining the advantages big data brings. Furthermore, the surging demand for data has environmental implications; by 2020, 12 percent of the world’s energy consumption will be taken by our digital ecosystem, and this is expected to grow annually at approximately 7 percent until 2030.
Data storage hindering business growth
Since the costs associated with exporting large volumes of data from the cloud to an on-premise data center are frequently prohibitive, businesses are opting to store log collected data locally in the cloud, which requires a considerable amount of space. In an attempt to reduce some of the large volumes of data held, administrators may be forced to decide which logs to erase and which to keep. However, while this approach can help to reduce storage space and costs, it is inefficient, time-consuming and prone to human error, meaning valuable and irreplaceable information can be lost from the log data set.
In addition, as log data is collected from a wide range of systems and variety of vendors, such as load balancers, other network appliances, servers, databases and service enablers, it lacks a common schema and structure, and can differ from system to system. This is further compounded by the fact that the developers of applications running on these systems decide which events to log, creating huge inconsistencies, and potentially abstracting the “bigger picture.” It is also impossible to access information in real time due to the time it takes to collect data, and with 99 percent of IT and business decision makers noticing an increasing pace of change in today’s connected world, being unable to act in real time presents a major obstacle to success.
Getting smart with data
To address these challenges, a fundamental change in approach is required.
Forward-thinking enterprises are therefore looking beyond log data, and instead are adopting a smart data approach, which distills the essence of the traffic flows, also known as wire data, that traverse their service delivery infrastructure. These traffic flows include IP packets, segments, sessions and application data streams. The intelligence that is derived from these flows at the source of the instrumentation points, and is compressed into metadata. As a result, businesses have access to the valuable information they need to gain meaningful and actionable insights, and can ensure that only the most relevant data is kept. This in turn leads to very high levels of compression, and dramatically reduces storage costs by only holding valuable information. In addition, unlike log data, smart data is normalized, organized, structured, service-contextual and available in real time. Further efficiencies are then driven by the fact that all data is processed, optimized and contextualized at the source, with some of it converted to metadata in real time. This enables data to be rapidly compressed, substantially reducing the volume of data stored by order of magnitude or more. This efficiency also enables businesses to store smart data for extended periods of time for forensic and back-in-time analysis of past incidents and events.
Another key advantage of smart data is its consistency. While log data is based on information selected by programmers and engineers with a specific perspective of their domain in mind, continuous monitoring of the wire data that traverses the key service performance indicators enables businesses to obtain full granularity. Rather than simply having a select snapshot of sampled information, they instead have complete access to data that is continuously produced based on analysis of all wire data, which is then contextualized to provide real-time, actionable insights across the entire IT infrastructure.
However, while log data storage across all systems for extended period of time may be prohibitive for the vast majority of use cases, using log data in the right context to accomplish a specific IT service, operations or business management task can be very efficient. In a case of an incident management related to service performance degradation for example, once the intelligence from smart data has identified the service level root cause based on triage of complex dependencies across the service delivery infrastructure, log data can be effectively used for analysis of the “last mile.” In this case, the log data can be pulled in the context from the system that was identified as the incident root cause to analyze system performance. The amount of log data in this instance will be manageable, and the intelligence derived from this data will complement the insight obtained from the smart data analysis. Through this approach, businesses can gain maximum visibility into their networks, and thus have much-needed insight into service delivery and business operations.
Powering businesses to advance faster
With smart data proven to reduce storage size, improve coherency and structure, and provide consistent real-time intelligence, its advantages are clear. With more than one-third of IT professionals listing “moving faster” as their top goal for 2018, such time savings will help businesses that plan to harness big data efficiently to achieve a competitive edge. When combined with the processing and storage related cost savings which enable organizations to store data for extended periods of time for forensic and back-in-time analysis, as well as the wealth of deep insights smart data can provide to service, operations and business management, we expect it to quickly become a mandatory weapon in digital enterprises’ arsenals.
This article is published as part of the IDG Contributor Network. Want to Join?