ETL is slowing down real-time data analytics

The data transformation tool known as ETL, or extract, transfer and load, is slowing down companies’ ability to do real-time data analysis, costing those companies business opportunities and making their analytics inefficient. 

That is the result of a survey of 502 IT professionals conducted by IDC on behalf of InterSystems Corp., a high-performance database management vendor. The survey also found that Changed Data Capture (CDC) technology is also slowing companies down and impeding their ability to do real-time data analysis. 

ETL is a process that has been around since the 1970s. It is used in data transformation to prepare it for storage and analysis in a data warehouse. It’s especially popular in business intelligence, the forerunner of big data analytics. But it can be a long, CPU-intensive process—and that’s the problem. 

The study found that nearly two-thirds of data moved via ETL was at least five days old by the time it reached an analytics database, clearly useless for any real-time analytics. When it comes to CDC, which is supposed to be a real-time data replication technology, the survey found that on average, it takes 10 minutes or more to move 65 percent of CDC data into an analytics database. That’s better but still not suitable for real-time work, and it does not say how large that database is. With big data, data sets are only getting larger. 

Leave a Reply

Your email address will not be published. Required fields are marked *