Unless you have spent the past couple of years living on the moon, you will be aware that big data is big news. Headlines have been screaming at us about how it is transforming businesses, and consultants are sagely warning that those who fail to get on board the big data bandwagon will be left behind. All this might have an air of hyperbole about it, but it is nevertheless absolutely true.

The vast amounts of data that are available today provide unprecedented insights into customer behaviour, sales patterns and lots more. The caveat has always been that you need to know how to unlock it, and that is where the machine learning guys modestly take a small step forward.

However, that is only part of the story. Any database developer in London will tell you that the other thing you need to be able to do is somehow store and access it all.

Big data has led to some wholesale changes in database software.

Bridging the gap

Today, data is flying in at businesses at greater speed and in higher volumes than anything we could have imagined a few years ago. It is also far more diverse, and it is part of a new data world that is beyond anything traditional relational database systems were designed to handle.

Take American-based marketing agency Harte Hanks as an example. Up until about 2013, it used a traditional database on a Microsoft SQL Server. But as time went by and big data became the watchword, the company’s systems simply couldn’t process the data quickly enough. Sean Iannuzzi is Harte Hanks’ Head of Technology and Development. He explained: “If you keep buying servers, you can only keep going so far. We wanted to make sure we had a platform that could scale outward.”

The company sought to avoid causing widespread disruption through switching to something completely new like Hadoop, the bespoke big data platform. So instead, it chose something called Splice Machine, which essentially puts a SQL database on top of Hadoop, providing existing systems with a conduit to interact with it.

This is just one example of how innovative systems can bridge the gap between the databases we have been using for years and the new big data platforms.

Big data is only one challenge

Finding a way to handle big data is only one challenge being faced by database developers. Workloads, for example, have also evolved. Go back ten years, and websites were mostly static, while today, we are all operating in live web service environments. This places new scalability demands on databases.

Also, the way companies use the data they have stored has fundamentally changed. In the past, it was all about processing transactions, analyzing sales figures and so on. Today, companies expect their data to do so much more, monitoring every action of a customer to identify patterns that might drive future strategy. Data is fundamental to modern businesses, we all know that – but the databases on which it is stored and managed need to develop stepwise.