Data migration is the process of moving data from one system to another. While this might seem pretty straightforward, there are complexities associated with the process, from data quality issues in the source to incompatible data models to technical challenges.

In the context of the extract/transform/load (ETL) process, any data migration will involve at least the transform and load steps. This means that extracted data needs to go through a series of functions in preparation, after which it can be loaded into a target location.

Organisations undertake data migrations for a number of reasons. They might need to overhaul an entire system, upgrade databases, establish a new data warehouse, or merge new data from an acquisition or other source. Data migration is also necessary when deploying another system that sits alongside existing applications or when replacing an old legacy system with a new one.

The term “big data” refers to data that is so large, fast or complex that it’s difficult or impossible to process using traditional methods.

There are three defining properties that can help break down the term. Dubbed the three Vs; volume, velocity, and variety, these are key to understanding how we can measure big data and just how very different ‘big data’ is to old fashioned data

Volume: Big data is about volume. Volumes of data that can reach unprecedented heights in fact. It’s estimated that 2.5 quintillion bytes of data is created each day, and as a result, there will be 40 zettabytes of data created by 2020 – which highlights an increase of 300 times from 2005. As a result, it is now not uncommon for large companies to have Terabytes – and even Petabytes – of data in storage devices and on servers. This data helps to shape the future of a company and its actions, all while tracking progress.

Velocity: Velocity essentially measures how fast the data is coming in. Some data will come in in real-time, whereas other will come in fits and starts, sent to us in batches. And as not all platforms will experience the incoming data at the same pace, it’s important not to generalise, discount, or jump to conclusions without having all the facts and figure

Variety: Data comes in all types of formats – from structured, numeric data in traditional databases to unstructured text documents, emails, videos, audios, stock ticker data and financial transactions.