Four elements of a successful data migration

Data Migrations are difficult

A data migration is more than just a technical exercise to move data from one system to another. Often these projects take longer and cost more than we would like. How can we mitigate the risks inherent in a data migration and ensure that we are successful?

In this series of posts we will present the four elements that contribute to the success or failure of data migration projects.

These are 

• Data
• Methods
• Technology
• People

We can arrange these elements into a pyramid to show how they relate to each other. Technology and People form the base of the pyramid. Data sits at the top supported by Methods which ties together the Technology and people.

The Data Migration Pyramid

Data  

To be successful in our migration project we must have a very good understanding of the data involved. This means that we must explore the source and target systems to find all the objects and fields that need to be migrated and we must explore the data itself to assess its quality. Our understanding of the scope of the data and its quality will give us the basis for our estimate of the complexity and effort involved in the migration. We are also responsible for ensuring the security of the data during the migration. For this we must understand the access controls for the data in each of the locations in which it will reside. We must then also understand where the data is, where it will be processed during the migration, and the target location of the data.

We need to know what data objects and fields we are migrating. We need to have a measure of the quality of that data. We need to know the source of the data and the target location for it. We also need to understand how we will keep the data secure and comply with data protection legislation during the migration.

 

Methods

We need to understand how the migration project will be executed and this is the scope of the method element. The first aspect to consider is the contract. Should we commit to deliver the entire migration for a single price in a fixed timescale? Or should we break the project into several sections for discovery, design and implementation? Once we have decided on the structure of the project we then need to define the design artefacts that will be used for the construction of the migration process. With the design in place we move on to the build phase and so we need a pattern for how to build the migration processes. Finally for this element we need to define our testing strategy and decide how to support it within our design and implementation. The purpose of the testing will be to prove that we have not degraded the data quality of the migrated data, migrated all records correctly or if some have failed provide an audit trail showing why they have failed.

 

Technology

The technology element covers gaining a good understanding for the source and target systems. We must identify all the sources of data for the migration, these may not be evident at first from the client's requirements. Often as the discovery progresses we find that there are data that need to be pulled from other systems. We need to decide on the tools that we will use for the migration. Such as choosing between doing the migration manually using csv files, excel and the Salesforce data loader tool, or using an ETL package such as Talend with a staging database. Finally for the technology element we need to understand all the limitations of the source and target systems and our tools in relation to the data that we are migrating. We need to check that the source, target and tools can support the data volumes and timescales of our project. 

 

People

People are critical to the success of the data migration. We need to find the right people with the right skills for the project. We need to identify the Data Owner who will accept the result of the migration and the Data Subject Matter Expert who will help us to understand the data and its quality. We need to find the right project team members with expert level skills in the tools we are using and experience in migrating data into our target system. 

Summary

Data migrations can be very difficult, but if we remember the four key elements of data, methods, technology, and people we can mitigate a great deal of the risk and make our migrations successful.