3 data migration challenges (and techniques to solve them)

Moving data from one software system to another is always a challenge, but when it comes to migrating content management data, the fact that you are moving tens to hundreds of millions of records between systems makes this type of migration particularly complex. With complexity comes problems, which usually lead to long delays in the migration process and huge headaches for you and your team.

To avoid these issues during your own migration, we’ve identified three of the most common data migration challenges you might face.

Challenge 1: Complexity of source data

Due to the huge volume of records that you have to move, you may experience various issues with your source data. It’s never as easy as moving information from content management system A to system B, especially when you consider that you will need to perform analysis and configuration to handle any complexity of source data. Here are some things that add to this complexity, and techniques for solving those challenges:

1) Data transformations: Each mainframe stores data in a certain way and if you are using an outdated data storage format you may need to transform the data if you want to store it in a modern database (like Oracle or SQL Server). To learn more about three specific areas to consider regarding your data transformations, take a look at this article.

2) Codified fields: Storing 30-40 digit request numbers in a single field may be standard practice in your current content management system, but if you are migrating to a new system, it may be useful to divide these types of coded fields in order to that they are easier to use. By doing this, you will be able to quickly see the relevant information (like sequence number, member ID, date, etc.) being scanned instead of having to look at a long number and decode this information in your head.

3) Data normalization: If you have been using the same content management system for many years, your organization may have the same data stored in several different places. Going through the process of finding all the places where a data is stored and making sure that it is only stored once – in the right place – in the new content management system is known as data normalization.

Challenge 2: Data loss or corruption

Let’s face it: All data loss or corruption can be a major problem. Your organization could be harmed if you lose even one record! Here are two strategies to avoid any unexpected issues with migrating your data that could lead to data loss or corruption:

1) Reconcile your accounts during migration and testing. You should know before the start of the data migration process how many records are imported and how many records you need to produce in the new system (keeping in mind that this is not always 1: 1, as some records from your current content management system may be duplicate). If the output does not match the expected number, you will need to do some research to find out why.

2) Use tools to help validate the migrated data. When you move data from one content management system to another, you need to make sure that the documents that you have migrated to the new system meet your expectations. For example, are states found in state fields? Are there the correct number of characters in each claim number field? There are a number of utilities and commercial services available that can help monitor the veracity of each data field.

Challenge 3: test and validate the data in depth

You rarely find an IT team that can afford testing and quality assurance, but content management migration demands it. With content management data migrations of $ 500,000 or more, the cost of failure is just too high. Here are some recommended data validation and testing techniques:

1) Consider any data events that could have impacted data quality. For example, if you remember that your content management system encountered a problem in March 2013, write it down and test all 3/13 records separately to ensure data quality.

2) Test a large volume of data for quality assurance. We recommend extracting at least 10-20% of your data to make sure you are covering a wide range. If you are moving data over decades, consider removing about 5% of each year in addition to sample data from your profile.

3) Test early and test often. As soon as you can start testing (even if it’s just a subset of the configuration and code you’ll be using to migrate), do so. Continue to run tests throughout your data migration. Testing should be an ongoing process, not something that begins after the last document has been migrated to your new content management system.

Data migration is a big business, so start thinking about these challenges and the solutions you will use to overcome them early on!

We also recommend that you involve the technical team members from the start so that they can contribute to your risk analysis. If you are capable of it, it is certainly worth teaming up with a team of experts who are familiar with data migration and who can help you with the process. Good luck!


Sean N. Ayres