According to Cory Janssen, Data Migration scenarios are routine IT activities, and most organizations migrate data on a quarterly basis. However, there is nothing routine about Data Migration, and there are a tremendous number of issues that arise when a migration is designated as a purely “IT activity”. When 83% of data migration projects either fail or exceed their budgets and schedules, how does a responsible organization avoid or minimize the risk of being part of those statistics?
The answer is not a simple one, but a combination of different aspects that need to be factored in and managed as early as inception. There are common pitfalls that, once identified and properly addressed, will pave the road for a less painful Data Migration Project. These pitfalls include:
IT Project Syndrome: the common view that Data Migration is a Technical Problem will lead, wrongly, to a technical solution. It’s true that the extraction, transformation, load and testing are technical tasks. However, many aspects of the migration process require the input of the owners of that information. Identification of the right sources, the approach for purging unneeded data, defining transformation and business rules, deciding how to handle duplicated data, as well as accepting the test scenarios: these issues often require the input and guidance of business users.
Migrate First, Test later. More frequently than not, the testing component of a Data Migration is delayed, if not totally thrown out the window. However, by designing a data testing strategy early on, the team can better understand what the acceptance criteria will be, and more easily determine the desired outcome.
No Data Governance body. In order for a Data Migration to be successful, it is important that all data stockholders are involved and participating in all data related decisions, and the team has a clear set of procedures to follow. These are both crucial elements in project implementation, yet are often skipped in the real world of migrations. Creating a regulatory body early in the project will help reduce risks like data duplication, and will help improve data homogenization.
Lack of the right tool chain. While a Data Migration is not an IT project, it does not mean that it is an easy task; there are complex technical challenges that cannot be overlooked. In order to maintain order during a medium-sized project, the use of the right tools and technology is necessary for:
- Source Version Control (Git, SVN, TFS)
- Use of a Database as a Data repository (RDBMS, Hadoop, NoSQL)
- Centralized and shared business and data mapping rules
- Automating extractions and Loads
- Exposing transformation results to business users for review
- Report data defects (Bugzilla, Jira, Confluence)
No MDM initiative. Data Migration is an excellent occasion for implementing Master Data Management initiatives within an organization. Often Master Data is treated lackadaisically, and stored without rigor, as if the information will never change. However, in a complex environment with different source systems, it is just a matter of time for a Master Data Domain to be outdated unless there is a plan in place for it to be not only refreshed, but in sync with the information in the source systems.
No Landing Area separate from Staging and Load Area. Instead of accessing the legacy source systems Databases, decoupling the extraction process from Staging and Load can be achieved in a natural way by using a distinct Landing Area. Doing this provides a way to troubleshoot transformation bugs without having to go back to source systems.
No Data Versioning. When you’re implementing an iterative approach, don’t just delete the previous version of the system. Each iteration should be instead labeled and versioned accordingly. This way you can reference back to a previous version any time it’s needed. This approach will help to troubleshoot business rules, as well as help test past rules against new or changed rules.
A Data Migration project is an important endeavor. It deals with a crucial asset of each business: its information. Depending on the business reason behind the initiative, data migration can contribute to even bigger transformative projects, like an ERP implementation or integrating a recently acquired business. It is vital to approach it with the right mindset and discipline. If you do that, your team can routinely learn from its mistakes and avoid being part of that 83% of failed data migration projects.
