Recently I was working on data migration for sites, to migrate two bespoke sites to wordpress and an very old edirectory site to latest edirectory platform. I would like to share my experiences on it with you.
Compare to pure web page coding it is really scary to do data migration work because you have to know not only the business behind it but also the knowledge of network, operating system and database as well. I would like to say it is not an easy job.
First of all, understanding the business requirement which is helpful to let you know which data needed which does not, it can save lots of time.
Second, learn the data hard. Usually i learn the code first, how the code operate on database and how the database tables linked by which relationship, is it 1 to 1 or 1 to many etc.
Third, start to make hand dirty. Write the migration script step by step. Initially I wrote functional code, for example if migrating on articles, so what will be included in a article? article content, title, comments, images, authors, attachment, custom fields etc. By thinking and doing this way, a function called articleImport() will be created and it will have sub functions inside like imageImport(), pdfImport() etc.
Fourth, debug on migrate script. This step is quite important. for the data migration debug, some factors need be considered. first, the resource consuming of script. After migration script start running, we do not want to make the live site slower due to resource download like images, pdfs etc, so it is important to make decision on how much processes that i will use and how long it will run based on the calculation of total data and estimate the total time needed from the perspective of business. Second, is the script steady enough for long time running? in my work, a script may run 10 hours even 30 hours, and definitely i will not be there on weekends or midnight, so let it go and observe the error log to see whether is there anything to make it stopped unexpectedly. Finally, check the data. Is the data integrity? is the data correct? I usually check the new web pages, system backend and communicate with people on it to ask them whether the data they want is there.
By the above approach, I successfully migrated 3 sites which have more than ten thousands posts, images, pdf, attachments, thousands of listings and accounts information to new platforms.