Migrating case data between applications or projects

Migrating case data between applications is a process that involves a few advanced features, so read the instructions carefully and only undertake it if you feel confident in the use of the features involved.  Here are a few tips based on some experience with migration, but make sure you think out the process for yourself, as the case structure of your particular application may require different steps. 

Why would I want to migrate case data from one app to another?

• You have two applications in your project space (or separate project spaces), that are really versions of the same application, one old and one new.

• You need the old cases available to the new app for the users and/or reports to access.

But wait, why would I have two apps in the first place?

• You undertake a substantial redesign of an app

• The substantial redesign involves changing/renaming case properties or case types/structure

• This redesign take some time to build, test, and deploy, during which you are still supporting users using the old app 

Why would I not want to create a second app when I am making improvements to my app?

• You are only making small/medium changes to your app.  You can still build, test, and deploy a new version of an application while an old version is in use by managing this on the app deploy page of CommCareHQ.

• You cannot migrate Form data from one app/project space to another, so the new app will be missing any form data submitted before the switch over.

• This case migration process can be difficult to do without error, meaning that you should avoid it unless really necessary. 

Implementation considerations when migrating case data

• Ideally make sure that no users are using the old app anymore (and that all of their phones have synced!)  Otherwise, you'll have to do the process again for any outstanding cases after these conditions have been met.

• Ideally complete this process before the new app is deployed, perhaps during training for the new app, so that users will have access to all their old cases when they start to use the new app.

General steps for migrating data from one app to another

Note: These steps are not meant to be comprehensive, but rather guidelines for advanced users.  Refer to https://dimagi.atlassian.net/wiki/x/5SbKfw and https://dimagi.atlassian.net/wiki/x/TAjKfw for more details.

1. Download all of the case data from the old app using the case exporter.

2. Download a few cases (can be tests) from the new app so that you have the file format (column names) needed for the new app.

3. Transform the data from the old app into the new format

   1. If your properties have not changed, this is very easy - you don't have to do anything.

   2. If the property names have changed but no calculations are involved in the transformation, you can probably just rename headings in excel

   3. If choice values for multiple choice questions have changed, use Export Form Contents to ensure you are using the correct values as specified in the new application

   4. If your transformations are complex (ie have calculations), it is strongly recommended that you do not do the transformation ad-hoc in excel.  As with all data analysis, you want to remove yourself as much as possible from the analysis/transformations, as all humans are far more prone to mistakes than computers.  Some options:

      1. Use excel but be careful to have the raw data in one tab and use calculations referencing that raw data to create the transformed file.  (Make sure to copy-paste the values to another file for the upload in step 4, as the case-importer can't handle the calculation references).

      2. Use a scripting language outside of excel to make the transformations (you'll probably want to use csv format for the export for this).

   5. Keep the old case id as a property for the new cases (perhaps called old_case_id).  This will make it transparent in the data exports from the new app where these cases came from.  It will also make sure the data is labeled by a unique marker that can help you figure out what happened if you end up making a mistake in the migration process or if you need a unique label for lining up any case relationships (child-parent cases for instance).

   6. Copy a few of the open cases in your file and add them as extra lines assigned to a test user.  This will help you test on the app that the cases appear correctly in the new app.

4. Once you are confident in the new data file, use the case importer to upload the cases to the new app.

5. Check that the new data is correct

   1. Download the case export from the new app and do a spot comparison with the case export from the old app.

   2. Log in to the new app as the test user you added cases for in step 3e.  Fill forms against those cases and check that everything is working well in the new app.

   3. Look at any custom or report builder reports you have for this project and check that everything is looking good. 

Managing case ownership changes

If you are migrating to a new project space, or if your new application has a different case sharing design or hierarchy, you will also want to make sure that the cases you import are assigned to the right users, groups, or locations.

Please refer to Organization Data Management | Assigning Cases to Locations When Using the Excel Importer