Rebuilding Project Space Components Between CommCare Clouds
In some cases, a project may be interested in moving core elements of one project space to another. Before undertaking such a large task, users are encouraged to check whether this process is a good fit for their requirements. Step by step instructions are provided for the elements that can be recreated in a new project space.
Overview
When signing up for CommCare, you can select the cloud location that best fits your organization's needs. You can see a full list of cloud locations here: CommCare Cloud Hosting Locations Once created, a project space cannot be transferred from one hosting location to another.
CommCare does not support directly transferring a project space to a new cloud. However, programs may be able to achieve similar outcomes by manually recreating parts of the original project space in a new one on a different cloud location. This involves exporting and importing core components, such as applications, case types, and data structures as well as importing active case data.
While this process is technically feasible, it is complex and labor-intensive. Programs should be cautious and only pursue this approach if they have the technical capacity to manage the risks and ensure data integrity throughout moving components.
Please note: Form data cannot be migrated between project spaces.
There isn’t a one stop solution, but you can move components from one project space to another in CommCare. This page details how.
What Can I Move Between Project Spaces?
The table below gives a high-level overview of what CommCare components you can or acnnot move between project spaces. These are described in greater detail below.
Ability to Move | Component |
|---|---|
Can’t Move |
|
Export / Import |
|
Manual Recreation |
|
Is This Process Right For Me?
The first question to ask is “Is this process worth the effort?” If you can avoid this process or create a new project space, we strongly recommend doing so. It is usually much easier and more reliable to open up a new project space when you begin a new project. This is especially true for smaller or newer projects, where starting fresh is often the better option. If you are seriously considering moving components over to a new project space, look at the instructions below and try to estimate the amount of time and coordination it would take for your team to make this move. That may help you weigh the cost / benefit of whether this process is worth the effort.
If you do decide to move components--whether to change cloud locations or merge project spaces-- make sure you’re prepared for the effort involved. This process requires manual work, technical skill, and careful coordination. While it preserves live case data, it does not include historical form submissions. You’ll need to manually set up components, run multiple test migrations to ensure data accuracy, rebuild any form-based reports, reconfigure integrations, and clearly communicate with field teams so they can reinstall apps and follow the new timelines. All of this has to be done in careful sequence, so if you have many team members lending a hand, they will need to work together closely to ensure each step happens in the right order.
This process may work for you if… | This process may not work for you if.. |
|---|---|
|
|
Step-by-Step Instructions
This guide provides step-by-step instructions to help organizations easily transition individual components of their CommCare project space from one cloud location to another, on their own.
Getting Set Up
In the following instructions, we’ll be referring to both the “Source” project space and the “Destination” project space. The source project space is the project space that you are moving away from-- the old space you are trying to shut down. The destination proejct space is the new project space you are setting up, where you want your users to start working.
In order to get ready to move components from one project space to another, make sure you take the following steps:
Sign into your account and open the source project space.
Sign into your account and open the destination project space.
Prepare a secure, local folder to save data files from CommCare during the process.
Below is a summary of what you’ll be able to transition.
First, Understand What You Can’t Transition
Forms (Raw Submissions)
Individual form submissions (XML) cannot be recreated in the new project space. Form history and metadata will remain on the original cloud location. If needed, you can optionally extract form data via the API or through form exports, but this is suitable only for archiving or reporting — it cannot be reloaded into the new project space.
Historical SMS
Historical SMS messages cannot be recreated in the new project space. If required, you can extract SMS history via the API for archival or reporting purposes only. SMS interactions will remain linked to the original cloud location and will not carry over to the new project space.
Elements You Can Import and Export Between Project Spaces
Applications
If you are copying an application between two project spaces on the same cloud, follow the instructions to Copy or Delete an Application .
If you are copying an application between two different cloud locations, follow the instructions below to copy applications from the source space to the destination space.
Copy or Delete an Application | Copy an Application to a Different Server
Users, Locations and Groups
Mobile users, locations and case sharing groups are interlinked in CommCare and therefore should be moved in tandem. This is divided into two sections - data preparation and data upload.
Data Preparation
1. Instructions for Mobile Users
Use the Bulk Download Mobile Workers feature to download all mobile users from the source space. Repeat the same process in the destination space to get a blank template.
Bulk Download Mobile Workers
The filter "Filter and Download Mobile Workers" option allows for the downloading of mobile workers using a select set of filter options. The below steps can be taken to download mobile workers.
Click on the "Download Mobile Workers" button on the mobile workers page.
You will be redirected to a page with filter options.
You have the option to download all details of a Mobile Workers by clicking on "All" next to "Columns" or just the usernames by choosing "Only Usernames".
Then click on the "Download" button once satisfied with your filter options.
Copy paste the exact names in the destination space file from the source space file downloaded above.
Ensure that the user_id column is blank in the destination space file as CommCare will automatically assign new IDs to these entities.
2. Instructions for Organization (Locations)
Replicate the exact organization structure from the source space in the destination space. See Setting Up Organization Levels & Structure | Step 1: Set Your Organization Levels for step wise instructions. You will need to rebuild the entire structure (including all of the same settings) for every level in your organizational hierarchy.
Download the organization structure from the source space and remove all location_ids from the Excel file. Remove rows for any locations that are no longer active. Save all changes to the Excel file.
Ensure that the location_id column is blank in the destination space file as CommCare will automatically assign new IDs to these entities.
3. Instructions for Case Sharing Groups
Create case sharing groups in the destination space with the same values as in the source space. See Mobile Worker Groups | Create Groups for step wise instructions.
Note that case sharing groups cannot be downloaded separately in CommCare. They are mapped in the mobile workers excel file.
Data Upload
In the destination space, take the Excel file you just cleaned, and conduct a bulk upload. See Setting Up Organization Levels & Structure | Bulk Upload your Organization Structure for information on bulk uploads. This will create all of the locations in your new project space.
Upload the excel file prepared for the destination space for mobile users in the destination space. See Create and Manage CommCare Mobile Workers | Use Bulk Upload to create multiple mobile workers at once
Cross-check the mapping of mobile workers to their respective locations in the source space and replicate for all mobile workers in the destination space.
Lookup Tables
Download existing lookup tables from the source space. See Lookup Tables | Downloading Table for step wise instructions.
Duplicate the excel file and save as lookup table files for the destination space.
Ensure that the UID column is blank in the destination space file as CommCare will automatically assign new IDs to these entities.
Upload lookup tables in the destination space. See Lookup Tables | Upload Table for step wise instructions.
Importing Case Data
Why Import Case Data?
When rebuilding project components in a new project space, the main reason to import case data is so that your mobile users can seamlessly continue working with their same set of cases. Once you have imported all of the case data correctly, this means that your mobile users can login to their new application in the destination project space, sync, and see all of the data that they had on their previous project space. They can continue collecting data as they would have before, with almost no interruption to their day.
General Process
When rebuilding or importing data into the destination project space, you need to export all of your case data from the source space, and you need to import it into your destination space. When you import it into the new system, you need to update the case’s owner to ensure it gets assigned to its new owner in the destination space. The case owner will either be a mobile user, a location, or a case-sharing group.
In addition, we recommend saving the old case id of the preivous case, and the old owner id. This allows you to easily look at data from your source space and destination space and link records to each other (particularly if you are putting both set of data into a database).
Methods for Exporting and Importing
There are several ways that you can go about export and import the data from your source space into your destination space. Depending on the amount of case types that you need to bring over from your source space, there are two different methods you can use for achieving this.
Simple method: The first method involves simple vlookups, and is appropriate if 1) you are migrating a small number of case types (for instance, 1-3 case types), and 2) if all cases are owned by the same type of entity (e.g. they are all owned by mobile users, or all owned by locations).
Advanced method: The second method involves creating a main mapping file, and saving several different files within one folder. By utilizing Excel formulas, you are then able to map the correct case owners from all of your location, mobile user, and case sharing files into your case data files, preparing them to import. This method is much more complex, but works well if you have either several different case types to import to the destination (4+), or if you have different entities that own cases (e.g. one case type might be owned by mobile users, while another is owned by locations). In this situation, using the more advanced mapping file will save you time in the long run.
Simple Method
The instructions below outline the process for exporting and importing Case Data. We run through two different scenarios:
Cases owned by mobile workers or case sharing groups
Cases owned by locations
For more information on exporting cases in CommCare, please see Case Data Export.
Cases Owned by Mobile Workers or Case Sharing Groups
Note: this method assumes that your mobile workers and case sharing groups have the exact same username / group name in the new destination project space as they had in the old source space.
Export Data and Clean it for Import
Download your case data from the source project space. Make sure to include the field owner_name in your case data export.
Delete all columns containting metadata.
Rename the case_id column old_caseid.
Copy and paste the owner_name column, and in the new, pasted column, rename it old_owner_name.
Import Data into Destination Project Space
Navigate to Import Cases via Excel in the new project space and import cases.
When you import the cases, the fields old_case_id and old_owner_name will show as new properties. That is fine-- you want to create these properties (see screenshot below).
This is what the case import mapping screen would look like - where the old_case_id and old_owner_name columns are being created and the owner_name column is mapping to the new project space owners in the excel file.
Important: Make sure the correct and corresponding case type is chosen and the excel column named caseid is chosen for mapping. Do not forget to select “Create new records if there is no matching case”. This will ensure that CommCare creates new cases when the file is imported and maps them to the case type selected.
Note: this method will only work if the username of your mobile users is unique, and no other location or case sharing group has the same name. If that is not true, follow instructions to do a vlookup from the mobile user name to the commcare-user caseid for the mobile users in your destintation space.
Cases Owned by Locations
Export Case Data and Clean it for Import
Download your case data from the source project space. Make sure to include the field owner_id in your case data export.
Delete all columns containting metadata.
Rename the case_id column old_caseid.
Copy and paste the owner_id column, and in the new, pasted column, rename it old_owner_name.
Map Old and New Locations
Download your entire Organizational Structure in the source project space.
[If your cases are only owned by one level in your organization structure, skip this step] In a new tab of your Organizations file, copy and paste all of the location_ids and site_codes from all your different location types into one sheet.
In your case data file, add a column that is called location_site_code. Do a vlookup to pull in the location site_code based on the owner_id column listed. The formula for the vlookup is:
=VLOOKUP(AT2,'[cs-sandbox_locations (4).xlsx]all_locations'!$A:$B,2,FALSE)
the field with owner_id in it
name of the source project space organization structure file
name of tab with all locations in itDownloading and cleaning the export file, and bringing in site_codeNow Download the Organizational Structure from the destination project space.
Go through the same step of creating a tab with All Locations in it. Once you have done that, switch the location_id and site_code columns so that the site_code is in column A. If you skip this step, your vlookup will not work.
Go back to your case data file. Add a column called destination_owner_id. Do a vlookup to pull in the owner_id of the location from the destination project space based on the site_code.
=VLOOKUP(AU2,'[menstrual-hygiene_locations.xlsx]all locations'!$A:$B,2,FALSE)
the field with owner_id in it
name of the destination project space organization structure file
name of tab with all locations in itRename that column owner_id, and remove the old column with the site_code. This file is now ready to be uploaded.
Import Data into Destination Project Space
Navigate to Import Cases via Excel in the new project space and import cases.
When you import the cases, the fields old_case_id and old_owner_id will show as new properties. That is fine-- you want to create these properties (see screenshot below).
This is what the case import mapping screen would look like - where the old_case_id and old_owner_id columns are being created and the owner_id column is mapping to the new project space’s id in the Excel file.
Important: Make sure the correct and corresponding case type is chosen and the excel column named caseid is chosen for mapping. Do not forget to select “Create new records if there is no matching case”. This will ensure that CommCare creates new cases when the file is imported and maps them to the case type selected.
Advanced Method
The process for exporting and importing Case Data is quite extensive. Please click the instructions below to access this process.
See the video below for stepwise instructions on the Advanced method of case data import and export.
For more information on exporting cases in CommCare, please see Case Data Export.
Mapping IDs
See the video below for a detailed walkthrough.
Download Users and Locations for both spaces and save them in the same folder path. Create a new xlsx file called mapping_ids and save it in the same folder path as well.
For example: Saved Folder: /path/Working Files/
old_space_users.xlsx | new_space_users.xlsx |
old_space_locations.xlsx | new_space_locations.xlsx |
mapped_ids.xlsx | |
Create a new sheet in the locations files called all_locations and copy-paste the first two columns from each sheet (append column data) so that all location names and codes are available in a single sheet in the workbook.
In this example, all the values from column A and B (location_id and site_code) in the sheets called state, district and ward were copied over to a new sheet called all_locations.
Check the format of all the five Excel (xlsx) files.
Create the following sheet in the mapped_ids file. See this template file as an example:
Map user, location and groups from the old space to the new space and save values in the mapped_ids.xlsx file.
Column Name | Column Header | Formula Used |
A | username | =[old_space_users.xlsx]users!A2 |
B | old_user_id | =[old_space_users.xlsx]users!G2 |
C | new_user_id | =XLOOKUP(A2, [new_space_users.xlsx]users!A:A, [new_space_users.xlsx]users!G:G, "")
(this is looking up the value of username in column A in column A of the new space users sheet and retrieving the corresponding user_id in column G) |
D |
|
|
E | location_name | =[old_space_locations.xlsx]all_locations!B2 |
F | old_location_id | =[old_space_locations.xlsx]all_locations!A2 |
G | new_location_id | =XLOOKUP(E2, [new_space_locations.xlsx]all_locations!B:B, [new_space_locations.xlsx]all_locations!A:A, "")
(this is looking up the value of location_name in column E in column B of the new space locations sheet and retrieving the corresponding user_id in column A) |
H |
|
|
I | old_group_name | =[old_space_users.xlsx]groups!B2 |
J | old_group_id | =[old_space_users.xlsx]groups!A2 |
K | new_group_id | =XLOOKUP(I2, [new_space_users.xlsx]groups!B:B, [new_space_users.xlsx]groups!A:A, "")
(this is looking up the value of old_group_name in column I in column B of the new space groups sheet and retrieving the corresponding group_id in column A) |
Drag the formulas down until all values are populated and then copy-paste the entire dataset as values only.
Here’s an example of what the populated mapped_ids.xlsx file would look like (note this is transposed for easier viewing here)
Case Data Migration across spaces
See the video below for a detailed walkthrough.
Export case data from the old space by creating case export for each case type.
Make sure that the case_id is part of the export for all exports created and All Data and the entire time frame is chosen at the time of export.
Download all the exports in the same folder path as the mapping_ids.xlsx file.
Prepare a case import file for the new project space.
Important: Identify the exports for individual case types (with no child cases) and parent cases first. In Excel, use Save-as to save a copy of the file as a case import file for the new project space.
Saved Folder: /path/Working Files/
old_space_users.xlsx | new_space_users.xlsx |
old_space_locations.xlsx | new_space_users.xlsx |
mapped_ids.xlsx | |
old_space_[case_type1].xlsx | new_space_[case_type1].xlsx |
old_space_[case_type2].xlsx | new_space_[case_type2].xlsx |
old_space_[case_typeN].xlsx | new_space_[case_typeN].xlsx |
In the new_space_[case_type1].xlsx make the following edits:
Column Name | Column Header | Column Value |
A | caseid | Blank |
B | old_case_id | Existing case_ids from old space |
C | owner_id | =IFERROR( XLOOKUP(S2, mapped_ids.xlsx!B:B, mapped_ids.xlsx!C:C), IFERROR(XLOOKUP(S2, mapped_ids.xlsx!F:F, mapped_ids.xlsx!G:G), XLOOKUP(S2, mapped_ids.xlsx!J:J,mapped_ids.xlsx!K:K)))
(This formula will look for the owner_id in the userid column then the location id column and finally group id column to match with the new space ids.) |
D | old_owner_id | Existing owner_ids from old space |
E onwards | All other case properties | Same as in source file |
This will update the owner_id column to the new id (whether it is the user id, location id or group id) based on the lookup value in column C.
Navigate to Import Cases via Excel in the new project space and import cases.
This is what the case import mapping screen would look like - where the old_caseid and old_owner_id columns are being created and the owner_id column is mapping to the new space ids in the excel file.
Important: Make sure the correct and corresponding case type is chosen and the excel column named caseid is chosen for mapping. Do not forget to select “Create new records if there is no matching case”. This will ensure that CommCare creates new cases when the file is imported and maps them to the case type selected.
Repeat this process for all individual case types.
Important: Make sure you are importing case data for all individual case types that are not linked to any child cases first. In the second iteration, import all case types that are parent cases to any child cases. The process will remain the same.
The following documentation is relevant for parent and child case mapping when moving case data from one space to another.
Parent and Child Case Data Migration across spaces
See the video below for a detailed walkthrough.
Export child case data from the old space by creating case export for each case type that is a child case to another parent case type.
Ensure that along with caseid and owner_id, parent_id is selected in this export.
Make sure that the case_id is part of the export for all exports created and All Data and the entire time frame is chosen at the time of export.
Export parent case data from the new space by creating case export for each case type that is a parent case to a child case type.
Make sure that the case_id is part of the export for all exports created and All Data and the entire time frame is chosen at the time of export.
Prepare a case import file for the new project space.
In Excel, use Save-as to save a copy of the file as a case import file for the new project space.
Saved Folder: /path/Working Files/
old_space_users.xlsx | new_space_users.xlsx |
old_space_locations.xlsx | new_space_users.xlsx |
mapping_ids.xlsx | |
old_space_[case_type1].xlsx | new_space_[case_type1].xlsx |
old_space_[case_type2].xlsx | new_space_[case_type2].xlsx |
old_space_[case_typeN].xlsx | new_space_[case_typeN].xlsx |
old_space_[child_case_type1].xlsx | new_space_[child_case_type1].xlsx |
old_space_[child_case_type2].xlsx | new_space_[child_case_type2].xlsx |
old_space_[child_case_typeN].xlsx | new_space_[child_case_typeN].xlsx |