Setting up a Data Pipeline via CommCare
- Rishabh Rath
- Gillian Javetski
- Kai Cowger
No two CommCare projects are identical. Therefore, users may encounter varying requirements for data processing and visualization needs that could change over time. This page presents various options for CommCare data processing and analysis to help users setup robust data pipelines via CommCare.
Every CommCare project will eventually need to interpret the Case and Form data collected by mobile workers. The design of a data pipeline involves two fundamental questions: the method of export (e.g., basic export interface) and the automation of analysis (e.g., VBA queries). Both of these aspects should be regularly reassessed in projects planning large-scale data analysis (exporting 50,000+ rows at a time).
Method of Export | Scale (rows) | Requirements | Documentation |
|---|---|---|---|
Data Exports | 0 - 50,000 | Excel | https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143954661 |
Daily Saved Exports | 0 - 100,000 | Excel | https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143951612 |
OData Feeds | 0 - 500,000 | PowerBI/Tableau Desktop | https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2198503499 |
CommCare Data Export Tool | 0 - 1,000,000+ | Excel, Python, programming knowledge, database knowledge, server setup | https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143955952 |
Data Exports
The basic data exports interface in CommCare allows exporting up to 50,000 rows of case and form data submitted by mobile workers. CommCare offers robust data exporting features with the ability to customise exports as needed.
See https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143954661 for more.
Data Saved Exports
CommCare can be used to setup Daily Saved Exports as well. These are pre-compiled exports, allowing users to download fresh data immediately from CommCare HQ instead of waiting for a new export to be generated. This method can handle up to 500,000 rows.
See https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143951612 for more.
OData Feeds
OData feeds provide another way to export CommCare data, supporting volumes between 10,000 to 500,000 rows. However, this requires use of PowerBI or Tableau Desktop software. OData feeds can be setup for incremental refresh on Power BI as well for larger data sets.
See https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2282717354 for more.
The CommCare Data Export Tool can write exported data directly to an Excel file, handling between 10,000 to 500,000 rows. This requires having an Excel license and setting up the Data Export Tool. For very large data volumes from 50,000 up to over 1 million rows, the CommCare Data Export Tool can write exports to a configured database instead of Excel. This has additional requirements like the Data Export Tool setup, an installed database, and ideally a dedicated server to run it on.
See https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143955952 for more.