This page contains guidance, definitions and suggestions for how to interpret the system metadata property fields in CommCare data.
...
Types of Data in CommCare
Case vs. Form Data
In the CommCare data model, a case is anything that we track over time, while form data is any information collected about a case at a specific point in time through a CommCare form. Ultimately form data is the source of all case data, but not all form data is case data. Here is a breakdown of the differences between the two.
| Case Data | Form Data |
|---|---|
| Only those data fields that are designated as “case properties” | All data fields captured while completing a form in the application |
| Case properties are updated each time a new form is submitted (e.g. whether symptomatic) | Form properties from a new submission will not change the previous submission |
| Case data only shows the most recent value for each case property | Form data includes all historical information submitted using the application |
| Each row in a case data export is an individual case | Each row in a form data export is an individual form submission |
Metadata
Another category of data is metadata: CommCare automatically assigns meta properties as needed to cases and form submissions, and can then turn around and read them. Metadata is used in CommCare to document processes such as when a form was submitted, or when a case was opened.
...
| Field Name | Description | Other Notes |
|---|---|---|
| userID | CommCare-generated alphanumeric ID for a particular user. | |
| @xmlns | This defines the URI which specifies the structure of metadata, which is http://openrosa.org/jr/xforms | |
| @name | The text readable name of the form. | |
| App Version | Version of the CommCare application used to submit the form. | This number corresonds to the list of versions in the Application Builder |
| deviceID | The ID of the device from which the form was submitted. | This number changes if you reinstall CommCare on the same phone. This used to be the phone's IMEI code in older CommCare versions though starting CommCare 2.48, we use a completely random identifier to populate this field. This change was done to meet Google requirements for Android 10+ devices. |
| location | ID of the relevant location | |
| app_id | Unique ID of the application | |
| build_id | ||
| @version | Version of the specific form. | Each time you save a change, the app builder increments this version count but this doesn't update the App Version column (above) until you build and update to that version of the app. |
| state | Should be "XFormInstance" | |
| last_sync_token | ||
| partial_submission | This should always be FALSE. | |
| edited_on | If the form was edited, displays the system date/time of that edit | |
| submit_ip | IP from which the form was submitted. |
...
| Reserved Case Properties | Case List Explorer | Case List Report | Case Data Page | Report Builder | Export Case Data | Data Export Tool | API | Data Forwarding |
|---|---|---|---|---|---|---|---|---|
| @case_id | @case_id | Case ID | caseid | case_id | case_id | case_id | ||
| @case_type | @case_type | Case Type | Case Type | case_type | case_type | case_type | case_type | |
| @owner_id | @owner_id | Owner | owner id | owner_id | properties.owner_id | owner_id | owner_id | |
| @status | @status | Status | ||||||
| case_name | case_name | Name | Name | name | name | properties.case_name | case_name | case_id |
| date_opened | date_opened | Created Date | Opened On | opened on | opened_date | properties.date_opened | date_opened | date_opened |
| last_modified | last_modified | Modified Date | Modified On | modified on | last_modified_date | date_modified | date_modified | date_modified |
| closed | Case Closed | closed | closed | closed | closed | |||
| closed_on | Closed On | closed on | closed_date | date_closed | date_closed | date_closed | ||
| closed_by_user_id | closed_by_user_id | |||||||
| closed_by_username | closed_by_username | |||||||
| last_modified_by_user_id | user_id | |||||||
| last_modified_by_user_username | Last Submitter | last_modified_by_user_username | ||||||
| opened_by_user_id | opened_by_user_id | opened_by | ||||||
| opened_by_username | Created By | opened_by_username | ||||||
| owner_name | Owner | Case Owner | owner_name | |||||
| server_last_modified_date | server_last_modified_date | server_date_modified | server_date_modified | server_date_modified | ||||
| #case/parent_id* | indices.parent.case_id |
Notes
- Any blank means the property may not be accessible from this area of the platform.
...
More information on each of these interfaces for accessing data in CommCare appears below.
Reserved Case Properties
Certain system case properties are reserved and will always be created automatically in each application, in each form, and for each case. App builders may access these properties through the Case Management settings page for forms, as well in App Properties when building a form. Since this metadata is what the application needs to reference in order to function properly, they can be considered the official names of these properties, but may be labeled differently elsewhere (such as in exports, as seen in the above table). They are sometimes designated with an "Info” tag for their data type as depicted here.
...
These are available to the application “on the phone”, which refers to the section of the casedb database that is accessible to the user and is available to the CommCare app on their smart phone or Web Apps session. This is where a representation of each case and its metadata lives to be read and written to through the application’s XPath expressions that the app builders write in order to work with the specific XML nodes/questions available during the session.
Case Data Page
The Case List Report and Case List Explorer are two related tools to navigate the dataset from within CommCare HQ. They will allow you to access the Case Data Page and Case Changes to review any case through the CommCare back end user interface (UI). This can be helpful for quick reference and querying or troubleshooting data without leaving the Commcare UI. Data can also be “cleaned” here, meaning that corrections can be made on a one off basis. Finally, all of this data can also be downloaded using the available export buttons.
Report Builder
The Report Builder allows you to build basic web-based reports using the data collected in your application in the following formats:
...
This data can also be downloaded to CSV and Excel. View more information about the Report Builder on this Confluence help page.
Export Data
These exports are set up manually, and can be configured to display properties using their default labels in the exports header, or with modified headings. Find out more here:
- Export case data (latest-available data for a case)
- Export form data (historical answers to form questions, multiple rows per individual)
And also see De-Identifying Data Exports: data fields can be manually marked as sensitive, and once configured the de-identified reports can be saved and downloaded again in the future.
A clarification on the difference between "---" and blank values in form exports can be found here.
Data Export Tool
The Data Export Tool is also called commcare-export. Based on python script filters, it provides a way to run automated updates to large datasets stored in a MySQL SQL database or a file.
The Data Export Tool manual and detailed setup instructions are available here, along with an outline of the applicable system properties in the export field reference.
Application Programming Interface (API)
This is a way to make calls to CommCare for data from another software program.
The most common use case for the case investigation and contact tracing application is for integration with laboratory systems or disease reporting systems. The API can be used to submit data into CommCare or get data in realtime from CommCare. Requesting data requires sending a request to a URL endpoint, and requesting data in batches. The two most common endpoints are “List Cases” and “List Forms.” The data which is returned in response to the request is in JSON format.
One way to explore the format of the API response payloads is to use the API Explorer. This tool is designed for programmers to explore the data structures and data types that can be received through the API. It requires the user to have API access enabled in at least one domain. The tool will use this API permission to send and receive real API payloads using a web browser. If this tool is used on a production domain, it will return production data into the browser window. Be thoughtful when demonstrating this tool to external audiences as it can raise questions about data security. Do not use this tool to request data from a production domain while screen-sharing or presenting to an audience, unless you have specific permission and are in a HIPAA-compliant video conference session.
Learn more about this tool at the following link.
- https://confluencedimagi-dev.dimagiatlassian.comnet/wiki/display/commcarepublic/CommCare+HQ+APIs#CommCareHQAPIs-APIExplorer
- Also, see this Confluence help page for more information on API case data.
Data Forwarding
Data forwarders are a lower touch way to manage data integrations with CommCare by relying on these set-it-and-forget-it data exchanges. The Case Data Forwarder is the one responsible for sending cases to new venues.
See the documentation on the Case Data and other Data Forwarding services here.