Metadata Glossary
- Mengji Chen
- Gillian Javetski
- Kai Cowger
This page contains guidance, definitions and suggestions for how to interpret the system metadata property fields in CommCare data.
Types of Data in CommCare
Case vs. Form Data
In the CommCare data model, a case is anything that we track over time, while form data is any information collected about a case at a specific point in time through a CommCare form. Ultimately form data is the source of all case data, but not all form data is case data. Here is a breakdown of the differences between the two.
Case Data | Form Data |
|---|---|
Only those data fields that are designated as “case properties” | All data fields captured while completing a form in the application |
Case properties are updated each time a new form is submitted (e.g. whether symptomatic) | Form properties from a new submission will not change the previous submission |
Case data only shows the most recent value for each case property | Form data includes all historical information submitted using the application |
Each row in a case data export is an individual case | Each row in a form data export is an individual form submission |
Metadata
Another category of data is metadata: CommCare automatically assigns meta properties as needed to cases and form submissions, and can then turn around and read them. Metadata is used in CommCare to document processes such as when a form was submitted, or when a case was opened.
App-Specific Fields
In general, the bulk of a form export will have these types of column headers, unless modified in the export configuration:
Data displayed in the format: form.field_name (for instance, form.edd). Data entered by the mobile worker.
Data displayed in the format: form.case.update.field_name (for instance, form.case.update.edd). Data entered by the mobile worker that then updated a case.
Data displayed in the format: form.case.create.field_name (for instance, form.case.create.case_name). Data created by the system during the creation of a new case.
Basic Form Metadata
When creating or modifying form export you will find certain fields are automatically included:
Field Name | Description | Other Notes |
|---|---|---|
number | Number of the row of data; mostly useful for matching with other tabs for repeat groups | |
formid | Unique id for the form submission |
|
completed_time | The time when the form was submitted (based on device time) |
|
started_time | The time when the form was initially opened (based on device time) |
|
username | Username of the user who submitted the form |
|
received_on | Time that the form was received by the server |
|
Form Name
The name of your form in the actual XML of the form. This name is used on some reports on CommCareHQ. In most scenarios the mobile phone and CommCareHQ reports use the name you give the form in the App Builder instead.
Form ID
The form ID is the question ID of the largest group in the form, which is there by default and contains all the questions in the form. When you create a form in the Form Builder, this group is always named "data". You'll notice when you refer to questions in a display condition, the syntax is always /data/question_id. This is because you are writing out the path to where the question is located in the form, and the question is always inside the "data" group.
Forms created in other tools often have a different form ID. This is fine. We warn against changing the Form ID here because it will not automatically change the Form ID in existing references in your logic conditions. If you change the Form ID, you must manually change any existing logic references.
Advanced Form Metadata
Attention: Advanced Questions
If you're looking for a metadata field (like external_id) and cannot find it in an export, these fields can often be found by choosing "Show Advanced Questions" in the Form or Case Export configuration page.
Field Name | Description | Other Notes |
|---|---|---|
userID | CommCare-generated alphanumeric ID for a particular user. |
|
@xmlns | This defines the URI which specifies the structure of metadata, which is https://getodk.github.io/xforms-spec/ |
|
@name | The text readable name of the form. |
|
App Version | Version of the CommCare application used to submit the form. | This number corresonds to the list of versions in the Application Builder |
deviceID | The ID of the device from which the form was submitted. | This number changes if you reinstall CommCare on the same phone. This used to be the phone's IMEI code in older CommCare versions though starting CommCare 2.48, we use a completely random identifier to populate this field. This change was done to meet Google requirements for Android 10+ devices. |
location | ID of the relevant location |
|
app_id | Unique ID of the application |
|
build_id |
|
|
@version | Version of the specific form. | Each time you save a change, the app builder increments this version count but this doesn't update the App Version column (above) until you build and update to that version of the app. |
state | Should be "XFormInstance" |
|
last_sync_token |
|
|
partial_submission | This should always be FALSE. |
|
edited_on | If the form was edited, displays the system date/time of that edit |
|
submit_ip | IP from which the form was submitted. |
|
If you need to reference this form metada from within a form in your app, you can do so using this format: /data/@name (for form name)
Case-Related Metadata
There are also a number of fields which are unique to cases and their exports, some of which are listed here:
Name | Description | Example |
|---|---|---|
@case_id | A random machine generated UUIDv4 code that uniquely identifies this case. | a1d6916d-3d3c-4b90-bd86-64de292c6ce7 |
@case_type | The CommCare Case Type specific to the data model of your project configuration. | pregnancy |
@owner_id | Specific to your project configuration, this is the UUIDv4 Machine ID of the entity which owns this case. It can be considered the address where the case lives in CommCare, and will be either a CommCare User, a Location, or a Group. Only users who have visibility into that area of the organization structure will be able to access it. | 313d21e6-ae44-4fe5-899e-6926b43b969c |
@status | Shows if the case is open or closed from a system perspective. Closed cases are effectively archived and not accessible from the application, however, they will appear on CommCare HQ anywhere that case data is displayed or exported. Values: ‘open’, ‘closed’. | closed |
case_name | A human readable name for the entity being kept track of with this case object. For app builders, under the form Case Management settings page, this property will be called “name”. What this is set to will be specific to your project configuration. | Rose |
closed | Closely related to @status above, this property also describes whether a case is closed or not at the system level and is accessible from many locations in CommCare that @status is not. Default values: ‘true’, ‘false’. | false |
date_closed | Date and time case was marked as closed and archived at the system level. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC). | 2011-12-20T15:09:47.2Z |
date_opened | Date and time the record was created in CommCare, whether through an integration or by a user creating the record manually. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC). | 2020-08-20T13:33:57.548737Z |
external_id | External ID associated with the case. | 123456 |
last_modified | Also labeled in some exports as "date_modified", it is the date/timestamp that the user's app or web browser last sent data for this case to the server. It uses the clock on the user's device, so it's less reliable than server_date_modified. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC). | 2011-12-13T15:09:47.1Z |
opened_by_user_id | The UUID system identifier for the user that opened the case. | f309c76478b041b19448b6f7dad17291 |
opened_by_username | The username of the user that opened the case. | testuser1 |
owner_name | User name of case owner, including domain. | |
properties | List of all editable case properties, including both special predefined properties and user-defined dynamic properties. |
|
server_date_modified | Date/timestamp that the server received the data. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC). | 2020-08-20T13:33:57.548737Z |
Metadata Case Properties by Interface Display/Output
Since there is some variability in how the same property is labeled depending on where it is pulled from, the following table of system metadata properties can serve as a crosswalk to help identify each property as it appears in various interfaces.
Reserved Case Properties | Case List Explorer | Case List Report | Case Data Page | Report Builder | Export Case Data | Data Export Tool | API | Data Forwarding |
|---|---|---|---|---|---|---|---|---|
@case_id | @case_id |
| Case ID |
| caseid | case_id | case_id | case_id |
@case_type | @case_type | Case Type | Case Type |
| case_type | case_type | case_type | case_type |
@owner_id | @owner_id |
| Owner | owner id | owner_id | properties.owner_id | owner_id | owner_id |
@status | @status | Status |
|
|
|
|
|
|
case_name | case_name | Name | Name | name | name | properties.case_name | case_name | case_id |
date_opened | date_opened | Created Date | Opened On | opened on | opened_date | properties.date_opened | date_opened | date_opened |
last_modified | last_modified | Modified Date | Modified On | modified on | last_modified_date | date_modified | date_modified | date_modified |
| closed |
|
| Case Closed | closed | closed | closed | closed |
| closed_on |
| Closed On | closed on | closed_date | date_closed | date_closed | date_closed |
| closed_by_user_id |
|
|
| closed_by_user_id |
|
|
|
| closed_by_username |
|
|
| closed_by_username |
|
|
|
|
|
|
|
| last_modified_by_user_id |
|
| user_id |
| last_modified_by_user_username |
| Last Submitter |
| last_modified_by_user_username |
|
|
|
| opened_by_user_id |
|
|
| opened_by_user_id | opened_by |
|
|
| opened_by_username | Created By |
|
| opened_by_username |
|
|
|
| owner_name | Owner |
| Case Owner | owner_name |
|
|
|
| server_last_modified_date |
|
|
| server_last_modified_date | server_date_modified | server_date_modified | server_date_modified |
#case/parent_id* |
|
|
|
|
| indices.parent.case_id |
|
|
Notes
- Any blank means the property may not be accessible from this area of the platform.
* The structure and language of indexed properties are variable and subject to change based on the project configuration.
More information on each of these interfaces for accessing data in CommCare appears below.
Reserved Case Properties
Certain system case properties are reserved and will always be created automatically in each application, in each form, and for each case. App builders may access these properties through the Case Management settings page for forms, as well in App Properties when building a form. Since this metadata is what the application needs to reference in order to function properly, they can be considered the official names of these properties, but may be labeled differently elsewhere (such as in exports, as seen in the above table). They are sometimes designated with an "Info” tag for their data type as depicted here.
These are available to the application “on the phone”, which refers to the section of the casedb database that is accessible to the user and is available to the CommCare app on their smart phone or Web Apps session. This is where a representation of each case and its metadata lives to be read and written to through the application’s XPath expressions that the app builders write in order to work with the specific XML nodes/questions available during the session.
Case Data Page
The Case List Report and Case List Explorer are two related tools to navigate the dataset from within CommCare HQ. They will allow you to access the Case Data Page and Case Changes to review any case through the CommCare back end user interface (UI). This can be helpful for quick reference and querying or troubleshooting data without leaving the Commcare UI. Data can also be “cleaned” here, meaning that corrections can be made on a one off basis. Finally, all of this data can also be downloaded using the available export buttons.
Report Builder
The Report Builder allows you to build basic web-based reports using the data collected in your application in the following formats:
List: Table with a basic list of data (from forms or cases)
Summary: Summarized aggregate data based on a chosen case property or form question
Map: Locations plotted on a map
This data can also be downloaded to CSV and Excel. View more information about the Report Builder on this page.
Export Data
These exports are set up manually, and can be configured to display properties using their default labels in the exports header, or with modified headings. Find out more here:
https://dimagi.atlassian.net/wiki/x/vCzKfw (latest-available data for a case)
https://dimagi.atlassian.net/wiki/x/Ly3Kfw (historical answers to form questions, multiple rows per individual)
And also see https://dimagi.atlassian.net/wiki/x/QS7Kfw : data fields can be manually marked as sensitive, and once configured the de-identified reports can be saved and downloaded again in the future.
A clarification on the difference between "---" and blank values in form exports can be found here.
Data Export Tool
The Data Export Tool is also called commcare-export. Based on python script filters, it provides a way to run automated updates to large datasets stored in a SQL database or a file. Please read more here: CommCare Data Export Tool (DET)
Application Programming Interface (API)
This is a way to make calls to CommCare for data from another software program.
The most common use case for the case investigation and contact tracing application is for integration with laboratory systems or disease reporting systems. The API can be used to submit data into CommCare or get data in realtime from CommCare. Requesting data requires sending a request to a URL endpoint, and requesting data in batches. The two most common endpoints are “List Cases” and “List Forms.” The data which is returned in response to the request is in JSON format.
One way to explore the format of the API response payloads is to use the API Explorer. This tool is designed for programmers to explore the data structures and data types that can be received through the API. It requires the user to have API access enabled in at least one domain. The tool will use this API permission to send and receive real API payloads using a web browser. If this tool is used on a production domain, it will return production data into the browser window. Be thoughtful when demonstrating this tool to external audiences as it can raise questions about data security. Do not use this tool to request data from a production domain while screen-sharing or presenting to an audience, unless you have specific permission and are in a HIPAA-compliant video conference session.
Learn more about this tool at the following link.https://dimagi.atlassian.net/wiki/x/BjTKfw
Data Forwarding
Data forwarders are a lower touch way to manage data integrations with CommCare by relying on these set-it-and-forget-it data exchanges. The Case Data Forwarder is the one responsible for sending cases to new venues.
See the documentation on the Case Data and other Data Forwarding services here.