Metadata Glossary





This page contains guidance, definitions and suggestions for how to interpret the system metadata property fields in CommCare data. 

Types of Data in CommCare

Case vs. Form Data

In the CommCare data model, a case is anything that we track over time, while form data is any information collected about a case at a specific point in time through a CommCare form.  Ultimately form data is the source of all case data, but not all form data is case data. Here is a breakdown of the differences between the two.

Case Data

Form Data

Case Data

Form Data

Only those data fields that are designated as “case properties”

All data fields captured while completing a form in the application

Case properties are updated each time a new form is submitted (e.g. whether symptomatic)

Form properties from a new submission will not change the previous submission

Case data only shows the most recent value for each case property

Form data includes all historical information submitted using the application

Each row in a case data export is an individual case

Each row in a form data export is an individual form submission

Metadata

Another category of data is metadata: CommCare automatically assigns meta properties as needed to cases and form submissions, and can then turn around and read them. Metadata is used in CommCare to document processes such as when a form was submitted, or when a case was opened.

App-Specific Fields

In general, the bulk of a form export will have these types of column headers, unless modified in the export configuration:

    1. Data displayed in the format: form.field_name (for instance, form.edd). Data entered by the mobile worker.

    2. Data displayed in the format: form.case.update.field_name (for instance, form.case.update.edd). Data entered by the mobile worker that then updated a case.

    3. Data displayed in the format: form.case.create.field_name (for instance, form.case.create.case_name). Data created by the system during the creation of a new case.

Basic Form Metadata

When creating or modifying form export you will find certain fields are automatically included:

Field Name

Description

Other Notes

Field Name

Description

Other Notes

number

Number of the row of data; mostly useful for matching with other tabs for repeat groups

See Repeat Data in Form Exports

formid

Unique id for the form submission



completed_time

The time when the form was submitted (based on device time)



started_time

The time when the form was initially opened (based on device time)



username

Username of the user who submitted the form



received_on

Time that the form was received by the server




Advanced Form Metadata

Attention: Advanced Questions

If you're looking for a metadata field (like external_id) and cannot find it in an export, these fields can often be found by choosing "Show Advanced Questions" in the Form or Case Export configuration page.



Field Name

Description

Other Notes

Field Name

Description

Other Notes

userID

CommCare-generated alphanumeric ID for a particular user.



@xmlns

This defines the URI which specifies the structure of metadata, which is http://openrosa.org/jr/xforms



@name

The text readable name of the form.



App Version

Version of the CommCare application used to submit the form.

This number corresonds to the list of versions in the Application Builder

deviceID

The ID of the device from which the form was submitted.

This number changes if you reinstall CommCare on the same phone. This used to be the phone's IMEI code in older CommCare versions though starting CommCare 2.48, we use a completely random identifier to populate this field. This change was done to meet Google requirements for Android 10+ devices.

location

ID of the relevant location



app_id

Unique ID of the application



build_id





@version

Version of the specific form.

Each time you save a change, the app builder increments this version count but this doesn't update the App Version column (above) until you build and update to that version of the app. 

state

Should be "XFormInstance"



last_sync_token





partial_submission

This should always be FALSE.



edited_on

If the form was edited, displays the system date/time of that edit



submit_ip

IP from which the form was submitted.



If you need to reference this form metada from within a form in your app, you can do so using this format: /data/@name (for form name)

There are also a number of fields which are unique to cases and their exports, some of which are listed here: 

Name

Description

Example

Name

Description

Example

@case_id

A random machine generated UUIDv4 code that uniquely identifies this case.

a1d6916d-3d3c-4b90-bd86-64de292c6ce7

@case_type

The CommCare Case Type specific to the data model of your project configuration.

pregnancy

@owner_id

Specific to your project configuration, this is the UUIDv4 Machine ID of the entity which owns this case. It can be considered the address where the case lives in CommCare, and will be either a CommCare User, a Location, or a Group. Only users who have visibility into that area of the organization structure will be able to access it.

313d21e6-ae44-4fe5-899e-6926b43b969c

@status

Shows if the case is open or closed from a system perspective. Closed cases are effectively archived and not accessible from the application, however, they will appear on CommCare HQ anywhere that case data is displayed or exported. Values: ‘open’, ‘closed’.

closed

case_name

A human readable name for the entity being kept track of with this case object. For app builders, under the form Case Management settings page, this property will be called “name”. What this is set to will be specific to your project configuration.

Rose

closed

Closely related to @status above, this property also describes whether a case is closed or not at the system level and is accessible from many locations in CommCare that @status is not. Default values: ‘true’, ‘false’.

false

date_closed

Date and time case was marked as closed and archived at the system level. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC).

2011-12-20T15:09:47.2Z

date_opened

Date and time the record was created in CommCare, whether through an integration or by a user creating the record manually. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC).

2020-08-20T13:33:57.548737Z

external_id

External ID associated with the case.

123456

last_modified

Also labeled in some exports as "date_modified", it is the date/timestamp that the user's app or web browser last sent data for this case to the server. It uses the clock on the user's device, so it's less reliable than server_date_modified. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC).

2011-12-13T15:09:47.1Z

opened_by_user_id

The UUID system identifier for the user that opened the case.

f309c76478b041b19448b6f7dad17291

opened_by_username

The username of the user that opened the case.

testuser1

owner_name

User name of case owner, including domain.

jdoe@example.commcarehq.org

properties

List of all editable case properties, including both special predefined properties and user-defined dynamic properties.



server_date_modified

Date/timestamp that the server received the data. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC).

2020-08-20T13:33:57.548737Z



Metadata Case Properties by Interface Display/Output

Since there is some variability in how the same property is labeled depending on where it is pulled from, the following table of system metadata properties can serve as a crosswalk to help identify each property as it appears in various interfaces. 

Reserved Case Properties

Case List Explorer

Case List Report

Case Data Page

Report Builder

Export Case Data

Data Export Tool

API

Data Forwarding

Reserved Case Properties

Case List Explorer

Case List Report

Case Data Page

Report Builder

Export Case Data

Data Export Tool

API

Data Forwarding

@case_id

@case_id



Case ID



caseid

case_id

case_id

case_id

@case_type

@case_type

Case Type

Case Type



case_type

case_type

case_type

case_type

@owner_id

@owner_id



Owner

owner id

owner_id

properties.owner_id

owner_id

owner_id

@status

@status

Status













case_name

case_name

Name

Name

name

name

properties.case_name

case_name

case_id

date_opened

date_opened

Created Date

Opened On

opened on

opened_date

properties.date_opened

date_opened

date_opened

last_modified

last_modified

Modified Date

Modified On

modified on

last_modified_date

date_modified

date_modified

date_modified



closed





Case Closed

closed

closed

closed

closed



closed_on



Closed On

closed on

closed_date

date_closed

date_closed

date_closed



closed_by_user_id







closed_by_user_id









closed_by_username







closed_by_username

















last_modified_by_user_id





user_id



last_modified_by_user_username



Last Submitter



last_modified_by_user_username









opened_by_user_id







opened_by_user_id

opened_by







opened_by_username

Created By





opened_by_username









owner_name

Owner



Case Owner

owner_name









server_last_modified_date







server_last_modified_date

server_date_modified

server_date_modified

server_date_modified

#case/parent_id*











indices.parent.case_id





Notes 

- Any blank means the property may not be accessible from this area of the platform.

* The structure and language of indexed properties are variable and subject to change based on the project configuration.

More information on each of these interfaces for accessing data in CommCare appears below.

Reserved Case Properties

Certain system case properties are reserved and will always be created automatically in each  application, in each form, and for each case. App builders may access these properties through the Case Management settings page for forms, as well in App Properties when building a form. Since this metadata is what the application needs to reference in order to function properly, they can be considered the official names of these properties, but may be labeled differently elsewhere (such as in exports, as seen in the above table). They are sometimes designated with an "Info” tag for their data type as depicted here.

These are available to the application “on the phone”, which refers to the section of the casedb database that is accessible to the user and is available to the CommCare app on their smart phone or Web Apps session. This is where a representation of each case and its metadata lives to be read and written to through the application’s XPath expressions that the app builders write in order to work with the specific XML nodes/questions available during the session.

Case Data Page

The Case List Report and Case List Explorer are two related tools to navigate the dataset from within CommCare HQ. They will allow you to access the Case Data Page and Case Changes to review any case through the CommCare back end user interface (UI). This can be helpful for quick reference and querying or troubleshooting data without leaving the Commcare UI. Data can also be “cleaned” here, meaning that corrections can be made on a one off basis. Finally, all of this data can also be downloaded using the available export buttons.

Report Builder

The Report Builder allows you to build basic web-based reports using the data collected in your application in the following formats:

  • List: Table with a basic list of data (from forms or cases)

  • Summary: Summarized aggregate data based on a chosen case property or form question

  • Map: Locations plotted on a map

This data can also be downloaded to CSV and Excel. View more information about the Report Builder on this Confluence help page.

Export Data

These exports are set up manually, and can be configured to display properties using their default labels in the exports header, or with modified headings. Find out more here:

And also see De-Identifying Data Exports: data fields can be manually marked as sensitive, and once configured the de-identified reports can be saved and downloaded again in the future.

A clarification on the difference between "---" and blank values in form exports can be found here.

Data Export Tool

The Data Export Tool is also called commcare-export. Based on python script filters, it provides a way to run automated updates to large datasets stored in a SQL database or a file.

The Data Export Tool manual and detailed setup instructions are available here, along with an outline of the applicable system properties in the export field reference.

Application Programming Interface (API)

This is a way to make calls to CommCare for data from another software program.

The most common use case for the case investigation and contact tracing application is for integration with laboratory systems or disease reporting systems. The API can be used to submit data into CommCare or get data in realtime from CommCare. Requesting data requires sending a request to a URL endpoint, and requesting data in batches. The two most common endpoints are “List Cases” and “List Forms.” The data which is returned in response to the request is in JSON format. 

One way to explore the format of the API response payloads is to use the API Explorer. This tool is designed for programmers to explore the data structures and data types that can be received through the API. It requires the user to have API access enabled in at least one domain. The tool will use this API permission to send and receive real API payloads using a web browser. If this tool is used on a production domain, it will return production data into the browser window. Be thoughtful when demonstrating this tool to external audiences as it can raise questions about data security. Do not use this tool to request data from a production domain while screen-sharing or presenting to an audience, unless you have specific permission and are in a HIPAA-compliant video conference session. 

Learn more about this tool at the following link.

Data Forwarding

Data forwarders are a lower touch way to manage data integrations with CommCare by relying on these set-it-and-forget-it data exchanges. The Case Data Forwarder is the one responsible for sending cases to new venues.

See the documentation on the Case Data and other Data Forwarding services here.