Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 38 Next »



This page contains guidance, definitions and suggestions for how to interpret the system metadata property fields in CommCare data. 

Types of Data in CommCare

Case vs. Form Data

In the CommCare data model, a case is anything that we track over time, while form data is any information collected about a case at a specific point in time through a CommCare form.  Ultimately form data is the source of all case data, but not all form data is case data. Here is a breakdown of the differences between the two.

Case DataForm Data
Only those data fields that are designated as “case properties”All data fields captured while completing a form in the application
Case properties are updated each time a new form is submitted (e.g. whether symptomatic)Form properties from a new submission will not change the previous submission
Case data only shows the most recent value for each case propertyForm data includes all historical information submitted using the application
Each row in a case data export is an individual caseEach row in a form data export is an individual form submission

Metadata

Another category of data is metadata: CommCare automatically assigns meta properties as needed to cases and form submissions, and can then turn around and read them. Metadata is used in CommCare to document processes such as when a form was submitted, or when a case was opened.

App-Specific Fields

In general, the bulk of a form export will have these types of column headers, unless modified in the export configuration:

    1. Data displayed in the format: form.field_name (for instance, form.edd). Data entered by the mobile worker.

    2. Data displayed in the format: form.case.update.field_name (for instance, form.case.update.edd). Data entered by the mobile worker that then updated a case.

    3. Data displayed in the format: form.case.create.field_name (for instance, form.case.create.case_name). Data created by the system during the creation of a new case.

Basic Form Metadata

When creating or modifying form export you will find certain fields are automatically included:

Field NameDescriptionOther Notes
numberNumber of the row of data; mostly useful for matching with other tabs for repeat groupsSee Repeat Data in Form Exports
formidUnique id for the form submission
completed_timeThe time when the form was submitted (based on device time)
started_timeThe time when the form was initially opened (based on device time)
usernameUsername of the user who submitted the form
received_onTime that the form was received by the server


Advanced Form Metadata

Attention: Advanced Questions

If you're looking for a metadata field (like external_id) and cannot find it in an export, these fields can often be found by choosing "Show Advanced Questions" in the Form or Case Export configuration page.


Field NameDescriptionOther Notes
userIDCommCare-generated alphanumeric ID for a particular user.
@xmlnsThis defines the URI which specifies the structure of metadata, which is http://openrosa.org/jr/xforms
@nameThe text readable name of the form.
App VersionVersion of the CommCare application used to submit the form.This number corresonds to the list of versions in the Application Builder
deviceIDThe ID of the device from which the form was submitted.This number changes if you reinstall CommCare on the same phone. This used to be the phone's IMEI code in older CommCare versions though starting CommCare 2.48, we use a completely random identifier to populate this field. This change was done to meet Google requirements for Android 10+ devices.
locationID of the relevant location
app_idUnique ID of the application
build_id

@versionVersion of the specific form.Each time you save a change, the app builder increments this version count but this doesn't update the App Version column (above) until you build and update to that version of the app. 
stateShould be "XFormInstance"
last_sync_token

partial_submissionThis should always be FALSE.
edited_onIf the form was edited, displays the system date/time of that edit
submit_ipIP from which the form was submitted.

If you need to reference this form metada from within a form in your app, you can do so using this format: /data/@name (for form name)

Case-Related Metadata

There are also a number of fields which are unique to cases and their exports, some of which are listed here: 

NameDescriptionExample
@case_idA random machine generated UUIDv4 code that uniquely identifies this case.a1d6916d-3d3c-4b90-bd86-64de292c6ce7
@case_typeThe CommCare Case Type specific to the data model of your project configuration.pregnancy
@owner_idSpecific to your project configuration, this is the UUIDv4 Machine ID of the entity which owns this case. It can be considered the address where the case lives in CommCare, and will be either a CommCare User, a Location, or a Group. Only users who have visibility into that area of the organization structure will be able to access it.313d21e6-ae44-4fe5-899e-6926b43b969c
@statusShows if the case is open or closed from a system perspective. Closed cases are effectively archived and not accessible from the application, however, they will appear on CommCare HQ anywhere that case data is displayed or exported. Values: ‘open’, ‘closed’.closed
case_nameA human readable name for the entity being kept track of with this case object. For app builders, under the form Case Management settings page, this property will be called “name”. What this is set to will be specific to your project configuration.Rose
closedClosely related to @status above, this property also describes whether a case is closed or not at the system level and is accessible from many locations in CommCare that @status is not. Default values: ‘true’, ‘false’.false
date_closedDate and time case was marked as closed and archived at the system level. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC).2011-12-20T15:09:47.2Z
date_openedDate and time the record was created in CommCare, whether through an integration or by a user creating the record manually. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC).2020-08-20T13:33:57.548737Z
external_idExternal ID associated with the case.123456
last_modifiedAlso labeled in some exports as "date_modified", it is the date/timestamp that the user's app or web browser last sent data for this case to the server. It uses the clock on the user's device, so it's less reliable than server_date_modified. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC).2011-12-13T15:09:47.1Z
opened_by_user_id
The UUID system identifier for the user that opened the case.f309c76478b041b19448b6f7dad17291
opened_by_usernameThe username of the user that opened the case.testuser1
owner_nameUser name of case owner, including domain.jdoe@example.commcarehq.org
propertiesList of all editable case properties, including both special predefined properties and user-defined dynamic properties.
server_date_modifiedDate/timestamp that the server received the data. Default format: 'YYYY-MM-DDThh:mm:ss.sZ' (UTC).2020-08-20T13:33:57.548737Z


Metadata Case Properties by Interface Display/Output

Since there is some variability in how the same property is labeled depending on where it is pulled from, the following table of system metadata properties can serve as a crosswalk to help identify each property as it appears in various interfaces. 

Reserved Case PropertiesCase List ExplorerCase List ReportCase Data PageReport BuilderExport Case DataData Export ToolAPIData Forwarding
@case_id@case_id
Case ID
caseidcase_idcase_idcase_id
@case_type@case_typeCase TypeCase Type
case_typecase_typecase_typecase_type
@owner_id@owner_id
Ownerowner idowner_idproperties.owner_idowner_idowner_id
@status@statusStatus





case_namecase_nameNameNamenamenameproperties.case_namecase_namecase_id
date_openeddate_openedCreated DateOpened Onopened onopened_dateproperties.date_openeddate_openeddate_opened
last_modifiedlast_modifiedModified DateModified Onmodified onlast_modified_datedate_modifieddate_modifieddate_modified

closed

Case Closedclosedclosedclosedclosed

closed_on
Closed Onclosed onclosed_datedate_closeddate_closeddate_closed

closed_by_user_id


closed_by_user_id



closed_by_username


closed_by_username







last_modified_by_user_id

user_id

last_modified_by_user_username
Last Submitter
last_modified_by_user_username



opened_by_user_id


opened_by_user_idopened_by


opened_by_usernameCreated By

opened_by_username



owner_nameOwner
Case Ownerowner_name



server_last_modified_date


server_last_modified_dateserver_date_modifiedserver_date_modifiedserver_date_modified
#case/parent_id*




indices.parent.case_id

Notes 

- Any blank means the property may not be accessible from this area of the platform.

* The structure and language of indexed properties are variable and subject to change based on the project configuration.

More information on each of these interfaces for accessing data in CommCare appears below.

Reserved Case Properties

Certain system case properties are reserved and will always be created automatically in each  application, in each form, and for each case. App builders may access these properties through the Case Management settings page for forms, as well in App Properties when building a form. Since this metadata is what the application needs to reference in order to function properly, they can be considered the official names of these properties, but may be labeled differently elsewhere (such as in exports, as seen in the above table). They are sometimes designated with an "Info” tag for their data type as depicted here.

These are available to the application “on the phone”, which refers to the section of the casedb database that is accessible to the user and is available to the CommCare app on their smart phone or Web Apps session. This is where a representation of each case and its metadata lives to be read and written to through the application’s XPath expressions that the app builders write in order to work with the specific XML nodes/questions available during the session.

Case Data Page

The Case List Report and Case List Explorer are two related tools to navigate the dataset from within CommCare HQ. They will allow you to access the Case Data Page and Case Changes to review any case through the CommCare back end user interface (UI). This can be helpful for quick reference and querying or troubleshooting data without leaving the Commcare UI. Data can also be “cleaned” here, meaning that corrections can be made on a one off basis. Finally, all of this data can also be downloaded using the available export buttons.

Report Builder

The Report Builder allows you to build basic web-based reports using the data collected in your application in the following formats:

  • List: Table with a basic list of data (from forms or cases)
  • Summary: Summarized aggregate data based on a chosen case property or form question
  • Map: Locations plotted on a map

This data can also be downloaded to CSV and Excel. View more information about the Report Builder on this Confluence help page.

Export Data

These exports are set up manually, and can be configured to display properties using their default labels in the exports header, or with modified headings. Find out more here:

And also see De-Identifying Data Exports: data fields can be manually marked as sensitive, and once configured the de-identified reports can be saved and downloaded again in the future.

A clarification on the difference between "---" and blank values in form exports can be found here.

Data Export Tool

The Data Export Tool is also called commcare-export. Based on python script filters, it provides a way to run automated updates to large datasets stored in a MySQL database or a file.

The Data Export Tool manual and detailed setup instructions are available here, along with an outline of the applicable system properties in the export field reference.

Application Programming Interface (API)

This is a way to make calls to CommCare for data from another software program.

The most common use case for the case investigation and contact tracing application is for integration with laboratory systems or disease reporting systems. The API can be used to submit data into CommCare or get data in realtime from CommCare. Requesting data requires sending a request to a URL endpoint, and requesting data in batches. The two most common endpoints are “List Cases” and “List Forms.” The data which is returned in response to the request is in JSON format. 

One way to explore the format of the API response payloads is to use the API Explorer. This tool is designed for programmers to explore the data structures and data types that can be received through the API. It requires the user to have API access enabled in at least one domain. The tool will use this API permission to send and receive real API payloads using a web browser. If this tool is used on a production domain, it will return production data into the browser window. Be thoughtful when demonstrating this tool to external audiences as it can raise questions about data security. Do not use this tool to request data from a production domain while screen-sharing or presenting to an audience, unless you have specific permission and are in a HIPAA-compliant video conference session. 

Learn more about this tool at the following link.

Data Forwarding

Data forwarders are a lower touch way to manage data integrations with CommCare by relying on these set-it-and-forget-it data exchanges. The Case Data Forwarder is the one responsible for sending cases to new venues.

See the documentation on the Case Data and other Data Forwarding services here.


  • No labels