Form Completion versus Form Submission and Timezones

A Note About Dates and Times

Dates and Times can be a little confusing, because there are so many of them.  However, by knowing which times are defined where, you can actually use this plethora of data to do some solid debugging:

• See how long a form took to fill out

• See if the time on the phone is set incorrectly

• See the time between when the form was finished and when the server received it, indicating potential network, GPRS or recharge issues

Here are parameters that are set on the server (ie, the server creates this time, which can usually be trusted):

• server_modified_on - the last time the case was modified (good to compare to date_modified, which is set on the phone) 

• received_on - time the form is received by the server 

Here are parameters that are set on the phone (ie, the phone sets the time):

• timeStart - time phone records that a form is started

• timeEnd - time phone records that a form is finished 

• date_modified - last time the case was modified 

What is the difference between Form Completion and Form Submission?

Each form submitted to CommCareHQ has three date-time stamps:

• Form Start - the time when a form was first opened, according to the phone's time (form|meta|timeStart)

• Form Completion - the time when a form was saved as complete, according to the phone's time (form|meta|timeEnd)

• Server Received ("Form Submission)" - the time when the form was received at the CommCareHQ server (received_on)

The main issue addressed here is the difference between the last two, and deciding which one you should use when viewing reports.

Why would Form Completion and Form Submission be different?

If you are working in an area with very good connectivity, and if the phone time is correct, then Form Completion and Form Submission time will be almost the same. That is, the form will be marked as complete and then immediately sent to the server.

However, sometimes the form completion and submission times may be quite far apart.  There are a couple of reasons that they could be different:

• Phone's time is wrong - form completion time could be either before or after form submission time

• Internet connectivity is limited (forms stay on the phone for a period of time before sending to the server) - form completion time is likely to be much earlier than form submission time

How do I know if they are different?

You can look at the CommCareHQ report "Form Completion vs Submission Trends" or look at the data in a form export.

Which one should I use?

It depends upon the situation in your project. In general, if user's phones are set by the network and tend to be correct, it may be more reliable to use the Form Completion time, especially if mobile users are in an area with poor connectivity. If, for example, a worker fills out 25 forms over the course of a month, but if they have to go somewhere at the end of the month to submit all of them, the submission data will not be reflect the user's actual pattern of form completion. If, however, your project is in an area with very good connectivity, it may be generally easier to use Form Submission.

To get a sense of whether there is a big difference between form completion and submission in your project, go to the reports tab and choose the "Form Completion vs Submission Trends" report.

Does this report use Form Completion or Form Submission times?

Timezones

A CommCare work space's time settings can be configured in the project settings, or at the time of creating a new project.

Most reports will be displayed in the timezone that is selected in the project settings.

Currently CommCareHQ strips all timezone information from all dates in forms. As a result CommCareHQ assumes in reports that dates which come from the phone (such as form completion time) are already in the project's timezone, whereas dates that come from the server (such as submission time) are in the server timezone. 

This leads to discrepancies when you submit from a device that is actually not in the project timezone (including all submissions on cloudcare or case importer, or reassign cases).

CommCareHQ does not do the time shifting in the exports, so data from phones that shows up in the exports will also look offset from the server data. In exports, data from the phone is in the phone's timezone, whereas data from the server such as submission date is in UTC.

How to handle project spaces with multiple countries

• Each web user on the project may set their own perferred timezone that overrides the project timezone when they're looking at reports. If the same user wanted to see India data in IST and South America data in the correct South American timezone, then they would have to change their own preferred timezone override between looking at one location's data and looking at the other. If there are certain people who only look at India data or only look at South America data then this could be helpful for them

• When you export data to excel, date/times will always be in UTC / +00:00 timezone / "zulu time", regardless of the project's timezone. It was decided to do this for exactly this reason: if you have different timezones at play, then as much as a pain as it is, it's still much easier if everything is consistently reported in this standard time than if it's unclear which timezone a particular date/time is in.