Page Comparison

Show export history

This command can be used to view the history of a particular export:

$ commcare-export-utils history --project X --query A.xlsx --output [SQL URL]


| Checkpoint Time            | Batch end date      | Export Complete | Project | Query Filename      | Query MD5                        | Key | CommCare HQ |
| -------------------------- | ------------------- | --------------- | ------- | ------------------- | -------------------------------- | --- | ----------- |
| 2018-08-29T15:18:04.299947 | 2014-08-30T04:30:48 | True            | X       | A.xlsx              | 9043f06f8cb4b26871ebd939609ca01e |     | prod        |
| 2018-08-29T15:18:01.553475 | 2014-08-29T09:45:32 | False           | X       | A.xlsx              | 9043f06f8cb4b26871ebd939609ca01e |     | prod        |
| 2018-08-29T15:17:59.300238 | 2014-08-28T05:13:17 | False           | X       | A.xlsx              | 9043f06f8cb4b26871ebd939609ca01e |     | prod        |
| 2018-08-29T15:17:57.151183 | 2014-08-27T14:36:38 | False           | X       | A.xlsx              | 9043f06f8cb4b26871ebd939609ca01e |     | prod        |
| 2018-08-29T15:17:54.914646 | 2014-08-26T15:41:11 | False           | X       | A.xlsx              | 9043f06f8cb4b26871ebd939609ca01e |     | prod        |
| 2018-08-29T15:17:52.771408 | 2014-08-25T21:46:47 | False           | X       | A.xlsx              | 9043f06f8cb4b26871ebd939609ca01e |     | prod        |

User defined export key for checkpointing

Normally the export tool uses the MD5 hash of the query file as the checkpoint key i.e. when starting up it will query the database using the MD5 hash to find the last checkpoint time.

...

Note that when running with a user supplied key the export tool will never consider the MD5 hash and therefore any re-processing of older data will need to be handled manually.

Converting a non-keyed export to a keyed export

To convert an export that has been using the MD5 hash as the key into one that uses a custom key run the following command:

...

Note that this will use the MD5 hash of the query file to find the checkpoints that need to updated.

Saving data as JSON

This feature is only available on Postgres databases.

You can save data as a "JSON" column. This is particularly useful when attempting to export all form or case data regardless of which fields are present - for example, if you are trying to create a simple data lake for your CommCare data.

To export all form contents as JSON, create a  column in a case export file with a "Source Field" of "form" and a data type column set to "json".

To export all case properties as JSON, create a column in a case export file with a "Source Field" of "properties" and a data type column set to "json".

This data can then be queried using Postgres JSON queries. For example:

select form->'@name' as form_name, form->'meta'->'username' as username, form->'meta'->'timeStart' as time_start from form_lake limit 5;

  form_name  |     username      |          time_start           
-------------+-------------------+-------------------------------
 "New Event" | "alice"           | "2017-06-24T16:24:07.676000Z"
 "New Event" | "bob"             | "2017-07-02T13:43:52.655000Z"
 "New Event" | "chelsea"         | "2017-07-04T07:51:12.528000Z"
 "New Event" | "daniel"          | "2017-07-09T06:10:48.591000Z"
 "New Event" | "edith"           | "2020-04-02T05:58:11.712000Z"

Next Page:CommCare Sync >