DET Advanced Features
- Simon Kelly
- Gillian Javetski
- Cory Zue
- Former user (Deleted)
These feature only apply to SQL exports
Show export history
This command can be used to view the history of a particular export:
$ commcare-export-utils history --project X --query A.xlsx --output [SQL URL]
| Checkpoint Time | Batch end date | Export Complete | Project | Query Filename | Query MD5 | Key | CommCare HQ |
| -------------------------- | ------------------- | --------------- | ------- | ------------------- | -------------------------------- | --- | ----------- |
| 2018-08-29T15:18:04.299947 | 2014-08-30T04:30:48 | True | X | A.xlsx | 9043f06f8cb4b26871ebd939609ca01e | | prod |
| 2018-08-29T15:18:01.553475 | 2014-08-29T09:45:32 | False | X | A.xlsx | 9043f06f8cb4b26871ebd939609ca01e | | prod |
| 2018-08-29T15:17:59.300238 | 2014-08-28T05:13:17 | False | X | A.xlsx | 9043f06f8cb4b26871ebd939609ca01e | | prod |
| 2018-08-29T15:17:57.151183 | 2014-08-27T14:36:38 | False | X | A.xlsx | 9043f06f8cb4b26871ebd939609ca01e | | prod |
| 2018-08-29T15:17:54.914646 | 2014-08-26T15:41:11 | False | X | A.xlsx | 9043f06f8cb4b26871ebd939609ca01e | | prod |
| 2018-08-29T15:17:52.771408 | 2014-08-25T21:46:47 | False | X | A.xlsx | 9043f06f8cb4b26871ebd939609ca01e | | prod |User defined export key for checkpointing
Normally the export tool uses the MD5 hash of the query file as the checkpoint key i.e. when starting up it will query the database using the MD5 hash to find the last checkpoint time.
This means that if the query file is changed the MD5 will also change the the tool will not be able to find the last checkpoint which will cause it to re-export all the data. In most cases this is what's desired since change to the query file require re-processing the data however in certain scenarios it may be more desirable to continue using the previous checkpoints.
For example, if a new column is added to a form export query file which represents a new field in the form, it is not necessary to re-process all the old data since that field doesn't exist anyway.
To accommodate these use cases the export tool can be made to ignore the MD5 hash of the query file and instead use a key supplied on the command line:
$ commcare-export --project X --query A.xlsx --output [SQL URL] --checkpoint-key my-key ...Note that when running with a user supplied key the export tool will never consider the MD5 hash and therefore any re-processing of older data will need to be handled manually.
Converting a non-keyed export to a keyed export
To convert an export that has been using the MD5 hash as the key into one that uses a custom key run the following command:
$ commcare-export-utils set-key --project X --query A.xlsx --output [SQL URL] --checkpoint-key my-keyNote that this will use the MD5 hash of the query file to find the checkpoints that need to updated.
Saving data as JSON
This feature is only available on Postgres databases.
You can save data as a "JSON" column. This is particularly useful when attempting to export all form or case data regardless of which fields are present - for example, if you are trying to create a simple data lake for your CommCare data.
To export all form contents as JSON, create a column in a case export file with a "Source Field" of "form" and a data type column set to "json".
To export all case properties as JSON, create a column in a case export file with a "Source Field" of "properties" and a data type column set to "json".
This data can then be queried using Postgres JSON queries. For example: