The 1st stage differential upload is to upload a .gpg file to add, update or delete records. The existing UPDATE_MODE and STATUS fields are used to give information to do differential upload.
1.Organizations allowed for DIFF upload- The registries that upload donors with GRID. GRID is compulsory for DIFF.
- All registries and CBBs upload CBUs
2.How to create the .gpg file 2.Expected behavior
Differential Update cases | File level | DONOR Behavior (GRID is compulsory) | CBU Behavior |
---|
Upload frequency | Upload frequency must be more than 15mins for each DIFF upload. Otherwise, only one file in each 15mins will be processed. |
|
|
|
|
|
| Add new records when STATUS is "AV" or "TU" or "RS". |
| - Records with duplicated ID in the upload file will be rejected.
- Records with duplicated GRID in the upload file will be rejected.
- Records without GRID will be rejected.
But there is no rejection message, under implementation in next release.
- Invalid records will be rejected.
- Valid records with not existing GRID will be added
| - Invalid records will be rejected.
- Valid records with not existing ID will be added
| Update existing records when STATUS is "AV" or "TU" or "RS" |
| - Records with duplicated ID in the upload file will be rejected.
- Records with duplicated GRID in the upload file will be rejected.
- Invalid records will be rejected, and existing record with same ID and GRID in database will be deleted
- Valid records with existing GRID will be updated
| - Records with duplicated ID in the upload file will be rejected.
- Invalid records will be rejected, and existing record with same ID and GRID will be deleted in the database.
- Valid records with existing ID in DB will be updated
| Deleted records when STATUS is "DE" |
| - Record exists in database will be deleted.
- Record does not exist in the database will be ignored. And no warning message in the report.
| - Record exists in database will be deleted.
- Record does not exist in the database will be ignored.And no warning message in the report.
| Upload records threshold limitation | Less than 100K for each DIFF upload. (Can request for more than 50K 100K DIFF upload if needed, please contact support@wmda.info) |
|
|
3. Business validation rulesReference number | Validation level | Validation Source | Date rule is valid | Effective since XSD version number | Field Name | Error in field | Reported Validation message | Action |
---|
9 | File |
|
| 2.1 | UPDATE_MODE | Invalid update mode | Your file has been rejected as the UPDATE_MODE must be equal to "FULL" or "DIFF". | Reject file | 222 | File | WMDA |
| 2.1 | UPDATE_MODE | Invalid update mode for multiple inventories | Your file has been rejected as we have identified mixed update modes in your XML inventories. | Reject file |
| File | WMDA |
| 2.1 | N/A | Too many record provided for DIFF upload | File could not be processed due to DIFF upload exceeded record count threshold: 50000 | Reject file |
4. Errors and warnings in the processing reportReport Code Block |
---|
theme | Confluence |
---|
title | Example report of Differential upload |
---|
| File: ION-0999-D.gpg 2019-09-09 11:39:22
Pool(s): 0999
Content Type: D
Update Mode: DIFF
Start processing: 2019-09-09 11:50:00
Schema version: 2.1
Total records processed: 14
Total records with warnings: 6
Total records rejected: 4
Total valid records: 7
Total updated records: 4
Total new records: 4
Total deleted records: 2
List of Records with duplicated ID or GRID:
POOL: 0999
IDs:
TD-000004
GRIDs:
1234000000000203420
W | 0999 | TD-000002 | N/A | (Warning) GRID 774800006001853603 must be 19 characters.
R | 0999 | TD-000002 | N/A | GRID (Record Rejected) GRID is a mandatory field.
W | 0999 | TD-000003 | N/A | (Warning) GRID 7748000060018E53612 checksum is not correct.
R | 0999 | TD-000003 | N/A | GRID (Record Rejected) GRID is a mandatory field.
W | 0999 | TD-000010 | 1234000000000001031 | STAT_END_DATE (Warning) Status end date cannot be > 5yrs in the future.
W | 0999 | TD-000010 | 1234000000000001031 | STAT_REASON (Warning) Status reason cannot be provided with status AV or RS.
R | 0999 | TD-000013 | 999900000TD00001129 | BIRTH_DATE (Record Rejected) BIRTH_DATE is a mandatory field.
W | 0999 | TD-000014 | N/A | (Warning) GRID must be 19 characters.
R | 0999 | TD-000014 | N/A | GRID (Record Rejected) GRID is a mandatory field.
Processing finished at: 2019-09-09 11:53:55
Total processing time: 0 minutes. |
The processing report for differential upload has 3 parts. - 1st part is the summary of the file upload, with more details of updated, new, and deleted records.
- 2nd part is the duplication in the uploaded file, data uploader should clean up the duplication.
- 3rd part is the rejections and warnings. Data uploader should clean up all of them.
Summary numbers: We define some cases: Cases |
|
---|
DE non exist, ignored | This is the records with "DE" status, but they do not exist yet in WMDA database. This will be ignored. | GRID Missing, rejected | GRID is compulsory for DIFF and FULL from Dec, 17, 2019. So the message is Warning (W) type, and also Rejection (R) type. Example: W | 0999 | TD-000014 | N/A | (Warning) GRID must be 19 characters. R | 0999 | TD-000014 | N/A | GRID (Record Rejected) GRID is a mandatory field. | Calculation | Total records = Total records rejected+Total valid records+Total deleted records+DE non exist |
New Record Errors/warnings: |