The 1st stage differential upload is to upload a .gpg file to add, update or delete records. The existing UPDATE_MODE and STATUS fields are used to give information to do differential upload.
1.Organizations allowed for DIFF upload
- The registries that upload donors with GRID. GRID is compulsory for DIFF.
- All registries and CBBs upload CBUs
2.How to create the .gpg file
- Following the same process in the FULL upload user guide Search & Match Service Data Submission Information to prepare the differential upload .gpg file.
- The DIFF upload needs include all the fields for a DONOR/CBU exactly like the FULL upload.
Changes comparing to current FULL upload (Will later be updated in the FULL upload user guide)
Field Identifier Required Description Type Length Comment UPDATE_MODE Yes Update mode of the inventory, i.e. FULL or DIFF updateModeType 4 "FULL" means for full upload
"DIFF" means differential uploadSTATUS Yes Status of the donor/CBU statusType 2 statusType: "AV" ,"TU" ,"RS", "DE" *
AV = Available for transplantation purposes
TU = Temporarily unavailable
RS = Reserved
DE = Deleted, permanently unavailable**DE is only supported in DIFF upload, will be rejected in FULL upload.
2.Expected behavior
| Differential Update cases | File level | DONOR Behavior (GRID is compulsory) | CBU Behavior |
|---|---|---|---|
| Upload frequency | Upload frequency must be more than 15mins for each DIFF upload. Otherwise, only one file in each 15mins will be processed. | ||
| Add new records when STATUS is "AV" or "TU" or "RS". |
|
| |
| Update existing records when STATUS is "AV" or "TU" or "RS" |
|
| |
| Deleted records when STATUS is "DE" |
|
| |
| Upload records threshold limitation | Less than 50K for each DIFF upload. (Can request for more than 50K DIFF upload if needed, please contact support@wmda.info) |
3. Business validation rules
| Reference number | Validation level | Validation Source | Date rule is valid | Effective since XSD version number | Field Name | Error in field | Reported Validation message | Action |
|---|---|---|---|---|---|---|---|---|
| 9 | File | 2.1 | UPDATE_MODE | Invalid update mode | Your file has been rejected as the UPDATE_MODE must be equal to "FULL" or "DIFF". | Reject file | ||
| 222 | File | WMDA | 2.1 | UPDATE_MODE | Invalid update mode for multiple inventories | Your file has been rejected as we have identified mixed update modes in your XML inventories. | Reject file | |
| File | WMDA | 2.1 | N/A | Too many record provided for DIFF upload | File could not be processed due to DIFF upload exceeded record count threshold: 50000 | Reject file |
4. Errors and warnings in the processing report
Report
File: ION-0999-D.gpg 2019-09-09 11:39:22 Pool(s): 0999 Content Type: D Update Mode: DIFF Start processing: 2019-09-09 11:50:00 Schema version: 2.1 Total records processed: 14 Total records with warnings: 6 Total records rejected: 4 Total valid records: 7 Total updated records: 4 Total new records: 4 Total deleted records: 2 List of Records with duplicated ID or GRID: POOL: 0999 IDs: TD-000004 GRIDs: 1234000000000203420 W | 0999 | TD-000002 | N/A | (Warning) GRID 774800006001853603 must be 19 characters. R | 0999 | TD-000002 | N/A | GRID (Record Rejected) GRID is a mandatory field. W | 0999 | TD-000003 | N/A | (Warning) GRID 7748000060018E53612 checksum is not correct. R | 0999 | TD-000003 | N/A | GRID (Record Rejected) GRID is a mandatory field. W | 0999 | TD-000010 | 1234000000000001031 | STAT_END_DATE (Warning) Status end date cannot be > 5yrs in the future. W | 0999 | TD-000010 | 1234000000000001031 | STAT_REASON (Warning) Status reason cannot be provided with status AV or RS. R | 0999 | TD-000013 | 999900000TD00001129 | BIRTH_DATE (Record Rejected) BIRTH_DATE is a mandatory field. W | 0999 | TD-000014 | N/A | (Warning) GRID must be 19 characters. R | 0999 | TD-000014 | N/A | GRID (Record Rejected) GRID is a mandatory field. W | 0999 | 9999000000020000629 | (Warning) Record with DE status does not exist. Processing finished at: 2019-09-09 11:53:55 Total processing time: 0 minutes.
The processing report for differential upload has 4 parts.
- 1st part is the summary of the file upload.
- 2nd part is the duplication in the uploaded file, data uploader should clean up the duplication.
- 3rd part is the rejections and warnings. Data uploader should clean up all of them.
Summary numbers:
We define some cases:
DE non exist: This is the records with "DE" status, but they do not exist yet in WMDA database. This will be ignored.
GRID Missing: (Warning) GRID must be 19 characters.
Example:
W | 0999 | TD-000014 | N/A | (Warning) GRID must be 19 characters.
GRID is compulsory for DIFF, but not for FULL upload yet. So the message is Warning (W) type, but for Rejection (R) type yet. Will be improved after implement GRID as compulsory for FULL donor upload in November, 2019.
Total records = Total records rejected+Total valid records+Total deleted records+DE non exist
Total valid records = Total valid records+Total new records
Total records rejected=Records with duplicated ID or GRID in part 2+ R in part 3 + GRID Missing
Record Errors/warnings:
| Type | Message | Description |
|---|---|---|
| RecordWarning | Record with DE status does not exist. | When a record is with STATUS "DE" in differential upload file, but this record dose not exists in database |