| Show If | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Panel | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Project Name | Project Differentials | Author | Project manager / owner | Submitted | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Project group members: | Approval status | APPROVED | Project office supporter: | Approval date | Status |
| Progress Bar | Vectors (Formerly: SP Progress bar) | ||||
|---|---|---|---|---|
|
Start date:
End date:
Brief description of project, scope and aimed deliverables
Currently only full datasets can be uploaded or downloaded. Since only small amounts of donors/cbu actually change or get added/deleted, this is a wasteful exercise, both in capacity for validation, data transfer, and time. The differentials project aims to set up a method to allow for only partial data-uploads that contain only the differentials between the old and the new dataset.
Please note that Differential data-exchange means sending or receiving only the records that have been added, deleted or changed. This in contrary with the term Differences where is meant the (visual) indication if a record has been changed.
Scope includes both UPLOAD (GCD2 ingress) as DOWNLOAD/EXPORT of datasets (GCD2 egress)
1st stage: provide the Differential upload by uploading a Differential File. The whole process is similar to current FULL upload.
2nd stage: An API real time differentials update may be implemented.
Deliverables are
- Description and pro's and cons for various methods that allows for differential UPLOADS.
- Technical design of possible solution
- Implemented solution.
| borderColor | lightgray |
|---|---|
| borderWidth | 1 |
| borderStyle | solid |
| title | Differential Survey |
| borderColor | lightgray |
|---|---|
| borderWidth | 1 |
| borderStyle | solid |
| title | Differential upload User Guide |
The 1st stage differential upload is to upload a .gpg file to add, update or delete records. The existing UPDATE_MODE and STATUS fields are used to give information to do differential upload.
| Table of Contents |
|---|
1.Organizations allowed for DIFF upload
- The registries that upload donors with GRID. GRID is compulsory for DIFF.
- All registries and CBBs upload CBUs
2.How to create the .gpg file
- Following the same process in the FULL upload user guide Search & Match Service Data Submission Information to prepare the differential upload .gpg file.
- The DIFF upload needs include all the fields for a DONOR/CBU exactly like the FULL upload.
Changes comparing to current FULL upload (Will later be updated in the FULL upload user guide)
Field Identifier Required Description Type Length Comment UPDATE_MODE Yes Update mode of the inventory, i.e. FULL or DIFF updateModeType 4 "FULL" means for full upload
"DIFF" means differential uploadSTATUS Yes Status of the donor/CBU statusType 2 statusType: "AV" ,"TU" ,"RS", "DE" *
AV = Available for transplantation purposes
TU = Temporarily unavailable
RS = Reserved
DE = Deleted, permanently unavailable**DE is only supported in DIFF upload, will be rejected in FULL upload.
2.Expected behavior
| Differential Update cases | File level | DONOR Behavior (GRID is compulsory) | CBU Behavior |
|---|---|---|---|
| Upload frequency | Upload frequency must be more than 15mins for each DIFF upload. Otherwise, only one file in each 15mins will be processed. | ||
| Add new records when STATUS is "AV" or "TU" or "RS". |
|
| |
| Update existing records when STATUS is "AV" or "TU" or "RS" |
|
| |
| Deleted records when STATUS is "DE" |
|
| |
| Upload records threshold limitation | Less than 100K for each DIFF upload. (Can request for more than 100K DIFF upload if needed, please contact support@wmda.info) |
3. Business validation rules
| Reference number | Validation level | Validation Source | Date rule is valid | Effective since XSD version number | Field Name | Error in field | Reported Validation message | Action |
|---|---|---|---|---|---|---|---|---|
| 9 | File | 2.1 | UPDATE_MODE | Invalid update mode | Your file has been rejected as the UPDATE_MODE must be equal to "FULL" or "DIFF". | Reject file | ||
| 222 | File | WMDA | 2.1 | UPDATE_MODE | Invalid update mode for multiple inventories | Your file has been rejected as we have identified mixed update modes in your XML inventories. | Reject file | |
| File | WMDA | 2.1 | N/A | Too many record provided for DIFF upload | File could not be processed due to DIFF upload exceeded record count threshold: 50000 | Reject file |
4. Errors and warnings in the processing report
Report
| Code Block | ||||
|---|---|---|---|---|
| ||||
File: ION-0999-D.gpg 2019-09-09 11:39:22 Pool(s): 0999 Content Type: D Update Mode: DIFF Start processing: 2019-09-09 11:50:00 Schema version: 2.1 Total records processed: 14 Total records with warnings: 6 Total records rejected: 4 Total valid records: 7 Total updated records: 4 Total new records: 4 Total deleted records: 2 List of Records with duplicated ID or GRID: POOL: 0999 IDs: TD-000004 GRIDs: 1234000000000203420 W | 0999 | TD-000002 | N/A | (Warning) GRID 774800006001853603 must be 19 characters. R | 0999 | TD-000002 | N/A | GRID (Record Rejected) GRID is a mandatory field. W | 0999 | TD-000003 | N/A | (Warning) GRID 7748000060018E53612 checksum is not correct. R | 0999 | TD-000003 | N/A | GRID (Record Rejected) GRID is a mandatory field. W | 0999 | TD-000010 | 1234000000000001031 | STAT_END_DATE (Warning) Status end date cannot be > 5yrs in the future. W | 0999 | TD-000010 | 1234000000000001031 | STAT_REASON (Warning) Status reason cannot be provided with status AV or RS. R | 0999 | TD-000013 | 999900000TD00001129 | BIRTH_DATE (Record Rejected) BIRTH_DATE is a mandatory field. W | 0999 | TD-000014 | N/A | (Warning) GRID must be 19 characters. R | 0999 | TD-000014 | N/A | GRID (Record Rejected) GRID is a mandatory field. Processing finished at: 2019-09-09 11:53:55 Total processing time: 0 minutes. |
The processing report for differential upload has 3 parts.
- 1st part is the summary of the file upload, with more details of updated, new, and deleted records.
- 2nd part is the duplication in the uploaded file, data uploader should clean up the duplication.
- 3rd part is the rejections and warnings. Data uploader should clean up all of them.
Summary numbers:
We define some cases:
| Cases | |
|---|---|
| DE non exist, ignored | This is the records with "DE" status, but they do not exist yet in WMDA database. This will be ignored. |
| GRID Missing, rejected | GRID is compulsory for DIFF and FULL from Dec, 17, 2019. So the message is Warning (W) type, and also Rejection (R) type. Example: W | 0999 | TD-000014 | N/A | (Warning) GRID must be 19 characters.
|
| Calculation | Total records = Total records rejected+Total valid records+Total deleted records+DE non exist |
New Record Errors/warnings:
| Type | Message | Description |
|---|---|---|
...