Brief description of project, scope and aimed deliverables
Currently only full datasets can be uploaded or downloaded. Since only small amounts of donors/cbu actually change or get added/deleted, this is a wasteful exercise, both in capacity for validation, data transfer, and time. The differentials project aims to set up a method to allow for only partial data-uploads that contain only the differentials between the old and the new dataset.
Please note that Differential data-exchange means sending or receiving only the records that have been added, deleted or changed. This in contrary with the term Differences where is meant the (visual) indication if a record has been changed.
Scope includes both UPLOAD (GCD2 ingress) as DOWNLOAD/EXPORT of datasets (GCD2 egress)
1st stage: provide the Differential upload by uploading a Differential File. The whole process is similar to current FULL upload.
2nd stage: An API real time differentials update may be implemented.
- Description and pro's and cons for various methods that allows for differential UPLOADS.
- Technical design of possible solution
- Implemented solution.