2.0 Message Ordering

The general reasons behind the need for message ordering are:

• guaranteeing the order of messages 
  • controlling the correct order is important to stay in sync between sending and receiving registry
  • some messages only make sense if processed in the correct order
• making transparent (no) messages are missing
  • in a perfect world this would not be needed and the current timestamps + reference codes would be sufficient. However e.g. in the case that the receiving registry collects messages from the queue, but looses those messages afterwards in a fully automated system, they will not know (in all cases) messages are missing, and even if they do, they might not know which ones. 
  • proper sequencing will make error handling and issue tracing easier

There are several possibilities to realize this:

Option A):

intrinsic ordering only:

Sender Responsibility: The sender is responsible for ensuring the desired order of calls to the Connect APIs.  If api calls are to be scheduled / batched, it is suggested that the sender maintain a queue of outgoing calls / messages so that message sequence is persistent in the event of system failure.

Receiver Responsibility:  The receiver is responsible for monitoring the availableMessages queue and process retrieval calls to the Connect APIs in the order specified.

Pro:

• no additional information needed (ordering based on timestamps and incoming order SMC service)

Con:

• only implicit ordering
• no explicit indicator on missing messages on receiver side

Option B):

ordering via local sequence number:

Sender Responsibility: The sender has to add a receiver specific sequence to each and every message.  

Receiver Responsibility: The sequence number would be part of each message itself . The receiver has to check that the sequence number is correct (no number missing) and only process the messages if that is ok. Otherwise go back to the central system and claim the missing message.

Pro:

• most detailed information as the number is part of the whole communication proess

Con:

• alot of overhead in the communication
• extra work at each registry
• each registry has to maintain a sequence number for each new registry (peer to peer behaviour)
  • extra work for each registry when a new registry is added to the system
• error prone as independent implementations are needed and no control by the centralized system

Option C):

central sequence number provided by SMC:

option C1: one sequence per receiving registry

option C2: one number per sending / receiving registry combination

Sender Responsibility:  

The sender is responsible for ensuring the desired order of calls to the Connect APIs.  If api calls are to be scheduled / batched, it is suggested that the sender maintain a queue of outgoing calls / messages so that message sequence is persistent in the event of system failure. Make sure the next message is sent only when the status of the previous one has been marked OK by SMC.

Central system:

Central system provides the sequence number as part of the meta block as answer in each connect message (messages to another Registry).

Option: not add it to the message_response (and others)?

Receiver Responsibility: 

The sequence number would be part of the meta block. The receiver has to check that the sequence number is correct (no numbers missing) and only process the messages if that is ok. Otherwise go back to the central system and claim the missing message.

Pro:

• avoid peer-to peer problem
• least effort needed on registry side
• receiver can re-request "missing" message
  • can be automated, no help needed by WMDA staff
• allows for different levels of details (C1 vs. C2)

Con:

• additional effort on WMDA side
  • new endpoint for re-request specific message
  • handling of the sequences
  • storing registry to registry messages centralized for a time to make re-request possible. 

2.0.1

How to collect all messages at keep the sequence:

Options:

to allow external registries to access those messages there are several options:

A) 

"the current suggested situation":

• 1) call /availableMessages endpoint and then collect the messages from the endpoints with waiting messages
• 2) or collect messages from all endpoints and accept that some are empty

To keep the ordering (aside from timestamp) the above suggested sequence per receiving registry should be introduced.

The registries has to separate collecting from processing in order to keep the sequence of processing right.

B)

"meta-queue":

• 1) introduce the sequence per receiving registry to /availableMessages endpoint and list all messages per specific queue, not only give the sum
• 2) collect those messages based on the sequence number by going round the specific endpoints

This would allow a registry to collect and process messages in the right order without storing them locally and sort them there

C)

"All-in-One Queue":

• provide a new endpoint that makes use of the "oneOf" tag and provides all messages for external registries
• here the sequence is not needed for ordering anymore but still needed to realize missing messages

This would allow a registry to call just one endpoint to collect all messages.

Limitations:

Option C) doesn't work for all applications/tooling, like defining custom API connectors in the 365 PowerPlatform. In those cases the same static structure for all call responses on a specific endpoint is not only IBP but also required.

As on WMDA side (propably) all messages are stored in one queue/store anyhow (check with Mark) options A-C) will only effect the exposure of the messages to the registries and not the internal workings on WMDA side.

So implementing the options next to each other might give the registries the choice to pick there prefered solution and might still be fine for WMDA?!?

Recommendation:

implement B and C

As a consequence of implementing B) all individual endpoint will have an optional parameter "sequenceNumber" that will function as a filter so that you only get the message with that sequence from the endpoint, without that parameter the endpoints will function as is.

2.1 Rules

• The Match-Connect technical specifications consist of the API documentation and the documents listed below. When developing a Match-Connect implementation, all content of the following documents is relevant and needs to be considered:
  • The Semantics of Match-Connect Messages
  • Match-Connect Data Dictionary
• A Donor_DIFF block is embedded in every result message, i.e., TYP_RES, IDM_RES, SMP_RES, RSV_RES, or NO_RES message, as the donor record changes in its source.
  • More generally: every time data on a match list provided by donor upload changes, a DIFF upload becomes necessary to update the master data.
• The term" donor" in this document usually means "adult donor", "cord blood unit", or "adult donor cryopreserved unit." 
• The institution paying (INST_PAY) has to be of the institution type (INST_TYPE) FIN.
• Empty/blank/missing fields in the payload mean deletion. 
  • Consequently, empty/blank integer or float values must be provided as null values.
• The WMDA nomenclature files (http://hla.alleles.org/wmda/index.html) must be used for all HLA fields. The central hub and/or the receiving registry may reject messages with invalid values. The details are described in the WMDA Guidelines for use of HLA Nomenclature (https://www.nature.com/articles/bmt201393; follow the link" Supplementary information").
• The Match-Connect semantics for the WMDA-approved additional codes are defined as follows: 

• HLA changes of a patient must not cancel requests for the patient. The requests remain under the full responsibility of the patient registry.
• Duplicate requests on the same day: This issue becomes complicated if SMP_REQs are concerned - sometimes users want to" correct" their previous request (i.e., forgot to request quantity and product). The correct way to do this is to cancel the erroneous first request and send the second one later. However, this procedure might also confuse if not carried out on the same working day. In doubt, a phone call helps to sort things out.
• MSG_DEN, WARNING, and REMARK content from the message response messages as well as the TXT_MSG messages, have to be presented to the responsible user (search coordinator) by the local Match-Connect administrator in an appropriate manner.
• Don't send information multiple times - WARNINGs are to be used if it happens in order to find errors.
• A donor can have only one open request at a time. Multiple *_REQ messages for the same patient-donor pair are invalid and must be rejected by the receiving registry. If a request has to be corrected, it has to be canceled (REQ_CAN) and subsequently re-requested ( *_REQ)
• Multiple *_RES or *_ARR messages for the same request (i.e., the same message ID) are regarded as updates and should therefore differ from the previously sent one.
• Do not change the IDs of patients, donors, or addresses: use the primary keys of your database tables prefixed by your unique registry code (in EMDIS, known as hub code) as Match-Connect IDs.
• Return the data as you've received it. For example, return the original data with the answer if you had to change incoming data due to national rules.
• Announce changes in your national system, for example, sending/accepting more Match-Connect messages.
• Provide SMP_RES or NO_RES for all samples received, even if the donor is reported as unavailable in the meantime. Closing a request formally by sending an appropriate result message is essential (for result reminders and invoicing) because the donor often shows up as AV again.

2.2 How to read the new semantics if you know EMDIS

As one can see from the first sentence above, the API documentation is part of the new documentation set. This API documentation is called "swagger" for shortness. It is located at this publicly available address: https://apispecs.wmda.info/

At this address one can find both Search & Match API specifications and Connect API specifications:

Since the communication is now split to communication with WMDA's Search & Match and messaging with other registries, each API request in the Semantics may be represented in two styles:

1. Communication with S&M
   
   1. Most of the Patient API endpoints and all Search API endpoints consist of such messages.
2. Messaging with other registries
   
   
   1. One Patient API and all Connect APIs are represented in this style.

Wherever suitable, we included EMDIS names of the messages in parentheses, e.g. 5.1 - Extended Typing Request (TYP_REQ).

In many cases, the current implementation of EMDIS messages will require a conversion of the flat EMDIS message structure into the nested (multi-level) structure of the API messages. All the fields in API have better names: e.g., REQ_DATE becomes requestDate.

EMDIS rules from the EMDIS Semantics are modified accordingly to the new reality and are added to each message, where applicable.

The table below shows recommended sets of API endpoints to be implemented depending on your use of services:

Service in use API endpoint \	Comment	Recommended for Patient & Search	Recommended for Patient & Search, download Registry	Mandatory for Connect	Mandatory for Connect, Download Registry	Comment
SEARCH & MATCH
PATIENTS
​/patients [POST]	Create a patient	yes	no	yes (PR)	no
​/patients [PUT]	Update an existing patient	no	no	yes (PR)	no
​/patients [GET]	Retrieved patients list	no	no	no	no	Search and Match Service Graphical User Interface (GUI) content, optional for participating organisations
​/patients​/{wmdaId} [GET]	Retrieve	no	no	no	no	Search and Match Service GUI content, optional for participating organisations
/patients/status [PUT]		yes	no	no	no
​/patients​/user [PUT]		no	no	no	no	Search and Match Service GUI content, optional for participating organisations
SEARCH
/searches [POST]		yes	no	yes (PR)	no	for recommendations see semantics
/searches/patientSearches/{wmdaId} [GET]		yes	no	yes (PR)	no
/searches/refreshAllPatientSearches [POST]		no	no	no	no	Search and Match Service GUI content, optional for participating organisations
/searches/{searchId} [GET]		no	no	no	no	highly recommended
/searches/refreshAllPatientSearches [POST]		no	no	no	no
/searches/searchResults/donors [POST]		yes	no	yes (PR)	no	at least one of those is mandatory,
/searches/searchResults/cbus [POST]
/searches/{searchId}/searchResults/adcu [GET]
​/searches​/{searchId}​/searchResults​/registries [GET]		no	no	no	no	Search and Match Service GUI content, optional for participating organisations
/searches/{searchId}/searchResults/cbbs [GET]		no	no	no	no	Search and Match Service GUI content, optional for participating organisations
/searches/selected/donors [POST]		no	no	no	no	Search and Match Service GUI content, optional for participating organisations
​/searches​/selected​/cbus [POST]		no	no	no	no	Search and Match Service GUI content, optional for participating organisations
/searches/donors/{searchResultsId} [GET]		no	no	no	no	Search and Match Service GUI content, optional for participating organisations
​/searches​/cbus​/{searchResultsId} [GET]		no	no	no	no	Search and Match Service GUI content, optional for participating organisations
DASHBOARD
for all		no	no	no	no	Search and Match Service GUI content, optional for participating organisations
CONNECT API
ATTACHMENT
​/api​/v1​/attachmentTicket [POST]				no	no	Optional for participating organisations
/api/v1/attachmentDownloadURL [POST]				no	no	Optional for participating organisations
​/api​/v1​/attachmentDownloadedNotification [POST]				no	no	Optional for participating organisations
RETRIEVE
​/api​/v1​/availableMessages [POST]				yes	yes	check with Mark
ADMIN
/api/v1/alertRetrieve [GET]				no	no	recommended
/api/v1/alertUpdateRetrieve [GET]				no	no	recommended
PATIENT
/api/v1/updateRegisteredPatient [POST]				yes (PR)	yes (PR)
​/api​/v1​/updateRegisteredPatientRetrieve [GET]				yes (DR)	yes (DR)
PING
​/api​/v1​/pingRequest [POST]				no	no	Optional for participating organisations
/api/v1/pingRetrieve [POST]				no	no	Optional for participating organisations
TEXT MESSAGE
/api/v1/textMessageRequest [POST]				no	no	Optional for participating organisations
​/api​/v1​/TextMessageRetrieve [GET]				no	no	Optional for participating organisations
REQUEST
/api/v1/genericRequestRequest [POST]				no	no
/api/v1/genericRequestRetrieve [GET]				no	no
/api/v1/extendedTypingRequestRequest [POST]				yes (PR)	yes (PR)
/api/v1/extendedTypingRequestRetrieve [POST]				no	no	recommended for DCs with incompletely typed donors
/api/v1/extendedTypingResponseRequest [POST]				no	no	recommended for DCs with incompletely typed donors
/api/v1/extendedTypingResponseRetrieve [POST]				yes (PR)	yes (PR)
​/api​/v1​/sampleRequestRequest [POST]				yes (PR)	yes (PR)
​/api​/v1​/sampleRequestRetrieve [POST]				yes (DR, CBB)	yes (DR, CBB)
​/api​/v1​/sampleInfoRequest [POST]				yes (DR, CBB)	yes (DR, CBB)
​/api​/v1​/sampleInfoRetrieve [POST]				yes (PR)	yes (PR)
​/api​/v1​/sampleArrivalRequest [POST]				yes (DR, CBB)	yes (DR, CBB)
​/api​/v1​/sampleArrivalRetrieve [POST]				yes (PR)	yes (PR)
/api/v1/sampleResponseRequest [POST]				yes (PR)	yes (PR)
/api/v1/sampleResponseRetrieve [POST]				yes (DR, CBB)	yes (DR, CBB)
/api/v1/infectiousDiseaseMarkerRequest [POST]				yes (PR)	yes (PR)
/api/v1/infectiousDiseaseMarkerRequestRetrieve [POST]				yes (DR)	yes (DR)
​/api​/v1​/infectiousDiseaseMarkerResultRequest [POST]				yes (DR)	yes (DR)
/api/v1/infectiousDiseaseMarkerResultRetrieve [POST]				yes (PR)	yes (PR)
​/api​/v1​/reservationRequestRequest [POST]				yes (PR)	yes (PR)	under discussion in user group
​/api​/v1​/reservationRequestRetrieve [POST]				yes (DR)	yes (DR)
​/api​/v1​/reservationResponseRequest [POST]				yes (DR)	yes (DR)
/api/v1/reservationResponseRetrieve [POST]				yes (PR)	yes (PR)
​/api​/v1​/reservationReleaseRequest [POST]				yes (PR)	yes (PR)
​/api​/v1​/reservationReleaseRetrieve [POST]				yes (DR)	yes (DR)
/api/v1/requestCancellationRequest [POST]				yes (PR)	yes (PR)
/api/v1/requestCancellationRetrieve [POST]				yes (DR)	yes (DR)
​/api​/v1​/requestRejectedRequest [POST]				yes (DR)	yes (DR)
/api/v1/requestRejectedRetrieve [POST]				yes (PR)	yes (PR)
​/api​/v1​/resultReminderRequest [POST]				no	no	recommended
/api/v1/resultReminderRetrieve [POST]				no	no	recommended
GENERAL
/api/v1/messageResponse [POST]				yes	yes
​/api​/v1​/messageResponseRetrieve [POST]				yes	yes
/api/v1/messageAcknowledgementRequest [POST]						all replaced by the messageResponse
/api/v1/messageAcknowledgementRetrieve [POST]
​/api​/v1​/warningRequest [POST]
/api/v1/warningRetrieve [POST]
​/api​/v1​/messageDenialRequest [POST]
/api/v1/messageDenialRetrieve [POST]
CBU
/api/v1/cordBloodUnitReportRequestRequest	on hold			no	no	should be generalised to CBU and ADCUs, etc...
/api/v1/cordBloodUnitReportRequestRetrieve	on hold			yes (CBB)	yes (CBB)
/api/v1/cordBloodUnitReportResponseRequest	on hold			no	no
​/api​/v1​/cordBloodUnitReportResponseRetrieve	on hold			yes (CBB)	yes (CBB)

2.3 API authentication