• 2To be updated
• 3Purpose
• 4Using the REST API
  • 4.1SAMPLE Request
• 5Upload
  • 5.1Suggestion 1:
  • 5.2Suggestion 2:
• 6FAQ
• 7Download
  • 7.1Download full dataset

To be updated

Purpose

Upload and download files in Data Manager can be done using various mechanisms : 

• Web-based interface at https://sandbox-datamanager.wmda.info (please note the URL for production is https://datamanager.wmda.info )
• REST API, using commands like cURL, or libraries like Restsharp

This document describes the Data Manager mechanism via the REST API method, where users can upload a PGP encrypted XML file.

# example use:
curl --netrc-file mycredentialsfile https://xyz.com

# the mycredentialsfile contains :
machine <xyz.com> login <johndoe@xyz.com> password <uf9873o^9ufwa>

Using the REST API

Step 1: Request the credential which can be used to generate the Bear Token 

Step 2: Test to use the API in WMDA 

To test the API you may use a tool like Restlet or Postman, a plugin for Chrome browsers to test API calls before you deploy. The server responds with a HTTP code 200 in when successful, but the response body may contain detailed error messages.

In case the upload succeeded, an XML response with meta information of the upload is generated. You may use this for own logging purposes. For more info you can read the full API specification.

• Server name should be https://staging-dataupload.wmda.info
• For production you will need https://dataupload.wmda.info

SAMPLE Request

First test the connection by using the right path and fetching the metadata : it should result in a response 200 OK.

• Look carefully at the path : here we use ion0999, that should be replaced by your ION.
• You may notice that we add the filename to the path: depending on the library you may need to add that

Upload

Now we know the path is correct and we can do an upload using cURL: 

Below is the invalid to deploy, currently it is still available in PRODUCTION till production maintenance. But not available any more in STAGING dataupload. 

curl -i -X PUT -H "Authorization:Basic d21.....uZT=="   -T "./test.pgp"  'https://staging-dataupload.wmda.info/api/v2/io/ION0999/'

Below are suggestions with more secure headers:

Suggestion 1:

For all the files, script as below is suggested for file larger than 128K.

curl -i -X PUT -H "Authorization:Basic d21.....uZT==" -H "Content-Type: multipart/form-data" --data-binary "@/path/to/file/ION-0999-D.gpg"  "https://staging-dataupload.wmda.info/api/v2/io/ION0999/" -H "X-File-Name:ION-0999-D.gpg"

curl -i -X PUT -H "Authorization:Basic d21.....uZT==" -H "Content-Type: multipart/form-data" --data-binary "@c:/path/to/file/ION-0999-D.gpg"  "https://staging-dataupload.wmda.info/api/v2/io/ION0999/" -H "X-File-Name:ION-0999-D.gpg"

Suggestion 2:

For files that encrypted and identified as PGP/GPG MESSAGE (open the encrypted file and you can see this info), then -F and -H options also work. -F already included "Content-Type: multipart/form-data", so no need to provide it any more.

For binary file, extra header information will be added to the content of the file when file is uploaded, and then it can not be decrypted any more, so please use suggestion 1 to upload. 

curl -i -X PUT -H "Authorization:Basic d21.....uZT==" -F "file="@/path/to/file/ION-0999-D.gpg" "https://staging-dataupload.wmda.info/api/v2/io/ION0999/" -H "X-File-Name:ION-0999-D.gpg"

curl -i -X PUT -H "Authorization:Basic d21.....uZT==" -F "file="@c:/path/to/file/ION-0999-D.gpg" "https://staging-dataupload.wmda.info/api/v2/io/ION0999/" -H "X-File-Name:ION-0999-D.gpg"

FAQ

Question : My report states : File could not be processed due to a file decryption error, when using the API

Answer :

Download

Download reports

You may use the API also to fetch your reports . Unfortunately, it is not one line cmd, and need 2 steps. 

1.Fetch the files list and get the file names.

Use the following curl and endpoint to fetch an array with filedescriptors: url should use double quote or no quote in windows env:

curl -H "Authorization:Basic d21.....uZT==" https://staging-dataupload.wmda.info/api/v2/fs/reports-ion0999/?children=f

You will get the result in xml format, and you can get json format,  use the cmd  below:

curl -H "Authorization:Basic d21.....uZT==" https://staging-dataupload.wmda.info/api/v2/fs/reports-ion0999/?children=f&format=json

2.  Fetch all the reports or the one you need.

Note, you need to use "io" instead of "fs" in the path.

curl -O GET -H "Authorization:Basic d21.....uZT==" 'https://staging-dataupload.wmda.info/api/v2/io/reports-ion0999/$filename' -o '$filename.txt'

In windows cmd, the -o dose not work, and the cmd is as below:

curl -H "Authorization:Basic d21.....uZT==" https://staging-dataupload.wmda.info/api/v2/io/reports-ion0999/$filename > $filename.txt

Download archived files

Download archived files will be the same as download reports. And 2 steps as above are needed.

The example URL for archive folder is as below:

https://staging-dataupload.wmda.info/api/v2/fs/archive-ion0999/?children=f

In RESTLET :

                       

Click on Code to get raw CURL syntax:

Response should be 200:

Download full dataset

For those registries that are permitted to use the full dataset the statement below will fetch that. Please change the ION workspace to match your ION.

curl -O GET -H "Authorization:Basic .............." 'https://dataupload.wmda.info/api/v2/io/downloads/ION1804/bmdw4data.zip.gpg' -o 'wmda_data_v22.zip.gpg'

In windows cmd, the -o dose not work, and the cmd is as below:

curl -H "Authorization:Basic .............." "https://dataupload.wmda.info/api/v2/io/downloads/ION1804/bmdw4data.zip.gpg" > wmda_data_v22.zip.gpg 

We noticed sometimes the download maybe terminated because of some temp network limitation, and the "-C -" option can be used to continue the download.
-v is for more details of the track info

curl -v -H "Authorization:Basic .............." -o 'wmda_data_v22.zip.gpg' -C - 'https://dataupload.wmda.info/api/v2/io/downloads/ION1804/wmda_data_v22.zip.gpg'