How to import SOAP API to Azure data lake

Question

Hi,

I am a recent graduate and Azure data engineering enthusiastic.
As a part of small project, I had a requirement to load organization data as a SOAP API (WSDL url example: https://www.w3schools.com/xml/tempconvert.asmx?WSDL) and loading it into Azure data lake.
I could not find a proper resource for it.

Any suggestions are welcome.

I appreciate your time and consideration.

Regards,
RJ

Answer 1

A data lake is just a repository for data, it has no abilities to call an API. If you want to copy data from your API to the data lake you would need to create some sort of process to do this. You could look at using Azure Functions or Logic Apps to do this, but you would need to build this.

Answer 2

@RJ Azure Data Lake is a cloud platform designed to support big data analytics. It provides unlimited storage for structured, semi-structured or unstructured data. It can be used to store any type of data of any size. As @Sam Cogan mentioned you cannot call an API from azure data lake. To store the data to data lake you need to have the middleware which can read the data from your SOAP API and store the response/data in the azure data lake.

There are couple of services that you can used as the middleware:

• Create logic app custom connector (SOAP) that will be used as SOAP to REST endpoint. Now you can use this REST endpoint as the Request/Response in your logic app workflow to get the response/data and use Azure Data Lake connector to store the data in Azure Data Lake.
• Create logic app custom connector (SOAP) that will be used as SOAP to REST endpoint. Now use azure function trigger (you need to verify whether any trigger can be used for your scenario to trigger the functions) that have the business logic to call the REST endpoint and use Azure Data lake SDK to perform the operation on Azure data lake. Instead of azure function you can use azure webapps or any other service where your custom code will be running to offload this functionality.
• Instead of using logic app custom connectors you can also use APIM to import your SOAP API and expose them as your service and middleware will have the functionality of communicating with REST endpoint and Azure Data lake.
• You can directly call your SOAP service and use Azure Data Lake SDK to store the data and the middleware can be deploy in azure function, webaps etc.