Importing/Exporting Data from SQL Server in Live Endpoint

Cameron Grimshaw-Jones 11

Hello,

I'm currently migrating an experiment & webservice over from AzureML Classic to the new AzureML.

One of the actions the classic webservice did was export data into SQL Server during the run. In the new Azure ML, I have created a new pipeline with an export data module (essentially replicating how it was in classic), however, after reading the docs I'm told this would be removed in a live endpoint? https://learn.microsoft.com/en-us/azure/machine-learning/algorithm-module-reference/export-data

I need to store the data that is output from the pipeline somewhere. What are the best practice solutions to do this?

Give me a shout if you need any more info / I've not explained the problem well enough!

Thanks,

romungi-MSFT 41,961 Reputation points Microsoft Employee

2021-11-03T07:03:06.433+00:00

@Cameron Grimshaw-Jones You can use the convert to dataset module to convert the output of a pipeline to designer's internal format. I think the inference pipeline removes the export data module because of timeout issues to ensure there is no long running operation to export data with an endpoint. Using the convert to dataset module might help you but I am not sure if the same functionality as classic service to pull data from a SQL server will work with a inference pipeline.
Cameron Grimshaw-Jones 11 Reputation points

2021-11-03T10:33:58.02+00:00

Thanks - i'll give that a go, that might solve the issue of exporting the data. Would you expect another Azure service (possibly ADF?) to move that dataset across to SQL Server, or is there a better way you know of?

In terms of importing data directly from SQL server - that seems a little more tricky. I guess the best practice way would be just to pass all the data that's required directly in the web service input?
romungi-MSFT 41,961 Reputation points Microsoft Employee

2021-11-10T07:55:04.917+00:00

You can use the Azure ML pipeline to move the data from a blob to Azure SQL database.
Yes, using the required input to your endpoint should help it return a response faster rather than query a database which could timeout the request to the endpoint.