question

CameronGrimshawJones-8148 avatar image
0 Votes"
CameronGrimshawJones-8148 asked romungi-MSFT commented

Importing/Exporting Data from SQL Server in Live Endpoint

Hello,

I'm currently migrating an experiment & webservice over from AzureML Classic to the new AzureML.

One of the actions the classic webservice did was export data into SQL Server during the run. In the new Azure ML, I have created a new pipeline with an export data module (essentially replicating how it was in classic), however, after reading the docs I'm told this would be removed in a live endpoint? https://docs.microsoft.com/en-us/azure/machine-learning/algorithm-module-reference/export-data

I need to store the data that is output from the pipeline somewhere. What are the best practice solutions to do this?

Give me a shout if you need any more info / I've not explained the problem well enough!

Thanks,

azure-machine-learning
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@CameronGrimshawJones-8148 You can use the convert to dataset module to convert the output of a pipeline to designer's internal format. I think the inference pipeline removes the export data module because of timeout issues to ensure there is no long running operation to export data with an endpoint. Using the convert to dataset module might help you but I am not sure if the same functionality as classic service to pull data from a SQL server will work with a inference pipeline.


0 Votes 0 ·

Thanks - i'll give that a go, that might solve the issue of exporting the data. Would you expect another Azure service (possibly ADF?) to move that dataset across to SQL Server, or is there a better way you know of?



In terms of importing data directly from SQL server - that seems a little more tricky. I guess the best practice way would be just to pass all the data that's required directly in the web service input?

0 Votes 0 ·
romungi-MSFT avatar image romungi-MSFT CameronGrimshawJones-8148 ·

You can use the Azure ML pipeline to move the data from a blob to Azure SQL database.
Yes, using the required input to your endpoint should help it return a response faster rather than query a database which could timeout the request to the endpoint.


0 Votes 0 ·

0 Answers