Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,182 questions
Import Data component in Azure ML Designer
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 45%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGA%3C/text%3E%3C/svg%3E)
Gustavo Avino
0
Reputation points
Hi, I am using the import component to import a csv from a personal datastore in Azure Machine Learning, this job is scheduled to run everyday, the idea is to automate the csv updates and connect to the endpoints of the different models to obtain new predictions, the problem is that the import Data does not read the updated files, so they return the same csv that it has been read the first time. But If you look in the datastore the file is updated, so the the problem is in the "import" component. As you can see the first job lasts 12 minutes, but the second job with updates only lasts 11 seconds and does not read the csv updated. How can I solve this?
{count} votes
-
romungi-MSFT 48,681 Reputation points Microsoft Employee2023-09-13T07:18:10.69+00:00 @Gustavo Avino As per the import data reference page if your source data changes, you can refresh the dataset and add new data by rerunning Import Data. Are you adding a different version file on the original or just updating the file in the datastore?
-
Gustavo Avino 0 Reputation points
2023-09-20T14:36:36.43+00:00 Hi @romungi-MSFT , I am updating the file. The file is located in an Azure file share inside Azure ML
I have a job scheduled to run every day, but once the job runs the import data component on the pipeline does not import the updated data. But if you make a new pipeline from zero, the import data works, so the problem is in the reruns of the jobs I suppose.
How can I solve this? -
romungi-MSFT 48,681 Reputation points Microsoft Employee
2023-09-21T12:32:23.73+00:00 @Gustavo Avino Did you try adding a pipeline parameter to the import data component? I think the behavior you are seeing is expected, if you need to re-run the pipeline it will pickup the same data. To use variable data you have to use the pipeline parameter to pass the newer version of the same. See this section where options 2 and 3 will help.
-
Gustavo Avino 0 Reputation points
2023-09-25T11:24:10.5533333+00:00 @romungi-MSFT Yes I tried and this error jumps, Is this because the datastore is in a fileshare?
I also tried to create a data asset and then add a pipeline parameter to the process
The job works, but it does not show the data updated, it is weird because If I update the data in the fileshare datastore the data asset automatically gets updated too, but the version is still the version 1, and in the job the data is shown without the update
-
romungi-MSFT 48,681 Reputation points Microsoft Employee
2023-09-26T11:51:35.7433333+00:00 @Gustavo Avino If that particular run is re-run the data will be the same but if a new run is created it should pick up the latest version if you have configured to use the pipeline parameter.
-
Gustavo Avino 0 Reputation points
2023-09-26T13:37:37.61+00:00 @romungi-MSFT So if I am using a schedule to re-run the job is not going to work? or I have to create a new job every time I want to use updated data?
-
romungi-MSFT 48,681 Reputation points Microsoft Employee
2023-09-28T13:44:25.8766667+00:00 @Gustavo Avino Rerunning a particular job id will pick up the same data. Starting a new pipeline run where you have defined pipeline parameter as part of input should pick up the version passed.
Sign in to comment
Sign in to answer