Automatically download csv file from a public website using Azure

Question

Hi All,

I need to download a CSV file which is publicly available and this file gets updated regularly.
I have designed a flow as per my requirements but i have to download the file manually.

Is there any way to make a process which download the file automatically and store it in ADLS location?

Which tool i can use for this?

Answer 1

@Sarvesh Pandey I shared this conundrum with a coworker, who referred me to this article:

[https://community.powerbi.com/t5/Power-Query/Downloading-stock-data-from-NSE-India/td-p/1805031

Of particular interest:

I also found that there is a python package (nsepy) that scraps the data from NSE site.

I'm thinking try using this package in either Data Factory Custom Activity, or Azure Synapse Python Notebook. Both should be able to write to Azure Data Lake.

Answer 2

Hi @MartinJaffer-MSFT ,

I think i need support help to build the flow.

Currently i am trying to the read the JSON file in Azure Databricks with Standard compute configuration but it's taking too much time to read.

I am using below command -

pd.read_json('[https://www1.nseindia.com/live_market/dynaContent/live_analysis/pre_open/all.json')

and file size is 436kb.

I also used Command Prompt got this error message -

  File "C:\Users\sarve\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response