Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,624 questions
How to copy today's date file from server hosted on WebDAV protocol using HTTP connection in Azure Data Factory?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 22%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Deepak Shiv
0
Reputation points
There are list of files placed on server which is based on WebDAV protocol. I want to copy today's date file into Azure blob storage using copy activity. For example if today's date is 01-Mar-2024 i.e. 20240301, below highlighted file should
So for achieving this I created HTTP connection and in relative url path I gave filename path with wildcard characters as @concat('src/order/orderExport-Invoice-',formatDateTime(utcnow(), 'yyyyddMM'),'*.xml')'
But HTTP connection unable to recognize wildcard characters and unable to convert into actual filename, getting below error:
Is there any way through which we can dynamically copy the file for today's date from server(WebDAV) using HTTP connection or other method?
Azure Data Factory
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache 20,366 Reputation points Moderator2024-03-04T20:57:06.81+00:00 Did you try to access with direct filename (hardcoded)? was it succeeded?
Did you try to remove the Asterisk(*) from the Expression?
@concat('src/order/orderExport-Invoice-',formatDateTime(utcnow(), 'yyyyddMM'),'.xml')'
Your Expression is resulting in the following output, which has an Asterisk * character, so please double check your expression.
Please store the File path in a Variable and do let us know how was it constructed before using it inside the copy Activity.
@concat('src/order/orderExport-Invoice-',formatDateTime(utcnow(), 'yyyyddMM'),'.xml') -
Deepak Shiv 0 Reputation points
2024-03-05T06:39:35.0533333+00:00 Thank you SaBo-MSFT for your reply!!
Yes as I mentioned previously, we were providing full name of file as hardcoded in Relative URL path and copying the file successfully. So no issues if we are providing hardcode file name.
But as we need to automate this process, we want to provide file name dynamically in relative URL path.
The complete file name is in this format i.e. "orderExport-Invoice-20240301-081010101.xml".
Now it's easy to evaluate 20240301(YYYYMMDD as currentdate) using function utcnow().
But for getting remaining part i.e. "081010101.xml" we just provide *.xml as wildcard characters so that it pick the rest of the content but HTTP doesn't convert wildcard characters.
Yes I tried to remove the Asterisk(*) from characters the HTTP connection gives message as:
No file found with file name "orderExport-Invoice-20240301.xml" becuase actual file name on server is "orderExport-Invoice-20240301-081010101.xml"
Is is there any way through which we can copy the current file from the WebDAV server?
Any help on this is appreciated. Thanks in advance!!
-
QuantumCache 20,366 Reputation points Moderator
2024-03-05T06:54:52.5666667+00:00 @Deepak Shiv How many XML files will be created in a particular day, is it single or multiple? Can we use the Get Metadata to Extract the info of the latest file for today?
-
Deepak Shiv 0 Reputation points
2024-03-05T07:27:28.05+00:00 One file for each day.
Get Metadata doesn't support HTTP connections
-
Deepak Shiv 0 Reputation points
2024-03-06T09:28:32.7033333+00:00 Hi SaBo-MSFT, any update or idea how to achieve this, please ?
-
QuantumCache 20,366 Reputation points Moderator
2024-03-07T19:50:31.8466667+00:00 @Deepak Shiv In the copy activity do you see the option for Prefix?
Did you try the Prefix option? looks like the file name has a pre-defined prefix format.
@concat('orderExport-Invoice-',formatDateTime(utcnow(), 'yyyyMMdd'),'-') -
Deepak Shiv 0 Reputation points
2024-03-08T12:36:09.66+00:00 @SaBo-MSFT, We don't have option of file prefix in "HTTP" linked service. It's available for delimited files.
-
Deepak Shiv 0 Reputation points
2024-03-13T16:11:48.74+00:00 Hello @SaBo-MSFT any update on the issue or should we consider it as limitation of Azure Data Factory ?
-
QuantumCache 20,366 Reputation points Moderator
2024-03-13T17:17:02.3266667+00:00 @Deepak Shiv Please submit a technical support ticket in this matter.
Sign in to comment
Sign in to answer