Data Factory XML validation(Copy Activity) throwing User Configuration error

Question

Data Factory XML validation(Copy Activity) throwing User Configuration error

Partha Das 286

Hi,

I'm using Azure Data Factory Copy activity to validate one xml file against an XSD file.

I was mainly referring to the link "https://techcommunity.microsoft.com/t5/azure-data-factory-blog/azure-data-factory-adds-support-for-xml-format/bc-p/1533021"

I created one sample XML file and generated xsd using Visual Studio. I attached both files(changed the extension of xsd to txt for the shake of uploading)

books.xml books.txt

Please refer below my source configuration

SourceConfig

XML and XSD are stored in same blob storage

BlobContainer

And I'm getting below error while running the pipeline

Error

I'm unable to figure out, where I'm making the mistake.

Please help.

Also please let me know, what will be the configuration for sink? My requirement is to validate the xml and write the errors in any csv file(if any)

Regards,

Partha

Accepted answer

0 additional answers

Your answer

Answer 1

KranthiPakala-MSFT 46,627 Microsoft Employee

Hi @Partha Das ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

As per my understanding you are trying to validate your XML file against XSD and then copy data to your desired sink location. Please correct me if I'm wrong anywhere.

When I looked at the error message noticed that the schema validation is failing. And then I opened your XML file attached above in the question and noticed that XML file doesn't have XSD reference and hence the validation is failing as there no reference to your XSD in your XML file. Please see below (left side is your file data and right side what I updated referencing the XSD)

User's image

Resolution: Please add XSD reference in your XML file to validate schema. I have placed xsd in same location as xml file for testing. After adding XSD reference, your XML file will look like below:

<?xml version="1.0"?>
<books xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
       xsi:noNamespaceSchemaLocation="books.xsd">
   <book id="bk001">
      <author>Writer</author>
      <title>The First Book</title>
      <genre>Fiction</genre>
      <price>44.95</price>
      <pub_date>2000-10-01</pub_date>
      <review>An amazing story of nothing.</review>
   </book>
   <book id="bk002">
      <author>Poet</author>
      <title>The Poet's First Poem</title>
      <genre>Poem</genre>
      <price>24.95</price>
      <review>Least poetic poems.</review>
   </book>
</books>

The above will overcome your error message.

To your second question, in Copy activity mapping section, please configure as below so that data will be copied accordingly. First select the collection reference to $['books']['book'] and then click on Import Schemas which will import the columns as shown below and map it to your desired destination columns and run the pipeline.

User's image

For testing I copied the data to a blob as txt file and sample looks like below:

User's image

Hope this helps.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Partha Das 286 Reputation points

2023-03-03T10:24:32.4366667+00:00

hi @KranthiPakala-MSFT thanks for your detailed explanation. It worked for me. But I found it is throwing only the first error. I need all the errors to be thrown at a time. Because if I have 100 errors in the xml, I can't run the pipeline 100 times to fix all the errors. Please suggest.

Regards,

Partha
KranthiPakala-MSFT 46,627 Reputation points Microsoft Employee

2023-03-03T23:39:49.97+00:00

@Partha Das Glad to know above my response was helpful and you are unblocked.

Azure Data Factory has a default behavior of failing on the first error it encounters with schema validation.
Unfortunately, it does not validate the whole file and provide a list of errors.

As it seems to be new feature request, we recommend you to please go ahead and log a feature request in the feedback portal here: https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

Product group does monitor and review the requests and based on the priority of requests, they can plan for implementation in future . Once you log the feature request you will also be notified on the status of the request when respective product acknowledges it.

Once you have logged the feature request in the above feedback portal, please do share link for the same here, so that we can share it with internal teams to review it.

Hope this info helps.

Thank you

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
KranthiPakala-MSFT 46,627 Reputation points Microsoft Employee

2023-03-07T01:05:17.3266667+00:00

@Partha Das Just checking to see if you have got a chance to log a feature request. If so, could you please share the link here so that we can share it with respective product team.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
KranthiPakala-MSFT 46,627 Reputation points Microsoft Employee

2023-03-08T20:19:44.1633333+00:00

@Partha Das Checking to see if you have any further questions. Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Share via

Data Factory XML validation(Copy Activity) throwing User Configuration error

0 additional answers

Your answer