Skip Line Count in Mapping Data Flow`

DavidPendleton-7654 26

I am attempting to read a fixed-width text file in a mapping data flow.

I've set up my dataset with no delimiter and no schema. I've set up my source to use the dataset and skip the first 2 rows:

User's image

After parsing the incoming data (using substring, etc.) later in the data flow, I am writing the data out to a CSV file. The mapping in the Sink is present and complete and matches the parsed columns from earlier in the data flow. The CSV values are written correctly.

Unfortunately, the first two rows of the source file are also parsed and included. I don't want them in the output file.

Am I overlooking something here?

DavidPendleton-7654 26 Reputation points

2023-03-22T15:34:51.7133333+00:00

After additional testing, it appears that the "Skip line count" setting is ignored if the column delimiter in the dataset is set to "No delimiter." I can't think of a good reason for this behavior, but that's how it works.
KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2023-03-23T20:21:03.29+00:00

Hi @DavidPendleton-7654 ,

Thanks for using Microsoft Q&A forum and posting your query.

I was able to reproduce the problem and notice the same behavior. Hence reaching out to internal team to get better understanding of this inconsistency. Will keep you posted as soon as we hear back from the team.

Thank you
KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2023-03-27T18:54:08.52+00:00

@DavidPendleton-7654 This seems to a bug and as per my conversation with internal team, it is recommended to file a support ticket for deeper investigation on the product side and take necessary actions.

If you have a support plan, could you please file a support ticket and share the SR# with me here for tracking it. In case if you don't have a support plan, please let me know so that I can work with you offline in creating a support ticket.
DavidPendleton-7654 26 Reputation points

2023-03-28T13:42:18.7133333+00:00

I do have a support plan through my employer; however, I cannot create a ticket because I do not have a RunID or ErrorCode to complete the ticket creation.
KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2023-03-28T18:50:02.28+00:00

@DavidPendleton-7654 Thanks for the response. Please try to trigger the pipeline against a sample where you have only single record instead of an array. Once you have error message and the runID, please use them to file a support ticket.

Let me know if you have further queries.

Thank you
DavidPendleton-7654 26 Reputation points

2023-03-28T19:21:41.3966667+00:00

I don't get an error message. The setting is ignored and the data flow/pipeline completes successfully.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2023-03-29T07:33:52.02+00:00

@DavidPendleton-7654 Just had a conversation with respective product team and got confirmation that they have opened a work item to fix the issue. I do not have a definite ETA for the fix but it should be done in before end of April as it needs some testing.

But in the meantime, please do workaround the problem, by using some special character (Note: please ensure that character doesn't exist in the source data ) as delimiter in the dataset configuration. This should help skip the lines as configured.

Please don't open a support ticket as a work item is being logged from product side and will be taken care in few weeks.

Hope this info helps. Please let me know if you have any questions.

Thank you

Share via

Skip Line Count in Mapping Data Flow`

Your answer