SAP CDC connector doesn't work properly with special characters in TEXT fields

Question

SAP CDC connector doesn't work properly with special characters in TEXT fields

Dmytro Honcharuk 50

We try to ingest data from SAP system using SAP CDC connector with SLT. In some of the tables in SAP we have TEXT columns which may consist special characters. In particular, we have newline '\n' character in such columns. And when data comes to Azure the input stream move the data after this character to another row. So even in Derived Column transformation we cannot replace this character or to perform some similar operation.

In staging file that we receive the data is moved to another row separated by this newline character.

When performing Copy Activity from SAP HANA for replicated table we receive this data in proper way as a string with this special character inside it - this is the expected behavior that we try to have with SAP CDC as well.

Is there any way how we can avoid such situation or somehow to preprocess the data?

Accepted answer

1 additional answer

Your answer

Answer 1

QuantumCache 20,366 Moderator

Hello @Dmytro Honcharuk, Thanks for reaching out on this forum,

Updated 7/14/2023: From Original Poster: Resolution: @Dmytro Honchar

Was able to resolve it by adding ‘enableMultiLineRow: true’ in the mapping data flow code.
The solution for this will be adding “ " enableMultiLineRow:true,", ” into the dataflow json code, on source properties inside scriptLines. In this case string is interpreted as expected

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I have reposted your solution in case you'd like to "Accept " the answer.

Please try considering some of the below suggestions!

Check the encoding of the special characters in the TEXT fields. Make sure that the encoding is supported by the SAP CDC connector. You may need to convert the encoding of the special characters before ingesting the data.
Try using a different data type for the TEXT fields in SAP. For example, you could use a VARCHAR field instead of a TEXT field. This may help avoid issues with special characters.
Consider using a pre-processing step to clean up the data before ingesting it into Azure. For example, you could use a script to replace the newline character with a different character that is supported by the SAP CDC connector.

You may also let us know if you need further help in this matter and we are more than happy to help further!

Dmytro Honcharuk 50 Reputation points

2023-06-22T13:41:10.42+00:00

Thank you for response. Could you please tell me where I can find a documentation regarding encoding that is supported by the SAP CDC connector?
Dmytro Honcharuk 50 Reputation points

2023-06-27T10:25:23.2766667+00:00

Sorry, didn't found any docs regarding encoding for SAP CDC connector. Could you please help with this information?
QuantumCache 20,366 Reputation points Moderator

2023-07-04T00:26:33.3566667+00:00

I was also not able to find any documentation on the specifics about the encoding in SAP CDC connector.

Workaround can be to use a "Derived Column" transformation to your Data Flow and create a new column that replaces newline characters with a different character or string. You can use the following expression in the "Derived Column" transformation:

replace(column_name_with_newline, '\n', '<replacement_string>')
Dmytro Honcharuk 50 Reputation points

2023-07-04T08:42:51.53+00:00

Thank you for response. I have tried this and it doesn't work. The input stream already process the string as a separate row. So derived column transformation consider string with newline as two separate rows.
QuantumCache 20,366 Reputation points Moderator

2023-07-05T21:35:51.7333333+00:00

Hello @Dmytro Honcharuk
Please send an email to azcommunity@microsoft.com with the below details, so that we can work closely on this matter.

Your Email Subject : Attn Satish Boddu

Your Email Body content must include the below:

Thread URL: Link to this thread.

Your Azure Subscription ID: <This is must to be provided to us>
Power BI - Service Account 0 Reputation points

2023-07-11T06:22:48.3+00:00

Hi @SatishBoddu-MSFT we are currently facing the same issue when loading data from S/4 with the SAP CDC connector. There is a line break in the source system and the SAP CDC connector puts that information on a separate record in Azure. Is there any solution on this and/or should I open a separate support ticket with Microsoft? Thx
QuantumCache 20,366 Reputation points Moderator

2023-07-11T06:24:47.6533333+00:00

Hello @Power BI - Service Account,
Please send an email to with the below details, so that we can work closely on this matter.
QuantumCache 20,366 Reputation points Moderator

2023-07-12T17:12:14.4333333+00:00

Thanks for the email.

Answer 2

Dmytro Honcharuk 50

The solution for this will be adding “ " enableMultiLineRow:true,", ” into the dataflow json code, on source properties inside scriptLines. In this case string is interpreted as expected

QuantumCache 20,366 Reputation points Moderator

2023-08-01T07:44:49.8433333+00:00

Thank you @Dmytro Honcharuk for sharing the resolution.

I have updated the above response, with the resolution provided from your side.

please click "Accept Answer" . So that we can close this case. Thanks for understanding.

Share via

SAP CDC connector doesn't work properly with special characters in TEXT fields

1 additional answer

Your answer