Azure Data Factory issue with interpreting url strings with "extra charachter" before and after

Question

Azure Data Factory issue with interpreting url strings with "extra charachter" before and after

Michele Tangorra 0

Hello. I am using azure data factory, in particular a web activity. In this activity I have to pass a json containing some adf global params. The problem is when the adf global parameter is a url. In such case, Adf automatically interprets the urls as link, hence if I want to get this:
{"param": {"type": "string", "value": "https\:------"}}, I get this {"param": {"type": "string", "value": " https\:------ "}}

Those are interpreted by the call as extra spaces, however not in ADF, since I tried with the replace function to remove, there is no way (so not sure what parameters are there) Same happens in the get function, when I get parameter, if value is url is always surrounded by those chars. I want to to avoid that, how coan I do?

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-12-10T17:29:55.4866667+00:00

@Michele Tangorra

Thanks for using MS Q&A platform and posting your query.

It seems you’re encountering an issue with Azure Data Factory (ADF) automatically adding extra spaces around URLs in JSON parameters.

Please check the below steps:

Ensure that any special characters in your URL are properly escaped. This might help ADF interpret the URL correctly without adding extra spaces.

Use ADF’s string functions to manipulate the URL string. For example, you can use the replace function to remove any unwanted characters. However, you’ve mentioned that this hasn’t worked for you, so it might be worth double-checking the syntax and ensuring there are no hidden characters.

Encode the URL using Base64 before passing it as a parameter. This can prevent ADF from misinterpreting the URL. You can use the base64 function in ADF to encode and decode the URL as needed.

I hope these steps help you resolve the issue. Please Let me know if issue persists.
Michele Tangorra 0 Reputation points

2024-12-11T15:30:54.4433333+00:00

Hello, thanks for the explanation. After deep analysis, I realized that yes, the url is surrounded by those chars when fetched and parsed by ADF, but when provided in the api call, the extra characters are not computed actually. Probably, in a test, I copy-pasted the json as was parsed by ADF (with spaces), and manually inject into the api call, hence the spaces in my test. So yes, that parsing is misleading, put passing it through the calls, actually is not really existing for the ADF. I pass here the images of of that strange behave, however, that is not a problem as described above :)
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-12-11T18:48:05.7066667+00:00

@Michele Tangorra

Thanks for sharing the image and the additional details! It looks like the issue with the extra characters or spaces around the URL in ADF is more about how it’s displayed during parsing rather than how it’s actually processed in API calls.

From the image you provided, it seems that when the URL is included in the JSON, there are no extra spaces around it, which aligns with your observation that the issue doesn’t affect the actual API calls.

You should always treat the parsed JSON as a representation of the data, not the raw data itself. When making API calls, manually remove any extra characters that might have been added by the parser.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-12-12T17:37:33.86+00:00

@Michele Tangorra We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Michele Tangorra 0 Reputation points

2024-12-14T15:07:14.34+00:00

Hi @phemanth yes as I shared above, even if the url are displayed with that space, when passing the response json to the next api call where I use it as body to deploy ADF global variables, then are actually not there.

This means that the display is misleading, but fortunately the spaces are not passed into the calls.

I think the problem I had is that I did the first call, then I copy the output result manually, and for test purpose I copied back manually to the next call. Since the display has that spaces, manually copying the response to the next call brings the spaces, hence that was my issue.

But again, fortunately, that is only a display issue: when I setup the process correctly to pass the response of first call to second call, still, even if displaying these spaces, the call actually produces the adf global variable deployed with no extra spaces.

All about displaying, all in all :)

So thank you, I mean, that displaying is misleading, we can somehow see as a bug, but only as visual issue fortunately.

Thanks a lot for your responses
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-12-16T10:55:45.49+00:00

@Michele Tangorra Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

1 answer

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-12-10T17:29:55.4866667+00:00

@Michele Tangorra

Thanks for using MS Q&A platform and posting your query.

It seems you’re encountering an issue with Azure Data Factory (ADF) automatically adding extra spaces around URLs in JSON parameters.

Please check the below steps:

Ensure that any special characters in your URL are properly escaped. This might help ADF interpret the URL correctly without adding extra spaces.

Use ADF’s string functions to manipulate the URL string. For example, you can use the replace function to remove any unwanted characters. However, you’ve mentioned that this hasn’t worked for you, so it might be worth double-checking the syntax and ensuring there are no hidden characters.

Encode the URL using Base64 before passing it as a parameter. This can prevent ADF from misinterpreting the URL. You can use the base64 function in ADF to encode and decode the URL as needed.

I hope these steps help you resolve the issue. Please Let me know if issue persists.
Michele Tangorra 0 Reputation points

2024-12-11T15:30:54.4433333+00:00

Hello, thanks for the explanation. After deep analysis, I realized that yes, the url is surrounded by those chars when fetched and parsed by ADF, but when provided in the api call, the extra characters are not computed actually. Probably, in a test, I copy-pasted the json as was parsed by ADF (with spaces), and manually inject into the api call, hence the spaces in my test. So yes, that parsing is misleading, put passing it through the calls, actually is not really existing for the ADF. I pass here the images of of that strange behave, however, that is not a problem as described above :)
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-12-11T18:48:05.7066667+00:00

@Michele Tangorra

Thanks for sharing the image and the additional details! It looks like the issue with the extra characters or spaces around the URL in ADF is more about how it’s displayed during parsing rather than how it’s actually processed in API calls.

From the image you provided, it seems that when the URL is included in the JSON, there are no extra spaces around it, which aligns with your observation that the issue doesn’t affect the actual API calls.

You should always treat the parsed JSON as a representation of the data, not the raw data itself. When making API calls, manually remove any extra characters that might have been added by the parser.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-12-12T17:37:33.86+00:00

@Michele Tangorra We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Michele Tangorra 0 Reputation points

2024-12-14T15:07:14.34+00:00

Hi @phemanth yes as I shared above, even if the url are displayed with that space, when passing the response json to the next api call where I use it as body to deploy ADF global variables, then are actually not there.

This means that the display is misleading, but fortunately the spaces are not passed into the calls.

I think the problem I had is that I did the first call, then I copy the output result manually, and for test purpose I copied back manually to the next call. Since the display has that spaces, manually copying the response to the next call brings the spaces, hence that was my issue.

But again, fortunately, that is only a display issue: when I setup the process correctly to pass the response of first call to second call, still, even if displaying these spaces, the call actually produces the adf global variable deployed with no extra spaces.

All about displaying, all in all :)

So thank you, I mean, that displaying is misleading, we can somehow see as a bug, but only as visual issue fortunately.

Thanks a lot for your responses
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-12-16T10:55:45.49+00:00

@Michele Tangorra Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Answer 1

Hi @Michele Tangorra

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Ask: Hello. I am using azure data factory, in particular a web activity. In this activity I have to pass a json containing some adf global params. The problem is when the adf global parameter is a url. In such case, Adf automatically interprets the urls as link, hence if I want to get this: {"param": {"type": "string", "value": "https:------"}}, I get this {"param": {"type": "string", "value": " https:------ "}}

Those are interpreted by the call as extra spaces, however not in ADF, since I tried with the replace function to remove, there is no way (so not sure what parameters are there) Same happens in the get function, when I get parameter, if value is url is always surrounded by those chars. I want to to avoid that, how coan I do?

Solution: While working with API responses and deploying ADF global variables, the issue arises due to misleading display formatting of spaces in the JSON response. Even though spaces may appear visually in the output, they are not actually present when the response is programmatically passed to the next API call.The root cause of the issue was manual copying of the JSON response for testing purposes. Copy-pasting manually retained the displayed spaces, which led to errors. However, when the process is automated—passing the API response directly to the next call—the spaces are ignored, and ADF global variables are deployed correctly without any extra spaces.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-12-16T19:19:08.1266667+00:00

@Michele Tangorra Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Azure Data Factory issue with interpreting url strings with "extra charachter" before and after

1 answer

Your answer