Share via

Wrong URL encoding in Data Flow leading to illegal character error in query

VINCENZO SALVO 0 Reputation points
2024-07-24T15:59:26.0666667+00:00

Hi, I have a REST API dataset. The official APIs (https://me.sap.com/notes/3463893) require relative URLs with white spaces, for example:

DATA_COLLECTION?$filter=DC_PARAMETER_NAME eq 'TAGGY_SN'

In this case, I have encountered misleading behavior:

  1. Using this dataset as a source in a copy activity within a pipeline works as expected
  2. However, using the same dataset as a source in Data Flow leads to an illegal character error:

"*Job failed due to reason: at Source 'SnInfoSapDmDataFlow': Illegal character in query at index 41: DATA_COLLECTION?$filter=DC_PARAMETER_NAME eq 'TAGGY_SN'"
*
I have tried encoding with %20 :

DATA_COLLECTION?$filter=DC_PARAMETER_NAME%20eq%20'TAGGY_SN'

however, I was not able to execute the data flow without issues.
(the data flow works with URLs not requiring white spaces)

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,187 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Amira Bedhiafi 20,101 Reputation points
    2024-07-24T17:44:22.09+00:00

    Try encoding single quotes as %27:

    DATA_COLLECTION?$filter=DC_PARAMETER_NAME%20eq%20%27TAGGY_SN%27

    In your Data Flow, you can use a Derived Column transformation to construct the query string dynamically. This way, you can manipulate the URL string within the Data Flow before it is sent to the source.