Can you read a file content entriely within Data Factory

Question

Is there a way to read contents of a file entirely within a Azure Data Factory. Lets say the file is a simple text file, stored in Data Lake Gen2 as a Blob.
Please note, i am not looking to Copy the file or transform the file, instead i just need to read the file content entirely within the ADF pipeline.

I can easily do the same within a Logic App using "Get Blob", but it seems I haven't found a way in ADF.

Answer 1

@AbhijitNath-2966

There are multiple ways to get data into the pipeline and available as activity output.

Lookup activity is the easiest, but it has volume limitations.

A web activity is another method. Most Azure services have a REST API. By pointing the web activity at the REST API endpoint for the service in question, you can fetch the data. Using web activity is harder than the Lookup, because the authentication is not as in-built.

For Gen2 / Blob, the process is to get a Bearer Token, then use it in the Authorization header of subsequent calls.

The first step is identical to Step 5 in using OAuth with REST connector.

The remaining steps are similar, except using Web Activity instead of Copy Activity.
To get the data, the URL should look like
GET https://{accountName}.{dnsSuffix}/{filesystem}/{path}

You will need to add a header, 'Authorization' , and the value should use the token @concat('Bearer ',variables('token'))

Answer 2

Here is the simple way to do it

1. The data in the file must be stored in a single line as ADF does not provide separate data types for .txt files. Converting the file data into a single line is simple and can be done in Notepad++. Please see the article for reference https://community.notepad-plus-plus.org/topic/14791/how-to-make-all-data-in-one-line
2. The column delimiter and the row delimiter in the dataset should be set to those values that will not be present in the file like pipe '|' and '\n' respectively. This will not divide the file data into array elements