Reading PDF/DOC Encoded Files ,Decoding them and Loading into ADLS

Riya Bakshi 0

I want to read encoded(base 64) pdf /doc files from salesforce,decode the files and load them back into ADLS Location in the same format(PDF/DOC).I want to achieve this either using Databricks or ADF.Kindly suggest some way to do so.

phemanth 11,885 Reputation points Microsoft Vendor

2023-09-30T09:23:25.2533333+00:00
@Anonymous

Thanks for Approaching MicrosoftQ&A

To read encoded (base64) PDF/DOC files from Salesforce, decode the files, and load them back into ADLS location in the same format (PDF/DOC), you can use Azure Data Factory (ADF) or Databricks. Here are some steps to achieve this:Using Azure Data Factory:

Create a pipeline in ADF that retrieves the encoded PDF/DOC files from Salesforce using the Salesforce connector.

Use a Data Flow activity in ADF to decode the base64-encoded files. In the Data Flow, use a Derived Column transformation to add a new column that contains the decoded file content. Use the Base64Decode function to decode the base64-encoded file content. For example:

Base64Decode(MyEncodedFileContent)

Use a Sink transformation in the Data Flow to write the decoded files to ADLS in the same format (PDF/DOC). Set the file format in the Sink transformation to match the format of the original files.

Using Databricks:

Use the Azure Databricks connector to read the encoded PDF/DOC files from Salesforce. For example:

df = spark.read.format("com.databricks.spark.sqldw").option("url", jdbcUrl).option("forwardSparkAzureStorageCredentials", "true").option("dbTable", "SalesforceTable").load()

Use the base64 library in Python to decode the base64-encoded file content. For example:

import base64

Write the decoded files to ADLS in the same format (PDF/DOC). For example:

dbutils.fs.put("/mnt/adls/path/to/file.pdf", decoded_content, overwrite=True)

These are some general steps to read encoded PDF/DOC files from Salesforce, decode them, and load them back into ADLS location in the same format (PDF/DOC) using ADF or Databricks. The specific implementation may vary depending on the details of your use case.

for further queries please do let us know.
phemanth 11,885 Reputation points Microsoft Vendor

2023-10-03T09:30:13.4166667+00:00

@Anonymous We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Riya Bakshi 0 Reputation points

2023-10-04T07:11:27.9766667+00:00

Hi @phemanth , regarding your 2nd point to use dataflow activity, I tried using the same but the challenge with this is the source dataset cannot be binary format, the option is disabled. How do we handle this scenario?
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
phemanth 11,885 Reputation points Microsoft Vendor

2023-10-04T08:01:15.9166667+00:00

@Anonymous

if you want to use the Dataflow activity, you may need to convert the binary format to a supported format such as Delimited Text

Alternatively, you could try using a different type of dataset that supports binary format in the Dataflow activity, such as the Binary Stream dataset.

please confirm with us if issue persists.
phemanth 11,885 Reputation points Microsoft Vendor

2023-10-05T09:26:39.2433333+00:00

@Anonymous

just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more

Share via

Reading PDF/DOC Encoded Files ,Decoding them and Loading into ADLS

1 answer

Your answer