Oracle DB to ADLS using ADF

Question

Hello -

I have a requirement to create a data archival for on-prem oracle db. The DB contains attachments like excel,email etc. stored in DB storage as oracle Blobs for pega application. The data will be accessed by business users and we need an interface for users to view these files securely without downloading.

I think we should be able to use ADF to bring the data and attachments into ADLS.

1. Is it possible to read the attachments and ingest into ADLS - is it blob container or file share we should store ?
2. How do we allow user to query them and view them without the ability to download when they need to look at those attachments ?
3. do we need to use sharepoint or MS Purview API ?

Please could you give details on how to achieve this. Please dont give a chatgpt response.

Thank you !

Answer 1

Hey Sourav! It seems like you're looking to create a data archival solution for your on-prem Oracle DB, specifically to handle blobs that include attachments for the Pega application. You want to ingest this data into Azure Data Lake Storage (ADLS) using Azure Data Factory (ADF) and then provide an interface for users to view these files securely without downloading them. Let's break this down!

Here’s what you can do:

Ingesting Oracle BLOBs into ADLS:

• You can definitely use Azure Data Factory to copy data from your Oracle database (including BLOBs) into ADLS. It’s important to set up a Linked Service in ADF that connects to your Oracle database.
  • You need to create a connection to Azure Blob Storage for staging the data if ADF doesn't support direct copy to ADLS. Make sure you configure your ADF pipeline correctly to use the built-in staged copy option if needed.
  Choosing the Right Storage Type:

  - For storing the ingested data in ADLS, you typically work with a **Blob Container** rather than a file share. Blob Containers are designed specifically for handling unstructured data like BLOBs.
  
  **Providing Secure Access to Business Users**:
  
     - To allow users to query and view these attachments without downloading them, you might consider implementing a front-end application or using an existing solution like **SharePoint**. SharePoint can act as a secure document repository that allows users to view files directly in the browser without downloading.
  
        - Alternatively, utilizing the **Microsoft Purview API** could help manage and catalog the data, adding another layer of data governance.
  

Follow-Up Questions:

To provide more tailored advice, consider answering these:

1. What type of users will be accessing these files, and what level of access do they require?
2. Do you currently have any platform (like a web application) in mind for users to interact with the files?
3. Are there specific security requirements you need to meet for accessing these files?
4. Have you already set up Azure Data Factory and the necessary Linked Services for Oracle and ADLS?

I hope this gives you a solid foundation to start with! If you have more questions or need deeper insights into a specific area, feel free to ask!

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.