How to enable External Data Sources (EDS) Preview?
External Data Sources (EDS) is a capability in OSDU® that allows data from an OSDU® compliant external data source to be shared with an Azure Data Manager for Energy resource. EDS is designed to pull specified data (metadata) from OSDU-compliant data sources via scheduled jobs while leaving associated dataset files (LAS, SEG-Y, etc.) stored at the external source for retrieval on demand.
For more information about External Data Sources (EDS), see The OSDU Forum 2Q 2022 Newsletter - EDS.
Note
OSDU community shipped EDS as a preview feature in M18 Release, and it is available as a preview feature on Azure Data Manager for Energy in Developer tier only.
Important
Limit your Identity Provider (IdP) token to read operations only.
Protect your search and delivery wrapper APIs from unauthorized access. As a best practice, ensure only Azure Data Manager for Energy can access data through your wrapper APIs. This helps with data discovery and delivery for EDS Preview.
Prerequisites
Create a new or use an existing key vault to store secrets managed by OSDU® secret service. To learn how to create a key vault with the Azure portal, see Quickstart: Create a key vault using the Azure portal.
Important
Your key vault must exist in the same tenant and subscription as your Azure Data Manager for Energy resource.
Tip
When you create the key vault, select Enable purge protection (enforce a mandatory retention period for deleted vaults and vault objects).
In the Access configuration tab, under Permission model select Vault access policy.
Give permission to Azure Data Manager for Energy Service Principal (SPN) on key vault (existing or new). How to give access to Azure Data Manager for Energy SPN - Assign an Azure Key Vault access policy (CLI) | Microsoft Learn.
To enable EDS Preview, create an Azure support request
To enable External Data Sources Preview on your Azure Data Manager for Energy, create an Azure support ticket with the following information:
- Subscription ID
- Azure Data Manager for Energy developer tier resource name
- Data partition name (the data partition in which EDS needs to be enabled)
- Key Vault name (created in Prerequisites)
Note
EDS does not have multi data partition support
We notify you once EDS preview is enabled in your Azure Data Manager for Energy resource.
Known issues
- Below issues are specific to OSDU® M18 release:
- EDS ingest DAG results in failures when the data supplier’s wrapper Search service is unavailable.
- EDS Dataset service response provides an empty response when data supplier’s Dataset wrapper service is unavailable.
- Secret service responds with 5xx HTTP response code instead of 4xx in some cases. For example,
- When an application tries to recover a deleted secret, which isn't deleted.
- When an application tries to get an invalid deleted secret.
Limitations
Some EDS capabilities like Naturalization, Reverse Naturalization, Reference data mapping are unavailable in the M18 OSDU® release (available in later releases), and hence unavailable in Azure Data Manager for Energy M18 release. These features are available once we upgrade to subsequent OSDU® milestone release.