Connect to managed ingestion sources

Lakeflow Connect managed ingestion connections store authentication credentials for your data sources in Unity Catalog. Any user with the USE CONNECTION privilege on the connection can create managed ingestion pipelines from sources like Salesforce and SQL Server.

An admin user must create the connection if the users who will create pipelines are non-admin users or will use programmatic interfaces. These interfaces require that users specify an existing connection when they create a pipeline.

Alternatively, admin users can create a connection and a pipeline at the same time in the data ingestion UI. See Managed connectors in Lakeflow Connect.

Ingestion vs. federation connections

Lakeflow Connect offers fully-managed ingestion connectors for enterprise apps and database sources. Lakehouse Federation allows you to query external data sources without moving your data. You can create connections for both ingestion and query federation in Catalog Explorer.

Privilege requirements

The user privileges required to connect to a managed ingestion source depend on the interface you choose:

Scenario Supported interfaces Required user privileges
An admin user creates a connection and an ingestion pipeline at the same time. Data ingestion UI
  • CREATE CONNECTION on the metastore
  • USE CATALOG on the target catalog
  • (SaaS apps) USE SCHEMA and CREATE TABLE on an existing schema or CREATE SCHEMA on the target catalog
  • (Databases) USE SCHEMA, CREATE TABLE, and CREATE VOLUME on an existing schema or CREATE SCHEMA on the target catalog
An admin user creates a connection for non-admin users to create pipelines with. Admin:
  • Catalog Explorer

Non-admin:
  • Data ingestion UI
  • Declarative Automation Bundles
  • Databricks APIs
  • Databricks SDKs
  • Databricks CLI
 Admin:
  • CREATE CONNECTION on the metastore

Non-admin:
  • USE CONNECTION on an existing connection
  • USE CATALOG on the target catalog
  • (SaaS apps) USE SCHEMA and CREATE TABLE on an existing schema or CREATE SCHEMA on the target catalog
  • (Databases) USE SCHEMA, CREATE TABLE, and CREATE VOLUME on an existing schema or CREATE SCHEMA on the target catalog

Create a connection

For step-by-step instructions to create a Unity Catalog connection for a specific managed ingestion source, see the connector's "Create a connection" page in the managed connector documentation.

Next steps

After you create a connection to your managed ingestion source in Catalog Explorer, any user with the USE CONNECTION privilege on the connection can create an ingestion pipeline in the following ways:

  • Ingestion wizard (supported connectors only)
  • Declarative Automation Bundles
  • Databricks APIs
  • Databricks SDKs
  • Databricks CLI

For pipeline creation instructions, see the managed connector documentation.

  • Last updated on