Community connectors in Lakeflow Connect

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.

Community connectors are open-source connectors that extend Lakeflow Connect to sources without managed connector support. The community builds and maintains them. This page shows how to use a community connector to ingest data from a supported source into Azure Databricks.

To build your own connector for a source that isn't supported yet, see Build a custom connector. You can run a custom connector in your own workspace without contributing it to the community.

How community connectors work

Community connectors are built on the LakeflowConnect interface, which wraps the Spark Python Data Source API. Each connector handles authentication, schema discovery, and incremental data reads so you can create, configure, and run an ingestion pipeline backed by Lakeflow pipelines.

When you use a community connector, Azure Databricks clones the connector source code from a GitHub repository into a workspace directory that you specify. The pipeline then reads the connector source code at runtime and executes the ingestion logic against the configured source.

Supported sources

The community regularly adds new connectors. For the latest list of supported sources, see the Add data UI (Data Ingestion) in your Azure Databricks workspace or the Lakeflow Community Connectors repository on GitHub.

Requirements

  • A Azure Databricks workspace with Unity Catalog enabled
  • A connection for the source you want to ingest, or permissions to create a connection
  • Write access to a catalog and schema for the ingested tables

Create an ingestion pipeline

To ingest data using a community connector:

  1. In the sidebar of your Azure Databricks workspace, click +New > Add or upload data, then select the source under Community connectors.

  2. Click + Create connection or select an existing connection, then click Next.

  3. For Pipeline name, enter a name for the pipeline.

  4. For Event log location, enter a catalog name and a schema name. Azure Databricks stores the pipeline event log here. Ingested tables are also written here by default.

  5. For Root path, enter your workspace path (for example, /Workspace/Users/<your-email>/connectors). Azure Databricks clones and stores the connector source code here.

  6. Click Create pipeline.

  7. In the pipeline editor, open ingest.py and update the objects field to include the tables you want to ingest. For example:

    from databricks.labs.community_connector.pipeline import ingest

pipeline_spec = {
    "connection_name": "my_stripe_connection",  # Required: UC connection name
    "objects": [
        {"table": {"source_table": "charges"}},
        {"table": {"source_table": "customers",
                   "destination_table": "stripe_customers"}},
    ],
}

ingest(spark, pipeline_spec)

  8. Run the pipeline manually or schedule it.

Pipeline configuration options

You can configure the following options in ingest.py:

Option Description
connection_name Required. The name of the connection that stores authentication credentials for the source.
objects Required. A list of tables to ingest. Each entry has the format {"table": {"source_table": "..."}}. You can also specify an optional destination_table inside the table object.
destination_catalog The catalog where ingested tables are written. Defaults to the catalog set during pipeline creation.
destination_schema The schema where ingested tables are written. Defaults to the schema set during pipeline creation.
scd_type The slowly changing dimension strategy: SCD_TYPE_1, SCD_TYPE_2, or APPEND_ONLY. Defaults to SCD_TYPE_1.
primary_keys Override the default primary keys for a table. Provide a list of column names.

Considerations

  • Community connectors are under active development. Interfaces and behavior are subject to change.
  • Databricks doesn't maintain community connectors. They're not backed by Databricks SLAs and don't guarantee forward compatibility.

Submit feedback

Report bugs and submit feature requests in the Lakeflow Community Connectors repository.

  • Last updated on