Oracle integrated CDC connector limitations

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.

This page lists limitations and considerations for Oracle ingestion using Databricks Lakeflow Connect.

General database connector limitations

The limitations in this section apply to all database connectors in Lakeflow Connect. Keep reading for connector-specific limitations.

  • When you run a scheduled pipeline, alerts don't trigger immediately. Instead, they trigger when the next update runs.

  • When a source table is deleted, the destination table is not automatically deleted. You must delete the destination table manually. This behavior is not consistent with Spark Declarative Pipelines on Lakeflow behavior.

  • The staging catalog can't be a foreign catalog.

  • During source maintenance periods, Databricks might not be able to access your data.

  • If a source table name conflicts with an existing destination table name, the pipeline update fails.

  • Multi-destination pipeline support is API-only.

  • You can optionally rename a table that you ingest. If you rename a table in your pipeline, it becomes an API-only pipeline, and you can no longer edit the pipeline in the UI.

  • If you select a column after a pipeline has already started, the connector does not automatically backfill data for the new column. To ingest historical data, manually run a full refresh on the table.

  • Databricks can't ingest two or more tables with the same name in the same pipeline, even if they come from different source schemas.

  • The source system assumes that the cursor columns are monotonically increasing.

  • Gateway snapshots can't be resumed. If you update the pipeline while a snapshot is in progress (for example, by adding new tables), the current snapshot is canceled and a new snapshot starts. The new snapshot includes the union of the tables from the canceled snapshot and any newly added tables.

  • The connector ingests raw data without transformations. Use downstream Spark Declarative Pipelines on Lakeflow pipelines for transformations.

Authentication

  • The connector supports basic (username and password) authentication only.

Database variations

  • The connector supports Oracle versions 12c and above (12c, 18c, 19c, 21c, 23ai, and 26ai).
  • The connector supports Amazon RDS for Oracle and Oracle running on Amazon EC2 or Azure virtual machines (VMs) or Oracle cloud (OCI). The connector also supports on-premises Oracle using Azure ExpressRoute, AWS Direct Connect, or VPN. For details about cross-cloud connectivity, see Network connectivity.
  • Oracle Real Application Clusters (RAC) is not supported, including Exadata in a RAC configuration. Connect to a single-instance Oracle database.
  • The following Oracle deployments are also not supported: physical standby, Oracle Autonomous Database, and multi-tenant Amazon RDS for Oracle instances.
  • Transparent Data Encryption (TDE) encrypted data is supported only when the encryption wallet (keystore) is open. With a closed wallet, LogMiner reports Unsupported Type for encrypted columns.

Source data

  • Unsupported data types: XML (XMLTYPE), JSON, BFILE, VECTOR, BOOLEAN, and spatial data types.
  • BLOB, CLOB, and NCLOB values are partially supported. LogMiner delivers most values inline with the SQL statement in the change log, and the connector reads them directly from the change. Large-object values that are written out of line, and that LogMiner therefore does not include in the redo record, are not captured. See Large object (LOB) columns.
  • LogMiner ignores any table that contains BFILE, nested tables, VARRAY collections, identity columns, temporal validity columns, PKREF columns, or PKOID columns. See LogMiner limitations.
  • LogMiner requires tables or column names to not exceed 30 characters.
  • Oracle stores unquoted identifiers in uppercase. The source_catalog (service name), source_schema, and source_table names are case-sensitive and must match how Oracle stores them.

Schema evolution

The connector automatically handles the following schema changes:

  • New columns. When a new column appears in the source, Databricks ingests it on the next pipeline run.
  • Deleted columns. When a column is deleted from the source, the connector marks it inactive in the destination rather than dropping it. If another column later appears with a name that conflicts with the inactive column, the pipeline fails. In this case, run a full refresh of the table or manually drop the inactive column.
  • Data type widening (for example, NUMBER(10) to NUMBER(20), or VARCHAR2(50) to VARCHAR2(100)).
  • New tables, if you ingest the entire schema.

The following schema changes are not handled automatically and require a full refresh of the affected tables:

  • Column renames. A rename is applied as a column drop and add, so historical values are not carried over under the new name.
  • Table renames.
  • Non-widening (narrowing) data type changes.
  • Adding a NOT NULL constraint to an existing column.

To recover from these changes without manual intervention, enable auto full refresh, which automatically full-refreshes the affected tables. See Auto full refresh policy.

Pipelines

  • The Oracle integrated CDC connector uses an integrated CDC pipeline that combines extraction and application into a single pipeline. See Create an integrated CDC pipeline for Oracle.
  • If Oracle purges archive logs before the pipeline can process them, perform a full refresh on the affected tables.

Staging

  • The staging catalog cannot be a foreign catalog.

Tables

  • Last updated on