Oracle Eloqua connector limitations

Important

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow Connect for Oracle Eloqua from the Previews page. See Manage Azure Databricks previews.

The managed Oracle Eloqua connector in Lakeflow Connect reingests five tables in full on every run, doesn't capture records deleted from incremental tables until you run a full refresh, and can miss activities that Oracle Eloqua records late.

General software as a service (SaaS) connector limitations

The limitations in this section apply to all SaaS connectors in Lakeflow Connect.

• When you run a scheduled pipeline, alerts don't trigger immediately. Instead, they trigger when the next update runs.
• When a source table is deleted, the destination table is not automatically deleted. You must delete the destination table manually. This behavior is not consistent with Spark Declarative Pipelines on Lakeflow behavior.
• During source maintenance periods, Databricks might not be able to access your data.
• If a source table name conflicts with an existing destination table name, the pipeline update fails.
• Multi-destination pipeline support is API-only.
• You can optionally rename a table that you ingest. If you rename a table in your pipeline, it becomes an API-only pipeline, and you can no longer edit the pipeline in the UI.
• If you select a column after a pipeline has already started, the connector does not automatically backfill data for the new column. To ingest historical data, manually run a full refresh on the table.
• Databricks can't ingest two or more tables with the same name in the same pipeline, even if they come from different source schemas.
• The source system assumes that the cursor columns are monotonically increasing.
• The connector ingests raw data without transformations. Use downstream Spark Declarative Pipelines on Lakeflow pipelines for transformations.

Connector-specific limitations

The limitations in this section apply to the Oracle Eloqua connector.

• Expect every pipeline run to reingest the option_list, event_registration_registrant, custom_object, contact_list_contact, and contact_segment tables in full. These tables don't support incremental ingestion, so each run reads all of their records from Oracle Eloqua, and the destination tables reflect inserts, updates, and deletes as of that run.
• Run a full refresh of an incremental table to remove records deleted in Oracle Eloqua. Incremental tables capture inserted and updated records only, so deleted records remain in the destination table until the next full refresh. Activities of a deleted contact also remain in the activity_* tables until a full refresh.

Activity tables

• Expect late-arriving activities to be missed. Each sync rereads the last hour of activities, or the last day for activity_email_clickthrough, three days for activity_external_activity, and seven days for activity_bounceback. Activities that Oracle Eloqua records later than that, including external activities backdated by more than three days, are missed until you run a full refresh. Oracle Eloqua records bouncebacks hours or days late and doesn't document a maximum delay.
• activity_bounceback ingests activities up to one hour before each sync starts, because Oracle Eloqua records bouncebacks asynchronously. The other activity tables ingest activities up to the start of the sync.
• An activity table with no rows rereads its full history, back to start_datetime, on every sync. To limit the cost, set a recent start_datetime.
• Oracle Eloqua returns at most 5,000,000 records per activity export, and each export covers 30 days by default. If a table produces more than about 160,000 records per day, set a shorter activity_window_step on that table. See Connector options.
• In activity_page_view and activity_web_visit, ContactId is the contact linked to the visitor when the activity was recorded. To get the visitor's current contact, join VisitorId to the visitor table.

Contact list and segment membership tables

• contact_list_contact skips empty contact lists. It also skips a list whose member count in Oracle Eloqua hasn't yet caught up after members were added, until the count updates.
• contact_segment exports every segment on every sync, so its sync duration grows with the number of segments in your instance. Oracle Eloqua applies a soft limit of 2,000 exports per hour to each instance, so schedule contact_segment to run daily and at a different time than contact_list_contact.
• After a sync, Oracle Eloqua blocks deleting an exported contact list or segment for up to two hours.
• If a contact list or segment is deleted while a sync is running, that sync fails, and the next sync recovers. If Oracle Eloqua can't export a segment, the contact_segment table fails until you fix the segment in Oracle Eloqua.