SurveyMonkey connector limitations

Important

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow Connect for SurveyMonkey from the Previews page. See Manage Azure Databricks previews.

The managed SurveyMonkey connector in Lakeflow Connect doesn't capture deletes for its incremental tables and reingests eight of its 11 tables in full on every run.

General software as a service (SaaS) connector limitations

The limitations in this section apply to all SaaS connectors in Lakeflow Connect.

• When you run a scheduled pipeline, alerts don't trigger immediately. Instead, they trigger when the next update runs.
• When a source table is deleted, the destination table is not automatically deleted. You must delete the destination table manually. This behavior is not consistent with Spark Declarative Pipelines on Lakeflow behavior.
• During source maintenance periods, Databricks might not be able to access your data.
• If a source table name conflicts with an existing destination table name, the pipeline update fails.
• Multi-destination pipeline support is API-only.
• You can optionally rename a table that you ingest. If you rename a table in your pipeline, it becomes an API-only pipeline, and you can no longer edit the pipeline in the UI.
• If you select a column after a pipeline has already started, the connector does not automatically backfill data for the new column. To ingest historical data, manually run a full refresh on the table.
• Databricks can't ingest two or more tables with the same name in the same pipeline, even if they come from different source schemas.
• The source system assumes that the cursor columns are monotonically increasing.
• The connector ingests raw data without transformations. Use downstream Spark Declarative Pipelines on Lakeflow pipelines for transformations.

Connector-specific limitations

The limitations in this section apply to the SurveyMonkey connector.

• To ingest other SurveyMonkey objects, such as webhooks, users, workgroups, response rollups, or benchmarks, use another ingestion method. The connector supports only the 11 prebuilt tables, so these objects aren't ingested.
• Expect every pipeline run to reingest all records in the survey_collectors, collectors, survey_folders, contacts, contact_lists, contact_list_contacts, collector_messages, and collector_recipients tables. These tables don't support incremental ingestion, so each run reads them in full. On accounts with many surveys or collectors, these full reads can use most of the app's daily request limit. To stay under the limit, run the pipeline less often, leave out tables you don't need, or ask SurveyMonkey for a higher limit. See SurveyMonkey API rate limit exceeded.
• To ingest older data, set an earlier start_datetime connector option before the first sync. The surveys, survey_details, and survey_responses tables read only records modified on or after start_datetime, which defaults to 365 days before the first sync, so older records aren't ingested. See Connector options.
• To remove records deleted in SurveyMonkey, run a full refresh. The connector doesn't capture deletes for the surveys, survey_details, and survey_responses tables, so deleted records remain in the destination tables.
• To ingest more than 25 responses per survey, use a paid SurveyMonkey plan. During the 90-day draft period, an app authorized by a Basic plan account ingests only the first 25 responses for each survey, without an error.
• Leave the survey_folders table out of the pipeline for connections with the Data Region set to ca. The SurveyMonkey survey folders endpoint is unavailable for Canadian users, so a pipeline update that includes this table fails for these connections. See the SurveyMonkey survey folders API reference.
• Expect no survey_details row for surveys that SurveyMonkey can't return in full, such as surveys with more than 1,000 questions. The connector skips these surveys, so their pages and questions aren't ingested, and other surveys sync normally. See the SurveyMonkey response limits.