Microsoft OneDrive connector limitations

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page by opting into the Lakeflow Connect for OneDrive For Business preview. See Manage Azure Databricks previews.

This page lists limitations and considerations for ingestion from Microsoft OneDrive using Databricks Lakeflow Connect.

Connector-specific limitations

• During unstructured ingestion using binary file, each file's content is loaded into memory as a single record, so files larger than 100 MB can cause the update to fail (for example, with an out-of-memory error or by exceeding the 2 GB limit on binary columns in Delta). To prevent this, exclude large files using a row_filter on the length column in table_configuration. For example, "row_filter": "length <= 104857600" skips files larger than 100 MB. For large unstructured files, ingesting managed FILE values is preferred. There is no file size limit for structured file formats.
• Unstructured (BINARYFILE and FILE) ingestion supports only SCD_TYPE_1 storage mode. Structured ingestion (CSV, JSON, XML, EXCEL, and other formats) supports only APPEND_ONLY storage mode. SCD type 2 is not supported.
• Schema evolution is supported only for CSV, JSON, XML, PARQUET, and AVRO. For EXCEL, ORC, BINARYFILE, and FILE, set schema_evolution_mode to NONE.

For supported formats and configuration parameters, see OneDrive connector reference.