Microsoft OneDrive connector reference

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page by opting into the Lakeflow Connect for OneDrive For Business preview. See Manage Azure Databricks previews.

This page contains reference material for the Microsoft OneDrive connector in Databricks Lakeflow Connect.

onedrive_options parameters

Set these options inside the connector_options.onedrive_options block of each table in your pipeline definition.

Parameter Type Required Description
url String Yes The URL of the OneDrive for Business My files drive, folder, file, or share link to ingest from. The URL must use a personal <tenant>-my.sharepoint.com host. See Read files from OneDrive for URL format details.
entity_type String No The type of entity to ingest. Supported values: FILE (ingest file content and metadata) and FILE_METADATA (ingest metadata only, without downloading file contents). The default is FILE. Use FILE_METADATA only with the BINARYFILE format.
file_ingestion_options Object Yes Controls file format and ingestion behavior. See file_ingestion_options parameters.

file_ingestion_options parameters

Set these options inside onedrive_options.file_ingestion_options.

Parameter Type Required Description
format String Yes The file format to ingest. Supported values: BINARYFILE, CSV, JSON, XML, EXCEL, PARQUET, AVRO, ORC, FILE (Beta). Use BINARYFILE or FILE for unstructured ingestion (PDFs, Office files, images). Use a structured format to parse file contents into rows. Structured formats apply only when entity_type is FILE. When entity_type is FILE_METADATA, use BINARYFILE.
file_filters Array of objects No Filters that restrict which files are ingested. Each filter object can contain one of the following keys:
  • path_filter (string): A glob pattern matched against file names or paths. Based on Spark path glob filter.
  • modified_before (string): A timestamp in YYYY-MM-DDTHH:mm:ss format. Only files modified before this time are ingested.
  • modified_after (string): A timestamp in YYYY-MM-DDTHH:mm:ss format. Only files modified after this time are ingested.
infer_column_types Boolean No Whether to infer the exact data types for structured and semi-structured data. The default is true.
schema_evolution_mode String No Controls how new columns in incoming files are handled. Modes match Auto Loader schema evolution modes. Supported values: ADD_NEW_COLUMNS_WITH_TYPE_WIDENING (default), ADD_NEW_COLUMNS, RESCUE, FAIL_ON_NEW_COLUMNS, NONE. Schema evolution applies only to CSV, JSON, XML, PARQUET, and AVRO. For other formats, set NONE. See How does Auto Loader schema evolution work?.
schema_hints String No Overrides inferred column types. Specify as a comma-separated list of column_name TYPE pairs, for example order_id INT, amount DOUBLE. See Override schema inference with schema hints.
ignore_corrupt_files Boolean No Whether to skip corrupt files during ingestion.
corrupt_record_column String No The column that stores malformed records for formats that support a corrupt-record column.
rescued_data_column String No The column that stores data that does not match the inferred or provided schema.
single_variant_column String No The column that stores an entire input record as a single VARIANT value for formats that support this option.
reader_case_sensitive Boolean No Whether column matching is case-sensitive when rescuing data.
format_options Object No Format-specific parsing options. Keys are standard Auto Loader format option names. See Format options.

table_configuration parameters

Set these options inside the table_configuration block of each table in your pipeline definition. table_configuration is a sibling of connector_options, not nested inside it.

Parameter Type Required Description
storage_mode String No The storage mode for the destination table. Supported values:
  • SCD_TYPE_1 (default and only supported mode for BINARYFILE and FILE): Replaces the current row when a file changes and removes the current row when a file is deleted.
  • APPEND_ONLY (default and only supported mode for structured formats): Appends new rows from new or updated files.

Because these are the defaults and the only supported values, setting storage_mode explicitly is optional. Setting scd_type throws an error. Use storage_mode instead.

Format options

The format_options block accepts standard Auto Loader format option keys for the selected format. Express each value as a string. For example:

"file_ingestion_options": {
  "format": "CSV",
  "format_options": {
    "header": "true",
    "sep": ","
  }
}

For format-specific option names and behavior, see Spark data source options.

Ingested data format

The schema of the destination table depends on the entity_type and format you configure.

FILE entity type with BINARYFILE format

When entity_type is FILE and format is BINARYFILE, each ingested file becomes one row with the following columns:

Field Type Description
path string The path of the file.
modificationTime timestamp The time the file was last modified.
length bigint The size of the file in bytes.
content binary The file content.
_file_id string The unique OneDrive identifier of the file.
_onedrive_metadata struct Contains OneDrive-specific metadata for the file. See _onedrive_metadata fields.
_file_metadata struct Standard file metadata added by Databricks during ingestion. See _file_metadata fields.

FILE entity type with FILE format

When entity_type is FILE and format is FILE, each ingested file becomes one row with the following columns:

Field Type Description
path string The path of the file.
size bigint The size of the file in bytes.
modification_time timestamp The time the file was last modified.
file FILE A managed reference to the copy of the source file in the configured Unity Catalog volume.
_file_id string The unique OneDrive identifier of the file.
_onedrive_metadata struct Contains OneDrive-specific metadata for the file. See _onedrive_metadata fields.
_file_metadata struct Standard file metadata added by Databricks during ingestion. See _file_metadata fields.

FILE entity type with a structured format

When entity_type is FILE and format is a structured format (CSV, JSON, XML, EXCEL, PARQUET, AVRO, or ORC), the destination table schema matches the schema of the source files, plus the _onedrive_metadata and _file_metadata columns. Columns are inferred from the file contents, subject to the schema_evolution_mode and schema_hints settings.

FILE_METADATA entity type

When entity_type is FILE_METADATA, file content is not downloaded. The destination table contains only the metadata columns from the _onedrive_metadata and _file_metadata structs described above, plus _file_id.

_onedrive_metadata fields

The _onedrive_metadata struct contains the following fields. A field can be null when Microsoft Graph does not return a value.

Field Type Description
id string The unique OneDrive identifier of the file.
site_id string The identifier of the personal SharePoint site that owns the OneDrive drive.
drive_id string The OneDrive drive identifier.
drive_type string The type of OneDrive drive.
parent_id string The identifier of the parent folder.
parent_name string The name of the parent folder.
parent_path string The path of the parent folder.
web_url string A link to the file in OneDrive.
mime_type string The MIME type of the file.
created_by_email string The email address of the user who created the file.
created_by_name string The name of the user who created the file.
created_timestamp timestamp The time the file was created in OneDrive.
last_modified_by_email string The email address of the last user who modified the file.
last_modified_by_name string The name of the last user who modified the file.
etag string The entity tag for the file.
ctag string The content tag for the file.
description string The description of the file.
additional_metadata variant Additional metadata returned by Microsoft Graph that is not represented by another field.

_file_metadata fields

The _file_metadata struct contains standard file metadata added by Databricks during ingestion:

Field Type Description
file_path string The source path of the file.
file_name string The name of the file.
file_size bigint The size of the file in bytes.
file_block_start bigint The start byte offset of the file block.
file_block_length bigint The length of the file block in bytes.
file_modification_time timestamp The time the file was last modified.
  • Last updated on