Ingest data from OneDrive

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page by opting into the Lakeflow Connect for OneDrive For Business preview. See Manage Azure Databricks previews.

This page shows how to create a managed Microsoft OneDrive ingestion pipeline using Lakeflow Connect. The pipeline ingests files from OneDrive into Delta tables and keeps them in sync with the source.

Before you begin

  • To create the ingestion pipeline, you must first meet the following requirements:

    • Your workspace must be enabled for Unity Catalog.

    • Serverless compute must be enabled for your workspace. See Serverless compute requirements.

    • To create a new connection, you must have CREATE CONNECTION privileges on the metastore. See Manage privileges in Unity Catalog.

      If the connector supports UI-based pipeline authoring, an admin can create the connection and the pipeline at the same time by completing the steps on this page. However, if the users who create pipelines use API-based pipeline authoring or are non-admin users, an admin must first create the connection in Catalog Explorer. See Connect to managed ingestion sources.

    • To use an existing connection, you must have USE CONNECTION privileges or ALL PRIVILEGES on the connection object.

    • You must have USE CATALOG privileges on the target catalog.

    • You must have USE SCHEMA and CREATE TABLE privileges on an existing schema or CREATE SCHEMA privileges on the target catalog.

  • Use Databricks Runtime 19 or above. In an API pipeline definition, set channel to PREVIEW.

  • To ingest from OneDrive, you must first configure a supported authentication method and create a connection. See Create a OneDrive connection.

Create an ingestion pipeline

For a list of supported file formats, see file_ingestion_options parameters. For connector-specific limitations, see Microsoft OneDrive connector limitations.

  1. Import the following notebook into your Azure Databricks workspace:

    Get notebook

  2. Do not modify cell two.

  3. Modify cell three with your pipeline configuration details. See Microsoft OneDrive connector reference and Examples.

  4. Click Run all.

Running the notebook creates the pipeline but does not start it. To start and schedule the pipeline, see Common pipeline maintenance tasks.

Choose a source URL

Each object in a pipeline definition specifies its source in connector_options.onedrive_options.url. The URL must resolve to one of the following resources in a OneDrive for Business My files drive:

  • The drive root
  • A folder
  • A file
  • A valid share link

The URL must use a personal OneDrive host, such as <tenant>-my.sharepoint.com. The following URLs are not supported:

  • SharePoint site URLs (/sites/...)
  • Consumer OneDrive URLs, such as onedrive.live.com or 1drv.ms
  • Bare host URLs
  • Paths outside a personal drive

For URL examples and instructions to copy a URL from OneDrive, see Read files from OneDrive.

Examples

Use these examples to configure your pipeline. Replace each placeholder with values for your environment.

Ingest files as unstructured data

Ingest files from OneDrive as binary content. Each file becomes a row that contains the file's bytes and metadata. Use this approach for PDFs, Office documents, images, and other files you intend to process downstream, for example for RAG applications.

Note

Available in Beta, the FILE type stores a governed reference to each file and its metadata instead of the file's bytes, which improves read performance. To ingest files as the FILE type instead of BINARYFILE, see Ingest files as the FILE type (Beta).

pipeline_spec = {
  "name": "onedrive_binary_pipeline",
  "catalog": "main",
  "schema": "ingest_destination_schema",
  "ingestion_definition": {
    "connection_name": "<onedrive-connection>",
    "objects": [
      {
        "table": {
          "destination_catalog": "main",
          "destination_schema": "ingest_destination_schema",
          "destination_table": "drive_files",
          "connector_options": {
            "onedrive_options": {
              "entity_type": "FILE",
              "url": "https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents",
              "file_ingestion_options": {
                "format": "BINARYFILE",
                "schema_evolution_mode": "NONE"
              }
            }
          }
        }
      }
    ]
  },
  "channel": "PREVIEW"
}

json_payload = json.dumps(pipeline_spec, indent=2)
create_pipeline(json_payload)

Ingest files as the FILE type (Beta)

Important

This feature is in Beta.

Ingest files as the FILE type to copy each source file into a Unity Catalog volume and store a governed reference to each file and its metadata in the destination table. The destination table exposes a file column that you can pass to functions such as ai_parse_document and to user-defined functions (UDFs).

To use the FILE type, configure the following in your pipeline:

  • Set the pipeline channel to PREVIEW.
  • Set file_ingestion_options.format to FILE.
  • Set the databricks.filespace-preview table property to declare the volume Unity Catalog uses as managed storage.

For FILE requirements and behavior, see Ingest files as the FILE type.

pipeline_spec = {
  "name": "onedrive_file_pipeline",
  "catalog": "<catalog>",
  "schema": "<schema>",
  "ingestion_definition": {
    "connection_name": "<onedrive-connection>",
    "objects": [
      {
        "table": {
          "destination_catalog": "<catalog>",
          "destination_schema": "<schema>",
          "destination_table": "<destination-table>",
          "table_configuration": {
            "table_properties": {
              "databricks.filespace-preview": "/Volumes/<catalog>/<schema>/<volume>/"
            }
          },
          "connector_options": {
            "onedrive_options": {
              "entity_type": "FILE",
              "url": "https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents",
              "file_ingestion_options": {
                "format": "FILE",
                "schema_evolution_mode": "NONE"
              }
            }
          }
        }
      }
    ]
  },
  "channel": "PREVIEW"
}

json_payload = json.dumps(pipeline_spec, indent=2)
create_pipeline(json_payload)

Ingest structured files

Ingest structured files (for example, JSON files) from a OneDrive folder. Each row in the source files becomes a row in the destination table.

pipeline_spec = {
  "name": "onedrive_json_pipeline",
  "catalog": "main",
  "schema": "ingest_destination_schema",
  "ingestion_definition": {
    "connection_name": "<onedrive-connection>",
    "objects": [
      {
        "table": {
          "destination_catalog": "main",
          "destination_schema": "ingest_destination_schema",
          "destination_table": "json_files",
          "connector_options": {
            "onedrive_options": {
              "entity_type": "FILE",
              "url": "https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents/json_folder",
              "file_ingestion_options": {
                "format": "JSON",
                "schema_evolution_mode": "NONE"
              }
            }
          }
        }
      }
    ]
  },
  "channel": "PREVIEW"
}

json_payload = json.dumps(pipeline_spec, indent=2)
create_pipeline(json_payload)

Ingest file metadata only

Ingest file metadata (name, size, timestamps, path) without downloading file contents. Use this approach when you need an inventory of files without the overhead of ingesting their content.

Set entity_type to FILE_METADATA. FILE_METADATA ingestion must use the BINARYFILE format. Structured and semi-structured formats are not supported for metadata-only ingestion.

pipeline_spec = {
  "name": "onedrive_metadata_pipeline",
  "catalog": "main",
  "schema": "ingest_destination_schema",
  "ingestion_definition": {
    "connection_name": "<onedrive-connection>",
    "objects": [
      {
        "table": {
          "destination_catalog": "main",
          "destination_schema": "ingest_destination_schema",
          "destination_table": "file_metadata",
          "connector_options": {
            "onedrive_options": {
              "entity_type": "FILE_METADATA",
              "url": "https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents",
              "file_ingestion_options": {
                "format": "BINARYFILE",
                "schema_evolution_mode": "NONE"
              }
            }
          }
        }
      }
    ]
  },
  "channel": "PREVIEW"
}

json_payload = json.dumps(pipeline_spec, indent=2)
create_pipeline(json_payload)

Configure file ingestion options

The file_ingestion_options block controls how files are processed. All options are set inside the connector_options.onedrive_options.file_ingestion_options block in the pipeline definition. The following snippets show the file_ingestion_options fragment of a pipeline definition. For all parameters, see file_ingestion_options parameters.

File filters

Use file_filters to restrict which files are ingested from the source URL:

"file_ingestion_options": {
  "format": "CSV",
  "file_filters": [
    { "path_filter": "invoices/*.csv" },
    { "modified_after": "2026-01-01T00:00:00" }
  ]
}

Schema evolution and schema hints

Set schema_evolution_mode to control how new columns in incoming files are handled. Override inferred column types using schema_hints.

Schema evolution applies only to CSV, JSON, XML, PARQUET, and AVRO. For other formats (BINARYFILE, FILE, EXCEL, and ORC), set schema_evolution_mode to NONE.

"file_ingestion_options": {
  "format": "CSV",
  "schema_evolution_mode": "ADD_NEW_COLUMNS_WITH_TYPE_WIDENING",
  "schema_hints": "order_id INT, amount DOUBLE, ts TIMESTAMP"
}

For more information about schema evolution and hints, see Configure schema inference and evolution in Auto Loader.

Format-specific options

Pass format-specific options using format_options:

"file_ingestion_options": {
  "format": "CSV",
  "format_options": {
    "header": "true",
    "sep": ","
  }
}

Supported keys are the standard Auto Loader format options. See Spark data source options.

Common patterns

For advanced pipeline configurations, see Common patterns for managed ingestion pipelines.

Next steps

Start, schedule, and set alerts on your pipeline. See Common pipeline maintenance tasks.

Additional resources

  • Last updated on