Clean up processed files with Auto Loader

Note

cloudFiles.cleanSource is available in Databricks Runtime 16.4 and above.

Use cloudFiles.cleanSource to move or delete files from the source directory after they are processed. Removing processed files reduces storage costs and shortens the duration of future listing operations.

Mode	Description
OFF (default)	The files in the source directory are not moved or deleted.
MOVE	The files in the source directory are moved to the path specified by cloudFiles.cleanSource.moveDestination after the retention duration (default 30 days) defined by cloudFiles.cleanSource.retentionDuration elapses.
DELETE	The files in the source directory are deleted after the retention duration (default 30 days) defined by cloudFiles.cleanSource.retentionDuration elapses.

Additional option	Default	Valid values	Description
cloudFiles.cleanSource.retentionDuration	30 days	A CalendarInterval string such as 14 days, 2 weeks, or 1 month	Amount of time to wait before processed files become candidates for cleanup with clean source. Must be greater than 7 days for DELETE. No minimum restriction for MOVE.
cloudFiles.cleanSource.waitForCompletion	false	true, false	This option is available in Databricks Runtime 19 and above. Clean source is by default a best-effort operation. If the stream completes processing files before clean source is done moving or deleting files, the clean source operation is terminated. Setting cloudFiles.cleanSource.waitForCompletion forces the stream to stay alive until clean source is done moving or deleting files. This can increase the stream run time if there are a lot of files to delete. This only applies when the stream completes on its own (for example, an availableNow trigger draining all files). Manually stopping or canceling the stream terminates the clean source operation immediately, even when this option is set.
cloudFiles.cleanSource.moveDestination	None	A cloud storage or Unity Catalog volume path	Path to archive processed files to when cloudFiles.cleanSource is set to MOVE. This can be a cloud storage path or a Unity Catalog volume path (for example, /Volumes/my_catalog/my_schema/my_volume/archive/). The move location must: Not be a child of the source directory. If you place the move destination inside the source directory, the archived files are ingested again. Be in the same external location, volume, or DBFS mount as the source. Cross-bucket and cross-container moves are not supported and result in an error. Auto Loader must have write permissions to this directory.

Considerations before enabling cloudFiles.cleanSource

• Azure Databricks does not recommend using this option when multiple streams consume data from the same source directory. The fastest stream cleans up the files, so the slower streams never ingest them.
• Enabling this feature requires Auto Loader to maintain additional state in its checkpoint, which incurs performance overhead but enables improved observability through the cloud_files_state table-valued function. See cloud_files_state table-valued function.
• Clean source uses the current setting to decide whether to MOVE or DELETE a given file. For example, suppose that the setting was MOVE when the file was originally processed but was changed to DELETE when the file became a candidate for cleanup 30 days later. In this case, clean source deletes the file.
• Files are not guaranteed to be cleaned as soon as the cloudFiles.cleanSource.retentionDuration expires. To keep costs low, Auto Loader cleans up files concurrently with stream processing and terminates as soon as the stream processing is complete or is terminated. Files that were candidates for cleanup, but couldn't be cleaned during the stream processing are picked up the next time Auto Loader runs.

Notes about clean source

• Clean source only runs if there is a batch of files to process. It is not a background process that runs independently of ingestion. If there are no new files to ingest in the source directory, clean source does not start for the current stream run. As a result, if a stream stops receiving new files, files that have already passed their retention duration are not cleaned up until a later stream run processes a new batch.

  This batch requirement applies regardless of cloudFiles.cleanSource.waitForCompletion. That option only keeps the stream alive long enough to finish an in-progress cleanup within a run. It does not start clean source when there is no batch to process.

• If a file is ingested on the Nth stream run, the commit_time for the file is set on the N+1 stream run. commit_time must be set before clean source can determine whether a file is eligible for move or delete, so the earliest a file can become a candidate for cleanup is the N+2 stream run.

• Setting commit_time is necessary but not sufficient. A file is only cleaned once its retention duration has elapsed, measured from its commit_time. For example, with the default cloudFiles.cleanSource.retentionDuration of 30 days, a file processed today is not eligible for cleanup until 30 days after its commit_time is set. This holds regardless of how many stream runs occur in between. Both conditions must be met before the file is moved or deleted.