I'm using following in the autoloader options. .option("cloudFiles.maxFileAge", "90 days")\ .option("cloudFiles.backfillInterval", "1 day")\ Our data retention policy is 7 years. Shall I use maxFileAge as 7 years which is not good value? And, I want to use backfillInterval to load missing data for all time (Not limited to 90 days). Highly appreciate expert comments to select best values for these parameters. Thank you

@Hiran Amarathunga - Thanks for the question and using MS Q&A platform. It's great to see that you are using Azure Databricks Autoloader for your data ingestion needs. Regarding your question, it is not recommended to set the cloudFiles.maxFileAge to 7 years as it is a very large value and can cause performance issues. The recommended value for cloudFiles.maxFileAge is usually between 30 to 90 days, depending on your specific use case. As for the cloudFiles.backfillInterval , it is used to load missing data for a specific time period. If you want to load missing data for all time, you can set the value to a very large number, such as 100 years. However, keep in mind that setting a very large value can cause performance issues and increase the time it takes to load data. It is recommended to test different values for these parameters and monitor the performance of your Autoloader job to determine the best values for your specific use case. For more details, refer to Azure Databricks - Auto Loader options Hope this helps. Do let us know if you any further queries. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Set cloudFiles.maxFileAge and cloudFiles.backfillInterval values in Autoloader

Accepted answer

PRADEEPCHEEKATLA-MSFT 90,221 Reputation points Microsoft Employee

2024-07-02T03:00:47.9366667+00:00

@Hiran Amarathunga - Thanks for the question and using MS Q&A platform.

It's great to see that you are using Azure Databricks Autoloader for your data ingestion needs. Regarding your question, it is not recommended to set the cloudFiles.maxFileAge to 7 years as it is a very large value and can cause performance issues. The recommended value for cloudFiles.maxFileAge is usually between 30 to 90 days, depending on your specific use case.

As for the cloudFiles.backfillInterval, it is used to load missing data for a specific time period. If you want to load missing data for all time, you can set the value to a very large number, such as 100 years. However, keep in mind that setting a very large value can cause performance issues and increase the time it takes to load data.

It is recommended to test different values for these parameters and monitor the performance of your Autoloader job to determine the best values for your specific use case.

For more details, refer to Azure Databricks - Auto Loader options

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
Hiran Amarathunga 95 Reputation points

2024-07-02T06:59:45.0566667+00:00

@PRADEEPCHEEKATLA-MSFT Thanks for the answer.

But according to documentation, "By setting the backfillinterval, you can control how frequently the system checks for unprocessed files."

If we set cloudFiles.backfillInterval="7 years", how will it work?

Hiran Amarathunga 95 Reputation points

2024-07-02T07:01:53.94+00:00

Hiran Amarathunga 95 Reputation points

2024-07-02T07:02:25.7+00:00

1 day to backfill once a day - Will it load all history data once a day or will it load history data only for 90days?

PRADEEPCHEEKATLA-MSFT 90,221 Reputation points Microsoft Employee

2024-07-02T07:34:20.48+00:00

@Hiran Amarathunga - But according to documentation, "By setting the backfillinterval, you can control how frequently the system checks for unprocessed files."

If we set cloudFiles.backfillInterval="7 years", how will it work?

Setting cloudFiles.backfillInterval to a large value like 7 years will control how frequently the system checks for unprocessed files. However, it is important to note that setting such a large value can cause performance issues and increase the time it takes to load data.

When you set cloudFiles.backfillInterval to 7 years, the system will check for unprocessed files every 7 years. This means that if there are any files that were not processed during the previous 7 years, they will be processed during the next check. However, if you have a large number of unprocessed files, it can take a long time to process them all, which can impact the performance of your system.

In general, it is recommended to set cloudFiles.backfillInterval to a value that is appropriate for your data ingestion needs. If you need to load missing data for all time, you can set the value to a large number like 100 years, but please keep in mind the potential performance impact.

1 day to backfill once a day - Will it load all history data once a day or will it load history data only for 90days?

If you set cloudFiles.backfillInterval to 1 day, the system will check for unprocessed files every day. This means that if there are any files that were not processed during the previous day, they will be processed during the next check.

However, the cloudFiles.maxFileAge parameter determines how far back in time the system will look for unprocessed files. If you set cloudFiles.maxFileAge to 90 days, the system will only look for unprocessed files that are up to 90 days old. This means that if you have files that are older than 90 days and have not been processed yet, they will not be processed even if you set cloudFiles.backfillInterval to 1 day.

In summary, if you set cloudFiles.backfillInterval to 1 day and cloudFiles.maxFileAge to 90 days, the system will check for unprocessed files every day and process any files that are up to 90 days old. It will not load all historical data once a day, but rather process any unprocessed files that fall within the 90-day window.

Hope this helps. Do let us know if you any further queries.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Set cloudFiles.maxFileAge and cloudFiles.backfillInterval values in Autoloader

0 additional answers

Your answer