How to filter month wise data in ADF

Question

I am trying to filter data for particular month dynamically

Created Parameters while calling notebook from ADF

Created Widgets in databricks

Not able to get data for Jan month it is getting data for all months

Answer 1

@Srinivas Sairi

Thanks for reaching out to Microsoft Q&A.

Based on the information you provided, it seems like you're encountering an issue where your Azure Data Factory (ADF) pipeline isn't filtering data for the desired month even though you've set up parameters and widgets in Databricks.

Here are some possible reasons why this might be happening:

Incorrect Parameter Usage: Double-check how you're using the P_Reporting_period parameter in your Databricks notebook. Ensure you're referencing it correctly in your filter query.

Data Type Mismatch: Make sure the data type of the P_Reporting_period parameter matches the format of your date column in the data you're filtering.

Filter Logic Issue: Verify the logic you're using to filter the data based on the parameter value. You might need to adjust the syntax depending on the language used in your Databricks notebook (e.g., SQL, Python).

Here's a possible approach to troubleshoot the issue:

Examine the Databricks Notebook: Check how you're using the P_Reporting_period parameter in your filter query.

• Look for typos or inconsistencies in how you're referencing the parameter name.
• Ensure the date format used in the filter matches the format of the P_Reporting_period parameter value.

Test the Filter Logic: Try isolating the filter logic in your notebook. Create a sample dataset and apply the filter using a hardcoded value for the month (e.g., "Jan 2024"). If the filter works correctly with a static value, then the issue likely lies in how you're passing the parameter value from ADF.

Inspect Parameter Passing: Verify how you're passing the P_Reporting_period parameter value from ADF to your Databricks notebook.

• Make sure you're referencing the parameter correctly in the notebook code.
• Check the data type of the parameter being passed and ensure it aligns with the expected format in Databricks.

Hope this helps. Do let us know if you any further queries.