current_timestamp-1 in dataframe

Shambhu Rai 1,411 Reputation points
2023-11-29T04:40:44.8166667+00:00

Hi Expert

i wanted to fetch the data for last one hour in databricks dataframe

i tried below filter on top of select statement but gives a error integer ca not bee converted .. how we can get last one hour data in dataframe

.filter(df['Updatedtime'].between
( current_timestamp(),1 )
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,422 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,604 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,661 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,063 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,101 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Suba Balaji 11,206 Reputation points
    2023-11-29T09:21:40.4+00:00

    Hi
    Shambhu Rai

    Please try this expression

    filter(df['Updatedtime'].between( current_timestamp(),current_timestamp()-1)

    In your case, you are passing current_timestamp and 1 to the between function and hence you are getting the error. We need to pass two valid timestamps for it to work fine.

    Kindly try above expression and let us know if that works.

    Thanks