Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
A transform for timestamps to partition data into hours.
Note
This function can be used only in combination with DataFrameWriterV2.partitionedBy method.
Syntax
from pyspark.sql.functions import partitioning
partitioning.hours(col)
Parameters
| Parameter | Type | Description |
|---|---|---|
col |
pyspark.sql.Column or str |
Target date or timestamp column to work on. |
Examples
from pyspark.sql.functions import partitioning
df.writeTo("catalog.db.table").partitionedBy(
partitioning.hours("ts")
).createOrReplace()