הערה
הגישה לדף זה מחייבת הרשאה. באפשרותך לנסות להיכנס או לשנות מדריכי כתובות.
הגישה לדף זה מחייבת הרשאה. באפשרותך לנסות לשנות מדריכי כתובות.
Return a new DataFrame with duplicate rows removed, optionally only considering certain columns.
Syntax
dropDuplicates(subset: Optional[List[str]] = None)
Parameters
| Parameter | Type | Description |
|---|---|---|
subset |
list of column names, optional | List of columns to use for duplicate comparison (default All columns). |
Returns
DataFrame: DataFrame without duplicates.
Notes
For a static batch DataFrame, it just drops duplicate rows. For a streaming DataFrame, it will keep all data across triggers as intermediate state to drop duplicates rows. You can use withWatermark to limit how late the duplicate data can be and the system will accordingly limit the state. In addition, data older than watermark will be dropped to avoid any possibility of duplicates.
Examples
from pyspark.sql import Row
df = spark.createDataFrame([
Row(name='Alice', age=5, height=80),
Row(name='Alice', age=5, height=80),
Row(name='Alice', age=10, height=80)
])
df.dropDuplicates().show()
# +-----+---+------+
# | name|age|height|
# +-----+---+------+
# |Alice| 5| 80|
# |Alice| 10| 80|
# +-----+---+------+
df.dropDuplicates(['name', 'height']).show()
# +-----+---+------+
# | name|age|height|
# +-----+---+------+
# |Alice| 5| 80|
# +-----+---+------+