If df is the name of your DataFrame, there are two ways to get unique rows:
df2 = df.distinct()
df2 = df.drop_duplicates()
When you load data into dataframes in databricks how can you make sure the rows in dataframes are not duplicated.
In SQL you can handle by using unique constraint on the tables. how this can be handle in dataframes to ensure rows are not duplicated.