Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Returns the intersection of two binary representations of Datasketches ThetaSketch objects, using a Datasketches Intersection object.
Syntax
from pyspark.sql import functions as sf
sf.theta_intersection(col1, col2)
Parameters
| Parameter | Type | Description |
|---|---|---|
col1 |
pyspark.sql.Column or str |
The first Theta sketch. |
col2 |
pyspark.sql.Column or str |
The second Theta sketch. |
Returns
pyspark.sql.Column: The binary representation of the intersected ThetaSketch.
Examples
Example 1: Get intersection of two Theta sketches
from pyspark.sql import functions as sf
df = spark.createDataFrame([(1,1),(2,2),(3,2),(3,3)], "struct<v1:int,v2:int>")
df = df.agg(
sf.theta_sketch_agg("v1").alias("sketch1"),
sf.theta_sketch_agg("v2").alias("sketch2")
)
df.select(sf.theta_sketch_estimate(sf.theta_intersection(df.sketch1, "sketch2"))).show()
+-----------------------------------------------------------+
|theta_sketch_estimate(theta_intersection(sketch1, sketch2))|
+-----------------------------------------------------------+
| 3|
+-----------------------------------------------------------+