Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Aggregate function: returns the slope of the linear regression line for non-null pairs in a group, where y is the dependent variable and x is the independent variable.
Syntax
from pyspark.sql import functions as sf
sf.regr_slope(y, x)
Parameters
| Parameter | Type | Description |
|---|---|---|
y |
pyspark.sql.Column or str |
The dependent variable. |
x |
pyspark.sql.Column or str |
The independent variable. |
Returns
pyspark.sql.Column: the slope of the linear regression line for non-null pairs in a group.
Examples
Example 1: All pairs are non-null
from pyspark.sql import functions as sf
df = spark.sql("SELECT * FROM VALUES (1, 1), (2, 2), (3, 3), (4, 4) AS tab(y, x)")
df.select(sf.regr_slope("y", "x")).show()
+----------------+
|regr_slope(y, x)|
+----------------+
| 1.0|
+----------------+
Example 2: All pairs' x values are null
from pyspark.sql import functions as sf
df = spark.sql("SELECT * FROM VALUES (1, null) AS tab(y, x)")
df.select(sf.regr_slope("y", "x")).show()
+----------------+
|regr_slope(y, x)|
+----------------+
| NULL|
+----------------+
Example 3: All pairs' y values are null
from pyspark.sql import functions as sf
df = spark.sql("SELECT * FROM VALUES (null, 1) AS tab(y, x)")
df.select(sf.regr_slope("y", "x")).show()
+----------------+
|regr_slope(y, x)|
+----------------+
| NULL|
+----------------+