TableValuedFunction.python_worker_logs

A Python-feldolgozóktól gyűjtött naplók adatkeretét adja vissza.

Szemantika

spark.tvf.python_worker_logs()

Paraméterek

Ez a függvény nem vesz fel paramétereket.

Visszatérítések

pyspark.sql.DataFrame: A Python-feldolgozók naplóit tartalmazó DataFrame.

Példák

import pyspark.sql.functions as sf
import logging

@sf.udf("string")
def my_udf(x):
    logger = logging.getLogger("my_custom_logger")
    logger.warning("This is a warning")
    return str(x)

spark.conf.set("spark.sql.pyspark.worker.logging.enabled", "true")
spark.range(1).select(my_udf("id")).show()

+----------+
|my_udf(id)|
+----------+
|         0|
+----------+

import pyspark.sql.functions as sf
import logging

@sf.udf("string")
def my_udf(x):
    logger = logging.getLogger("my_custom_logger")
    logger.warning("This is a warning")
    return str(x)

spark.conf.set("spark.sql.pyspark.worker.logging.enabled", "true")
spark.range(1).select(my_udf("id")).show()
spark.tvf.python_worker_logs().select(
    "level", "msg", "context", "logger"
).show(truncate=False)

+-------+-----------------+---------------------+----------------+
|level  |msg              |context              |logger          |
+-------+-----------------+---------------------+----------------+
|WARNING|This is a warning|{func_name -> my_udf}|my_custom_logger|
+-------+-----------------+---------------------+----------------+