parquet （DataFrameReader）

載入 Parquet 檔案並回傳結果為 DataFrame.

語法

parquet(*paths, **options)

參數

參數	類型	說明
*paths	str	一個或多個檔案路徑可以讀取 Parquet 檔案。

退貨

DataFrame

Examples

將 DataFrame 寫入 Parquet 檔案並讀取回來。

import tempfile
df = spark.createDataFrame(
    [(10, "Alice"), (15, "Bob"), (20, "Tom")], schema=["age", "name"])

with tempfile.TemporaryDirectory(prefix="parquet") as d:
    df.write.mode("overwrite").format("parquet").save(d)
    spark.read.parquet(d).orderBy("name").show()
    # +---+-----+
    # |age| name|
    # +---+-----+
    # | 10|Alice|
    # | 15|  Bob|
    # | 20|  Tom|
    # +---+-----+

讀取多個 Parquet 檔案並合併結構。

import tempfile
df = spark.createDataFrame(
    [(10, "Alice"), (15, "Bob"), (20, "Tom")], schema=["age", "name"])
df2 = spark.createDataFrame([(70, "Alice"), (80, "Bob")], schema=["height", "name"])

with tempfile.TemporaryDirectory(prefix="parquet1") as d1:
    with tempfile.TemporaryDirectory(prefix="parquet2") as d2:
        df.write.mode("overwrite").format("parquet").save(d1)
        df2.write.mode("overwrite").format("parquet").save(d2)

        spark.read.option(
            "mergeSchema", "true"
        ).parquet(d1, d2).select(
            "name", "age", "height"
        ).orderBy("name", "age").show()
        # +-----+----+------+
        # | name| age|height|
        # +-----+----+------+
        # |Alice|NULL|    70|
        # |Alice|  10|  NULL|
        # |  Bob|NULL|    80|
        # |  Bob|  15|  NULL|
        # |  Tom|  20|  NULL|
        # +-----+----+------+