FileType

Applies to: check marked yes Databricks Runtime 18 LTS and above

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.

FileType is the PySpark type for the SQL FILE type, a reference to an unstructured file and its metadata. Use it to declare FILE parameters and return types in Python user-defined functions (UDFs). In Python, a FILE value is a FileRef object, which holds the file's metadata and reads its bytes.

For the SQL type reference and conceptual overview, see FILE type and FILE type and unstructured data. For file-processing UDF examples, see Process files with UDFs.

Note

The FILE type isn't supported on serverless notebooks. It is supported on notebooks attached to serverless Databricks SQL warehouses.

Import

from pyspark.sql.types import FileType, FileRef

FileType

FileType is a subclass of pyspark.sql.types.DataType. Use it as the parameter or return type of a UDF, or as a field type in a schema.

FileType doesn't specify MANAGED or EXTERNAL. Those qualifiers apply only to a FILE table column, where the column determines whether a reference is stored as managed or external. A UDF passes and returns FILE references, and the target column decides how each reference is stored when you write the result to a table.

You can use FileType in the following ways, which also apply to SQL and Scala UDFs and to SQL stored procedures:

  • As a top-level type.
  • Nested inside a StructType or an ArrayType.
  • As the value type of a MapType, but not as a MapType key.
  • Inside a VariantType, but only for external files.

When a query returns a FILE column to Python, either inside a UDF or through DataFrame.collect(), each value is a FileRef.

FileRef

A FileRef is the Python value for a FILE. It holds the file's metadata and has methods to read the file's bytes and to create new references.

Attributes

Attribute Type Description
uri str The URI of the file. Always set.
offset int An offset into the file, in bytes.
size int The size of the file in bytes.
content_type str The MIME type of the file, when known.
checksum str An integrity token for the file's bytes, of the form <algorithm>:<digest>. For the recognized algorithms, see Checksums.

Methods

Method Description
as_local_file() Returns a pathlib.Path to the file. Pass the result to any library that accepts a path. Available on Azure Databricks compute (notebooks and UDF workers). Not available on a Databricks Connect client, such as an IDE or local application that runs your code outside Azure Databricks compute.
open() Opens the file for binary reading and returns a file object. The caller closes it. Has the same compute requirement as as_local_file().
from_bytes(content, path=None, content_type=None) Class method. Uploads content (a bytes value) to the Unity Catalog volume path given by path and returns a FileRef. Fails if a file already exists at the target path. Supported only inside a UDF.
from_local_file(local_file, path=None, content_type=None) Class method. Uploads a local file to a Unity Catalog volume and returns a FileRef. The path and content_type parameters behave as in from_bytes. Supported only inside a UDF.

Example

In the following code, a scalar UDF receives a FILE value as a FileRef, opens the image, and returns its dimensions:

from pyspark.sql.functions import col, udf
from pyspark.sql.types import StringType
from PIL import Image

@udf(returnType=StringType())
def image_resolution(file):
    with Image.open(file.as_local_file()) as img:
        return f"{img.width}x{img.height}"

spark.read.table("images").select(image_resolution(col("photo"))).display()

For more file-processing UDF examples, including generating files with a UDTF, see Process files with UDFs. For general UDF authoring, see Python scalar user-defined functions (UDFs) and Python user-defined table functions (UDTFs).

Limitations

Using FileType in PySpark has the following limitations:

  • PySpark doesn't support declaring FILE MANAGED or FILE EXTERNAL table columns or ingesting files in bulk. Use SQL for those operations. PySpark only supports FILE, as FileType, in UDFs and file reads.
  • as_local_file() and open() require cluster-side access, so they aren't available on a Databricks Connect client. Call them in a UDF or on cluster compute instead.
  • For from_bytes and from_local_file, Python UDFs infer content_type from the path extension, whereas Scala UDFs infer it from the file's magic bytes.
  • Returning a FileRef from a UDF that writes to a FILE MANAGED column isn't supported.
  • Last updated on