create_streaming_table

Use the create_streaming_table() function in a pipeline to create a target table for records output by streaming operations, including create_auto_cdc_flow(), create_auto_cdc_from_snapshot_flow(), and append_flow output records.

Note

The create_target_table() and create_streaming_live_table() functions are deprecated. Databricks recommends updating existing code to use the create_streaming_table() function.

Syntax

from pyspark import pipelines as dp

dp.create_streaming_table(
  name = "<table-name>",
  comment = "<comment>",
  spark_conf={"<key>" : "<value", "<key" : "<value>"},
  table_properties={"<key>" : "<value>", "<key>" : "<value>"},
  path="<storage-location-path>",
  partition_cols=["<partition-column>", "<partition-column>"],
  cluster_by_auto = <bool>,
  cluster_by = ["<clustering-column>", "<clustering-column>"],
  schema="schema-definition",
  expect_all = {"<key>" : "<value", "<key" : "<value>"},
  expect_all_or_drop = {"<key>" : "<value", "<key" : "<value>"},
  expect_all_or_fail = {"<key>" : "<value", "<key" : "<value>"},
  row_filter = "row-filter-clause"
)

Parameters

Parameter	Type	Description
name	str	Required. The table name.
comment	str	A description for the table.
spark_conf	dict	A list of Spark configurations for the execution of this query
table_properties	dict	A dict of table properties for the table.
path	str	A storage location for table data. If not set, use the managed storage location for the schema containing the table.
partition_cols	list	A list of one or more columns to use for partitioning the table.
cluster_by_auto	bool	Enable automatic liquid clustering on the table. This can be combined with cluster_by and define the columns to be use as initial clustering keys, followed by monitoring and automatic key selection updates based on the workload. See Automatic liquid clustering.
cluster_by	list	Enable liquid clustering on the table and define the columns to use as clustering keys. See Use liquid clustering for tables.
schema	str or StructType	A schema definition for the table. Schemas can be defined as a SQL DDL string or with a Python StructType.
expect_all, expect_all_or_drop, expect_all_or_fail	dict	Data quality constraints for the table. Provides the same behavior and uses the same syntax as expectation decorator functions, but implemented as a parameter. See Expectations.
row_filter	str	(Public Preview) A row filter clause for the table. See Publish tables with row filters and column masks.