Data engineering best practices

The following articles provide best practices for data engineering in Azure Databricks.

• Optimize join performance in Azure Databricks
• Data modeling
• Configure RocksDB state store on Azure Databricks
• Asynchronous state checkpointing for stateful queries
• Asynchronous progress tracking
• Production considerations for Structured Streaming
• Clean and validate data with batch or stream processing
• Observability in Azure Databricks for jobs, Lakeflow Spark Declarative Pipelines, and Lakeflow Connect
• Fan-in and fan-out architecture in Lakeflow Spark Declarative Pipelines
• Best practices for Lakeflow Spark Declarative Pipelines

For links to other best practices articles, including CI/CD workflows best practices, see Best practice.