Data engineering best practices

The following articles provide best practices for data engineering in Azure Databricks.

• Optimize join performance in Azure Databricks
• Data modeling
• Configure RocksDB state store on Azure Databricks
• Asynchronous state checkpointing for stateful queries
• Asynchronous progress tracking
• Production considerations for Structured Streaming
• Run multiple Structured Streaming queries on the same cluster
• Clean and validate data with batch or stream processing
• Observability in Azure Databricks for jobs, Lakeflow pipelines, and Lakeflow Connect
• Fan-in and fan-out architecture in Lakeflow pipelines
• Best practices for Lakeflow pipelines

For links to other best practices articles, including developer and CI/CD workflows best practices, see Best practice.