Azure SQL Database & Azure Databricks

jase jackson USA 201 Reputation points
2021-02-21T19:24:00.787+00:00

Are there scenarios (time/cost) where it is more efficient to replicate SQL stored procedures using databricks. To clarify; you may have a stored procedure that takes 15 minutes in SQL (level P4) whereas using Azure Databricks would offer a quicker processing time within the same or similar costing?

Happy to clarify if the above too vague

Thanks

Azure SQL Database
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,978 questions
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 80,251 Reputation points Microsoft Employee
    2021-02-22T11:09:38.08+00:00

    Hello @jase jackson USA ,

    There are scenarios where spark will be more efficient than standard SQL. For example, calculating a moving average in a large dataset will be always faster in spark (regardless of spark SQL or pyspark) than standard SQL. So the performance benefit of databricks is coming from the efficient usage of spark.

    Deeper understanding of the queries & their plans, the data volumes and parallel execution possibilities for the query are some of the factors that'll come into play here. So, it's difficult to tell if performance will be better straight off the bat. Secondly, there may be some redevelopment effort involved here as well given syntax changes from SQL to Spark SQL.

    Hope this helps. Do let us know if you any further queries.

    ------------

    Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments

0 additional answers

Sort by: Most helpful