Copy activity vs notebook which is more perfromant

Dheeraj Kumar Reddy Jogi 21 Microsoft Employee

I need information around the performance and cost incurred while performing copy of data from one location to another in adlsgen2 using the copy activity vs synapse notebook. Can you please recommend which will be a better approach along with details.

AnnuKumari-MSFT 30,361 Reputation points Microsoft Employee

2022-12-07T08:29:44.153+00:00

Hi @Dheeraj Kumar Reddy Jogi ,
Just checking in to see if the below answer helped. Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.
AnnuKumari-MSFT 30,361 Reputation points Microsoft Employee

2023-01-31T06:48:25.63+00:00

Hi Dheeraj Kumar Reddy Jogi ,

Just following up to see if the below answer helped. Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.

1 answer

AnnuKumari-MSFT 30,361 Reputation points Microsoft Employee

2022-12-05T15:59:32.41+00:00
Hi @Dheeraj Kumar Reddy Jogi ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

As I understand your query, you want to know which is a better option in terms of money and performance: Copy activity or notebook. Please let me know if that is not the ask.

Performance: Notebook takes an additional time to spin up the spark cluster which is why copy activity is more performant when your source and sink is ADLS as in your case. For other datastores, it depends on the copy method you opt for: Click here to know tips to achieve best performance using copy activity in Synapse pipelines.

Pricing: Pricing might vary based on the DIU used in copy activity, kind of IR used etc. Similarly in case of notebook, it depends on the size of spark pool.

Explore the pricing calculator here. Additionally ,You can raise support ticket here directly to get more details on pricing related queries which is free of cost .

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.
Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
Please sign in to rate this answer.
Dheeraj Kumar Reddy Jogi 21 Reputation points Microsoft Employee

2022-12-07T17:48:50.633+00:00

IIn Similar to the cost report that was sent, If possible can you please share the estimate for using the apache spark pool instead of SQL pool

AnnuKumari-MSFT 30,361 Reputation points Microsoft Employee

2022-12-09T08:40:00.75+00:00

Hi @Dheeraj Kumar Reddy Jogi ,

If you need a data warehouse, you can create a dedicated SQL pool, which lets you run SQL queries on structured, relational tables. If you want a data lake, then you can create a Spark pool, which lets you use Spark to query both structured and unstructured data.

Both of these types of pools are clusters of virtual machines. The code in the notebook would run on the Spark pool cluster and would store the results on the SQL pool cluster. If you’ve ever accidentally left a cluster of virtual machines running, you’ll know that it can get very expensive very quickly. Microsoft provides some mechanisms to help you avoid that. First, when you create a Spark pool, if you enable Autoscale, you specify the minimum and maximum number of VMs in the cluster. Then the pool automatically scales up and down within that range based on the requirements of the workloads you run on it. You can also enable Auto-pause, which will stop the cluster if it has been inactive for a given period of time. By default, it auto-pauses after 15 minutes.

AnnuKumari-MSFT 30,361 Reputation points Microsoft Employee

2022-12-09T08:40:19.017+00:00

---Continuation ---
SQL pools have similar features, but the underlying architecture is much different. First, it’s important to know the difference between dedicated and serverless SQL pools. Dedicated SQL pool is the new name for Azure SQL Data Warehouse, which is a service that’s been around for quite a few years. It provides both a compute cluster and storage.

When you create a dedicated SQL pool, you specify how many DWUs (or Data Warehousing Units) to allocate. DWUs set the amount of CPU, memory, and I/O in the compute cluster. You can only increase or decrease the number of DWUs manually because there’s no autoscaling feature. Storage space is provided by Azure Storage, so it scales independently from the compute cluster. If you don’t need to run a SQL pool all the time, you can manually pause it when it’s not in use. When it’s paused, you won’t pay for the compute cluster, but you’ll still pay for the storage being used by the data warehouse.

However in serveless sql pool , you don't have to pay for the compute resources in them. You only have to pay for the amount of data processed by your queries. Second, you don’t even have to create them because when you create an Azure Synapse Workspace, it’ll automatically create a serverless SQL pool as well.

---------
Hope it helps.
Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well
Sign in to comment