Azure Synapse Managed Private Endpoint Data Transfer Cost

Question

Azure Synapse Managed Private Endpoint Data Transfer Cost

E P 0

We have a Synapse workspace under one subscription (dev) with a managed private endpoint to a Storage Account in another subscription (production). Data from production is processed in serverless Spark jobs.

We ~~tried to replicate~~ replicated the Synapse setup from dev to the production subscription, and it's functionally ok. But we are seeing data transfer costs. These kind of costs we don't see in the dev subscription, even though we were processing the same amount of data.

We are trying to understand the difference and would appreciate any help.

EDIT: one thing we've found different is that, in the workspace in the production subscription, there's no "fqdns" in the default private endpoint. This one has been created using Bicep and the fields has not been set. The dev one was created using the Azure Portal.

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-04T07:44:59.85+00:00

@E P - Thanks for the question and using MS Q&A platform.

Based on the information you provided, it seems that the difference in data transfer costs between your dev and production subscriptions may be related to the private endpoint configuration.

In your dev subscription, the private endpoint may have been created with default settings that include fully qualified domain names (FQDNs) for the storage account. This means that data transfers between the Synapse workspace and the storage account are routed through the private endpoint, which is not charged for data transfer.

However, in your production subscription, the private endpoint may have been created without FQDNs, which means that data transfers between the Synapse workspace and the storage account are not routed through the private endpoint. Instead, they are routed through the public internet, which may incur data transfer costs.

To confirm this, you can check the private endpoint configuration in both subscriptions and compare them. You can also try adding FQDNs to the private endpoint in the production subscription and see if that reduces the data transfer costs.

I hope this helps! Let me know if you have any further questions.
E P 0 Reputation points

2023-09-04T15:21:26.0566667+00:00

We thought that too. As we couldn't find a way [0] to add the "dfs" FQDN to the default private endpoint we removed it and added a new one, which got the FQDN set.

However, there was no reduction in cost.

Thanks.

[0] Is it possible to add fqdns without recreating the private endpoint?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-08T05:41:24.3333333+00:00
@E P - Yes, it is possible to add FQDNs to an existing private endpoint without recreating it. You can do this by updating the private DNS zone that is associated with the private endpoint.

To add FQDNs to an existing private endpoint, you can follow these steps:

Open the Azure portal and navigate to the private DNS zone that is associated with the private endpoint.

Click on the "Record sets" tab and then click on the "Add" button to add a new record set.

In the "Add record set" blade, enter the FQDN that you want to add and select the appropriate record type (e.g., A or CNAME).

Click on the "Create" button to create the new record set.

Once you have added the FQDN to the private DNS zone, you should be able to access the private endpoint using the new FQDN.

However, please note that adding FQDNs to an existing private endpoint may not necessarily reduce data transfer costs. Other factors, such as the location of your Synapse workspace and Storage Account, may also affect data transfer costs.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-11T08:30:11.36+00:00

@E P - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
E P 0 Reputation points

2023-09-11T13:39:12.1+00:00

No. Maybe I wasn't clear, but I was talking about a managed privated endpoint, which we do not have access.

In the case of the costs, we've found that it was correct. Private endpoint has costs.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-13T10:29:27.9266667+00:00

@E P - Can you please share more details on what do you mean by - managed privated endpoint, which we do not have access?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-18T08:29:06.1266667+00:00

@E P - Just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
E P 0 Reputation points

2023-09-18T19:45:22.8466667+00:00

"Managed Private Endpoint, which we do not have access" - The managed private endpoint that is created by Synapse when the corresponding option is enabled during the workspace creation.

And, so far, we still have doubts about all this, functionally and regarding costs. We are redirecting them to the appropriate client's channels. I'll keep here posted if we get some positive progress.

Thanks.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-25T05:24:29.5833333+00:00

@E P - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
E P 0 Reputation points

2023-09-25T17:18:31.06+00:00

What we found is that, when accessing a restricted storage account, there is a functional difference between running a "standalone" Spark job and running it using an activity in a pipeline. The standalone requires the (manage) private endpoint. When using a pipeline, the access is authorized solely through the workspace managed identity.

That explains the cost difference. However, I couldn't find a clear statement in the documentation.

I think we can close this.

1 answer

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-04T07:44:59.85+00:00

@E P - Thanks for the question and using MS Q&A platform.

Based on the information you provided, it seems that the difference in data transfer costs between your dev and production subscriptions may be related to the private endpoint configuration.

In your dev subscription, the private endpoint may have been created with default settings that include fully qualified domain names (FQDNs) for the storage account. This means that data transfers between the Synapse workspace and the storage account are routed through the private endpoint, which is not charged for data transfer.

However, in your production subscription, the private endpoint may have been created without FQDNs, which means that data transfers between the Synapse workspace and the storage account are not routed through the private endpoint. Instead, they are routed through the public internet, which may incur data transfer costs.

To confirm this, you can check the private endpoint configuration in both subscriptions and compare them. You can also try adding FQDNs to the private endpoint in the production subscription and see if that reduces the data transfer costs.

I hope this helps! Let me know if you have any further questions.
E P 0 Reputation points

2023-09-04T15:21:26.0566667+00:00

We thought that too. As we couldn't find a way [0] to add the "dfs" FQDN to the default private endpoint we removed it and added a new one, which got the FQDN set.

However, there was no reduction in cost.

Thanks.

[0] Is it possible to add fqdns without recreating the private endpoint?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-08T05:41:24.3333333+00:00

@E P - Yes, it is possible to add FQDNs to an existing private endpoint without recreating it. You can do this by updating the private DNS zone that is associated with the private endpoint.

To add FQDNs to an existing private endpoint, you can follow these steps:

Open the Azure portal and navigate to the private DNS zone that is associated with the private endpoint.

Click on the "Record sets" tab and then click on the "Add" button to add a new record set.

In the "Add record set" blade, enter the FQDN that you want to add and select the appropriate record type (e.g., A or CNAME).

Click on the "Create" button to create the new record set.

Once you have added the FQDN to the private DNS zone, you should be able to access the private endpoint using the new FQDN.

However, please note that adding FQDNs to an existing private endpoint may not necessarily reduce data transfer costs. Other factors, such as the location of your Synapse workspace and Storage Account, may also affect data transfer costs.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-11T08:30:11.36+00:00

@E P - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
E P 0 Reputation points

2023-09-11T13:39:12.1+00:00

No. Maybe I wasn't clear, but I was talking about a managed privated endpoint, which we do not have access.

In the case of the costs, we've found that it was correct. Private endpoint has costs.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-13T10:29:27.9266667+00:00

@E P - Can you please share more details on what do you mean by - managed privated endpoint, which we do not have access?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-18T08:29:06.1266667+00:00

@E P - Just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
E P 0 Reputation points

2023-09-18T19:45:22.8466667+00:00

"Managed Private Endpoint, which we do not have access" - The managed private endpoint that is created by Synapse when the corresponding option is enabled during the workspace creation.

And, so far, we still have doubts about all this, functionally and regarding costs. We are redirecting them to the appropriate client's channels. I'll keep here posted if we get some positive progress.

Thanks.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-09-25T05:24:29.5833333+00:00

@E P - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
E P 0 Reputation points

2023-09-25T17:18:31.06+00:00

What we found is that, when accessing a restricted storage account, there is a functional difference between running a "standalone" Spark job and running it using an activity in a pipeline. The standalone requires the (manage) private endpoint. When using a pipeline, the access is authorized solely through the workspace managed identity.

That explains the cost difference. However, I couldn't find a clear statement in the documentation.

I think we can close this.

Answer 1

@E P - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: We have a Synapse workspace under one subscription (dev) with a managed private endpoint to a Storage Account in another subscription (production). Data from production is processed in serverless Spark jobs.

We ~~tried to replicate~~ replicated the Synapse setup from dev to the production subscription, and it's functionally ok. But we are seeing data transfer costs. These kind of costs we don't see in the dev subscription, even though we were processing the same amount of data.

We are trying to understand the difference and would appreciate any help.

Solution: What we found is that, when accessing a restricted storage account, there is a functional difference between running a "standalone" Spark job and running it using an activity in a pipeline. The standalone requires the (manage) private endpoint. When using a pipeline, the access is authorized solely through the workspace managed identity.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Share via

Azure Synapse Managed Private Endpoint Data Transfer Cost

1 answer

Your answer