Is ADF (and Synapse Analytics) going to being deprecated in favor of "Fabric"

David Beavon 981

Lately I have started seeing tons of advertising for Microsoft "Fabric" (formerly known as the Power BI service portal). This advertising appears to be sprinkled into lots of the documentation for ADF and Synapse, and seems to be redirecting customers to use "Fabric" rather than ADF and Synapse. Moreover, it appears that new development in the original platforms seems to be slowing down. Instead of investing in the original platforms, it looks like the Microsoft developer teams are working on a wholesale migration of all their functionality into "Fabric".

Here is an example of where the docs are suggesting that customers should switch to Fabric:
User's image

https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities?tabs=data-factory

Based on what I can tell, new investment is no longer happening in the original platforms. There are many examples, especially on the Spark side of things. EG. For years customers have been asking the Synapse-Spark team to offer single-node clusters - but it was never prioritized. However that appears to already be available as a feature for "Fabric" notebooks. It seems likely that Microsoft will target their investments at the new Fabric SaaS rather than the original PaaS offerings. The original platforms may go into "mature" phase of life, aka maintenance mode. This is very similar to what happened with Azure Analysis Services.

Should Microsoft customers stop using ADF and Synapse Analytics for new development?

Please do not post AI or bot-generated answers to this question. They are unlikely to be helpful.

Amira Bedhiafi 27,051 Reputation points

2023-10-24T10:58:11.2466667+00:00

Nothing is communicated yet by Microsoft regarding this matter. Just keep in your mind that Microsoft Fabric is in preview and it is an all-in-one analytics solution for enterprises. It covers everything from data movement to data science, real-time analytics, business intelligence, and reporting.

But if I want only " data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale", I will use ADF and no need to invest in MF.

https://learn.microsoft.com/en-us/fabric/get-started/microsoft-fabric-overview
David Beavon 981 Reputation points

2023-10-24T13:11:59.4433333+00:00

When I read that this is an "all-in-one analytics solution for enterprises", it sounds exactly like how Synapse Analytics was once described.... And it seems to target almost the exact same customer audience. It is hard to believe that Microsoft has a long term vision for both products, if the end result is that there would be duplicated efforts and each of them would cannibalize the other's sales numbers.

I've already noticed that a lot more investment seems to be happening in the Power BI portal, rather than in Synapse or ADF. It is very frustrating for those of us who recently adopted these Azure PaaS platforms. It seems obvious that Microsoft has a favorite new toy and they are spending most of their time playing with that - at the expense of Synapse and ADF.

It might be helpful if you can describe the profile of an enterprise customer who should be using Synapse or ADF (rather than Fabric). If there is no target audience for those platforms anymore, then it seems clear that Microsoft is slowly abandoning those platforms and will no longer be making the critical improvements and bug fixes that customers have been waiting for.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
KranthiPakala-MSFT 46,597 Reputation points Microsoft Employee

2023-10-30T22:07:37.1233333+00:00

@David Beavon Thanks for raising this concern. As per conversation with internal team, ADF and synapse pipeline are not getting deprecated. There's hasn't been any statement from ADF or Synapse product team regarding shifting focus to Fabric. I even reached out to respective product owners to see if they have any info regarding your concern and will keep you posted as soon as I hear back from them.
David Beavon 981 Reputation points

2023-11-11T02:48:15.8566667+00:00

@KranthiPakala-MSFT
Do you have your own experiences of these? I think if you used them you would know what I'm talking about.

FYI, I have a number of bugs on this Synapse-Spark platform that I can't get fixed. I've heard it might be possible to get the bugs fixed on a "unified" support contract, but unfortunately we don't have one.

Some of the biggest problem are related to the virtualized networking. My network traffic is constantly broken (many times a day). Even internal ADF components like the "LSR" (linked service resolver) is struggling to connect to local directory resources in the East US region, for the purpose of authenticating the MSI of the Synapse workspace.

I suspect the network problems are related to the proprietary "managed vnet" technology. It is likely that many other users of the azure-managed-vnet-IR are also aware of these problems. Any new customers to the Synapse or ADF platform might consider avoiding that "managed vnet" functionality, and thereby sparing themselves a ton of unpredictable networking errors ("connection reset by peer").

Better yet, just avoid these two Azure platforms altogether if you can. It is clear from the docs that Microsoft is guiding new customers away from the pipelines and Spark pools in Synapse, and encouraging them to use the new "Fabric" instead.

Aside from the docs, my feedback is also based on two years of struggles with the ADF and Synapse platforms, and over a dozen support cases with CSS.

I should say that I have had GREAT experiences with some of the other platforms in Azure. My satisfaction in Azure has depended on learning which products to avoid. I've found that ADF in particular doesn't even offer 24x7 "professional" support if you run into a critical issue over the weekend.
KranthiPakala-MSFT 46,597 Reputation points Microsoft Employee

2023-11-13T18:03:05.11+00:00

@David Beavon I am sorry to hear that you have had difficulties with the ADF and Synapse platforms, but I appreciate your feedback and insights. It is important for us to hear from our customers about their experiences so that we can continue to improve our products and services. I understand that you have encountered several issues with the Synapse-Spark platform and have had difficulty getting them resolved. I have shared your feedback with the respective product owners for their review and will keep you updated on any response or action they may take.

In case if you have any active support case that is blocking you currently, please share the SR# with us here so that I can try my best to expedite the case.

Thank you
David Beavon 981 Reputation points

2023-11-13T18:22:23.0966667+00:00

@KranthiPakala-MSFT

You are more than welcome to try.

I think both CSS and PG are out of their depth at the moment.

They don't even seem to be monitoring their own telemetry, and they keep blaming outside parties without any evidence (like Cosmos DB or another John Doe, like the "Orchestration Team"). And worst of all, even when the problem becomes as severe as a twenty minute outage it still doesn't seem like a significant concern to them . It is obvious that things have gotten way out of control when a PG team starts to use the word "transient" to describe a twenty minute network outage!

I had reported these networking problems four weeks ago, yet haven't had the slightest hint of an RCA yet.

The support case I'm (unsuccessfully) trying to push at the moment is 2310130040012067. It is one of several cases that (I suspect) will have networking bugs as their root cause. The networking is virtualized (aka the underlying software and VM's that are used to compose a "managed vnet"). These virtualized networking bugs are probably isolated to ADF/Synapse. This is probably not a physical infrastructure problem or it would impact every other type of workload I have in East US.
KranthiPakala-MSFT 46,597 Reputation points Microsoft Employee

2023-11-13T21:02:37.3033333+00:00

@David Beavon Thank you for sharing the details and sorry for the inconvenience you have experienced with the support case.

I will reach out to the case owners to see if they can expedite the process and provide you with an update as soon as possible. I appreciate your patience and understanding as we work to resolve this issue.
David Beavon 981 Reputation points

2023-11-15T17:32:40.9966667+00:00

Hi @KranthiPakala-MSFT

I'm not sure what your position is at Microsoft... but I don't believe you would be able to get more traction with this ADF CSS if your name was Satya Nadella. This CSS team seems to have insurmountable troubles.

They haven't investigated the networking bug yet, even after a whole month has passed since my severe outage. The engineer keeps stalling. They try to tell me that they are engaging with one team after another. They have mentioned every possible Azure CSS and PG team by now - EXCEPT the team that would be responsible for the failed networking. They have even started inventing imaginary teams, which have absolutely nothing to do with my Spark issue (like the "orchestration team"). Until they find yet another team to blame, they will continue to tell me they are waiting on the PG that is responsible for "orchestration" (whatever that means). In the past they had even blamed the Cosmos DB team for some of my outages. That seems totally implausible to me. I'm pretty confident that any outages from Cosmos would be acknowledged and reported in the service health dashboard.

I really wish there was some level of accountability for the poor level of support. I suppose the best answer is to find a path that would move us away from these particular platforms which seem like they are already decaying (ADF and Synapse).
David Beavon 981 Reputation points

2023-11-15T17:36:41.3266667+00:00

Hi @KranthiPakala-MSFT If you are a Microsoft employee you might be more interested in the ICM. There has been more than one. I believe the current one is 435 ,, 606 , 040

Share via

Is ADF (and Synapse Analytics) going to being deprecated in favor of "Fabric"

Your answer