Re-usable spark frameworks

Question

Re-usable spark frameworks

Bhagavan-Azure, Pavan 1

Hi,

Do we have any pyspark/scala re-usable framework's (notebooks) for SCD Type 1 ,2,3 so that developer can simply pass the parameters like source table name, target table name, primary keys ........other parameters instead of building from scratch every time

Thanks,
Pavan

HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2022-11-17T19:39:18.997+00:00

Hello @Bhagavan-Azure, Pavan ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2022-11-30T21:59:53.01+00:00

Hello @Bhagavan-Azure, Pavan ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread .
Thanks
Himanshu

1 answer

Your answer

HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2022-11-17T19:39:18.997+00:00

Hello @Bhagavan-Azure, Pavan ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2022-11-30T21:59:53.01+00:00

Hello @Bhagavan-Azure, Pavan ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread .
Thanks
Himanshu

Answer 1

Hello @Bhagavan-Azure, Pavan ,
Thanks for the question and using MS Q&A platform.
As we understand the ask here is is there a framework / library to implement SCD , please do let us know if its not accurate.
At this time I don't think that there is anything like that whic you can use . But there are tons of resources on how to implament SCD on spark .
I have found that all these blogs code are longer as they are creating the dataframe to show the implemenation . The actual SCD code is very small and simple .

https://towardsdatascience.com/processing-a-slowly-changing-dimension-type-2-using-pyspark-in-aws-9f5013a36902

Please do let me if you have any queries.
Thanks
Himanshu

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
- If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Share via

Re-usable spark frameworks

1 answer

Your answer