This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 2%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDV%3C/text%3E%3C/svg%3E)
I want to build a multi-tenant deltalake based on a database which contains all data of all tenants. I am using Azure Synapse Pipelines and Spark Notebooks. In the database there is one table which contains all tenants. Besides that we have several other tables which are linked to the tenant table. I want to store data of each tenant in a different storage container. I have a table with organisation units. Each organisation unit belongs to a tenant. A tenant can have multiple organisation units. OrgUnitName TenantId Dept 1 1 Dept 1.1 1 Dept 2 1 Sales 2 Finance 2
I want to come to situation in my azure storage account that I have for each tenant a storage container. In that storage container, I want to create a delta table for each table in the source database. My first idea was to build a foreach loop in Pipelines over the different tenants. For each tenant I would call a spark notebook which would load each individual table. I have the impression that this process is very very slow. I would like to use the parallellism power of Spark in order to achieve this, but the "partitionBy"-functionality of a dataframe is always limited to one storage container. Any advice?
See mainly above. Everthing is explained there :-)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
@Dirk Vrancken Hello and welcome to Microsoft Q&A.
I understand you want to partition your Synapse dataframe over multiple containers.
I think you were onto something with "partitionBy". I was thinking of putting the write inside a "forEachPartition".
Doing a forEach in the Synapse Pipeline would be slowest if you chose the "sequential" option. Paralleled it would be less slow. I think you are right doing it all in spark would be faster, less overhead.
I've been trying to investigate whether the inside of "foreachParition" is writable, but I must be doing something wrong, as it won't even display/show/print for me. For a while i was looking at groupBy. Now I'm looking into RDD.
@Dirk Vrancken Did you find your own solution? If so, please share here with the community.
Hi @MartinJaffer-MSFT ,
At this moment it is solved by a simple For loop in the Spark Notebook script. I loop over every tenant, and for each tenant I can now use the normal "PartitionBy" operation.