Hi Praveen Sreeram:
Glad to know that you were able to figure this out and thanks for sharing the rootcause. By the way, Microsoft Q&A community has a policy that "The question author cannot accept their own answer, they can only accept answers by others.". Hence request you to accept this as an answer so we can close this thread. I am summarizing the issue and root cause you have found for it.
https://learn.microsoft.com/en-us/answers/support/accept-answer
Issue Summary: Permission error when connecting Azure Synapse Analytics - Dedicated Pool using the PySpark Code that runs from Azure Data Bricks using SQL Authentication. But the error gets resolved only when we provide db_owner permission to the user. For obvious reasons, we can't assign db_owner permission that user.
Rootcause: Permissions at Granular level needed to fix this
Fix provided by MS:
GRANT CREATE TABLE TO [<your_domain_user>@<your_domain_name>.com];
GRANT ALTER ON SCHEMA::<target_database_schema_name> TO [<your_domain_user>@<your_domain_name>.com];
--Make sure your user has ADMINISTER DATABASE BULK OPERATIONS permissions
GRANT ADMINISTER DATABASE BULK OPERATIONS TO [<your_domain_user>@<your_domain_name>.com];
--Make sure your user has INSERT permissions on the target table
GRANT INSERT ON <your_table> TO [<your_domain_user>@<your_domain_name>.com]
Additional permission if using copy command:
Provide CONTROL permission on Database Level