Moving multiple SQL server data warehouse databases from On premise to Azure
Dear Experts, I am new to Azure. I have few questions please help me. we have one SQL server dw database per each customer on premise for reporting (using SSRS). Each dw database consists few dimension tables, one fact and one staging table. …
Can't write Pandas df to ADLS from Synapse Notebook - No such file or directory exists.
I'm using mount point in synapse notebook as shown in MS Azure Docs. I'm using same mount point and same path to read and write spark dataframe and that is working fine but while operation with pandas I'm facing this issue. import pandas as pd …
How do I add a new string to the folder name? - Azure Synapse Analytics
I would like to add a new string to the Blob folder name when bulk copying data from S3 to Azure Blob Storage. The folder structure of S3 is shown in the following picture Folder Name/yyyy/mm/dd/files. I would like to copy it to Blob…
I would like to add the date and time string in addition to the source file name.
When copying a file from S3 to AzureBlobStorage, I would like to add the date and time string in addition to the source file name. In essence, the S3 folder structure looks like this data/yyyy/mm/dd/files *yyyy=2019-2022, mm=01-12, dd=01-31 …
Synapse Link for Dataverse can't be accessed in pipelines
Hi, we setup the link between dataverse and synapse, sync our table. fine, working. I can run SQL statements in Azure Synapse. but as soon as I try to connect a pipeline activity, like a simple copy, I can't. I have this error: …
Issue sinking from Azure Data Factory v2 to Azure Synapse Warehouse with Greek letters
We are trying to import data that contains Greek letters into Azure Synapse Analytics (Data Warehouse) via Azure Data Factory v2. We are getting replacement characters for the Greek characters. Collation for Data Warehouse:…
Create synapse credential for power bi
Hi, I wish to create a az synapse analytics credential for power bi that has a scheduled load
BlobServiceClient timeout
Problem: we are reaching a timeout error with the BlobServiceClient instead of receving a error that shows that the credentials are invalid: from azure.storage.blob import BlobServiceClient …
Is there a way to see how expression gets evalated in "if condition"
Hi, I have a very basic doubt. Suppose i have an If condition activity in my pipeline. It has an expression that gets evaluated to a boolean value. I want to see how the expression is interpreted or evaluated in run time, in other words whats…
How to Flatten A Complex Nested JSON Structure in Synapse
As the title suggests, I'm trying to take a complex JSON file, flatten it, and put it into a dedicated SQL pool table. The data is an output from the Shopify API orders endpoint. Here is a sample, it is one single order: [ { …
what-is-the-synapse lake database -security-setup-to-allow-reader
I have a requirement where i have to give read access to the user in Azure Data Lake Synapse , is there any way by which we can provide read access to the user ? As of now I have observed that we need to provide user admin access to the synapse…
Using XML in Synapse spark pool
Hi, Can anybody help with using the XML in Synapse spark pool with pyspark? I found some articles where they suggest a code like this would load the XML into a data frame, but I get an error when trying it: …
Failing select command of external statement
Msg 105019, Level 16, State 1, Line 12 External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_IsDirExist. Java exception message: HdfsBridge::isDirExist - Unexpected…
Dedicated SQL Pool - Can I import a sql server table or a sql server database into azure Synapse dedicated sql pool?
I have existing sql server tables. I would like to archive them in Azure Synapse Dedicated pool as it is said that one can use T-sql to query in Azure Synapse. My question is: Can I import sql server tables into Azure Synapse ( structure and…
Azure Synapse Analytics: Publishing error Error while publishing: Cannot read properties of undefined (reading 'constructor')
I tried to: -delete last changes -remove git and reenable it
How to refresh data in Synapse Lake Database using pipeline DataFlow
I have a pipeline set up in Synapse which runs 2 dataflow activities which is supposed to 1. copy data from Fhir as source to Azure SQL Server as sink and 2. copy data from Azure SQL Server as source to Azure Lake as sink. First, is this the best…
synapse spark pool - pyspark load a subset of xml files from given folder
I am limited to use 4 vCores / 32 GB - 3 to 10 nodes configuration of apache spark pool. I am trying to load all xml files from given folder with below code: …
Flowlets question
Hi, I am exploring the usage of flowlets and wonder how I should implement the following scenario: I have a dataflow which reads a source with 10 columns, one of these columns contains a customer number for which I want a flowlet to execute some logic on…
Data trasnfer across teants using flows
Hi All, I want to copy data from Kusto(MSFT tenant) and ingest to ADLS Gen2(PME tenant). How to do it securely? Can I do it without using App Registration?
LinkedService in Deleting state but not actually being deleted
I'm running the following command via a Azure Powershell job in my Release Pipeline in Azure DevOps: Install-Module Az.Synapse -RequiredVersion 0.7.0 -Scope CurrentUser -Force Remove-AzSynapseLinkedService -WorkspaceName…