Azure Data Lake Analytics

Migration of ADLS gen1 to gen2 - U-SQL scripts to ADLS Gen2

We are in the process of migrating ADLS Gen1 to ADLS Gen2.  We understand that ADLA (Azure Data Lake Analytics) is not supported on ADLS Gen2. Can you please support us in how we can migrate U-SQL scripts to ADLS Gen2? Is there a quick workaround for…

Security concerns when linking Dataverse to Azure Synapse

Hi all, When linking Dataverse to Azure Synapse I noticed that it is required to have public network access enabled for the Synapse Workspace. The company I work for has a policy where we are required to use IP whitelisting for services. It is unclear…

SaveTable in Datalake Table of Azure Synapse from Synapse Spark pool

I have create a new lake database called DeltaTables and when I'm trying to write a spark dataframe into that Datalake it's showing me success message but I don't see any table created. where as If I write same dataframe in default lakedatabase which is…

move files to DFS

how can we tranfer only required files from emr cluster(pyspark code) to Microsoft DFS

Download read only file from azure data lake gen2

I am using azure data factory and I have to copy/download READ ONLY file residing in azure data lake gen2 to storage blob. Is it possible to download file (marked as READ ONLY) from Azure Data Lake gen2? Please advise

Data integration

Hi there, I'm seeking guidance on data integration. I'm looking to integrate different sources which have different structures. For example: 2 companies merging. Their data is of same concept but each field could be named differently between…

Not able to copy data using azcopy from one subscription to another

Hi, I have a scenario where I need to copy data from one storage account to another. Subscription A storage account AA to (=>) Subscription B storage account BB. AA -> BB This copy needs to be recursive as we have a lot of…

How to frame Dynamic Content Concatinate logic using Parameters in SQL Query - Azure Data Factory Pipeline

Hi Experts, Struckup in configuring the syntax for the below query. Sample SQL Query: select column1, substr(fst_path, (instr(fst_path, 'JJJ_image_lib') + LENGTH('JJJ_image_lib'))), fst_index, substr(bck_path, (instr(bck_path,…

How to read a Parquet data file in jupyter notebook with R kernel

I am very new to Azure ML. I have a registered data asset with name "test" with Data source as "workspaceblobstore" and there is a parquet file with name "userdata1.parquet" in the "test" dataset with version 2. I…

Aws aurora connector to azure

I would like to bring data from AWS aurora to Azure or Azure ADLS. Is there a connector from Aurora to Azure? What's best way to bring the data to Azure? This will be on a regular/daily basis.

U-SQL activity is loading partial data in ADLS using Azure Data factory

I am executing the U-SQL script using U-SQL activity inside Azure Data Factory, It is loading partial data into ADLS ( 25 GB/50 GB) And when I execute the script using Visual Studio (U-SQL enabled), it is loading complete data (50 GB/50 GB), even If…

Unable to view lake databases

Hello, I am unable to view the lake databases in our workspaces in synapse. We are still able to query the data. Receiving error code 500 seen below. I have ran the diagnostic session via PS as advised in the error code and the diagnostic session…

Invalid api-version exception when submitting job to Data Lake Analytics using Azure .NET SDK

I got below exception when I tried to submit a job to Azure Data Lake Analytics using .NET SDK my code is like this: var res = adlaJobClient.Job.Get(adlaAccountName, new Guid("cc75454e-5ab0-4cc0-84f0-35bd7b09b8a2")); error msg: …

Azure Data Lake Gen2

Hi, Is it so that the Azure Data Lake Gen2 only is supported in test environments? Do anyone know when it will be available for production environments? Thanks in advance.

Synapse create lake table from data lake giving error: Failed to execute query. Error: CREATE DATABASE failed. User database limit has been already reached (20).

I have an existing folder with delta log and parquet files. This was working up until yesterday. I could via workspace/database gui - add a table via "from data lake" and select the folder for that table and it previously would create schema…

Error 401: "Server failed to authenticate the request" while accesing data from Azure Synapse to ADLS Gens

Hi I get the following error when trying to run the following simple command in Synapse Studio df = spark.read.load('abfss://storagefs1@storage1.dfs.core.windows.net/somefilepath.parquet', format='parquet') It is important to mention : …

Process ADLS gen2 Files(csv or json ) using python (smaller files)

I could find the Azure documentation to create directory, upload , download files from the for ADLS gen2 data lake using python. Is there any Adlsfilesystem python library which can be used to read the files directly from …

How to create custom role in Azure Data Lake Analytics (ADLA) only for monitoring scenarios

From official doc about role-based access control in ADLA, the Data Lake Analytics Developer role could: Submit jobs. Monitor job status and the progress of jobs submitted by any user. See the U-SQL scripts from jobs submitted by any…

Azure synapse pipelineでのデータ変換について

azure synapseのパイプラインでExcelデータの.xlsxからcsv変換のやり方のやり方がわからないです。Microsoftのドキュメントが分かりにくく理解できないので、画像付きでご教授して頂きたいです。 Converted Japanese => English language: About data conversion in Azure synapse pipeline I don't know how to convert .xlsx to csv of…

I have a issue with data lake gen2 with multiple folders and i need to delete grathan 60 days files in gen with multiple folders