Microsoft Q&A

Azure Data Lake Analytics

78 questions

An Azure on-demand analytics job service.

Browse all Azure tags

78 questions with Azure Data Lake Analytics tags

Sort by: Updated
1 answer

Security concerns when linking Dataverse to Azure Synapse

Hi all, When linking Dataverse to Azure Synapse I noticed that it is required to have public network access enabled for the Synapse Workspace. The company I work for has a policy where we are required to use IP whitelisting for services. It is unclear…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,600 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
Azure Data Lake Analytics
asked 2023-01-11T10:12:55.5433333+00:00
Thijs van Haren 0 Reputation points
commented 2023-01-26T03:23:33.14+00:00
BhargavaGunnam-MSFT 7,311 Reputation points Microsoft Employee
1 answer

Data Transfer in real time between on-prem and cloud

Hello, i basically need my on-prem MS SQL db to be synced with cloud MS SQL DB at all times. As of now, i am thinking of using Azure Data factory but i fear it cannot perform near realtime data transfer. Other option is not use kafka to move data, but…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,199 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Data Lake Analytics
asked 2023-01-22T18:41:59.0333333+00:00
Rajshekar Mani 1 Reputation point
commented 2023-01-23T13:46:53.21+00:00
Rajshekar Mani 1 Reputation point
2 answers

SaveTable in Datalake Table of Azure Synapse from Synapse Spark pool

I have create a new lake database called DeltaTables and when I'm trying to write a spark dataframe into that Datalake it's showing me success message but I don't see any table created. where as If I write same dataframe in default lakedatabase which is…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,600 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
Azure Data Lake Analytics
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,199 questions
asked 2022-11-05T07:20:22.86+00:00
Mohammed Sohail 61 Reputation points
commented 2022-11-14T09:16:26.393+00:00
PRADEEPCHEEKATLA-MSFT 53,086 Reputation points Microsoft Employee
0 answers

Getting error "'CONNECTIONPROPERTY' is not a recognized built-in function name." when connecting to Parallel Data Warehouse in Azure Data Studio

Getting the before mentioned error when trying to connect Azure Data Studio to a PDW. To be more specific, I can connect since I can query the database but the object explorer in Azure Data Studio doesn't work. I have tried connecting in JetBrains…

Azure Data Lake Analytics
Azure SQL Database
asked 2022-04-29T23:36:34.31+00:00
Eduardo Aguilar 1 Reputation point
commented 2022-10-14T16:04:04.91+00:00
Richard Gonzalez 1 Reputation point
1 answer

move files to DFS

how can we tranfer only required files from emr cluster(pyspark code) to Microsoft DFS

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,199 questions
Azure Data Lake Analytics
asked 2022-09-21T11:09:04.59+00:00
commented 2022-10-12T06:29:11.21+00:00
PRADEEPCHEEKATLA-MSFT 53,086 Reputation points Microsoft Employee
1 answer

Download read only file from azure data lake gen2

I am using azure data factory and I have to copy/download READ ONLY file residing in azure data lake gen2 to storage blob. Is it possible to download file (marked as READ ONLY) from Azure Data Lake gen2? Please advise

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
Azure Data Lake Analytics
Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
248 questions
asked 2022-09-26T14:30:17.35+00:00
Pankaj Joshi 86 Reputation points
commented 2022-10-10T04:16:36.053+00:00
PRADEEPCHEEKATLA-MSFT 53,086 Reputation points Microsoft Employee
1 answer

Data integration

Hi there, I'm seeking guidance on data integration. I'm looking to integrate different sources which have different structures. For example: 2 companies merging. Their data is of same concept but each field could be named differently between…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,600 questions
Azure Data Lake Analytics
asked 2022-09-19T16:17:45.763+00:00
RJ 106 Reputation points
commented 2022-09-27T16:05:26.677+00:00
ShaikMaheer-MSFT 28,096 Reputation points Microsoft Employee
2 answers

Not able to copy data using azcopy from one subscription to another

Hi, I have a scenario where I need to copy data from one storage account to another. Subscription A storage account AA to (=>) Subscription B storage account BB. AA -> BB This copy needs to be recursive as we have a lot of…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,199 questions
Azure Data Lake Analytics
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
1,540 questions
asked 2022-09-22T14:57:53.387+00:00
Sumit Kumar Gupta 1 Reputation point
answered 2022-09-25T05:40:13.457+00:00
Vinodh247 1,996 Reputation points
1 answer One of the answers was accepted by the question author.

How to frame Dynamic Content Concatinate logic using Parameters in SQL Query - Azure Data Factory Pipeline

Hi Experts, Struckup in configuring the syntax for the below query. Sample SQL Query: select column1, substr(fst_path, (instr(fst_path, 'JJJ_image_lib') + LENGTH('JJJ_image_lib'))), fst_index, substr(bck_path, (instr(bck_path,…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,199 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Data Lake Analytics
Not Monitored
Not Monitored
Tag not monitored by Microsoft.
23,704 questions
asked 2022-09-07T13:18:37.46+00:00
Kumar 41 Reputation points
accepted 2022-09-23T08:16:06.763+00:00
Kumar 41 Reputation points
1 answer One of the answers was accepted by the question author.

How to read a Parquet data file in jupyter notebook with R kernel

I am very new to Azure ML. I have a registered data asset with name "test" with Data source as "workspaceblobstore" and there is a parquet file with name "userdata1.parquet" in the "test" dataset with version 2. I…

Azure Data Lake Analytics
asked 2022-09-17T13:05:00.24+00:00
Ankit19 Gupta 46 Reputation points
commented 2022-09-20T09:45:40.913+00:00
Ankit19 Gupta 46 Reputation points
1 answer

Aws aurora connector to azure

I would like to bring data from AWS aurora to Azure or Azure ADLS. Is there a connector from Aurora to Azure? What's best way to bring the data to Azure? This will be on a regular/daily basis.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
Azure Data Lake Analytics
asked 2022-09-14T17:56:46.98+00:00
Raj Krishnamoorthy 1 Reputation point Microsoft Employee
commented 2022-09-19T18:07:02.323+00:00
BhargavaGunnam-MSFT 7,311 Reputation points Microsoft Employee
0 answers

U-SQL activity is loading partial data in ADLS using Azure Data factory

I am executing the U-SQL script using U-SQL activity inside Azure Data Factory, It is loading partial data into ADLS ( 25 GB/50 GB) And when I execute the script using Visual Studio (U-SQL enabled), it is loading complete data (50 GB/50 GB), even If…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Data Lake Analytics
asked 2022-09-14T15:38:52.367+00:00
amikm 1 Reputation point
commented 2022-09-16T14:50:25.373+00:00
amikm 1 Reputation point
1 answer

Unable to view lake databases

Hello, I am unable to view the lake databases in our workspaces in synapse. We are still able to query the data. Receiving error code 500 seen below. I have ran the diagnostic session via PS as advised in the error code and the diagnostic session…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,600 questions
Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
248 questions
Azure Data Lake Analytics
asked 2022-09-08T10:58:09.587+00:00
Adam Nevin 26 Reputation points
commented 2022-09-15T08:21:42.083+00:00
PRADEEPCHEEKATLA-MSFT 53,086 Reputation points Microsoft Employee
2 answers

Invalid api-version exception when submitting job to Data Lake Analytics using Azure .NET SDK

I got below exception when I tried to submit a job to Azure Data Lake Analytics using .NET SDK my code is like this: var res = adlaJobClient.Job.Get(adlaAccountName, new Guid("cc75454e-5ab0-4cc0-84f0-35bd7b09b8a2")); error msg: …

Azure Data Lake Analytics
asked 2022-06-09T15:05:06.32+00:00
Bowen Yang 1 Reputation point
commented 2022-09-13T18:29:46.75+00:00
Jacob Teves 1 Reputation point
1 answer One of the answers was accepted by the question author.

Azure Data Lake Gen2

Hi, Is it so that the Azure Data Lake Gen2 only is supported in test environments? Do anyone know when it will be available for production environments? Thanks in advance.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
Azure Data Lake Analytics
asked 2022-08-31T07:39:49.553+00:00
Alexander 21 Reputation points
accepted 2022-09-08T11:04:58.29+00:00
Alexander 21 Reputation points
1 answer

Synapse create lake table from data lake giving error: Failed to execute query. Error: CREATE DATABASE failed. User database limit has been already reached (20).

I have an existing folder with delta log and parquet files. This was working up until yesterday. I could via workspace/database gui - add a table via "from data lake" and select the folder for that table and it previously would create schema…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,600 questions
Azure Data Lake Analytics
asked 2022-08-11T21:45:47.727+00:00
Leslie 21 Reputation points
commented 2022-09-05T23:23:30.187+00:00
KranthiPakala-MSFT 32,896 Reputation points Microsoft Employee
1 answer

Error 401: "Server failed to authenticate the request" while accesing data from Azure Synapse to ADLS Gens

Hi I get the following error when trying to run the following simple command in Synapse Studio df = spark.read.load('abfss://storagefs1@storage1.dfs.core.windows.net/somefilepath.parquet', format='parquet') It is important to mention : …

Azure Data Lake Analytics
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,600 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,388 questions
asked 2022-08-25T10:28:57.97+00:00
Samer Aamar 1 Reputation point
commented 2022-08-29T04:12:31.72+00:00
ShaktiSingh-MSFT 4,671 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Process ADLS gen2 Files(csv or json ) using python (smaller files)

I could find the Azure documentation to create directory, upload , download files from the for ADLS gen2 data lake using python. Is there any Adlsfilesystem python library which can be used to read the files directly from …

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
Azure Data Lake Analytics
asked 2022-07-09T10:11:04.36+00:00
Prakash14 111 Reputation points
accepted 2022-07-22T09:26:50.993+00:00
Prakash14 111 Reputation points
0 answers

How to create custom role in Azure Data Lake Analytics (ADLA) only for monitoring scenarios

From official doc about role-based access control in ADLA, the Data Lake Analytics Developer role could: Submit jobs. Monitor job status and the progress of jobs submitted by any user. See the U-SQL scripts from jobs submitted by any…

Azure Data Lake Analytics
Azure Role-based access control
Azure Role-based access control
An Azure service that provides fine-grained access management for Azure resources, enabling you to grant users only the rights they need to perform their jobs.
372 questions
asked 2022-06-29T09:02:01.017+00:00
Weiyi Wang 1 Reputation point
commented 2022-06-30T12:28:21.86+00:00
Weiyi Wang 1 Reputation point
1 answer One of the answers was accepted by the question author.

Azure synapse pipelineでのデータ変換について

azure synapseのパイプラインでExcelデータの.xlsxからcsv変換のやり方のやり方がわからないです。Microsoftのドキュメントが分かりにくく理解できないので、画像付きでご教授して頂きたいです。 Converted Japanese => English language: About data conversion in Azure synapse pipeline I don't know how to convert .xlsx to csv of…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,600 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
Azure Data Lake Analytics
Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
248 questions
asked 2022-06-06T07:12:23.427+00:00
ponponpon 481 Reputation points
commented 2022-06-27T05:52:54.317+00:00
ponponpon 481 Reputation points