Azure Data Lake Storage

I'm attempting to use Data Factory to transfer a CSV file from my computer to Azure Data Lake. I'm unable to select the path to the file I want to migrate from my local computer when I try to link the file system in the data factory's source tab.

DataLakeFileSystemClient listPath, NumberFormatException when using String

Hello, I am new to this and trying to list all files in a container recursively. ListPathsOptions options = new ListPathsOptions(); options.setRecursive(true); for (PathItem pathItem :fileSystemClient.listPaths(options, null)){ ... } Doing so result…

How to fix MSSQL_101 "INSERTABLE columns list or UPDATABLE column list or key column list is not properly passed to the stored procedure." in Data Flow Sink?

I am trying to load CSV rows from Data Lake into an Azure SQL Database. I use the upsert functionality of the data flow and parameterize it to be able to load the same data into different tables. this is how i designed my data flow: In the alter row…

how setup dfs insted blob endpoint for BlobContainerClient

Hi, In general I need to create app with java which will perform some operations on azure storage like upload file, append to file, rename, check if exist and so on. And IMPORTANT It has to communicate with DFS endpoint https://xxxx.dfs.core.windows.. …

how config azure adls gen2 to migrate from hdfs

Could you please helpme to migrate data from on-premise hadoop to adls gen2, how can i config mi azure tenant, security netowork...

Can't query partitioned tables - Azure Synapse Link for Dataverse

Hi, we have a problem with accessing partitioned tables in Azure Synapse Workspace that were previously created by ingesting tables from Power Platform via Azure Synapse Link for Dataverse. Snapshot folder and .csv files are normally created but can't be…

Unable to mount ADLS

Hi All, I am trying to mount ADLS to Azure Databricks but getting this error - "shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container scd in account…

How to mount dataset or datalake with read/write permission in Notebook of Azure Machine Learning Studio?

I've successfully mounted a dataset with read permission to notebook of my machine learning studio account. But when I try to write back to the datalake/dataset, it throughs "[Errno 30] Read-only file system". How can I mount the dataset with…

ADF Event based trigger not working for ADLS Gen 2 : when a file is manually uploaded or moved via adf pipeline or logic app trigger is working fine but when file is being moved via java code trigger is not working

I am using azure data factory where I have configured an event based trigger pointing to ADLS Gen 2 storage. The trigger expects a file to end with Completed-DS001_01.csv ("Blob Path Ends With") in the file location specified in "Blob…

Copying data from synapses serverless SQL pool view to tables in the pool

Hi , I have created views for all D365 F&O tables in synapses serverless SQL pool using the CDMUtil solution . I am looking to transform the data from views to table in the same or a different DB , i.e. cache mechanism, to reduce the cost of call the…

How to distribute the contents of a spark dataframe over different azure storage containers

I want to build a multi-tenant deltalake based on a database which contains all data of all tenants. I am using Azure Synapse Pipelines and Spark Notebooks. In the database there is one table which contains all tenants. Besides that we have several other…

Copy data from SFTP to ADLS Gen 2 via ADF

I have around 200GB of data in SFTP server in .gz format. I need to copy that into ADLS via ADF. Can it be done.? what would be dataset file format that needs to be taken. The file format should also be the same .gz format in adls. There is no need for…

About backup of azure data lake storage gen2

Hi There! Please tell me how to get a backup of azure data lake storage gen2 online (without interrupting the process on the target resource). We would like to restore the data in azure data lake storage gen2 in case we accidentally delete the…

This XML file does not appear to have any style information associated with it

Hi All, I am trying to download a CSV file from Azure blob storage. I encounter this error while opening in google chrome. Here is the URL. Any pointers on why I am not able to download …

Stream RealTime data from an On Prem Microsoft Server to a Azure Data Lake.

There is a need for Real Time data to be collected from a On Prem SQL Server. This means ADF is not suitable because it only does batch. Azure Streaming Analytics only connects to Azure SQL Server and not on prem. Has anybody else encountered this and…

Getting "put: Status code: -1 error code: null error message: InvalidAbfsRestOperationException" error

Team, Getting below error while ingesting data from our VM to ADLS using below command and find below details. Command: curl -vkS sftp://user@domain.com:/FilePath/ingestion/file.eot | hadoop fs…

CDMUtilSolution Console App error

Hi , I am trying to run CDMUtil_ConsoleApp for converting D365 F&O manifest to synapses DLL format , and I am getting below error . Please help to fix this issue .

ADF DataFlow : BadRequest error with any additional information

Hello, I have a strange error when I only want to use an Excel dataset hosted in ADSL2 as input to a Dataflow. The preview of the data at the Dataset level is not a problem : Unfortunately it becomes impossible at the Input stage of…

how to delete a folder in ADLS with different name

Hi, I have a pipeline who generate parquet file in ADLS gen2 with similar naming patternfor each folder every day. bust some times a bug cause an empty folder generate in all folders with different naming. I would like to use delete activity befor my…

Unable to create View in Azure synapses using CDMUtil_ConsoleApp

I had tried to create a view for D365 F&O data in synapses using CDMUtil_ConsoleApplication.exe to convert the manifest data to DLL . I got stuck with the below error . It has mentioned to create credential in manage identities and CREATE DATABASE…