Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,608 questions
0 answers
Efficient Log Handling and Data Retention in Azure Data Factory and Databricks
I need to create a solution to send logs from Azure Data Factory to the Databricks Unity Catalog. I'm considering the following structure: Whenever an activity run results in either failure or success, the corresponding log will be sent to Azure Logic…
asked 2024-07-17T19:44:42.0666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 31%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYD%3C/text%3E%3C/svg%3E)
Yohanna de Oliveira Cavalcanti
160
Reputation points
asked 2024-07-17T19:44:42.0666667+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
1 answer
How to connect (create a linked service) an ADF instance to an SFTP Server that is hosted on an Azure VM?
I am trying to get my ADF instance to connect to my SFTP Server that is hosted on an Azure VM but when testing the connection it always Times Out. The ADF and VM hosting the SFTP are on different VNets and this is because I personally don't have…
asked 2024-07-16T15:29:06.2833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 66%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAL%3C/text%3E%3C/svg%3E)
Andrew Long
0
Reputation points
commented 2024-07-17T19:42:26.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 91%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAN%3C/text%3E%3C/svg%3E)
Aravind Nuthalapati
75
Reputation points Microsoft Employee
1 answer
How to have back of everything for Azure Active Directory?
I want to have backup of everything from Azure Active Directory. How to do it whats the procedure please help me guide it. If 3rd software also can be used Please help. I use Azure active directory and Intune.
asked 2024-07-17T14:47:17.2833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 34%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)
TestUser
40
Reputation points
answered 2024-07-17T19:07:56.0633333+00:00
Dillon Silzer
56,121
Reputation points
2 answers
One of the answers was accepted by the question author.
Data Factory Logs to Databricks
I need to create a way to send logs from Data Factory to the Databricks Catalog. What is the most cost-effective and efficient method to achieve this?
asked 2024-07-16T19:26:32.8633333+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
commented 2024-07-17T18:39:54.08+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
0 answers
Captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks
Good morning, I need assistance in creating a project that captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks. The key requirements for this project are as follows: No Duplicate Logs: Ensuring that the logs are not…
asked 2024-07-17T15:17:27.44+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
edited the question 2024-07-17T15:35:23.9433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 16%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rakesh Gurram
5,080
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
How to Correctly Pass and Use Boolean Values from Datafactory to Databricks Notebook
How can I correctly pass a Boolean value from Datafactory to a Databricks notebook and use it in conditional logic? I configured a pipeline in Datafactory that calls a Databricks notebook. I attempted to pass a Boolean parameter from Datafactory as a…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 32%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
0 answers
Data Factory Logs --> Catolog Databricks
Good morning, I need assistance in creating a project that captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks. The key requirements for this project are as follows: No Duplicate Logs: Ensuring that the logs are not…
asked 2024-07-17T14:53:09.4466667+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
asked 2024-07-17T14:53:09.4466667+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
0 answers
ADF Data Flow - CDC - Schema Drift not working
Hi, I have setup ADF pipeline for data sync of tables between Azure SQL Database to SQL Managed Instance databases. I am using Data Flow and CDC to track changes and sink activity to replicate it to target. Since there are multiple tables involved, I am…
asked 2024-05-08T04:10:23.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 83%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Ashwani SRIVASTAVA
60
Reputation points
commented 2024-07-17T14:48:52.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 69%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERP%3C/text%3E%3C/svg%3E)
Raluca Pojar
0
Reputation points Microsoft Employee
4 answers
How do you access the Airflow CLI/API/DB in Workflow Orchestration Manager?
Working with Airflow, one often requires access to the Airflow CLI/API or even access to the underlying meta database. For example: to import or export connections start backfills manage Variables (using scripts) manage DB items that are…
asked 2024-07-03T10:33:38.4866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 19%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
aeteq
0
Reputation points
commented 2024-07-17T14:34:43.21+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,445
Reputation points Microsoft Vendor
5 answers
how to fetch data from Azure Active Directory(AD) by using either ADF or databricks
To fetch data from Azure Active Directory (AD) using either Azure Data Factory (ADF) or Azure Databricks, Pleae let me know in detail. thanks
asked 2024-07-04T10:41:44.7933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
30
Reputation points
commented 2024-07-17T14:10:25.3766667+00:00
Smaran Thoomu
12,445
Reputation points Microsoft Vendor
2 answers
How to flatten nested json array in ADF where the first node does not have key and starts with values which is not constant
I am trying to parse nested JSON in my pipeline to store the values such as table_name, lastSuccessfulWriteTimestamp, totalProcessedRecordsCount, dataFilesPath, schemaHistory_value1, schemaHistory_value2 from below JSON data using ADF dataflow activity.…
asked 2024-07-08T17:22:36.6666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 5%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Satish Hadapad
0
Reputation points
commented 2024-07-17T14:05:55.1833333+00:00
Smaran Thoomu
12,445
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
I wan use Show billing report by pipeline in ADF, and enabled this via CI/CD
I want enabled the feature "Show billing report by pipeline " in azure Data Factory, but I don't know how can I enabled this via Terraform deployment or ARM template deployment. How Can I do that ?
asked 2024-07-03T15:15:45.9433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 9%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Benjamin Baele
20
Reputation points Microsoft Employee
commented 2024-07-17T13:59:15.7466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 17%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
PK
0
Reputation points
1 answer
"Extra"."Api_Issue" row limit from Data Factory
Im working with the API from Data Factory and whas doing this query: select * from "Extra"."Api_Issue" Previously it was retrieving like 60K rows but from 24th of June it is only getting EXACTLY 12.400 each day from this table. Has it…
asked 2024-07-15T06:15:30.1833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 2%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EII%3C/text%3E%3C/svg%3E)
Ibai Iglesia Alonso
0
Reputation points
edited a comment 2024-07-17T13:58:23.24+00:00
Smaran Thoomu
12,445
Reputation points Microsoft Vendor
0 answers
ADF Global Parameters validation failed using NPM Run validate command
Below CICD YAML deployment code is getting failed "customCommand: 'run build validate $(Build.Repository.LocalPath) /subscriptions........" is getting failed due to usage of global parameters in various pipelines."with below…
asked 2024-07-17T09:31:23.53+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 77%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ6%3C/text%3E%3C/svg%3E)
JyotiranjanMangaraj-6157
0
Reputation points
edited a comment 2024-07-17T13:38:37.5366667+00:00
Smaran Thoomu
12,445
Reputation points Microsoft Vendor
0 answers
How to Handle Case Sensitivity in Dynamic JSON Input Keys ("A" vs "a") for Derived Columns in ADF Dataflow Pipeline?
I am working on a dataflow pipeline in ADF with an input dataset schema that includes a column 'A'. The dataflow processes various JSON files with dynamic schemas, where some JSON files use 'A' and others use 'a' as keys. Since the schema is case…
asked 2024-07-17T12:59:49.7733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 16%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHP%3C/text%3E%3C/svg%3E)
Hrithik Purwar
0
Reputation points
asked 2024-07-17T12:59:49.7733333+00:00
Hrithik Purwar
0
Reputation points
0 answers
SAS token generation by Databricks to access CSV files from ADLS container folder
Hi Team, There are some csv files zips inside the ADLS container folder. These zip files need to be downloaded for data correction. Downloading the file requires SAS token embedded with zip file path. Databricks has been used to generate the token and…
asked 2024-07-17T05:08:24.3033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 91%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Subhadip Roy
1
Reputation point
commented 2024-07-17T12:44:54.65+00:00
Vinodh247
13,146
Reputation points
0 answers
Upsert Data in SQL Server Using Synapse Notebook and Pyspark
How do I use pyspark in a synapse notebook to upsert data in SQL Server? I am able to read the table with the following code: df = spark.read.jdbc(url=jdbc_url, table="Dim.table_name", properties=properties) But I am not sure how to upsert…
asked 2024-07-17T12:14:43.41+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 69%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Aditya Singh
105
Reputation points
asked 2024-07-17T12:14:43.41+00:00
Aditya Singh
105
Reputation points
0 answers
How to use/pass comparison query operators (like Greater Than $gt/Less Than or Equal To $lte, etc) in Filter condition in Source Dataset of Azure COSMOS for Mongo DB in the COPY activity?
Hello Team, I am sourcing data from Azure COSMOS DB for Mongo DB and load the same to Azure SQL Server DB. The data has to be read/loaded incrementally from source in every run based on a Date column. Could you please help me on how I can pass/use the…
asked 2024-07-03T06:56:49.4266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 5%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Astha Chaturvedi
0
Reputation points
commented 2024-07-17T12:12:04.46+00:00
Smaran Thoomu
12,445
Reputation points Microsoft Vendor
2 answers
Convert UTC timezone to US
Hi All, I need to convert the timezone from UTC to the US timezone. Can anyone please tell me what function I need to use: convertTimezone or convertfromutc? This is because my sharepoint is in the US time…
asked 2024-07-14T16:57:51.3366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 24%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
ADF_Coder
0
Reputation points
commented 2024-07-17T11:50:30.3966667+00:00
Erland Sommarskog
106.2K
Reputation points MVP
1 answer
One of the answers was accepted by the question author.
Best Practices for Automating Pipeline Execution Data Collection in Azure Data Factory
Hello everyone, I am looking for the best practices to create an automated workflow for collecting execution data from Azure Data Factory (ADF) pipelines, storing this data in Azure Data Lake Storage Gen2, and consolidating it into a single table for…
asked 2024-07-16T14:23:15.0633333+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
accepted 2024-07-17T11:38:09.2833333+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points