Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,352 questions
0 answers
Issue with max block count in Azure Data Lake Gen2
Starting from 03/02/23 I have noticed that my Stream Analytics Job has been creating more files per day than before. Looking through the files' properties I discovered that every file maxes out at 10000 blocks and then the SA Job creates another one,…
asked 2023-04-12T14:00:18.7933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
commented 2024-02-01T16:01:56.7666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 70%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFM%3C/text%3E%3C/svg%3E)
Forlani Mauro
1
Reputation point
0 answers
How to hide a column in Azure Stream Analytics Data Preview
Is there anyways to hide the column in Azure Stream Analytics Data Preview? In my case, I want to hide column "properties" in data preview. Thanks!
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 48%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETN%3C/text%3E%3C/svg%3E)
0 answers
Difference between schema.history and database.history in mysql connector
I set up kafka, mysql and debezium in my computer successfully. However, i have still not understood about the difference between database.history.kafka.topic and schema.history.internal.kafka.topic. Below is my connector: Please help me explaining the…
asked 2024-01-16T13:56:59.92+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 93%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EET%3C/text%3E%3C/svg%3E)
Eddy Tran
40
Reputation points
edited the question 2024-01-25T04:43:02.8633333+00:00
PRADEEPCHEEKATLA-MSFT
77,901
Reputation points Microsoft Employee
0 answers
Azure Function concurrency issues
I have multiple azure functions running at the same time and creating multiple instances of each.The parameters are also passed in parallel that means at one time all the azure function urls will be hit at once.Below is an example of how I am…
asked 2023-12-22T10:35:13.3066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 93%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rafay Khan
20
Reputation points
commented 2024-01-02T11:37:44.65+00:00
Rafay Khan
20
Reputation points
0 answers
ASA run same code with different result?
The following 2 codes get different result: FROM IotBaseInput as event CROSS APPLY GetArrayElements(event.monoEvent) AS arrayElement CROSS APPLY GetRecordProperties(arrayElement.arrayvalue) AS recordProperty Where recordProperty.PropertyName !=…
asked 2023-12-20T03:39:25.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 14.000000000000002%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
jane
60
Reputation points
commented 2023-12-27T18:11:58.2733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
26,316
Reputation points Microsoft Employee
0 answers
Using multiple, mixed length Windows in Stream analytics query results in high (100%) SU memory utlization
I want to use Windows to collect both 3-hour and 1 day max, min, average and count of certain device messages arriving to iot hub. If I use both windows, as in code snippet, SU memory usage goes to 100%. If I use only one Window (does not matter which…
asked 2023-11-29T11:07:26.6166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 6%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETV%3C/text%3E%3C/svg%3E)
Tornikoski, Veli-Matti
0
Reputation points
commented 2023-12-08T10:59:55.67+00:00
Tornikoski, Veli-Matti
0
Reputation points
0 answers
Facing error while creating Azure Stream Analytics job service
Hi Team, I am getting error while deploying an Azure Stream Analytics instance from UI. I am not using any ARM template. It says "Your deployment failed" and throws the below error. { "code": "NotFound", …
asked 2023-11-17T15:41:25.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 15%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWM%3C/text%3E%3C/svg%3E)
WAQAR MEHDI
5
Reputation points
commented 2023-11-17T17:31:42.76+00:00
Sander van de Velde
28,956
Reputation points MVP
0 answers
Azure Stream Analytics output config json as managed identity to service bus queue
Hello I have an ASA job that is using managed identity to send messages to a service bus queue (this is working) however when trying to deploy changes the template that is generated expects parameters for SharedAccessPolicyName, which to my understanding…
asked 2023-11-15T18:56:35.8133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 59%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWC%3C/text%3E%3C/svg%3E)
Wilson, Chris
0
Reputation points
commented 2023-11-16T18:02:46.5933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache
20,031
Reputation points
0 answers
spark.readStream.text() text file from adls gen2 in databricks
file_path = "adl://<account-name>.dfs.core.windows.net/<container>/<path>/*.txt" streaming_df = spark.readStream \ .schema(schema) \ .text(file_path) query = streaming_df_transformed.writeStream \ …
asked 2023-10-05T15:48:37.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 31%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Sasidhar R. Kolli
20
Reputation points
commented 2023-10-17T18:56:28.1633333+00:00
BhargavaGunnam-MSFT
26,316
Reputation points Microsoft Employee
0 answers
Azure Stream Analytics Test Query Does not work correctly with "WITH" clauses
Here is my query: WITH Stage1 AS ( SELECT records.ArrayValue.time as Recordedtime, records.ArrayValue.properties.flows as flows FROM [insights-logs-networksecuritygroupflowevent] as n CROSS APPLY GetArrayElements(n.records) AS…
asked 2023-10-02T06:32:13.2633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 30%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUJ%3C/text%3E%3C/svg%3E)
Ulla Jutila
0
Reputation points
commented 2023-10-09T17:59:34.4533333+00:00
BhargavaGunnam-MSFT
26,316
Reputation points Microsoft Employee
0 answers
How can I force cast JSON-number values to float?
Exactly what the image shows. I have numbers that not always have decimal numbers and I need them to be float. As the image shows, this doesn't work today. What am I doing wrong?
asked 2023-08-23T20:44:57.3933333+00:00
Nikolaos Delis
5
Reputation points
commented 2023-09-18T09:15:47.0266667+00:00
PRADEEPCHEEKATLA-MSFT
77,901
Reputation points Microsoft Employee
0 answers
Low level mistake - mismatch in naming found in Stream Analytics
I have been using Stream Analytics. I got iothub-pj001, iothub-pj005 and iothub-pj009 in the Iothub and these are set as input in the Stream Analytics. I found that I cannot find the option of iothub-pj001 last night when I was trying to add a new…
asked 2023-09-12T01:35:22.99+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 75%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECI%3C/text%3E%3C/svg%3E)
Chiu I Keanu Leung
0
Reputation points
edited a comment 2023-09-13T07:22:53.8866667+00:00
Chiu I Keanu Leung
0
Reputation points
0 answers
Not able to use ML model to post and retrieve data
Hi I am using ml library model that results into below output on test page. False Alarm Prediction using Two Class Decision Forest [Predictive Exp.]' test returned ["8/10/2023 6:15:00 AM","User…
asked 2023-08-30T02:37:42.6366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 17%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Rafay Hassan Niazi
6
Reputation points
commented 2023-09-06T00:07:46.8066667+00:00
QuantumCache
20,031
Reputation points
0 answers
Setup edge hub input for azure stream analytics using the azure cli or python SDK
I am trying to create an azure stream analytics job using azure cli or python SDK with both input and output as edge hub. I tried using None and GatewayMessageBus as the input types but both of them did not work. It gives an error message: (BadRequest)…
asked 2023-08-20T21:48:11.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 42%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Rajiv
10
Reputation points
commented 2023-09-01T22:39:03.4+00:00
LeelaRajeshSayana-MSFT
13,471
Reputation points
0 answers
watermark keeps increasing and event are not processed in Stream Analytics
I have IoT package data coming in from Event Hub to Stream Analytics and then outputted to a Delta Lake before switching to Delta Lake, I had output as CSV and it worked fine, no watermark delays, resource utilization was stable and low. After switching…
asked 2023-08-08T08:38:14.5+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 32%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Audrius
1
Reputation point
commented 2023-08-29T04:38:03.6566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
0 answers
Azure Stream Analytic at-least-once guarantees is not 100% accurate
I have Azure Stream Analytic (ASA) that process data from the event hub as input. And this job is for the finance biz so it is heavily based on Event Delivery Guarantees at least once of Azure Stream Analytic. But recently we found out that some data…
asked 2023-08-22T01:58:01.4166667+00:00
Anonymous
commented 2023-08-23T05:14:09.71+00:00
PRADEEPCHEEKATLA-MSFT
77,901
Reputation points Microsoft Employee
0 answers
Connecting Streamanalytics to IoTHub
Hi all, We're facing the issue that since 2 days ag We're getting the problem that "Stream Analytics job has validation errors: Unable to connect to IoT Hub using connection string. Please make sure you have configured the IoT Hub input correctly. …
asked 2023-06-19T14:55:11.37+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 81%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGQ%3C/text%3E%3C/svg%3E)
Georgios - QU
0
Reputation points
edited a comment 2023-06-20T11:39:45.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 74%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
AshokPeddakotla-MSFT
27,801
Reputation points
0 answers
Connection failed when connectiong to a Storage account in Vnet
I am trying to use storage as an input for the Stream Analytics job. For the storage account the access for the job is enabled and Managed Identity has a Storage Blob Data Contributor role. Still, when adding input I get successful connection but…
asked 2023-04-17T15:15:52.9+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 70%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIV%3C/text%3E%3C/svg%3E)
Ivana Vrdoljak
0
Reputation points
commented 2023-05-03T19:02:48.5466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
0 answers
MS Stream scrubbs poorly
Is there a setting to make it scrub faster? Sort of hard to see what's going on when it takes 5 second to load a frame.
asked 2023-04-07T19:24:21.0233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 16%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E4%3C/text%3E%3C/svg%3E)
41844108
256
Reputation points
commented 2023-04-10T03:02:35.9566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 24%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
JimmyYang-MSFT
49,106
Reputation points Microsoft Vendor
0 answers
Why did scaling not help on delayed Stream Analytics outputs?
Hello :) We have an Azure Function that sends a lot of data in a single format to an Event Hub, which sends it to a Stream Analytics, which forwards the data to multiple outputs. This have worked very good for a long time, but a couple of days ago I…
asked 2023-01-18T09:04:37.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 64%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
mar32811
6
Reputation points
commented 2023-01-23T21:43:23.84+00:00
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee