DatabricksInstancePools log table is empty

Question

DatabricksInstancePools log table is empty

Frederik Larsen 76

Hi

I'm interested in creating some alert rules to catch long-running pools. But when I check the DatabricksInstancePools in LogAnalytics, it is empty. Is this a bug or am I doing something wrong?

I have enabled the category and it is sent to my Log Analytics Workspace:

User's image

I created a pool last month. And this pool is used frequently each day:

User's image

But the Query in Log Analytics does not return any records: User's image

Thanks!

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-20T00:18:31.1133333+00:00

Hello @Frederik Larsen ,

Welcome to the MS Q&A platform.

Could you please confirm if you are configured the "Instance pools" on the categories section while enabling the log analytics?

You need to enable this to log the events related to the pools.
Frederik Larsen 76 Reputation points

2023-01-20T23:21:29.7933333+00:00

Hi @Bhargava-MSFT . Yes I have. Tried to show that in the first picture I uploaded in the question:)
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-20T23:22:28.2066667+00:00

Hello @Frederik Larsen ,

Sorry about that. I will try to reproduce this from my end and get back to you with more details.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-21T00:43:14.56+00:00

Hello @Frederik Larsen ,

By configuring the "Instance pools," I captured the events on the log analytics workspace. I believe it could be your log analytics configuration issue.

Do you have any other environments where you can test this?

Please see the below screenshot for your reference.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-23T22:05:54.3+00:00

Hello @Frederik Larsen ,

Thank you for the reply. Yes, I can also not see any events other than create and delete actions.

Let me check with my internal team and get back to you with more details.

I appreciate your patience.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-25T18:07:55.07+00:00

Hello @Frederik Larsen ,
I am still waiting for an update from my internal team. I will get back to you as soon as I hear more details.
Frederik Larsen 76 Reputation points

2023-01-25T19:12:07.84+00:00

Hi @Bhargava-MSFT

Thanks for your screenshot. I can confirm, I can also get delete and create actions for my pool - they were just outside my initial time range.

But my initial challenge was to setup "alert rules to catch long-running pools". I'm not able to find logs for start, restart, termination of the instances. Can you see those activities?

Thanks

Answer accepted by question author

0 additional answers

Your answer

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-20T00:18:31.1133333+00:00

Hello @Frederik Larsen ,

Welcome to the MS Q&A platform.

Could you please confirm if you are configured the "Instance pools" on the categories section while enabling the log analytics?

You need to enable this to log the events related to the pools.
Frederik Larsen 76 Reputation points

2023-01-20T23:21:29.7933333+00:00

Hi @Bhargava-MSFT . Yes I have. Tried to show that in the first picture I uploaded in the question:)
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-20T23:22:28.2066667+00:00

Hello @Frederik Larsen ,

Sorry about that. I will try to reproduce this from my end and get back to you with more details.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-21T00:43:14.56+00:00

Hello @Frederik Larsen ,

By configuring the "Instance pools," I captured the events on the log analytics workspace. I believe it could be your log analytics configuration issue.

Do you have any other environments where you can test this?

Please see the below screenshot for your reference.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-23T22:05:54.3+00:00

Hello @Frederik Larsen ,

Thank you for the reply. Yes, I can also not see any events other than create and delete actions.

Let me check with my internal team and get back to you with more details.

I appreciate your patience.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-01-25T18:07:55.07+00:00

Hello @Frederik Larsen ,
I am still waiting for an update from my internal team. I will get back to you as soon as I hear more details.
Frederik Larsen 76 Reputation points

2023-01-25T19:12:07.84+00:00

Hi @Bhargava-MSFT

Thanks for your screenshot. I can confirm, I can also get delete and create actions for my pool - they were just outside my initial time range.

But my initial challenge was to setup "alert rules to catch long-running pools". I'm not able to find logs for start, restart, termination of the instances. Can you see those activities?

Thanks

Answer 1

Bhargava-MSFT 31,361 Microsoft Employee Moderator

Hello @Frederik Larsen ,

Here is an update on the issue.

Diagnostic logging for category instance pools will capture only actions associated with instance pools. These are: create, delete and edit.

To capture additional events beyond what is provided by instance pools, then you will need to turn on additional diagnostic logging categories.

If you intend to capture cluster logs, you will need to use "cluster configuration" from the advanced options

I hope this helps.

User's image

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community.

0 comments

Share via

DatabricksInstancePools log table is empty

0 additional answers

Your answer