DCR based custom tables are not working

Question

DCR based custom tables are not working

Sing Kit Cheng 96

Hello All,

I have a Linux server vm in Azure as my syslog server. My Meraki MX devices are sending logs to the syslog server into their individual folders, and they are flowing. I used to have custom logs in Azure Log Analytics that pulls these logs from the syslog server so we can run query on them. Since Microsoft deprecated these classic custom logs, they are no long working. So, I am trying to create new Data Collection Rule (DCR) based custom table to pull the logs from the syslog server.

I went the process of creating the custom table. During the process, a new DCR was created as well. It looks like they (the table and the rule) were created successfully because I didn't get any error but they are not showing anything when I run query against these tables. Looks like it ran but the results have been: No results found from the last 24 hours Try selecting another time range.

I'm not sure where the problem might be. Does anyone have any idea? Any help is appreciated.

Full disclosure, I am not a Linux or a command line person at all. So I would need detail step by step instructions if you have any suggestion for me.

Thank you in advance.

Kit

Naveena Patlolla 3,925 Reputation points Microsoft External Staff Moderator

2025-06-27T08:44:41.1366667+00:00

Hi Sing Kit Cheng

We are looking into it and will update you shortly.
Sing Kit Cheng 96 Reputation points

2025-06-27T12:18:50.34+00:00

Hello Naveena,

Thank you for letting me know.

Kit
Rahul Jorrigala 1,210 Reputation points Microsoft External Staff Moderator

2025-06-29T16:55:13.7133333+00:00
Hello @Sing Kit Cheng ,

Here are some key troubleshooting steps and considerations that might help:

Verify DCR Configuration:

Ensure the Data Collection Rule (DCR) is correctly linked to the Log Analytics Workspace and the custom table.

Confirm that the data source path in the DCR matches the actual log file paths on your Linux VM.

Check Encoding and Format:

Logs should be in UTF-8 encoding.

Ensure the log format matches the expected schema (e.g., timestamp and message fields).

Validate Agent Communication:

Run a query in Log Analytics to check if the Azure Monitor Agent is sending heartbeats

Heartbeat | where TimeGenerated > ago(1h)

Check DCR Association:

In the Azure portal, go to your custom table and click Manage Table to confirm that a DCR is associated.

If not, you may need to manually associate the DCR with the table.

Query with Extended Time Range:

Try querying with a broader time range

<YourCustomTable>_CL | where TimeGenerated > ago(48h) | order by TimeGenerated desc

Reference Link:

https://learn.microsoft.com/en-us/azure/azure-monitor/agents/data-sources-custom-logs
Please let me know if you face any challenge here, I can help you to resolve this issue further

If the comment was helpful, please click "Upvote"
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-06-30T04:04:46.76+00:00

@Sing Kit Cheng

Good day!

I just wanted to check if the above provided information worked for you or if you need any further assistance?

If the comment was helpful, please click "Upvote"
Sing Kit Cheng 96 Reputation points

2025-06-30T21:42:54.0933333+00:00

Hello Rahul Jorrigala,

Thank you for your comments. I think everything checked out except for one thing. The log files are being rotated because if the file get too big, then the server will run out of space and the logs stop coming in. I had this setup for a long time now and it was working with the classic custom logs fine. Is this a restriction with the new DCR based custom table? Anyway, how do I address the file size without rotating it?

I'm not a Linux guy at all so if you can, please give me detail step by step instructions if you have any for me.

Thank you

Kit
Rahul Jorrigala 1,210 Reputation points Microsoft External Staff Moderator

2025-07-08T09:28:25.14+00:00

Hello @Sing Kit Cheng,

Just want to check if the above Comment worked for you or else please let us know if any help, we are always here to help whenever you need us.

If the comment is helpful, please click "Upvote it"

Thankyou
Rahul Jorrigala 1,210 Reputation points Microsoft External Staff Moderator

2025-07-09T08:55:14.9133333+00:00

Hello Kit,

We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion.

If the comment is helpful, please click "Upvote it"

1 answer

Your answer

Naveena Patlolla 3,925 Reputation points Microsoft External Staff Moderator

2025-06-27T08:44:41.1366667+00:00

Hi Sing Kit Cheng

We are looking into it and will update you shortly.
Sing Kit Cheng 96 Reputation points

2025-06-27T12:18:50.34+00:00

Hello Naveena,

Thank you for letting me know.

Kit
Rahul Jorrigala 1,210 Reputation points Microsoft External Staff Moderator

2025-06-29T16:55:13.7133333+00:00

Hello @Sing Kit Cheng ,

Here are some key troubleshooting steps and considerations that might help:

Verify DCR Configuration:

Ensure the Data Collection Rule (DCR) is correctly linked to the Log Analytics Workspace and the custom table.

Confirm that the data source path in the DCR matches the actual log file paths on your Linux VM.

Check Encoding and Format:

Logs should be in UTF-8 encoding.

Ensure the log format matches the expected schema (e.g., timestamp and message fields).

Validate Agent Communication:

Run a query in Log Analytics to check if the Azure Monitor Agent is sending heartbeats

Heartbeat | where TimeGenerated > ago(1h)

Check DCR Association:

In the Azure portal, go to your custom table and click Manage Table to confirm that a DCR is associated.

If not, you may need to manually associate the DCR with the table.

Query with Extended Time Range:

Try querying with a broader time range

<YourCustomTable>_CL | where TimeGenerated > ago(48h) | order by TimeGenerated desc

Reference Link:

https://learn.microsoft.com/en-us/azure/azure-monitor/agents/data-sources-custom-logs
Please let me know if you face any challenge here, I can help you to resolve this issue further

If the comment was helpful, please click "Upvote"
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-06-30T04:04:46.76+00:00

@Sing Kit Cheng

Good day!

I just wanted to check if the above provided information worked for you or if you need any further assistance?

If the comment was helpful, please click "Upvote"
Sing Kit Cheng 96 Reputation points

2025-06-30T21:42:54.0933333+00:00

Hello Rahul Jorrigala,

Thank you for your comments. I think everything checked out except for one thing. The log files are being rotated because if the file get too big, then the server will run out of space and the logs stop coming in. I had this setup for a long time now and it was working with the classic custom logs fine. Is this a restriction with the new DCR based custom table? Anyway, how do I address the file size without rotating it?

I'm not a Linux guy at all so if you can, please give me detail step by step instructions if you have any for me.

Thank you

Kit
Rahul Jorrigala 1,210 Reputation points Microsoft External Staff Moderator

2025-07-08T09:28:25.14+00:00

Hello @Sing Kit Cheng,

Just want to check if the above Comment worked for you or else please let us know if any help, we are always here to help whenever you need us.

If the comment is helpful, please click "Upvote it"

Thankyou
Rahul Jorrigala 1,210 Reputation points Microsoft External Staff Moderator

2025-07-09T08:55:14.9133333+00:00

Hello Kit,

We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion.

If the comment is helpful, please click "Upvote it"

Answer 1

Rahul Jorrigala 1,210 Microsoft External Staff Moderator

Hello Kit,

1.DCRs and File Patterns:

DCRs use file patterns to locate and ingest logs. If your logs rotate and the file name changes (e.g., mylog.log becomes mylog.log.1), the DCR must be configured to match all possible rotated file names using wildcards like mylog*.log 1.

2.Record Delimiters and Time Format

DCRs rely on timestamp delimiters to identify new log entries. If rotated files lose this formatting or are truncated mid-record, ingestion may fail.

3.Transformations and Filtering

You can apply ingestion-time transformations to filter or reshape incoming data. This can help reduce file size and avoid unnecessary log entries.

Recommendations to Avoid File Rotation Issues

User's image
https://learn.microsoft.com/en-us/azure/azure-monitor/vm/data-collection-log-text

Please let me know if you face any challenge here, I can help you to resolve this issue further

If the comment was helpful, please click "Upvote"

Sing Kit Cheng 96 Reputation points

2025-07-09T13:52:57.5733333+00:00

I was able to get this resolved. Basically, I had to use the script to create the table first, then create the DCR (and able to reference the newly created table). Once the table and the DCR is created successfully, then it worked. Oh, while creating the DCR, one mistake I was making, for what I am trying to do anyway, was that I was choosing Time Frame instead of End-of-line as the Record delimiter. That made a difference as well. Thanks for the help from everyone here though, really appreciate it.

Kit
Rahul Jorrigala 1,210 Reputation points Microsoft External Staff Moderator

2025-07-09T14:00:40.8933333+00:00

Hello @Sing Kit Cheng ,

I noticed that the provided solution has been helpful for you. As per Microsoft's recommendation, please consider accepting the answer and upvoting it if the information provided was useful. This can greatly benefit other community members.

You can refer to this document for more details: https://learn.microsoft.com/en-us/answers/support/accept-answer Your support in this matter would be greatly appreciated and helpful to others.

Thanks
Sing Kit Cheng 96 Reputation points

2025-07-10T12:20:01.5133333+00:00

If I figured this out on my own, how do I accept my own comment as the solution?

Share via

DCR based custom tables are not working

1 answer

Your answer