Need to know if fluentd can be setup to filter logs from azure kubernetes service before sending it to log analytics workspace

Question

Hi,

I have a Kubernetes service setup in azure and have enabled insights for the same. I have created a log analytics workspace and a diagnostic setting to send all the logs to the log analytics workspace. In order for me to reduce the cost of data ingestion and implement log rotation i'm thinking of using fluentd.

So my approach would be to install the fluentd agent/docker image in the kubernetes service and filter it out before sending the logs to the log analytics workspace.

Is this approach even possible? If so please help me with few sources/reference links.

Answer 1

Hi, yes, your approach is valid and it's a common practice to use Fluentd in conjunction with Kubernetes for log collection and filtering. Fluentd is a flexible tool that has the necessary plugins to distribute logs to various third-party applications, such as databases or cloud services.

Here are the general steps for implementing this:

1. Install Fluentd using a DaemonSet. This will ensure that each node in your cluster runs a copy of the Fluentd pod. The Fluentd pod will collect logs from each node.
2. Configure Fluentd to suit your needs. You can specify where you want the logs to be stored and configure the format and content of the logs. You may also want to extract specific parts of the logs if necessary.
3. Send the logs to your Log Analytics workspace after Fluentd has collected and filtered them.

Note that these are just general guidelines, and the specifics will depend on your specific use case and requirements.Remember, the exact steps may vary depending on your specific situation and requirements. However, if you need more detailed instructions, there are various resources available online. For example, you can refer to the FluentD documentation or search for tutorials that provide step-by-step guidance.

I hope this helps! If you have any more questions, feel free to ask. 😊

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.