1 FilesystemIsReadOnly: Node condition ReadonlyFilesystem is now: True, reason: FilesystemIsReadOnly, message: "EXT4-fs (sda1): Remounting filesystem read-only" - This error in AKS cluster -

Question

1 FilesystemIsReadOnly: Node condition ReadonlyFilesystem is now: True, reason: FilesystemIsReadOnly, message: "EXT4-fs (sda1): Remounting filesystem read-only" - This error in AKS cluster -

Satish Chandra Akella 0

1 FilesystemIsReadOnly: Node condition ReadonlyFilesystem is now: True, reason: FilesystemIsReadOnly, message: "EXT4-fs (sda1): Remounting filesystem read-only" -

This error in AKS cluster - doesn't autoscale of Azure Kubernetes cluster takes care of the nodes that are not healthy. If AKS doesn't, we have backend Virtual machine scalesets - at that level the node should be removed and a healthy node should be added is it.

If the node goes to FilesystemReadonly - Should we drain the node and move resources to fix the issue ?

Anonymous

2025-03-03T04:53:51.1566667+00:00

Hi Satish Chandra Akella

Just checking in to see if you had a chance to review my comment on your question. Please let us know if it was helpful and feel free to reach out if you have any further queries.

If you found the information useful, please click "Upvote" on the post to let us know.

Thank You.

1 answer

Your answer

Anonymous

2025-03-03T04:53:51.1566667+00:00

Hi Satish Chandra Akella

Just checking in to see if you had a chance to review my comment on your question. Please let us know if it was helpful and feel free to reach out if you have any further queries.

If you found the information useful, please click "Upvote" on the post to let us know.

Thank You.

Answer 1

Anonymous

Hi Satish Chandra Akella,

Thank you for reaching out to us on the Microsoft Q&A forum.

The potential reasons for a node transitioning from a Ready state to a ReadonlyFilesystem state include the disk being full, which may trigger a switch to read-only mode, as well as any underlying corruption or hardware malfunctions affecting the node.

To address this issue, please consider the following troubleshooting steps: Verify the status of the node and provide a description of the node that is in the NotReady state. Attempt to restart the node. Additionally, try unmounting and then remounting the file system. Check the disk size if it is full df -h

For more information, please check: https://learn.microsoft.com/en-us/azure/aks/node-problem-detector

https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/linux-recovery-cannot-start-file-system-errors

If the information is helpful, please consider by clicking the "Upvote" on the post.

If you have any further queries, please let us know in the comment.

Thank you.

Anonymous

2025-03-04T02:04:44.6333333+00:00

Hi Satish Chandra Akella

Just checking in to see if you had a chance to review my Answer on your question. Please let us know if it was helpful and feel free to reach out if you have any further queries.

If you found the information useful, please click "Upvote" on the post to let us know.

Thank You.
Satish Akella 0 Reputation points

2025-03-19T06:18:38.3733333+00:00

Thanks for the details., We have monitoring in place that provides these error details, We are looking for a fix on how to prevent this error from occuring in the first place
Mounika Reddy Anumandla 6,585 Reputation points Microsoft External Staff Moderator

2025-03-31T09:37:57.68+00:00
Hi Satish Akella,

The Cluster Autoscaler only adds/removes nodes based on resource demand, not health issues. The affected node remains unless you take manual action or Node Auto-Repair or VMSS auto-healing is enabled.

VMSS monitors instances and can replace nodes marked as unhealthy. If a node is stuck in FilesystemReadonly, VMSS may remove it and create a new healthy instance, depending on its fault tolerance and scaling policies.

To prevent the FilesystemReadonly error in an Azure Kubernetes Service (AKS) cluster, you need to address the root causes that lead to this issue. Typically, this error occurs due to disk corruption, resource exhaustion, or incorrect configurations.

Enable AKS node auto-repair to ensure unhealthy nodes are automatically rebooted or replaced when issues occur. Auto-repair-AKS

At the Virtual Machine Scale Sets (VMSS) level, configure fault tolerance and scaling policies to automatically replace unhealthy nodes. Automatic instance repairs for Azure Virtual Machine Scale Sets

Prevent resource exhaustion by setting resource limits for CPU and memory in your pod specifications.

Ensure that the underlying disks used by VM Scale Sets are not overloaded. If disks are frequently remounted as read-only, consider resizing them or using higher-performance disks (e.g., Premium SSDs).

Try draining and deleting the node and see if VMSS replaces it. One user confirmed that pressure on the disks was detected and once that went away it went back to "ready". https://learn.microsoft.com/en-us/answers/questions/2152850/what-to-do-when-our-aks-shows-a-node-to-have-a-ext

Hope this helps!

Let me know if you have any further queries!
Mounika Reddy Anumandla 6,585 Reputation points Microsoft External Staff Moderator

2025-04-01T06:02:39.97+00:00

Hi Satish Akella,

Just checking in to see if you had a chance to review the response provided to your question.

If the information is helpful, please click "upvote" to let us know.

Share via

1 FilesystemIsReadOnly: Node condition ReadonlyFilesystem is now: True, reason: FilesystemIsReadOnly, message: "EXT4-fs (sda1): Remounting filesystem read-only" - This error in AKS cluster -

1 answer

Your answer