Linux hanging when writing to a disk with insufficient space

Sahithi Singam 1 Reputation point
2021-11-22T09:41:59.947+00:00

Writing to a disk with insufficient disk space from linux is resulting in below kernel backtrace and finally linux hang.
System is not coming from the hang state even after 30minutes of time.

===================================================
~ # df -h /opt
Filesystem Size Used Available Use% Mounted on
/dev/sda3 14.6G 14.6G 0 100% /opt
~ # sync

[ 246.942628] INFO: task XXXX:3446 blocked for more than 122 seconds.
[ 246.951586] Tainted: G OE 5.4.17 #6
[ 246.957719] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 246.964842] Call Trace:
[ 246.966546] __schedule+0x250/0x10d3
[ 246.969053] ? _raw_spin_unlock_irqrestore+0x1b/0x2f
[ 246.972334] ? __wake_up_common_lock+0x78/0x8d
[ 246.975369] ? preempt_count_add+0x6f/0xad
[ 246.978201] schedule+0x38/0xc1
[ 246.981718] jbd2_log_wait_commit+0x94/0xe8
[ 246.984575] ? wait_woken+0x80/0x74
[ 246.986967] __jbd2_journal_force_commit+0x54/0x9f
[ 246.990239] jbd2_journal_force_commit_nested+0x9/0x13
[ 246.993548] ext4_should_retry_alloc+0x43/0x78
[ 246.996803] ext4_write_begin+0x197/0x4e8
[ 247.001977] generic_perform_write+0x9a/0x173
[ 247.005670] __generic_file_write_iter+0x103/0x1b1
[ 247.011059] ext4_file_write_iter+0x129/0x34d
[ 247.016138] ? _raw_spin_unlock+0x11/0x25
[ 247.018895] new_sync_write+0xf5/0x165
[ 247.023474] __vfs_write+0x21/0x2c
[ 247.028499] __kernel_write+0x4f/0xf8
[ 247.032141] dump_emit+0x5c/0xa7
[ 247.034337] elf_core_dump+0x1276/0x13ec
[ 247.039234] do_coredump+0xb5a/0x1011
[ 247.043583] get_signal+0x816/0x864
[ 247.045921] ? __local_bh_enable_ip+0x32/0x78
[ 247.048412] do_signal+0x32/0x72b
[ 247.050359] ? recalc_sigpending+0x15/0x3d
[ 247.052864] ? __set_task_blocked+0x28/0x69
[ 247.055467] exit_to_usermode_loop+0x7e/0xf5
[ 247.058055] do_syscall_64+0xfa/0x116
[ 247.063135] entry_SYSCALL_64_after_hwframe+0x170/0x0
[ 247.067240] RIP: 0033:0x7ffff61d49db
[ 247.071936] Code: 41 c1 ef 1a 41 83 e5 3f 42 33 b4 bf 00 07 00 00 42 33 b4 af 00 05 00 00 44 31 e6 45 89 f4 41 c1 ec 1a 42 33 b4 a7 00 06 00 00 <45> 89 f4 41 c1 ec 02 41 83 e4 3f 42 33 34 a7 45 89 f4 41 c1 ee 12
[ 247.088897] RSP: 002b:00007fffef33ede0 EFLAGS: 00003293
[ 247.094538] RAX: 000000000000002b RBX: 0000000001924120 RCX: 00007ffff61d49dd
[ 247.103893] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000000009eb
[ 247.113327] RBP: 00007fffef33ee10 R08: 00007fffef362700 R09: 00007fffef362700
[ 247.119989] R10: 00007fffef33e1e0 R11: 0000000000003293 R12: 00007fffef33edf8
[ 247.125224] R13: 0000000000801000 R14: 0000000000000004 R15: 00007fffef362700
[ 247.132337] INFO: task YYYY:3812 blocked for more than 123 seconds.
[ 247.138950] Tainted: G OE 5.4.17 #6
[ 247.146673] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 247.153596] Call Trace:
[ 247.157538] __schedule+0x250/0x10d3
[ 247.160267] ? _raw_spin_unlock_irqrestore+0x1b/0x2f
[ 247.167586] ? __wake_up_common_lock+0x78/0x8d
[ 247.170560] ? preempt_count_add+0x6f/0xad
[ 247.173619] schedule+0x38/0xc1
[ 247.175810] jbd2_log_wait_commit+0x94/0xe8
[ 247.178899] ? wait_woken+0x80/0x74
[ 247.181246] __jbd2_journal_force_commit+0x54/0x9f
[ 247.184582] jbd2_journal_force_commit_nested+0x9/0x13
[ 247.188171] ext4_should_retry_alloc+0x43/0x78
[ 247.191124] ext4_write_begin+0x197/0x4e8
[ 247.193719] generic_perform_write+0x9a/0x173
[ 247.197275] __generic_file_write_iter+0x103/0x1b1
[ 247.200288] ? futex_wake+0x7b/0x14d
[ 247.202612] ext4_file_write_iter+0x129/0x34d
[ 247.205413] ? _copy_to_user+0x26/0x2d
[ 247.207938] new_sync_write+0xf5/0x165
[ 247.210244] __vfs_write+0x21/0x2c
[ 247.212308] vfs_write+0xac/0x197
[ 247.214348] ksys_write+0x84/0xb2
[ 247.216491] __x64_sys_write+0x15/0x17
[ 247.219049] do_syscall_64+0x48/0x116
[ 247.221436] entry_SYSCALL_64_after_hwframe+0x170/0x0
[ 247.224759] RIP: 0033:0x7ffff7bcd6fd
[ 247.227174] Code: cd 20 00 00 75 10 b8 01 00 00 00 0f 05 48 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 4e fd ff ff 48 89 04 24 b8 01 00 00 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 97 fd ff ff 48 89 d0 48 83 c4 08 48 3d 01
[ 247.243324] RSP: 002b:00007fffddb9a010 EFLAGS: 00003293 ORIG_RAX: 0000000000000001
[ 247.248478] RAX: ffffffffffffffda RBX: 000000000000006c RCX: 00007ffff7bcd6fd
[ 247.253498] RDX: 000000000000006c RSI: 000000002e972000 RDI: 0000000000000a32
[ 247.258136] RBP: 000000002e972000 R08: 000000002e97205b R09: 3120312020766f4e
[ 247.262893] R10: 2e38343a35313a37 R11: 0000000000003293 R12: 0000000000000a32
[ 247.267855] R13: 000000000000006c R14: 000000002e9092f0 R15: 000000002e9093a0
[ 247.272943] INFO: task ZZZZ:3984 blocked for more than 123 seconds.
[ 247.277368] Tainted: G OE 5.4.17 #6
[ 247.280774] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 247.285886] Call Trace:
[ 247.287696] __schedule+0x250/0x10d3
[ 247.290089] ? _raw_spin_unlock_irqrestore+0x1b/0x2f
[ 247.295260] ? __wake_up_common_lock+0x78/0x8d
[ 247.298364] ? preempt_count_add+0x6f/0xad
[ 247.300957] schedule+0x38/0xc1
[ 247.302896] jbd2_log_wait_commit+0x94/0xe8
[ 247.305610] ? wait_woken+0x80/0x74
[ 247.308499] __jbd2_journal_force_commit+0x54/0x9f
[ 247.312796] jbd2_journal_force_commit_nested+0x9/0x13
[ 247.316386] ext4_should_retry_alloc+0x43/0x78
[ 247.320234] ext4_write_begin+0x197/0x4e8
[ 247.325612] generic_perform_write+0x9a/0x173
[ 247.330583] __generic_file_write_iter+0x103/0x1b1
[ 247.335374] ext4_file_write_iter+0x129/0x34d
[ 247.340519] new_sync_write+0xf5/0x165
[ 247.345178] __vfs_write+0x21/0x2c
[ 247.349865] vfs_write+0xac/0x197
[ 247.352287] ksys_write+0x84/0xb2
[ 247.354490] __x64_sys_write+0x15/0x17
[ 247.359249] do_syscall_64+0x48/0x116
[ 247.361469] entry_SYSCALL_64_after_hwframe+0x170/0x0
[ 247.364930] RIP: 0033:0x7ffff0031a1d
[ 247.369763] Code: d5 2d 00 00 75 10 b8 01 00 00 00 0f 05 48 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 7e cc 01 00 48 89 04 24 b8 01 00 00 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 c7 cc 01 00 48 89 d0 48 83 c4 08 48 3d 01
[ 247.383905] RSP: 002b:00007fffceaad660 EFLAGS: 00003293 ORIG_RAX: 0000000000000001
[ 247.391065] RAX: ffffffffffffffda RBX: 000000000000019f RCX: 00007ffff0031a1d
[ 247.397507] RDX: 000000000000019f RSI: 00007ffff7dfd000 RDI: 0000000000000001
[ 247.402592] RBP: 00007ffff7dfd000 R08: 00007fffceaaf700 R09: 0000000000000000
[ 247.409318] R10: 0000000000030000 R11: 0000000000003293 R12: 00007ffff030a400
[ 247.413940] R13: 000000000000019f R14: 0000000000000001 R15: 00007fffceaad7c0

======================================================

Azure Disk Storage
Azure Disk Storage
A high-performance, durable block storage designed to be used with Azure Virtual Machines and Azure VMware Solution.
659 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.