* Tasks blocking forever with XFS stack traces @ 2019-11-05 7:27 Sitsofe Wheeler 2019-11-05 8:54 ` Carlos Maiolino 0 siblings, 1 reply; 10+ messages in thread From: Sitsofe Wheeler @ 2019-11-05 7:27 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, linux-fsdevel Hi, We have a system that has been seeing tasks with XFS calls in their stacks. Once these tasks start hanging with uninterruptible sleep any write I/O to the directory they were doing I/O to will also hang forever. The I/O they doing is being done to a bind mounted directory atop an XFS filesystem on top an MD device (the MD device seems to be still functional and isn't offline). The kernel is fairly old but I thought I'd post a stack in case anyone can describe this or has seen it before: kernel: [425684.110424] INFO: task kworker/u162:0:58843 blocked for more than 120 seconds. kernel: [425684.110800] Tainted: G OE 4.15.0-64-generic #73-Ubuntu kernel: [425684.111164] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kernel: [425684.111568] kworker/u162:0 D 0 58843 2 0x80000080 kernel: [425684.111581] Workqueue: writeback wb_workfn (flush-9:126) kernel: [425684.111585] Call Trace: kernel: [425684.111595] __schedule+0x24e/0x880 kernel: [425684.111664] ? xfs_map_blocks+0x82/0x250 [xfs] kernel: [425684.111668] schedule+0x2c/0x80 kernel: [425684.111671] rwsem_down_read_failed+0xf0/0x160 kernel: [425684.111675] ? bitmap_startwrite+0x9f/0x1f0 kernel: [425684.111679] call_rwsem_down_read_failed+0x18/0x30 kernel: [425684.111682] ? call_rwsem_down_read_failed+0x18/0x30 kernel: [425684.111685] down_read+0x20/0x40 kernel: [425684.111736] xfs_ilock+0xd5/0x100 [xfs] kernel: [425684.111782] xfs_map_blocks+0x82/0x250 [xfs] kernel: [425684.111823] xfs_do_writepage+0x167/0x6a0 [xfs] kernel: [425684.111830] ? clear_page_dirty_for_io+0x19f/0x1f0 kernel: [425684.111834] write_cache_pages+0x207/0x4e0 kernel: [425684.111869] ? xfs_vm_writepages+0xf0/0xf0 [xfs] kernel: [425684.111875] ? submit_bio+0x73/0x140 kernel: [425684.111878] ? submit_bio+0x73/0x140 kernel: [425684.111911] ? xfs_setfilesize_trans_alloc.isra.13+0x3e/0x90 [xfs] kernel: [425684.111944] xfs_vm_writepages+0xbe/0xf0 [xfs] kernel: [425684.111949] do_writepages+0x4b/0xe0 kernel: [425684.111954] ? fprop_fraction_percpu+0x2f/0x80 kernel: [425684.111958] ? __wb_calc_thresh+0x3e/0x130 kernel: [425684.111963] __writeback_single_inode+0x45/0x350 kernel: [425684.111966] ? __writeback_single_inode+0x45/0x350 kernel: [425684.111970] writeback_sb_inodes+0x1e1/0x510 kernel: [425684.111975] __writeback_inodes_wb+0x67/0xb0 kernel: [425684.111979] wb_writeback+0x271/0x300 kernel: [425684.111983] wb_workfn+0x1bb/0x400 kernel: [425684.111986] ? wb_workfn+0x1bb/0x400 kernel: [425684.111992] process_one_work+0x1de/0x420 kernel: [425684.111996] worker_thread+0x32/0x410 kernel: [425684.111999] kthread+0x121/0x140 kernel: [425684.112003] ? process_one_work+0x420/0x420 kernel: [425684.112005] ? kthread_create_worker_on_cpu+0x70/0x70 kernel: [425684.112009] ret_from_fork+0x35/0x40 kernel: [425684.112024] INFO: task kworker/74:0:9623 blocked for more than 120 seconds. kernel: [425684.112461] Tainted: G OE 4.15.0-64-generic #73-Ubuntu kernel: [425684.112925] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kernel: [425684.113438] kworker/74:0 D 0 9623 2 0x80000080 kernel: [425684.113500] Workqueue: xfs-cil/md126 xlog_cil_push_work [xfs] kernel: [425684.113502] Call Trace: kernel: [425684.113508] __schedule+0x24e/0x880 kernel: [425684.113559] ? xlog_bdstrat+0x2b/0x60 [xfs] kernel: [425684.113564] schedule+0x2c/0x80 kernel: [425684.113609] xlog_state_get_iclog_space+0x105/0x2d0 [xfs] kernel: [425684.113614] ? wake_up_q+0x80/0x80 kernel: [425684.113656] xlog_write+0x163/0x6e0 [xfs] kernel: [425684.113699] xlog_cil_push+0x2a7/0x410 [xfs] kernel: [425684.113740] xlog_cil_push_work+0x15/0x20 [xfs] kernel: [425684.113743] process_one_work+0x1de/0x420 kernel: [425684.113747] worker_thread+0x32/0x410 kernel: [425684.113750] kthread+0x121/0x140 kernel: [425684.113753] ? process_one_work+0x420/0x420 kernel: [425684.113756] ? kthread_create_worker_on_cpu+0x70/0x70 kernel: [425684.113759] ret_from_fork+0x35/0x40 Other directories on the same filesystem seem fine as do other XFS filesystems on the same system. -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Tasks blocking forever with XFS stack traces 2019-11-05 7:27 Tasks blocking forever with XFS stack traces Sitsofe Wheeler @ 2019-11-05 8:54 ` Carlos Maiolino 2019-11-05 9:32 ` Sitsofe Wheeler 0 siblings, 1 reply; 10+ messages in thread From: Carlos Maiolino @ 2019-11-05 8:54 UTC (permalink / raw) To: Sitsofe Wheeler; +Cc: linux-xfs Hi. On Tue, Nov 05, 2019 at 07:27:16AM +0000, Sitsofe Wheeler wrote: > Hi, > > We have a system that has been seeing tasks with XFS calls in their > stacks. Once these tasks start hanging with uninterruptible sleep any > write I/O to the directory they were doing I/O to will also hang > forever. The I/O they doing is being done to a bind mounted directory > atop an XFS filesystem on top an MD device (the MD device seems to be > still functional and isn't offline). The kernel is fairly old but I > thought I'd post a stack in case anyone can describe this or has seen > it before: > > kernel: [425684.110424] INFO: task kworker/u162:0:58843 blocked for > more than 120 seconds. > kernel: [425684.110800] Tainted: G OE > 4.15.0-64-generic #73-Ubuntu > kernel: [425684.111164] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > kernel: [425684.111568] kworker/u162:0 D 0 58843 2 0x80000080 > kernel: [425684.111581] Workqueue: writeback wb_workfn (flush-9:126) > kernel: [425684.111585] Call Trace: > kernel: [425684.111595] __schedule+0x24e/0x880 > kernel: [425684.111664] ? xfs_map_blocks+0x82/0x250 [xfs] > kernel: [425684.111668] schedule+0x2c/0x80 > kernel: [425684.111671] rwsem_down_read_failed+0xf0/0x160 > kernel: [425684.111675] ? bitmap_startwrite+0x9f/0x1f0 > kernel: [425684.111679] call_rwsem_down_read_failed+0x18/0x30 > kernel: [425684.111682] ? call_rwsem_down_read_failed+0x18/0x30 > kernel: [425684.111685] down_read+0x20/0x40 > kernel: [425684.111736] xfs_ilock+0xd5/0x100 [xfs] > kernel: [425684.111782] xfs_map_blocks+0x82/0x250 [xfs] > kernel: [425684.111823] xfs_do_writepage+0x167/0x6a0 [xfs] > kernel: [425684.111830] ? clear_page_dirty_for_io+0x19f/0x1f0 > kernel: [425684.111834] write_cache_pages+0x207/0x4e0 > kernel: [425684.111869] ? xfs_vm_writepages+0xf0/0xf0 [xfs] > kernel: [425684.111875] ? submit_bio+0x73/0x140 > kernel: [425684.111878] ? submit_bio+0x73/0x140 > kernel: [425684.111911] ? xfs_setfilesize_trans_alloc.isra.13+0x3e/0x90 [xfs] > kernel: [425684.111944] xfs_vm_writepages+0xbe/0xf0 [xfs] > kernel: [425684.111949] do_writepages+0x4b/0xe0 > kernel: [425684.111954] ? fprop_fraction_percpu+0x2f/0x80 > kernel: [425684.111958] ? __wb_calc_thresh+0x3e/0x130 > kernel: [425684.111963] __writeback_single_inode+0x45/0x350 > kernel: [425684.111966] ? __writeback_single_inode+0x45/0x350 > kernel: [425684.111970] writeback_sb_inodes+0x1e1/0x510 > kernel: [425684.111975] __writeback_inodes_wb+0x67/0xb0 > kernel: [425684.111979] wb_writeback+0x271/0x300 > kernel: [425684.111983] wb_workfn+0x1bb/0x400 > kernel: [425684.111986] ? wb_workfn+0x1bb/0x400 > kernel: [425684.111992] process_one_work+0x1de/0x420 > kernel: [425684.111996] worker_thread+0x32/0x410 > kernel: [425684.111999] kthread+0x121/0x140 > kernel: [425684.112003] ? process_one_work+0x420/0x420 > kernel: [425684.112005] ? kthread_create_worker_on_cpu+0x70/0x70 > kernel: [425684.112009] ret_from_fork+0x35/0x40 > kernel: [425684.112024] INFO: task kworker/74:0:9623 blocked for more > than 120 seconds. > kernel: [425684.112461] Tainted: G OE > 4.15.0-64-generic #73-Ubuntu > kernel: [425684.112925] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > kernel: [425684.113438] kworker/74:0 D 0 9623 2 0x80000080 > kernel: [425684.113500] Workqueue: xfs-cil/md126 xlog_cil_push_work [xfs] > kernel: [425684.113502] Call Trace: > kernel: [425684.113508] __schedule+0x24e/0x880 > kernel: [425684.113559] ? xlog_bdstrat+0x2b/0x60 [xfs] > kernel: [425684.113564] schedule+0x2c/0x80 > kernel: [425684.113609] xlog_state_get_iclog_space+0x105/0x2d0 [xfs] > kernel: [425684.113614] ? wake_up_q+0x80/0x80 > kernel: [425684.113656] xlog_write+0x163/0x6e0 [xfs] > kernel: [425684.113699] xlog_cil_push+0x2a7/0x410 [xfs] > kernel: [425684.113740] xlog_cil_push_work+0x15/0x20 [xfs] > kernel: [425684.113743] process_one_work+0x1de/0x420 > kernel: [425684.113747] worker_thread+0x32/0x410 > kernel: [425684.113750] kthread+0x121/0x140 > kernel: [425684.113753] ? process_one_work+0x420/0x420 > kernel: [425684.113756] ? kthread_create_worker_on_cpu+0x70/0x70 > kernel: [425684.113759] ret_from_fork+0x35/0x40 > > Other directories on the same filesystem seem fine as do other XFS > filesystems on the same system. The fact you mention other directories seems to work, and the first stack trace you posted, it sounds like you've been keeping a singe AG too busy to almost make it unusable. But, you didn't provide enough information we can really make any progress here, and to be honest I'm more inclined to point the finger to your MD device. Can you describe your MD device? RAID array? What kind? How many disks? What's your filesystem configuration? (xfs_info <mount point>) Do you have anything else on your dmesg other than these two stack traces? I'd suggest posting the whole dmesg, not only what you think is relevant. Better yet: http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F Cheers. > > -- > Sitsofe | http://sucs.org/~sits/ -- Carlos P.S. I'm removing Darrick and linux-fsdevel from CC to avoid spamming too many. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Tasks blocking forever with XFS stack traces 2019-11-05 8:54 ` Carlos Maiolino @ 2019-11-05 9:32 ` Sitsofe Wheeler 2019-11-05 10:36 ` Carlos Maiolino 0 siblings, 1 reply; 10+ messages in thread From: Sitsofe Wheeler @ 2019-11-05 9:32 UTC (permalink / raw) To: Carlos Maiolino; +Cc: linux-xfs Hi, On Tue, 5 Nov 2019 at 08:54, Carlos Maiolino <cmaiolino@redhat.com> wrote: > > Hi. > > On Tue, Nov 05, 2019 at 07:27:16AM +0000, Sitsofe Wheeler wrote: > > Hi, > > > > We have a system that has been seeing tasks with XFS calls in their > > stacks. Once these tasks start hanging with uninterruptible sleep any > > write I/O to the directory they were doing I/O to will also hang > > forever. The I/O they doing is being done to a bind mounted directory > > atop an XFS filesystem on top an MD device (the MD device seems to be > > still functional and isn't offline). The kernel is fairly old but I > > thought I'd post a stack in case anyone can describe this or has seen > > it before: > > > > kernel: [425684.110424] INFO: task kworker/u162:0:58843 blocked for > > more than 120 seconds. > > kernel: [425684.110800] Tainted: G OE > > 4.15.0-64-generic #73-Ubuntu > > kernel: [425684.111164] "echo 0 > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > kernel: [425684.111568] kworker/u162:0 D 0 58843 2 0x80000080 > > kernel: [425684.111581] Workqueue: writeback wb_workfn (flush-9:126) > > kernel: [425684.111585] Call Trace: > > kernel: [425684.111595] __schedule+0x24e/0x880 > > kernel: [425684.111664] ? xfs_map_blocks+0x82/0x250 [xfs] <snip> > > > > Other directories on the same filesystem seem fine as do other XFS > > filesystems on the same system. > > The fact you mention other directories seems to work, and the first stack trace > you posted, it sounds like you've been keeping a singe AG too busy to almost > make it unusable. But, you didn't provide enough information we can really make > any progress here, and to be honest I'm more inclined to point the finger to > your MD device. Let's see if we can pinpoint something :-) > Can you describe your MD device? RAID array? What kind? How many disks? RAID6 8 disks. > What's your filesystem configuration? (xfs_info <mount point>) meta-data=/dev/md126 isize=512 agcount=32, agsize=43954432 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1 spinodes=0 rmapbt=0 = reflink=0 data = bsize=4096 blocks=1406538240, imaxpct=5 = sunit=128 swidth=768 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=521728, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 > Do you have anything else on your dmesg other than these two stack traces? I'd > suggest posting the whole dmesg, not only what you think is relevant. Yes there's more. See a slightly elided dmesg from a longer run on https://sucs.org/~sits/test/kern-20191024.log.gz . > > Better yet: > > http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F Note most of the following was gathered from the currently not-hanging system: kernel: was 4.15.0-64-generic from Ubuntu 18.04 but we're now testing 5.0.0-32-generic xfsprogs version: xfs_repair version 4.9.0 CPUs: 80 cat /proc/meminfo MemTotal: 791232512 kB MemFree: 616987432 kB MemAvailable: 781352708 kB Buffers: 5520 kB Cached: 113300540 kB SwapCached: 0 kB Active: 28385760 kB Inactive: 85358040 kB Active(anon): 436084 kB Inactive(anon): 3476 kB Active(file): 27949676 kB Inactive(file): 85354564 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 31248380 kB SwapFree: 31248380 kB Dirty: 688 kB Writeback: 0 kB AnonPages: 436396 kB Mapped: 206652 kB Shmem: 6944 kB KReclaimable: 56047960 kB Slab: 58126044 kB SReclaimable: 56047960 kB SUnreclaim: 2078084 kB KernelStack: 22240 kB PageTables: 17552 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 426864636 kB Committed_AS: 4147112 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB Percpu: 61760 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 3245828 kB DirectMap2M: 100208640 kB DirectMap1G: 702545920 kB cat /proc/mounts sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=395591264k,nr_inodes=98897816,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=79123252k,mode=755 0 0 /dev/mapper/vgsys-root / xfs rw,relatime,attr2,inode64,noquota 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=66154 0 0 mqueue /dev/mqueue mqueue rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 tmpfs /tmp tmpfs rw,nosuid,nodev 0 0 /dev/md0 /boot ext2 rw,relatime 0 0 /dev/md126 /localdata xfs rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota 0 0 /dev/md126 /var/lib/docker xfs rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota 0 0 /dev/mapper/vgsys-home /home xfs rw,relatime,attr2,inode64,noquota 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 overlay /var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/XPFD5GLZ7YBMUP7S3E6W5OUE6A:/var/lib/docker/overlay2/l/GJVZ2MXOD5AOLUELAEYCSYCXLK:/var/lib/docker/overlay2/l/JEYWOT7MNNHX2DAE4AQ5XO674I:/var/lib/docker/overlay2/l/YAS2YWA4FTAWNEKRAJQY47TQDY,upperdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/diff,workdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/work,xino=off 0 0 overlay /localdata/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/XPFD5GLZ7YBMUP7S3E6W5OUE6A:/var/lib/docker/overlay2/l/GJVZ2MXOD5AOLUELAEYCSYCXLK:/var/lib/docker/overlay2/l/JEYWOT7MNNHX2DAE4AQ5XO674I:/var/lib/docker/overlay2/l/YAS2YWA4FTAWNEKRAJQY47TQDY,upperdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/diff,workdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/work,xino=off 0 0 nsfs /run/docker/netns/160ed5c707bb nsfs rw 0 0 overlay /var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ECXX2YJFYUMBVKTP7OTRSAJVWE:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/diff,workdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/work,xino=off 0 0 overlay /localdata/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ECXX2YJFYUMBVKTP7OTRSAJVWE:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/diff,workdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/work,xino=off 0 0 nsfs /run/docker/netns/cc8ad7e2cc51 nsfs rw 0 0 overlay /var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/S5DDQ53MEAP37J6723CYPVDTO6:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/diff,workdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/work,xino=off 0 0 overlay /localdata/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/S5DDQ53MEAP37J6723CYPVDTO6:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/diff,workdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/work,xino=off 0 0 nsfs /run/docker/netns/e892b0d9fdea nsfs rw 0 0 overlay /var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/2MTUND5M3MS3FZWZCZVXTBIB5K:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/diff,workdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/work,xino=off 0 0 overlay /localdata/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/2MTUND5M3MS3FZWZCZVXTBIB5K:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/diff,workdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/work,xino=off 0 0 nsfs /run/docker/netns/e9d00dfcaa30 nsfs rw 0 0 overlay /var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SLHFVMXTCIQY5TYHXX3XY2QUTX:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/diff,workdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/work,xino=off 0 0 overlay /localdata/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SLHFVMXTCIQY5TYHXX3XY2QUTX:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/diff,workdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/work,xino=off 0 0 nsfs /run/docker/netns/2d3a60de14ae nsfs rw 0 0 tmpfs /run/user/2266 tmpfs rw,nosuid,nodev,relatime,size=79123248k,mode=700,uid=2266,gid=501 0 0 tmpfs /run/user/2042 tmpfs rw,nosuid,nodev,relatime,size=79123248k,mode=700,uid=2042,gid=501 0 0 cat /proc/partitions major minor #blocks name 8 0 937692504 sda 8 16 937692504 sdb 8 32 937692504 sdc 8 48 937692504 sdd 8 64 234431064 sde 8 65 999424 sde1 8 66 1 sde2 8 69 233428992 sde5 8 80 234431064 sdf 8 81 999424 sdf1 8 82 1 sdf2 8 85 233428992 sdf5 9 126 5626152960 md126 9 0 998848 md0 9 1 233297920 md1 8 96 937692504 sdg 8 112 937692504 sdh 8 128 937692504 sdi 8 144 937692504 sdj 253 0 104857600 dm-0 253 1 31248384 dm-1 253 2 52428800 dm-2 cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid10] md1 : active raid1 sdf5[1] sde5[0] 233297920 blocks super 1.2 [2/2] [UU] bitmap: 1/2 pages [4KB], 65536KB chunk md0 : active raid1 sdf1[1] sde1[0] 998848 blocks super 1.2 [2/2] [UU] md126 : active raid6 sdj[6] sdg[3] sdi[2] sdh[7] sdc[4] sdd[0] sda[5] sdb[1] 5626152960 blocks level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU] bitmap: 0/7 pages [0KB], 65536KB chunk unused devices: <none> All disks are SATA Micron 5200 SSDs No Battery Backed Write Cache Workload: Mixture of compiles and later on accelerator device I/O through multiple docker containers. It usually takes days before the problem is triggered. I'm afraid I don't have the iostat/vmstat during the time of the problem recorded If there's key information missing that I can supply let me know and I'll try and get it to you. -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Tasks blocking forever with XFS stack traces 2019-11-05 9:32 ` Sitsofe Wheeler @ 2019-11-05 10:36 ` Carlos Maiolino 2019-11-05 11:58 ` Carlos Maiolino ` (2 more replies) 0 siblings, 3 replies; 10+ messages in thread From: Carlos Maiolino @ 2019-11-05 10:36 UTC (permalink / raw) To: Sitsofe Wheeler; +Cc: linux-xfs Hi Sitsofe. ... > <snip> > > > > > > Other directories on the same filesystem seem fine as do other XFS > > > filesystems on the same system. > > > > The fact you mention other directories seems to work, and the first stack trace > > you posted, it sounds like you've been keeping a singe AG too busy to almost > > make it unusable. But, you didn't provide enough information we can really make > > any progress here, and to be honest I'm more inclined to point the finger to > > your MD device. > > Let's see if we can pinpoint something :-) > > > Can you describe your MD device? RAID array? What kind? How many disks? > > RAID6 8 disks. > > > What's your filesystem configuration? (xfs_info <mount point>) > > meta-data=/dev/md126 isize=512 agcount=32, agsize=43954432 blks > = sectsz=4096 attr=2, projid32bit=1 > = crc=1 finobt=1 spinodes=0 rmapbt=0 > = reflink=0 > data = bsize=4096 blocks=1406538240, imaxpct=5 > = sunit=128 swidth=768 blks > naming =version 2 bsize=4096 ascii-ci=0 ftype=1 > log =internal bsize=4096 blocks=521728, version=2 > = sectsz=4096 sunit=1 blks, lazy-count=1 ^^^^^^ This should have been configured to 8 blocks, not 1 > Yes there's more. See a slightly elided dmesg from a longer run on > https://sucs.org/~sits/test/kern-20191024.log.gz . At a first quick look, it looks like you are having lots of IO contention in the log, and this is slowing down the rest of the filesystem. What caught my attention at first was the wrong configured log striping for the filesystem and I wonder if this isn't the responsible for the amount of IO contention you are having in the log. This might well be generating lots of RMW cycles while writing to the log generating the IO contention and slowing down the rest of the filesystem, I'll try to take a more careful look later on. I can't say anything if there is any bug related with the issue first because I honestly don't remember, second because you are using an old distro kernel which I have no idea to know which bug fixes have been backported or not. Maybe somebody else can remember of any bug that might be related, but the amount of threads you have waiting for log IO, and that misconfigured striping for the log smells smoke to me. I let you know if I can identify anything else later. Cheers. -- Carlos ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Tasks blocking forever with XFS stack traces 2019-11-05 10:36 ` Carlos Maiolino @ 2019-11-05 11:58 ` Carlos Maiolino 2019-11-05 14:12 ` Sitsofe Wheeler 2019-11-13 10:04 ` Sitsofe Wheeler 2 siblings, 0 replies; 10+ messages in thread From: Carlos Maiolino @ 2019-11-05 11:58 UTC (permalink / raw) To: Sitsofe Wheeler, linux-xfs Just to make clear my previous point: > > > What's your filesystem configuration? (xfs_info <mount point>) > > > > meta-data=/dev/md126 isize=512 agcount=32, agsize=43954432 blks > > = sectsz=4096 attr=2, projid32bit=1 > > = crc=1 finobt=1 spinodes=0 rmapbt=0 > > = reflink=0 > > data = bsize=4096 blocks=1406538240, imaxpct=5 > > = sunit=128 swidth=768 blks > > > naming =version 2 bsize=4096 ascii-ci=0 ftype=1 > > log =internal bsize=4096 blocks=521728, version=2 > > = sectsz=4096 sunit=1 blks, lazy-count=1 > ^^^^^^ This should have been > configured to 8 blocks, not 1 > 8 blocks here considering you've used default configuration, i.e. XFS defaults log stripe unit to 32KiB when the device strip is larger than 256KiB. -- Carlos ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Tasks blocking forever with XFS stack traces 2019-11-05 10:36 ` Carlos Maiolino 2019-11-05 11:58 ` Carlos Maiolino @ 2019-11-05 14:12 ` Sitsofe Wheeler 2019-11-05 16:09 ` Carlos Maiolino 2019-11-07 0:12 ` Chris Murphy 2019-11-13 10:04 ` Sitsofe Wheeler 2 siblings, 2 replies; 10+ messages in thread From: Sitsofe Wheeler @ 2019-11-05 14:12 UTC (permalink / raw) To: Carlos Maiolino; +Cc: linux-xfs On Tue, 5 Nov 2019 at 10:37, Carlos Maiolino <cmaiolino@redhat.com> wrote: > > > Hi Sitsofe. > > ... > > <snip> > > > > > > > > Other directories on the same filesystem seem fine as do other XFS > > > > filesystems on the same system. > > > > > > The fact you mention other directories seems to work, and the first stack trace > > > you posted, it sounds like you've been keeping a singe AG too busy to almost > > > make it unusable. But, you didn't provide enough information we can really make > > > any progress here, and to be honest I'm more inclined to point the finger to > > > your MD device. > > > > Let's see if we can pinpoint something :-) > > > > > Can you describe your MD device? RAID array? What kind? How many disks? > > > > RAID6 8 disks. > > > > > > What's your filesystem configuration? (xfs_info <mount point>) > > > > meta-data=/dev/md126 isize=512 agcount=32, agsize=43954432 blks > > = sectsz=4096 attr=2, projid32bit=1 > > = crc=1 finobt=1 spinodes=0 rmapbt=0 > > = reflink=0 > > data = bsize=4096 blocks=1406538240, imaxpct=5 > > = sunit=128 swidth=768 blks > > > naming =version 2 bsize=4096 ascii-ci=0 ftype=1 > > log =internal bsize=4096 blocks=521728, version=2 > > = sectsz=4096 sunit=1 blks, lazy-count=1 > ^^^^^^ This should have been > configured to 8 blocks, not 1 > > > Yes there's more. See a slightly elided dmesg from a longer run on > > https://sucs.org/~sits/test/kern-20191024.log.gz . > > At a first quick look, it looks like you are having lots of IO contention in the > log, and this is slowing down the rest of the filesystem. What caught my Should it become so slow that a task freezes entirely and never finishes? Once the problem hits it's not like anything makes any more progress on those directories nor was there very much generating dirty data. If this were to happen again though what extra information would be helpful (I'm guessing things like /proc/meminfo output)? > attention at first was the wrong configured log striping for the filesystem and > I wonder if this isn't the responsible for the amount of IO contention you are > having in the log. This might well be generating lots of RMW cycles while > writing to the log generating the IO contention and slowing down the rest of the > filesystem, I'll try to take a more careful look later on. My understanding is that the md "chunk size" is 64k so basically you're saying the sectsz should have been manually set to be as big as possible at mkfs time? I never realised this never happened by default (I see the sunit seems to be correct given the block size of 4096 but I'm unsure about swidth)... > I can't say anything if there is any bug related with the issue first because I > honestly don't remember, second because you are using an old distro kernel which > I have no idea to know which bug fixes have been backported or not. Maybe Very true. > somebody else can remember of any bug that might be related, but the amount of > threads you have waiting for log IO, and that misconfigured striping for the log > smells smoke to me. > > I let you know if I can identify anything else later. Thanks. -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Tasks blocking forever with XFS stack traces 2019-11-05 14:12 ` Sitsofe Wheeler @ 2019-11-05 16:09 ` Carlos Maiolino 2019-11-07 0:12 ` Chris Murphy 1 sibling, 0 replies; 10+ messages in thread From: Carlos Maiolino @ 2019-11-05 16:09 UTC (permalink / raw) To: Sitsofe Wheeler; +Cc: linux-xfs Hello. > > > <snip> > > > > > > > > > > Other directories on the same filesystem seem fine as do other XFS > > > > > filesystems on the same system. > > > > > > > > The fact you mention other directories seems to work, and the first stack trace > > > > you posted, it sounds like you've been keeping a singe AG too busy to almost > > > > make it unusable. But, you didn't provide enough information we can really make > > > > any progress here, and to be honest I'm more inclined to point the finger to > > > > your MD device. > > > > > > Let's see if we can pinpoint something :-) > > > > > > > Can you describe your MD device? RAID array? What kind? How many disks? > > > > > > RAID6 8 disks. > > > > > > > > > What's your filesystem configuration? (xfs_info <mount point>) > > > > > > meta-data=/dev/md126 isize=512 agcount=32, agsize=43954432 blks > > > = sectsz=4096 attr=2, projid32bit=1 > > > = crc=1 finobt=1 spinodes=0 rmapbt=0 > > > = reflink=0 > > > data = bsize=4096 blocks=1406538240, imaxpct=5 > > > = sunit=128 swidth=768 blks > > > > > naming =version 2 bsize=4096 ascii-ci=0 ftype=1 > > > log =internal bsize=4096 blocks=521728, version=2 > > > = sectsz=4096 sunit=1 blks, lazy-count=1 > > ^^^^^^ This should have been > > configured to 8 blocks, not 1 > > > > > Yes there's more. See a slightly elided dmesg from a longer run on > > > https://sucs.org/~sits/test/kern-20191024.log.gz . > > > > At a first quick look, it looks like you are having lots of IO contention in the > > log, and this is slowing down the rest of the filesystem. What caught my > > Should it become so slow that a task freezes entirely and never > finishes? Once the problem hits it's not like anything makes any more > progress on those directories nor was there very much generating dirty > data. > I am not sure how long you waited until you assumed it 'never finishes' :), but... I've seen systems starve to death due large amounts of RMW being generated, mainly when disks are slow or problematic. Since you are using SSDs and not spindles, I wonder if the SSDs write cycle might play a role here, but it's just a question I'm asking myself, I am not sure if it can play a role in making things even worse. In the file you sent above, you basically have: xfs-sync worker thread, which forced a log flush, and now is sleeping waiting for the flush to complete: Oct 24 16:27:11 <host> kernel: [115151.674164] Call Trace: Oct 24 16:27:11 <host> kernel: [115151.674170] __schedule+0x24e/0x880 Oct 24 16:27:11 <host> kernel: [115151.674175] schedule+0x2c/0x80 Oct 24 16:27:11 <host> kernel: [115151.674178] schedule_timeout+0x1cf/0x350 Oct 24 16:27:11 <host> kernel: [115151.674184] ? sched_clock+0x9/0x10 Oct 24 16:27:11 <host> kernel: [115151.674187] ? sched_clock+0x9/0x10 Oct 24 16:27:11 <host> kernel: [115151.674191] ? sched_clock_cpu+0x11/0xb0 Oct 24 16:27:11 <host> kernel: [115151.674196] wait_for_completion+0xba/0x140 Oct 24 16:27:11 <host> kernel: [115151.674199] ? wake_up_q+0x80/0x80 Oct 24 16:27:11 <host> kernel: [115151.674203] __flush_work+0x15b/0x210 Oct 24 16:27:11 <host> kernel: [115151.674206] ? worker_detach_from_pool+0xa0/0xa0 Oct 24 16:27:11 <host> kernel: [115151.674210] flush_work+0x10/0x20 Oct 24 16:27:11 <host> kernel: [115151.674250] xlog_cil_force_lsn+0x7b/0x210 [xfs] Oct 24 16:27:11 <host> kernel: [115151.674253] ? __switch_to_asm+0x41/0x70 Oct 24 16:27:11 <host> kernel: [115151.674256] ? __switch_to_asm+0x35/0x70 Oct 24 16:27:11 <host> kernel: [115151.674259] ? __switch_to_asm+0x41/0x70 Oct 24 16:27:11 <host> kernel: [115151.674262] ? __switch_to_asm+0x35/0x70 Oct 24 16:27:11 <host> kernel: [115151.674264] ? __switch_to_asm+0x41/0x70 Oct 24 16:27:11 <host> kernel: [115151.674267] ? __switch_to_asm+0x35/0x70 Oct 24 16:27:11 <host> kernel: [115151.674306] _xfs_log_force+0x8f/0x2a0 [xfs] Which wake up and wait for xlog_cil_push on the following thread: Oct 24 16:29:12 <host> kernel: [115272.503324] kworker/52:2 D 0 56479 2 0x80000080 Oct 24 16:29:12 <host> kernel: [115272.503381] Workqueue: xfs-cil/md126 xlog_cil_push_work [xfs] Oct 24 16:29:12 <host> kernel: [115272.503383] Call Trace: Oct 24 16:29:12 <host> kernel: [115272.503389] __schedule+0x24e/0x880 Oct 24 16:29:12 <host> kernel: [115272.503394] schedule+0x2c/0x80 Oct 24 16:29:12 <host> kernel: [115272.503444] xlog_state_get_iclog_space+0x105/0x2d0 [xfs] Oct 24 16:29:12 <host> kernel: [115272.503449] ? wake_up_q+0x80/0x80 Oct 24 16:29:12 <host> kernel: [115272.503492] xlog_write+0x163/0x6e0 [xfs] Oct 24 16:29:12 <host> kernel: [115272.503536] xlog_cil_push+0x2a7/0x410 [xfs] Oct 24 16:29:12 <host> kernel: [115272.503577] xlog_cil_push_work+0x15/0x20 [xfs] Oct 24 16:29:12 <host> kernel: [115272.503581] process_one_work+0x1de/0x420 Oct 24 16:29:12 <host> kernel: [115272.503584] worker_thread+0x32/0x410 Oct 24 16:29:12 <host> kernel: [115272.503587] kthread+0x121/0x140 Oct 24 16:29:12 <host> kernel: [115272.503590] ? process_one_work+0x420/0x420 Oct 24 16:29:12 <host> kernel: [115272.503593] ? kthread_create_worker_on_cpu+0x70/0x70 Oct 24 16:29:12 <host> kernel: [115272.503596] ret_from_fork+0x35/0x40 xlog_state_get_iclog_space() is waiting at: xlog_wait(&log->l_flush_wait, &log->l_icloglock); Which should be awaken by xlog_state_do_callback() which is called by xlog_state_done_syncing(), which in short, IIRC, comes from bio end io callbacks. I'm saying this by a quick look at the current code though. So, I think it ends up by your system waiting for journal IO completion to make progress (or maybe your storage stack missed an IO and XFS is stuck waiting for the completion which will never come, maybe it's deadlocked somewhere I am not seeing). I may very well be wrong though, it's been a while since I don't work on this part of the code, so, I'll just wait somebody else to give a different POV, other than risking pointing you in the wrong direction. I'll try to keep looking at it when I get some extra time, but well, at the end you will probably need to update your kernel to a recent version and try to reproduce the problem again, chasing bugs on old kernels is not really the scope of this list, but I'll give it some more thought when I get some extra time (if nobody sees what's wrong before :) > If this were to happen again though what extra information would be > helpful (I'm guessing things like /proc/meminfo output)? > > > attention at first was the wrong configured log striping for the filesystem and > > I wonder if this isn't the responsible for the amount of IO contention you are > > having in the log. This might well be generating lots of RMW cycles while > > writing to the log generating the IO contention and slowing down the rest of the > > filesystem, I'll try to take a more careful look later on. > > My understanding is that the md "chunk size" is 64k so basically > you're saying the sectsz should have been manually set to be as big as > possible at mkfs time? I never realised this never happened by default > (I see the sunit seems to be correct given the block size of 4096 but > I'm unsure about swidth)... Your RAID chunk is 512k according to the information you provided previously: md126 : active raid6 sdj[6] sdg[3] sdi[2] sdh[7] sdc[4] sdd[0] sda[5] sdb[1] 5626152960 blocks level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU] bitmap: 0/7 pages [0KB], 65536KB chunk And I believe you used default mkfs options when you created the filesystem, so xfs properly identified and automatically configured it for you: sunit=128 swidth=768 blks (128*4096) = 512Kib And the swidth matches exactly the amount of data disks I'd expect from your array, i.e. 6 Data Disks (+ 2 parity): (768/128) = 6 What I meant is the Log sunit is weird in your configuration, see the 'sunit' in the Log section. The XFS log stripe unit can range from 32Kib to 256Kib. And mkfs configures it to match the data sunit, UNLESS the data sunit is bigger tha 256KiB, which then mkfs will set it to 32KiB by default, which, in your case, using default mkfs configuration, I was expecting to see a log sunit=8 (8*4096) = 32KiB Maybe your xfsprogs version had any bug, or maybe somebody set it manually. I can't really say. -- Carlos ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Tasks blocking forever with XFS stack traces 2019-11-05 14:12 ` Sitsofe Wheeler 2019-11-05 16:09 ` Carlos Maiolino @ 2019-11-07 0:12 ` Chris Murphy 1 sibling, 0 replies; 10+ messages in thread From: Chris Murphy @ 2019-11-07 0:12 UTC (permalink / raw) To: Sitsofe Wheeler; +Cc: Carlos Maiolino, xfs list On Tue, Nov 5, 2019 at 2:13 PM Sitsofe Wheeler <sitsofe@gmail.com> wrote: > > My understanding is that the md "chunk size" is 64k so basically > you're saying the sectsz should have been manually set to be as big as > possible at mkfs time? I never realised this never happened by default > (I see the sunit seems to be correct given the block size of 4096 but > I'm unsure about swidth)... Check with # mdadm -E /dev/sdXY Chunk size depends on the version of mdadm at the time of creation. It's been a while since 512KiB was the default, leading to rather large full stripe write size, and a lot of RMW if a significant number of writes aren't full stripe writes. -- Chris Murphy ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Tasks blocking forever with XFS stack traces 2019-11-05 10:36 ` Carlos Maiolino 2019-11-05 11:58 ` Carlos Maiolino 2019-11-05 14:12 ` Sitsofe Wheeler @ 2019-11-13 10:04 ` Sitsofe Wheeler 2020-12-23 8:45 ` Sitsofe Wheeler 2 siblings, 1 reply; 10+ messages in thread From: Sitsofe Wheeler @ 2019-11-13 10:04 UTC (permalink / raw) To: Carlos Maiolino; +Cc: linux-xfs On Tue, 5 Nov 2019 at 10:37, Carlos Maiolino <cmaiolino@redhat.com> wrote: > > I can't say anything if there is any bug related with the issue first because I > honestly don't remember, second because you are using an old distro kernel which > I have no idea to know which bug fixes have been backported or not. Maybe > somebody else can remember of any bug that might be related, but the amount of > threads you have waiting for log IO, and that misconfigured striping for the log > smells smoke to me. > > I let you know if I can identify anything else later. So just to let anyone who might be following this know, going to a 5.0 kernel didn't solve the issue: Nov 12 16:45:02 <host> kernel: [27678.931551] INFO: task kworker/50:0:20430 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.931613] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.931667] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.931723] kworker/50:0 D 0 20430 2 0x80000080 Nov 12 16:45:02 <host> kernel: [27678.931801] Workqueue: xfs-sync/md126 xfs_log_worker [xfs] Nov 12 16:45:02 <host> kernel: [27678.931804] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.931814] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.931819] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.931823] schedule_timeout+0x1db/0x360 Nov 12 16:45:02 <host> kernel: [27678.931829] ? ttwu_do_activate+0x77/0x80 Nov 12 16:45:02 <host> kernel: [27678.931833] wait_for_completion+0xba/0x140 Nov 12 16:45:02 <host> kernel: [27678.931837] ? wake_up_q+0x80/0x80 Nov 12 16:45:02 <host> kernel: [27678.931843] __flush_work+0x15c/0x210 Nov 12 16:45:02 <host> kernel: [27678.931847] ? worker_detach_from_pool+0xb0/0xb0 Nov 12 16:45:02 <host> kernel: [27678.931850] flush_work+0x10/0x20 Nov 12 16:45:02 <host> kernel: [27678.931915] xlog_cil_force_lsn+0x7b/0x210 [xfs] Nov 12 16:45:02 <host> kernel: [27678.931920] ? __switch_to_asm+0x41/0x70 Nov 12 16:45:02 <host> kernel: [27678.931924] ? __switch_to_asm+0x35/0x70 Nov 12 16:45:02 <host> kernel: [27678.931928] ? __switch_to_asm+0x41/0x70 Nov 12 16:45:02 <host> kernel: [27678.931931] ? __switch_to_asm+0x35/0x70 Nov 12 16:45:02 <host> kernel: [27678.931935] ? __switch_to_asm+0x41/0x70 Nov 12 16:45:02 <host> kernel: [27678.931938] ? __switch_to_asm+0x35/0x70 Nov 12 16:45:02 <host> kernel: [27678.931992] ? xfs_log_worker+0x34/0x100 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932043] xfs_log_force+0x95/0x2e0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932047] ? __switch_to+0x96/0x4e0 Nov 12 16:45:02 <host> kernel: [27678.932051] ? __switch_to_asm+0x35/0x70 Nov 12 16:45:02 <host> kernel: [27678.932054] ? __switch_to_asm+0x41/0x70 Nov 12 16:45:02 <host> kernel: [27678.932058] ? __switch_to_asm+0x35/0x70 Nov 12 16:45:02 <host> kernel: [27678.932107] xfs_log_worker+0x34/0x100 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932111] process_one_work+0x1fd/0x400 Nov 12 16:45:02 <host> kernel: [27678.932114] worker_thread+0x34/0x410 Nov 12 16:45:02 <host> kernel: [27678.932120] kthread+0x121/0x140 Nov 12 16:45:02 <host> kernel: [27678.932123] ? process_one_work+0x400/0x400 Nov 12 16:45:02 <host> kernel: [27678.932127] ? kthread_park+0xb0/0xb0 Nov 12 16:45:02 <host> kernel: [27678.932132] ret_from_fork+0x35/0x40 Nov 12 16:45:02 <host> kernel: [27678.932146] INFO: task kworker/u161:0:46903 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.932200] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.932253] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.932309] kworker/u161:0 D 0 46903 2 0x80000080 Nov 12 16:45:02 <host> kernel: [27678.932316] Workqueue: writeback wb_workfn (flush-9:126) Nov 12 16:45:02 <host> kernel: [27678.932319] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.932323] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.932373] ? xfs_map_blocks+0xab/0x450 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932376] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.932380] rwsem_down_read_failed+0xe8/0x180 Nov 12 16:45:02 <host> kernel: [27678.932386] call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.932390] ? call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.932393] down_read+0x20/0x40 Nov 12 16:45:02 <host> kernel: [27678.932445] xfs_ilock+0xd5/0x100 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932492] xfs_map_blocks+0xab/0x450 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932497] ? wait_woken+0x80/0x80 Nov 12 16:45:02 <host> kernel: [27678.932503] ? blk_queue_split+0x10c/0x640 Nov 12 16:45:02 <host> kernel: [27678.932509] ? kmem_cache_alloc+0x15f/0x1c0 Nov 12 16:45:02 <host> kernel: [27678.932561] ? kmem_zone_alloc+0x6c/0xf0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932607] xfs_do_writepage+0x110/0x410 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932614] write_cache_pages+0x1bc/0x480 Nov 12 16:45:02 <host> kernel: [27678.932658] ? xfs_vm_writepages+0xa0/0xa0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932662] ? submit_bio+0x73/0x140 Nov 12 16:45:02 <host> kernel: [27678.932703] ? xfs_setfilesize_trans_alloc.isra.16+0x41/0x90 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932745] xfs_vm_writepages+0x6b/0xa0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.932750] do_writepages+0x4b/0xe0 Nov 12 16:45:02 <host> kernel: [27678.932754] __writeback_single_inode+0x40/0x330 Nov 12 16:45:02 <host> kernel: [27678.932756] ? __writeback_single_inode+0x40/0x330 Nov 12 16:45:02 <host> kernel: [27678.932759] writeback_sb_inodes+0x1e6/0x510 Nov 12 16:45:02 <host> kernel: [27678.932763] __writeback_inodes_wb+0x67/0xb0 Nov 12 16:45:02 <host> kernel: [27678.932766] wb_writeback+0x265/0x2f0 Nov 12 16:45:02 <host> kernel: [27678.932770] ? strp_read_sock+0x70/0xa0 Nov 12 16:45:02 <host> kernel: [27678.932772] wb_workfn+0x180/0x400 Nov 12 16:45:02 <host> kernel: [27678.932775] ? wb_workfn+0x180/0x400 Nov 12 16:45:02 <host> kernel: [27678.932779] process_one_work+0x1fd/0x400 Nov 12 16:45:02 <host> kernel: [27678.932783] worker_thread+0x34/0x410 Nov 12 16:45:02 <host> kernel: [27678.932787] kthread+0x121/0x140 Nov 12 16:45:02 <host> kernel: [27678.932790] ? process_one_work+0x400/0x400 Nov 12 16:45:02 <host> kernel: [27678.932794] ? kthread_park+0xb0/0xb0 Nov 12 16:45:02 <host> kernel: [27678.932799] ret_from_fork+0x35/0x40 Nov 12 16:45:02 <host> kernel: [27678.932843] INFO: task kworker/58:0:48437 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.932895] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.932948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.933002] kworker/58:0 D 0 48437 2 0x80000080 Nov 12 16:45:02 <host> kernel: [27678.933059] Workqueue: xfs-cil/md126 xlog_cil_push_work [xfs] Nov 12 16:45:02 <host> kernel: [27678.933061] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.933065] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.933117] ? xlog_bdstrat+0x37/0x70 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933120] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.933170] xlog_state_get_iclog_space+0x105/0x2d0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933174] ? wake_up_q+0x80/0x80 Nov 12 16:45:02 <host> kernel: [27678.933224] xlog_write+0x163/0x6e0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933273] xlog_cil_push+0x2a7/0x400 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933322] xlog_cil_push_work+0x15/0x20 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933326] process_one_work+0x1fd/0x400 Nov 12 16:45:02 <host> kernel: [27678.933329] worker_thread+0x34/0x410 Nov 12 16:45:02 <host> kernel: [27678.933334] kthread+0x121/0x140 Nov 12 16:45:02 <host> kernel: [27678.933337] ? process_one_work+0x400/0x400 Nov 12 16:45:02 <host> kernel: [27678.933341] ? kthread_park+0xb0/0xb0 Nov 12 16:45:02 <host> kernel: [27678.933345] ret_from_fork+0x35/0x40 Nov 12 16:45:02 <host> kernel: [27678.933354] INFO: task ninja:53333 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.933401] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.933453] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.933508] ninja D 0 53333 53327 0x000003a0 Nov 12 16:45:02 <host> kernel: [27678.933511] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.933515] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.933565] ? xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933568] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.933572] rwsem_down_read_failed+0xe8/0x180 Nov 12 16:45:02 <host> kernel: [27678.933577] call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.933580] ? call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.933631] ? xfs_trans_roll+0xe0/0xe0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933635] down_read+0x20/0x40 Nov 12 16:45:02 <host> kernel: [27678.933684] xfs_ilock+0xd5/0x100 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933730] xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933762] xfs_attr_get+0xbe/0x120 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933814] xfs_xattr_get+0x4b/0x70 [xfs] Nov 12 16:45:02 <host> kernel: [27678.933818] __vfs_getxattr+0x59/0x80 Nov 12 16:45:02 <host> kernel: [27678.933823] get_vfs_caps_from_disk+0x6a/0x170 Nov 12 16:45:02 <host> kernel: [27678.933827] ? legitimize_path.isra.31+0x2e/0x60 Nov 12 16:45:02 <host> kernel: [27678.933832] audit_copy_inode+0x6d/0xb0 Nov 12 16:45:02 <host> kernel: [27678.933837] __audit_inode+0x17b/0x2f0 Nov 12 16:45:02 <host> kernel: [27678.933840] filename_lookup+0x130/0x190 Nov 12 16:45:02 <host> kernel: [27678.933846] ? iomap_file_buffered_write+0x6e/0xa0 Nov 12 16:45:02 <host> kernel: [27678.933850] ? __check_object_size+0xdb/0x1b0 Nov 12 16:45:02 <host> kernel: [27678.933854] ? path_get+0x27/0x30 Nov 12 16:45:02 <host> kernel: [27678.933858] ? __audit_getname+0x97/0xb0 Nov 12 16:45:02 <host> kernel: [27678.933861] user_path_at_empty+0x36/0x40 Nov 12 16:45:02 <host> kernel: [27678.933864] ? user_path_at_empty+0x36/0x40 Nov 12 16:45:02 <host> kernel: [27678.933867] vfs_statx+0x76/0xe0 Nov 12 16:45:02 <host> kernel: [27678.933871] __do_sys_newstat+0x3d/0x70 Nov 12 16:45:02 <host> kernel: [27678.933876] ? syscall_trace_enter+0x1da/0x2d0 Nov 12 16:45:02 <host> kernel: [27678.933880] __x64_sys_newstat+0x16/0x20 Nov 12 16:45:02 <host> kernel: [27678.933884] do_syscall_64+0x5a/0x120 Nov 12 16:45:02 <host> kernel: [27678.933889] entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 12 16:45:02 <host> kernel: [27678.933892] RIP: 0033:0x7f0a393fc775 Nov 12 16:45:02 <host> kernel: [27678.933900] Code: Bad RIP value. Nov 12 16:45:02 <host> kernel: [27678.933902] RSP: 002b:00007ffdaaa336b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 Nov 12 16:45:02 <host> kernel: [27678.933905] RAX: ffffffffffffffda RBX: 00007ffdaaa33740 RCX: 00007f0a393fc775 Nov 12 16:45:02 <host> kernel: [27678.933906] RDX: 00007ffdaaa33740 RSI: 00007ffdaaa33740 RDI: 0000556f5acdf100 Nov 12 16:45:02 <host> kernel: [27678.933908] RBP: 00007ffdaaa336d0 R08: 00007ffdaaa33866 R09: 0000000000000036 Nov 12 16:45:02 <host> kernel: [27678.933909] R10: 00007ffdaaa33867 R11: 0000000000000246 R12: 00007ffdaaa33820 Nov 12 16:45:02 <host> kernel: [27678.933911] R13: 00007ffdaaa33840 R14: 0000000000000000 R15: 0000000057d60100 Nov 12 16:45:02 <host> kernel: [27678.933920] INFO: task c++:56846 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.933967] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.934019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.934074] c++ D 0 56846 56844 0x000003a0 Nov 12 16:45:02 <host> kernel: [27678.934077] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.934081] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.934084] ? __switch_to+0x309/0x4e0 Nov 12 16:45:02 <host> kernel: [27678.934087] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.934091] schedule_timeout+0x1db/0x360 Nov 12 16:45:02 <host> kernel: [27678.934093] ? __schedule+0x2c8/0x870 Nov 12 16:45:02 <host> kernel: [27678.934097] wait_for_completion+0xba/0x140 Nov 12 16:45:02 <host> kernel: [27678.934101] ? wake_up_q+0x80/0x80 Nov 12 16:45:02 <host> kernel: [27678.934104] __flush_work+0x15c/0x210 Nov 12 16:45:02 <host> kernel: [27678.934107] ? worker_detach_from_pool+0xb0/0xb0 Nov 12 16:45:02 <host> kernel: [27678.934111] flush_work+0x10/0x20 Nov 12 16:45:02 <host> kernel: [27678.934162] xlog_cil_force_lsn+0x7b/0x210 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934211] ? xfs_buf_lock+0xe9/0xf0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934261] xfs_log_force+0x95/0x2e0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934309] ? xfs_buf_find.isra.29+0x1fa/0x600 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934355] xfs_buf_lock+0xe9/0xf0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934399] xfs_buf_find.isra.29+0x1fa/0x600 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934442] xfs_buf_get_map+0x43/0x2b0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934496] xfs_trans_get_buf_map+0xec/0x170 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934534] xfs_da_get_buf+0xbd/0xf0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934574] xfs_dir3_data_init+0x6e/0x210 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934613] xfs_dir2_sf_to_block+0x12e/0x6e0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934649] ? xfs_dir2_sf_to_block+0x12e/0x6e0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934700] ? kmem_zone_alloc+0x6c/0xf0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934750] ? kmem_zone_alloc+0x6c/0xf0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934797] ? xlog_grant_head_check+0x54/0xf0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934838] xfs_dir2_sf_addname+0xd9/0x6c0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934843] ? __kmalloc+0x178/0x210 Nov 12 16:45:02 <host> kernel: [27678.934891] ? kmem_alloc+0x6c/0xf0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934930] xfs_dir_createname+0x182/0x1d0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.934979] xfs_rename+0x771/0x8d0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.935029] xfs_vn_rename+0xd3/0x140 [xfs] Nov 12 16:45:02 <host> kernel: [27678.935033] vfs_rename+0x383/0x920 Nov 12 16:45:02 <host> kernel: [27678.935037] do_renameat2+0x4ca/0x590 Nov 12 16:45:02 <host> kernel: [27678.935041] __x64_sys_rename+0x20/0x30 Nov 12 16:45:02 <host> kernel: [27678.935045] do_syscall_64+0x5a/0x120 Nov 12 16:45:02 <host> kernel: [27678.935050] entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 12 16:45:02 <host> kernel: [27678.935052] RIP: 0033:0x7f5dc9e25d37 Nov 12 16:45:02 <host> kernel: [27678.935057] Code: Bad RIP value. Nov 12 16:45:02 <host> kernel: [27678.935059] RSP: 002b:00007ffdcc39b908 EFLAGS: 00000213 ORIG_RAX: 0000000000000052 Nov 12 16:45:02 <host> kernel: [27678.935061] RAX: ffffffffffffffda RBX: 00007ffdcc39b934 RCX: 00007f5dc9e25d37 Nov 12 16:45:02 <host> kernel: [27678.935063] RDX: 000055aee60eb010 RSI: 000055aee60f02d0 RDI: 000055aee60f01c0 Nov 12 16:45:02 <host> kernel: [27678.935064] RBP: 00007ffdcc39b9d0 R08: 0000000000000000 R09: 000055aee61321c0 Nov 12 16:45:02 <host> kernel: [27678.935066] R10: 000055aee60eb010 R11: 0000000000000213 R12: 0000000000000004 Nov 12 16:45:02 <host> kernel: [27678.935068] R13: 0000000000006e90 R14: 000055aee60f00d0 R15: 0000000000006e90 Nov 12 16:45:02 <host> kernel: [27678.935071] INFO: task c++:56847 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.935118] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.935170] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.935269] c++ D 0 56847 56845 0x000003a0 Nov 12 16:45:02 <host> kernel: [27678.935287] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.935293] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.935347] ? xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.935356] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.935364] rwsem_down_read_failed+0xe8/0x180 Nov 12 16:45:02 <host> kernel: [27678.935372] call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.935381] ? call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.935436] ? xfs_trans_roll+0xe0/0xe0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.935448] down_read+0x20/0x40 Nov 12 16:45:02 <host> kernel: [27678.935501] xfs_ilock+0xd5/0x100 [xfs] Nov 12 16:45:02 <host> kernel: [27678.935551] xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.935588] xfs_attr_get+0xbe/0x120 [xfs] Nov 12 16:45:02 <host> kernel: [27678.935643] xfs_xattr_get+0x4b/0x70 [xfs] Nov 12 16:45:02 <host> kernel: [27678.935652] __vfs_getxattr+0x59/0x80 Nov 12 16:45:02 <host> kernel: [27678.935664] get_vfs_caps_from_disk+0x6a/0x170 Nov 12 16:45:02 <host> kernel: [27678.935675] audit_copy_inode+0x6d/0xb0 Nov 12 16:45:02 <host> kernel: [27678.935684] __audit_inode+0x17b/0x2f0 Nov 12 16:45:02 <host> kernel: [27678.935692] path_openat+0x38f/0x1700 Nov 12 16:45:02 <host> kernel: [27678.935702] ? insert_pfn+0x152/0x240 Nov 12 16:45:02 <host> kernel: [27678.935709] ? vmf_insert_pfn_prot+0x9b/0x120 Nov 12 16:45:02 <host> kernel: [27678.935716] do_filp_open+0x9b/0x110 Nov 12 16:45:02 <host> kernel: [27678.935723] ? __check_object_size+0xdb/0x1b0 Nov 12 16:45:02 <host> kernel: [27678.935734] ? path_get+0x27/0x30 Nov 12 16:45:02 <host> kernel: [27678.935746] ? __alloc_fd+0x46/0x170 Nov 12 16:45:02 <host> kernel: [27678.935756] do_sys_open+0x1bb/0x2d0 Nov 12 16:45:02 <host> kernel: [27678.935764] ? do_sys_open+0x1bb/0x2d0 Nov 12 16:45:02 <host> kernel: [27678.935773] __x64_sys_openat+0x20/0x30 Nov 12 16:45:02 <host> kernel: [27678.935783] do_syscall_64+0x5a/0x120 Nov 12 16:45:02 <host> kernel: [27678.935793] entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 12 16:45:02 <host> kernel: [27678.935801] RIP: 0033:0x7fe4f94a5c8e Nov 12 16:45:02 <host> kernel: [27678.935813] Code: Bad RIP value. Nov 12 16:45:02 <host> kernel: [27678.935818] RSP: 002b:00007ffc4c6e2550 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 Nov 12 16:45:02 <host> kernel: [27678.935828] RAX: ffffffffffffffda RBX: 000000000003a2f8 RCX: 00007fe4f94a5c8e Nov 12 16:45:02 <host> kernel: [27678.935836] RDX: 00000000000000c2 RSI: 0000557e925fdc80 RDI: 00000000ffffff9c Nov 12 16:45:02 <host> kernel: [27678.935841] RBP: 0000000000000000 R08: 00007ffc4c7250a0 R09: 00007ffc4c725080 Nov 12 16:45:02 <host> kernel: [27678.935847] R10: 0000000000000180 R11: 0000000000000246 R12: 0000557e925fdc80 Nov 12 16:45:02 <host> kernel: [27678.935854] R13: 0000557e925fdcda R14: 00007fe4f9551c80 R15: 8421084210842109 Nov 12 16:45:02 <host> kernel: [27678.935863] INFO: task c++:56849 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.935917] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.935971] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.936028] c++ D 0 56849 56848 0x000003a0 Nov 12 16:45:02 <host> kernel: [27678.936031] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.936035] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.936084] ? xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936087] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.936091] rwsem_down_read_failed+0xe8/0x180 Nov 12 16:45:02 <host> kernel: [27678.936095] call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.936099] ? call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.936149] ? xfs_trans_roll+0xe0/0xe0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936154] down_read+0x20/0x40 Nov 12 16:45:02 <host> kernel: [27678.936202] xfs_ilock+0xd5/0x100 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936250] xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936283] xfs_attr_get+0xbe/0x120 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936335] xfs_xattr_get+0x4b/0x70 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936340] __vfs_getxattr+0x59/0x80 Nov 12 16:45:02 <host> kernel: [27678.936344] get_vfs_caps_from_disk+0x6a/0x170 Nov 12 16:45:02 <host> kernel: [27678.936348] audit_copy_inode+0x6d/0xb0 Nov 12 16:45:02 <host> kernel: [27678.936352] __audit_inode+0x17b/0x2f0 Nov 12 16:45:02 <host> kernel: [27678.936356] path_openat+0x38f/0x1700 Nov 12 16:45:02 <host> kernel: [27678.936360] ? insert_pfn+0x152/0x240 Nov 12 16:45:02 <host> kernel: [27678.936364] ? vmf_insert_pfn_prot+0x9b/0x120 Nov 12 16:45:02 <host> kernel: [27678.936368] do_filp_open+0x9b/0x110 Nov 12 16:45:02 <host> kernel: [27678.936371] ? __check_object_size+0xdb/0x1b0 Nov 12 16:45:02 <host> kernel: [27678.936375] ? path_get+0x27/0x30 Nov 12 16:45:02 <host> kernel: [27678.936379] ? __alloc_fd+0x46/0x170 Nov 12 16:45:02 <host> kernel: [27678.936383] do_sys_open+0x1bb/0x2d0 Nov 12 16:45:02 <host> kernel: [27678.936386] ? do_sys_open+0x1bb/0x2d0 Nov 12 16:45:02 <host> kernel: [27678.936390] __x64_sys_openat+0x20/0x30 Nov 12 16:45:02 <host> kernel: [27678.936394] do_syscall_64+0x5a/0x120 Nov 12 16:45:02 <host> kernel: [27678.936399] entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 12 16:45:02 <host> kernel: [27678.936401] RIP: 0033:0x7fe5c6575c8e Nov 12 16:45:02 <host> kernel: [27678.936407] Code: Bad RIP value. Nov 12 16:45:02 <host> kernel: [27678.936409] RSP: 002b:00007fffbd824140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 Nov 12 16:45:02 <host> kernel: [27678.936411] RAX: ffffffffffffffda RBX: 000000000003a2f8 RCX: 00007fe5c6575c8e Nov 12 16:45:02 <host> kernel: [27678.936413] RDX: 00000000000000c2 RSI: 0000561423cc41c0 RDI: 00000000ffffff9c Nov 12 16:45:02 <host> kernel: [27678.936414] RBP: 0000000000000000 R08: 00007fffbd8f80a0 R09: 00007fffbd8f8080 Nov 12 16:45:02 <host> kernel: [27678.936416] R10: 0000000000000180 R11: 0000000000000246 R12: 0000561423cc41c0 Nov 12 16:45:02 <host> kernel: [27678.936417] R13: 0000561423cc4218 R14: 00007fe5c6621c80 R15: 8421084210842109 Nov 12 16:45:02 <host> kernel: [27678.936421] INFO: task c++:56851 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.936474] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.936535] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.936596] c++ D 0 56851 56850 0x000003a0 Nov 12 16:45:02 <host> kernel: [27678.936605] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.936615] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.936623] ? __switch_to_asm+0x41/0x70 Nov 12 16:45:02 <host> kernel: [27678.936631] ? __switch_to_asm+0x35/0x70 Nov 12 16:45:02 <host> kernel: [27678.936683] ? xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936692] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.936703] rwsem_down_read_failed+0xe8/0x180 Nov 12 16:45:02 <host> kernel: [27678.936712] ? __switch_to_asm+0x35/0x70 Nov 12 16:45:02 <host> kernel: [27678.936722] call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.936731] ? call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.936787] ? xfs_trans_roll+0xe0/0xe0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936796] down_read+0x20/0x40 Nov 12 16:45:02 <host> kernel: [27678.936852] xfs_ilock+0xd5/0x100 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936903] xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936943] xfs_attr_get+0xbe/0x120 [xfs] Nov 12 16:45:02 <host> kernel: [27678.936998] xfs_xattr_get+0x4b/0x70 [xfs] Nov 12 16:45:02 <host> kernel: [27678.937002] __vfs_getxattr+0x59/0x80 Nov 12 16:45:02 <host> kernel: [27678.937006] get_vfs_caps_from_disk+0x6a/0x170 Nov 12 16:45:02 <host> kernel: [27678.937012] ? inode_permission+0x63/0x1a0 Nov 12 16:45:02 <host> kernel: [27678.937016] audit_copy_inode+0x6d/0xb0 Nov 12 16:45:02 <host> kernel: [27678.937020] __audit_inode+0x17b/0x2f0 Nov 12 16:45:02 <host> kernel: [27678.937023] filename_parentat+0x147/0x190 Nov 12 16:45:02 <host> kernel: [27678.937028] ? radix_tree_lookup+0xd/0x10 Nov 12 16:45:02 <host> kernel: [27678.937032] ? __check_object_size+0xdb/0x1b0 Nov 12 16:45:02 <host> kernel: [27678.937036] ? path_get+0x27/0x30 Nov 12 16:45:02 <host> kernel: [27678.937040] do_renameat2+0xc6/0x590 Nov 12 16:45:02 <host> kernel: [27678.937042] ? do_renameat2+0xc6/0x590 Nov 12 16:45:02 <host> kernel: [27678.937046] ? __audit_syscall_entry+0xdd/0x130 Nov 12 16:45:02 <host> kernel: [27678.937050] __x64_sys_rename+0x20/0x30 Nov 12 16:45:02 <host> kernel: [27678.937054] do_syscall_64+0x5a/0x120 Nov 12 16:45:02 <host> kernel: [27678.937058] entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 12 16:45:02 <host> kernel: [27678.937060] RIP: 0033:0x7f83b31c6d37 Nov 12 16:45:02 <host> kernel: [27678.937065] Code: Bad RIP value. Nov 12 16:45:02 <host> kernel: [27678.937067] RSP: 002b:00007ffd60610948 EFLAGS: 00000213 ORIG_RAX: 0000000000000052 Nov 12 16:45:02 <host> kernel: [27678.937069] RAX: ffffffffffffffda RBX: 00007ffd60610974 RCX: 00007f83b31c6d37 Nov 12 16:45:02 <host> kernel: [27678.937070] RDX: 000056474e2f3010 RSI: 000056474e2f82d0 RDI: 000056474e2f81c0 Nov 12 16:45:02 <host> kernel: [27678.937074] RBP: 00007ffd60610a10 R08: 0000000000000000 R09: 000056474e33cc20 Nov 12 16:45:02 <host> kernel: [27678.937075] R10: 000056474e2f3010 R11: 0000000000000213 R12: 0000000000000004 Nov 12 16:45:02 <host> kernel: [27678.937077] R13: 0000000000003720 R14: 000056474e2f80d0 R15: 0000000000003720 Nov 12 16:45:02 <host> kernel: [27678.937081] INFO: task c++:56853 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.937127] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.937179] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.937234] c++ D 0 56853 56852 0x000003a0 Nov 12 16:45:02 <host> kernel: [27678.937237] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.937241] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.937289] ? xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.937292] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.937296] rwsem_down_read_failed+0xe8/0x180 Nov 12 16:45:02 <host> kernel: [27678.937301] ? xas_store+0x1e1/0x5b0 Nov 12 16:45:02 <host> kernel: [27678.937305] call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.937309] ? call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.937358] ? xfs_trans_roll+0xe0/0xe0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.937368] down_read+0x20/0x40 Nov 12 16:45:02 <host> kernel: [27678.937424] xfs_ilock+0xd5/0x100 [xfs] Nov 12 16:45:02 <host> kernel: [27678.937476] xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.937514] xfs_attr_get+0xbe/0x120 [xfs] Nov 12 16:45:02 <host> kernel: [27678.937571] xfs_xattr_get+0x4b/0x70 [xfs] Nov 12 16:45:02 <host> kernel: [27678.937581] __vfs_getxattr+0x59/0x80 Nov 12 16:45:02 <host> kernel: [27678.937605] get_vfs_caps_from_disk+0x6a/0x170 Nov 12 16:45:02 <host> kernel: [27678.937614] ? inode_permission+0x63/0x1a0 Nov 12 16:45:02 <host> kernel: [27678.937623] audit_copy_inode+0x6d/0xb0 Nov 12 16:45:02 <host> kernel: [27678.937631] __audit_inode+0x17b/0x2f0 Nov 12 16:45:02 <host> kernel: [27678.937638] filename_parentat+0x147/0x190 Nov 12 16:45:02 <host> kernel: [27678.937646] ? radix_tree_lookup+0xd/0x10 Nov 12 16:45:02 <host> kernel: [27678.937652] ? __check_object_size+0xdb/0x1b0 Nov 12 16:45:02 <host> kernel: [27678.937661] ? path_get+0x27/0x30 Nov 12 16:45:02 <host> kernel: [27678.937669] do_renameat2+0xc6/0x590 Nov 12 16:45:02 <host> kernel: [27678.937676] ? do_renameat2+0xc6/0x590 Nov 12 16:45:02 <host> kernel: [27678.937683] ? __audit_syscall_entry+0xdd/0x130 Nov 12 16:45:02 <host> kernel: [27678.937690] __x64_sys_rename+0x20/0x30 Nov 12 16:45:02 <host> kernel: [27678.937694] do_syscall_64+0x5a/0x120 Nov 12 16:45:02 <host> kernel: [27678.937699] entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 12 16:45:02 <host> kernel: [27678.937701] RIP: 0033:0x7faba8af6d37 Nov 12 16:45:02 <host> kernel: [27678.937705] Code: Bad RIP value. Nov 12 16:45:02 <host> kernel: [27678.937707] RSP: 002b:00007fff0e234358 EFLAGS: 00000213 ORIG_RAX: 0000000000000052 Nov 12 16:45:02 <host> kernel: [27678.937709] RAX: ffffffffffffffda RBX: 00007fff0e234384 RCX: 00007faba8af6d37 Nov 12 16:45:02 <host> kernel: [27678.937710] RDX: 0000564342653010 RSI: 00005643426577a0 RDI: 0000564342659c80 Nov 12 16:45:02 <host> kernel: [27678.937712] RBP: 00007fff0e234420 R08: 0000000000000000 R09: 0000564342699f20 Nov 12 16:45:02 <host> kernel: [27678.937713] R10: 0000564342653010 R11: 0000000000000213 R12: 0000000000000004 Nov 12 16:45:02 <host> kernel: [27678.937715] R13: 000000000000f160 R14: 0000564342658030 R15: 000000000000f160 Nov 12 16:45:02 <host> kernel: [27678.937719] INFO: task c++:56855 blocked for more than 120 seconds. Nov 12 16:45:02 <host> kernel: [27678.937765] Tainted: G OE 5.0.0-32-generic #34~18.04.2-Ubuntu Nov 12 16:45:02 <host> kernel: [27678.939382] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 12 16:45:02 <host> kernel: [27678.940988] c++ D 0 56855 56854 0x000003a0 Nov 12 16:45:02 <host> kernel: [27678.940991] Call Trace: Nov 12 16:45:02 <host> kernel: [27678.940995] __schedule+0x2c0/0x870 Nov 12 16:45:02 <host> kernel: [27678.941048] ? xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.941052] schedule+0x2c/0x70 Nov 12 16:45:02 <host> kernel: [27678.941055] rwsem_down_read_failed+0xe8/0x180 Nov 12 16:45:02 <host> kernel: [27678.941062] call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.941066] ? call_rwsem_down_read_failed+0x18/0x30 Nov 12 16:45:02 <host> kernel: [27678.941116] ? xfs_trans_roll+0xe0/0xe0 [xfs] Nov 12 16:45:02 <host> kernel: [27678.941121] down_read+0x20/0x40 Nov 12 16:45:02 <host> kernel: [27678.941169] xfs_ilock+0xd5/0x100 [xfs] Nov 12 16:45:02 <host> kernel: [27678.941217] xfs_ilock_attr_map_shared+0x34/0x40 [xfs] Nov 12 16:45:02 <host> kernel: [27678.941251] xfs_attr_get+0xbe/0x120 [xfs] Nov 12 16:45:02 <host> kernel: [27678.941303] xfs_xattr_get+0x4b/0x70 [xfs] Nov 12 16:45:02 <host> kernel: [27678.941307] __vfs_getxattr+0x59/0x80 Nov 12 16:45:02 <host> kernel: [27678.941312] get_vfs_caps_from_disk+0x6a/0x170 Nov 12 16:45:02 <host> kernel: [27678.941317] audit_copy_inode+0x6d/0xb0 Nov 12 16:45:02 <host> kernel: [27678.941322] __audit_inode+0x17b/0x2f0 Nov 12 16:45:02 <host> kernel: [27678.941325] path_openat+0x38f/0x1700 Nov 12 16:45:02 <host> kernel: [27678.941330] ? insert_pfn+0x152/0x240 Nov 12 16:45:02 <host> kernel: [27678.941334] ? vmf_insert_pfn_prot+0x9b/0x120 Nov 12 16:45:02 <host> kernel: [27678.941337] do_filp_open+0x9b/0x110 Nov 12 16:45:02 <host> kernel: [27678.941341] ? __check_object_size+0xdb/0x1b0 Nov 12 16:45:02 <host> kernel: [27678.941345] ? path_get+0x27/0x30 Nov 12 16:45:02 <host> kernel: [27678.941349] ? __alloc_fd+0x46/0x170 Nov 12 16:45:02 <host> kernel: [27678.941353] do_sys_open+0x1bb/0x2d0 Nov 12 16:45:02 <host> kernel: [27678.941356] ? do_sys_open+0x1bb/0x2d0 Nov 12 16:45:02 <host> kernel: [27678.941360] __x64_sys_openat+0x20/0x30 Nov 12 16:45:02 <host> kernel: [27678.941364] do_syscall_64+0x5a/0x120 Nov 12 16:45:02 <host> kernel: [27678.941369] entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 12 16:45:02 <host> kernel: [27678.941371] RIP: 0033:0x7f8bc5732c8e Nov 12 16:45:02 <host> kernel: [27678.941376] Code: Bad RIP value. Nov 12 16:45:02 <host> kernel: [27678.941379] RSP: 002b:00007ffe3e6fa850 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 Nov 12 16:45:02 <host> kernel: [27678.941382] RAX: ffffffffffffffda RBX: 000000000003a2f8 RCX: 00007f8bc5732c8e Nov 12 16:45:02 <host> kernel: [27678.941383] RDX: 00000000000000c2 RSI: 000055b674710770 RDI: 00000000ffffff9c Nov 12 16:45:02 <host> kernel: [27678.941385] RBP: 0000000000000000 R08: 00007ffe3e7960a0 R09: 00007ffe3e796080 Nov 12 16:45:02 <host> kernel: [27678.941386] R10: 0000000000000180 R11: 0000000000000246 R12: 000055b674710770 Nov 12 16:45:02 <host> kernel: [27678.941388] R13: 000055b6747107cd R14: 00007f8bc57dec80 R15: 8421084210842109 -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Tasks blocking forever with XFS stack traces 2019-11-13 10:04 ` Sitsofe Wheeler @ 2020-12-23 8:45 ` Sitsofe Wheeler 0 siblings, 0 replies; 10+ messages in thread From: Sitsofe Wheeler @ 2020-12-23 8:45 UTC (permalink / raw) To: linux-xfs; +Cc: Carlos Maiolino On Wed, 13 Nov 2019 at 10:04, Sitsofe Wheeler <sitsofe@gmail.com> wrote: > > Nov 12 16:45:02 <host> kernel: [27678.931551] INFO: task > kworker/50:0:20430 blocked for more than 120 seconds. > Nov 12 16:45:02 <host> kernel: [27678.931613] Tainted: G > OE 5.0.0-32-generic #34~18.04.2-Ubuntu > Nov 12 16:45:02 <host> kernel: [27678.931667] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Nov 12 16:45:02 <host> kernel: [27678.931723] kworker/50:0 D 0 > 20430 2 0x80000080 > Nov 12 16:45:02 <host> kernel: [27678.931801] Workqueue: > xfs-sync/md126 xfs_log_worker [xfs] > Nov 12 16:45:02 <host> kernel: [27678.931804] Call Trace: > Nov 12 16:45:02 <host> kernel: [27678.931814] __schedule+0x2c0/0x870 > Nov 12 16:45:02 <host> kernel: [27678.931819] schedule+0x2c/0x70 > Nov 12 16:45:02 <host> kernel: [27678.931823] schedule_timeout+0x1db/0x360 > Nov 12 16:45:02 <host> kernel: [27678.931829] ? ttwu_do_activate+0x77/0x80 > Nov 12 16:45:02 <host> kernel: [27678.931833] wait_for_completion+0xba/0x140 > Nov 12 16:45:02 <host> kernel: [27678.931837] ? wake_up_q+0x80/0x80 > Nov 12 16:45:02 <host> kernel: [27678.931843] __flush_work+0x15c/0x210 (Quite some time later) This issue went away after switching to an Ubuntu 18.04 5.3 LTS kernel. Later kernels (e.g. 5.4) have not manifested the issue either. -- Sitsofe ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-12-23 8:47 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-11-05 7:27 Tasks blocking forever with XFS stack traces Sitsofe Wheeler 2019-11-05 8:54 ` Carlos Maiolino 2019-11-05 9:32 ` Sitsofe Wheeler 2019-11-05 10:36 ` Carlos Maiolino 2019-11-05 11:58 ` Carlos Maiolino 2019-11-05 14:12 ` Sitsofe Wheeler 2019-11-05 16:09 ` Carlos Maiolino 2019-11-07 0:12 ` Chris Murphy 2019-11-13 10:04 ` Sitsofe Wheeler 2020-12-23 8:45 ` Sitsofe Wheeler
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).