* permanent XFS volume corruption @ 2017-05-11 14:39 Jan Beulich 2017-05-11 14:58 ` Eric Sandeen 2017-05-11 16:39 ` Eric Sandeen 0 siblings, 2 replies; 18+ messages in thread From: Jan Beulich @ 2017-05-11 14:39 UTC (permalink / raw) To: linux-xfs [-- Attachment #1: Type: text/plain, Size: 3546 bytes --] Hello, It is now on two systems that I'm getting XFS (sda1): corrupt dinode 576254627, has realtime flag set. ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64 IN.............d ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f Y.....|/Y....../ ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80 Y....../........ XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c. Caller xfs_iread+0xea/0x2e0 [xfs] CPU: 10 PID: 4418 Comm: smbd Not tainted 3.12.73-sp1-2017-04-26-jb #2 Hardware name: AMD Dinar/Dinar, BIOS RDN1506A 08/31/2014 0000000000000001 ffffffff81354083 ffffffffa03ea40a ffffffffa03a0952 0000000000000000 0000000000000075 ffff88042ea63300 ffff88042f508000 ffff88022efe7000 ffff88042f508028 0000000000000000 ffffffffa03e9b06 Call Trace: [<ffffffff81004e3b>] dump_trace+0x7b/0x310 [<ffffffff81004ad6>] show_stack_log_lvl+0xe6/0x150 [<ffffffff81005ddc>] show_stack+0x1c/0x50 [<ffffffff81354083>] dump_stack+0x6f/0x84 [<ffffffffa03a0952>] xfs_corruption_error+0x62/0xa0 [xfs] [<ffffffffa03e9b06>] xfs_iformat_fork+0x3b6/0x530 [xfs] [<ffffffffa03ea40a>] xfs_iread+0xea/0x2e0 [xfs] [<ffffffffa03a6538>] xfs_iget_cache_miss+0x58/0x1d0 [xfs] [<ffffffffa03a67c3>] xfs_iget+0x113/0x190 [xfs] [<ffffffffa03e5be8>] xfs_lookup+0xb8/0xd0 [xfs] [<ffffffffa03aaddd>] xfs_vn_lookup+0x4d/0x90 [xfs] [<ffffffff8110539d>] lookup_real+0x1d/0x60 [<ffffffff811064d2>] __lookup_hash+0x32/0x50 [<ffffffff8110a2a4>] path_lookupat+0x7f4/0x8b0 [<ffffffff8110a38e>] filename_lookup+0x2e/0x90 [<ffffffff8110abef>] user_path_at_empty+0x9f/0xd0 [<ffffffff81100678>] vfs_fstatat+0x48/0xa0 [<ffffffff8110081f>] SyS_newstat+0x1f/0x50 [<ffffffff81358d42>] system_call_fastpath+0x16/0x1b [<00007f141f4f0d35>] 0x7f141f4f0d34 XFS (sda1): Corruption detected. Unmount and run xfs_repair after a crash with a 4.11-based kernel. I didn't try xfs_repair-ing the volume in this second instance, as the result from doing so in the first instance was only permanent re-occurrence (and apparently spreading) of the problem. It may be interesting that xfs_check finds only this one corrupted inode, while the kernel also finds at least one more: XFS (sda1): corrupt dinode 104812066, has realtime flag set. ffff88042e88f200: 49 4e 41 ed 02 01 00 00 00 00 03 e8 00 00 00 64 INA............d ffff88042e88f210: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 ................ ffff88042e88f220: 59 01 c7 96 00 16 1e 50 59 14 0e a2 15 60 54 2f Y......PY....`T/ ffff88042e88f230: 59 14 0e a2 15 60 54 2f 00 00 00 00 00 00 00 8e Y....`T/........ XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c. Caller xfs_iread+0xea/0x2e0 [xfs] The kernel used after the crash doesn't appear to matter, all report the same issues, but I've experienced similar crashes in the past without ever having seen such corruption before. In any event I think there are two problems: The corruption itself (possibly an issue with recent enough upstream kernels only) and the fact that running xfs_repair doesn't help in these cases. I'm attaching xfs_check and xfs_metadump warning output for both affected volumes in this second instance. The full files xfs_metadump -agow produced can be provided upon request (500Mb and 80Mb uncompressed respectively). Thanks for any advice / fix, Jan [-- Attachment #2: sda1.chk --] [-- Type: application/octet-stream, Size: 478 bytes --] inode 576254627 bad rt block number 36019429, offset 0 bad nblocks 9 for inode 576254627, counted 0 block 2/2464997 type unknown not expected block 2/2464998 type unknown not expected block 2/2464999 type unknown not expected block 2/2465000 type unknown not expected block 2/2465001 type unknown not expected block 2/2465002 type unknown not expected block 2/2465003 type unknown not expected block 2/2465004 type unknown not expected block 2/2465005 type unknown not expected [-- Attachment #3: sda1.warn --] [-- Type: application/octet-stream, Size: 93 bytes --] xfs_metadump: invalid dqblk inode number (-1) xfs_metadump: invalid dqblk inode number (-1) [-- Attachment #4: sdb8.chk --] [-- Type: application/octet-stream, Size: 679 bytes --] inode 764 bad rt block number 683, offset 0 bad nblocks 12 for inode 764, counted 0 inode 268448674 bad rt block number 16779005, offset 0 bad nblocks 1 for inode 268448674, counted 0 block 0/683 type unknown not expected block 0/684 type unknown not expected block 0/685 type unknown not expected block 0/686 type unknown not expected block 0/687 type unknown not expected block 0/688 type unknown not expected block 0/689 type unknown not expected block 0/690 type unknown not expected block 0/691 type unknown not expected block 0/692 type unknown not expected block 0/693 type unknown not expected block 0/694 type unknown not expected block 1/1789 type unknown not expected [-- Attachment #5: sdb8.warn --] [-- Type: application/octet-stream, Size: 93 bytes --] xfs_metadump: invalid dqblk inode number (-1) xfs_metadump: invalid dqblk inode number (-1) ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-11 14:39 permanent XFS volume corruption Jan Beulich @ 2017-05-11 14:58 ` Eric Sandeen 2017-05-11 15:12 ` Jan Beulich 2017-05-11 16:39 ` Eric Sandeen 1 sibling, 1 reply; 18+ messages in thread From: Eric Sandeen @ 2017-05-11 14:58 UTC (permalink / raw) To: Jan Beulich, linux-xfs On 5/11/17 9:39 AM, Jan Beulich wrote: > Hello, > > It is now on two systems that I'm getting > > XFS (sda1): corrupt dinode 576254627, has realtime flag set. > ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64 IN.............d > ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ > ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f Y.....|/Y....../ > ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80 Y....../........ > XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c. Caller xfs_iread+0xea/0x2e0 [xfs] > CPU: 10 PID: 4418 Comm: smbd Not tainted 3.12.73-sp1-2017-04-26-jb #2 Well, pretty old... oh, ok but you think it came about after a 4.11 crash? > Hardware name: AMD Dinar/Dinar, BIOS RDN1506A 08/31/2014 > 0000000000000001 ffffffff81354083 ffffffffa03ea40a ffffffffa03a0952 > 0000000000000000 0000000000000075 ffff88042ea63300 ffff88042f508000 > ffff88022efe7000 ffff88042f508028 0000000000000000 ffffffffa03e9b06 > Call Trace: > [<ffffffff81004e3b>] dump_trace+0x7b/0x310 > [<ffffffff81004ad6>] show_stack_log_lvl+0xe6/0x150 > [<ffffffff81005ddc>] show_stack+0x1c/0x50 > [<ffffffff81354083>] dump_stack+0x6f/0x84 > [<ffffffffa03a0952>] xfs_corruption_error+0x62/0xa0 [xfs] > [<ffffffffa03e9b06>] xfs_iformat_fork+0x3b6/0x530 [xfs] > [<ffffffffa03ea40a>] xfs_iread+0xea/0x2e0 [xfs] > [<ffffffffa03a6538>] xfs_iget_cache_miss+0x58/0x1d0 [xfs] > [<ffffffffa03a67c3>] xfs_iget+0x113/0x190 [xfs] > [<ffffffffa03e5be8>] xfs_lookup+0xb8/0xd0 [xfs] > [<ffffffffa03aaddd>] xfs_vn_lookup+0x4d/0x90 [xfs] > [<ffffffff8110539d>] lookup_real+0x1d/0x60 > [<ffffffff811064d2>] __lookup_hash+0x32/0x50 > [<ffffffff8110a2a4>] path_lookupat+0x7f4/0x8b0 > [<ffffffff8110a38e>] filename_lookup+0x2e/0x90 > [<ffffffff8110abef>] user_path_at_empty+0x9f/0xd0 > [<ffffffff81100678>] vfs_fstatat+0x48/0xa0 > [<ffffffff8110081f>] SyS_newstat+0x1f/0x50 > [<ffffffff81358d42>] system_call_fastpath+0x16/0x1b > [<00007f141f4f0d35>] 0x7f141f4f0d34 > XFS (sda1): Corruption detected. Unmount and run xfs_repair > > after a crash with a 4.11-based kernel. Oh, hm. What caused that crash, do you have logs prior to it? > I didn't try xfs_repair-ing > the volume in this second instance, as the result from doing so in > the first instance was only permanent re-occurrence (and > apparently spreading) of the problem. It may be interesting that > xfs_check finds only this one corrupted inode, while the kernel > also finds at least one more: xfs_repair -n would be safe, what does it say? (mount/unmount first, to clear the log) > XFS (sda1): corrupt dinode 104812066, has realtime flag set. > ffff88042e88f200: 49 4e 41 ed 02 01 00 00 00 00 03 e8 00 00 00 64 INA............d > ffff88042e88f210: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 ................ > ffff88042e88f220: 59 01 c7 96 00 16 1e 50 59 14 0e a2 15 60 54 2f Y......PY....`T/ > ffff88042e88f230: 59 14 0e a2 15 60 54 2f 00 00 00 00 00 00 00 8e Y....`T/........ > XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c. Caller xfs_iread+0xea/0x2e0 [xfs] > > The kernel used after the crash doesn't appear to matter, all report > the same issues, but I've experienced similar crashes in the past > without ever having seen such corruption before. > > In any event I think there are two problems: The corruption itself > (possibly an issue with recent enough upstream kernels only) and > the fact that running xfs_repair doesn't help in these cases. I'm > attaching xfs_check and xfs_metadump warning output for both > affected volumes in this second instance. The full files > xfs_metadump -agow produced can be provided upon request > (500Mb and 80Mb uncompressed respectively). can you provide one or both compressed xfs_metadumps offline? (No need to post URL to the list) Thanks, -Eric > Thanks for any advice / fix, > Jan > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-11 14:58 ` Eric Sandeen @ 2017-05-11 15:12 ` Jan Beulich 2017-05-11 15:16 ` Eric Sandeen 0 siblings, 1 reply; 18+ messages in thread From: Jan Beulich @ 2017-05-11 15:12 UTC (permalink / raw) To: Eric Sandeen; +Cc: linux-xfs >>> On 11.05.17 at 16:58, <sandeen@sandeen.net> wrote: > On 5/11/17 9:39 AM, Jan Beulich wrote: >> It is now on two systems that I'm getting >> >> XFS (sda1): corrupt dinode 576254627, has realtime flag set. >> ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64 IN.............d >> ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f Y.....|/Y....../ >> ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80 Y....../........ >> XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file > .../fs/xfs/xfs_inode_fork.c. Caller xfs_iread+0xea/0x2e0 [xfs] >> CPU: 10 PID: 4418 Comm: smbd Not tainted 3.12.73-sp1-2017-04-26-jb #2 > > Well, pretty old... oh, ok but you think it came about after a > 4.11 crash? As said elsewhere, similar messages appear with 4.11 or other kernel versions I have installed on that box. >> Hardware name: AMD Dinar/Dinar, BIOS RDN1506A 08/31/2014 >> 0000000000000001 ffffffff81354083 ffffffffa03ea40a ffffffffa03a0952 >> 0000000000000000 0000000000000075 ffff88042ea63300 ffff88042f508000 >> ffff88022efe7000 ffff88042f508028 0000000000000000 ffffffffa03e9b06 >> Call Trace: >> [<ffffffff81004e3b>] dump_trace+0x7b/0x310 >> [<ffffffff81004ad6>] show_stack_log_lvl+0xe6/0x150 >> [<ffffffff81005ddc>] show_stack+0x1c/0x50 >> [<ffffffff81354083>] dump_stack+0x6f/0x84 >> [<ffffffffa03a0952>] xfs_corruption_error+0x62/0xa0 [xfs] >> [<ffffffffa03e9b06>] xfs_iformat_fork+0x3b6/0x530 [xfs] >> [<ffffffffa03ea40a>] xfs_iread+0xea/0x2e0 [xfs] >> [<ffffffffa03a6538>] xfs_iget_cache_miss+0x58/0x1d0 [xfs] >> [<ffffffffa03a67c3>] xfs_iget+0x113/0x190 [xfs] >> [<ffffffffa03e5be8>] xfs_lookup+0xb8/0xd0 [xfs] >> [<ffffffffa03aaddd>] xfs_vn_lookup+0x4d/0x90 [xfs] >> [<ffffffff8110539d>] lookup_real+0x1d/0x60 >> [<ffffffff811064d2>] __lookup_hash+0x32/0x50 >> [<ffffffff8110a2a4>] path_lookupat+0x7f4/0x8b0 >> [<ffffffff8110a38e>] filename_lookup+0x2e/0x90 >> [<ffffffff8110abef>] user_path_at_empty+0x9f/0xd0 >> [<ffffffff81100678>] vfs_fstatat+0x48/0xa0 >> [<ffffffff8110081f>] SyS_newstat+0x1f/0x50 >> [<ffffffff81358d42>] system_call_fastpath+0x16/0x1b >> [<00007f141f4f0d35>] 0x7f141f4f0d34 >> XFS (sda1): Corruption detected. Unmount and run xfs_repair >> >> after a crash with a 4.11-based kernel. > > Oh, hm. What caused that crash, do you have logs prior to it? Nothing at all in the logs; the crashes were hypervisor ones in both instances. >> I didn't try xfs_repair-ing >> the volume in this second instance, as the result from doing so in >> the first instance was only permanent re-occurrence (and >> apparently spreading) of the problem. It may be interesting that >> xfs_check finds only this one corrupted inode, while the kernel >> also finds at least one more: > > xfs_repair -n would be safe, what does it say? (mount/unmount > first, to clear the log) So are you saying "xfs_repair -n" is not the same as "xfs_check"? >> In any event I think there are two problems: The corruption itself >> (possibly an issue with recent enough upstream kernels only) and >> the fact that running xfs_repair doesn't help in these cases. I'm >> attaching xfs_check and xfs_metadump warning output for both >> affected volumes in this second instance. The full files >> xfs_metadump -agow produced can be provided upon request >> (500Mb and 80Mb uncompressed respectively). > > can you provide one or both compressed xfs_metadumps offline? > (No need to post URL to the list) Well, I have no idea where to upload it to, all the sites I know of only accept text kind of data, and I don't think that would be suitable here. (I'm sorry for my ignorance, but that's not something I ever had a need to do.) Jan ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-11 15:12 ` Jan Beulich @ 2017-05-11 15:16 ` Eric Sandeen 2017-05-11 15:40 ` Jan Beulich 0 siblings, 1 reply; 18+ messages in thread From: Eric Sandeen @ 2017-05-11 15:16 UTC (permalink / raw) To: Jan Beulich; +Cc: linux-xfs On 5/11/17 10:12 AM, Jan Beulich wrote: >>>> On 11.05.17 at 16:58, <sandeen@sandeen.net> wrote: >> On 5/11/17 9:39 AM, Jan Beulich wrote: >>> It is now on two systems that I'm getting >>> >>> XFS (sda1): corrupt dinode 576254627, has realtime flag set. >>> ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64 IN.............d >>> ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ >>> ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f Y.....|/Y....../ >>> ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80 Y....../........ >>> XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file >> .../fs/xfs/xfs_inode_fork.c. Caller xfs_iread+0xea/0x2e0 [xfs] >>> CPU: 10 PID: 4418 Comm: smbd Not tainted 3.12.73-sp1-2017-04-26-jb #2 >> >> Well, pretty old... oh, ok but you think it came about after a >> 4.11 crash? > > As said elsewhere, similar messages appear with 4.11 or other kernel > versions I have installed on that box. > >>> Hardware name: AMD Dinar/Dinar, BIOS RDN1506A 08/31/2014 >>> 0000000000000001 ffffffff81354083 ffffffffa03ea40a ffffffffa03a0952 >>> 0000000000000000 0000000000000075 ffff88042ea63300 ffff88042f508000 >>> ffff88022efe7000 ffff88042f508028 0000000000000000 ffffffffa03e9b06 >>> Call Trace: >>> [<ffffffff81004e3b>] dump_trace+0x7b/0x310 >>> [<ffffffff81004ad6>] show_stack_log_lvl+0xe6/0x150 >>> [<ffffffff81005ddc>] show_stack+0x1c/0x50 >>> [<ffffffff81354083>] dump_stack+0x6f/0x84 >>> [<ffffffffa03a0952>] xfs_corruption_error+0x62/0xa0 [xfs] >>> [<ffffffffa03e9b06>] xfs_iformat_fork+0x3b6/0x530 [xfs] >>> [<ffffffffa03ea40a>] xfs_iread+0xea/0x2e0 [xfs] >>> [<ffffffffa03a6538>] xfs_iget_cache_miss+0x58/0x1d0 [xfs] >>> [<ffffffffa03a67c3>] xfs_iget+0x113/0x190 [xfs] >>> [<ffffffffa03e5be8>] xfs_lookup+0xb8/0xd0 [xfs] >>> [<ffffffffa03aaddd>] xfs_vn_lookup+0x4d/0x90 [xfs] >>> [<ffffffff8110539d>] lookup_real+0x1d/0x60 >>> [<ffffffff811064d2>] __lookup_hash+0x32/0x50 >>> [<ffffffff8110a2a4>] path_lookupat+0x7f4/0x8b0 >>> [<ffffffff8110a38e>] filename_lookup+0x2e/0x90 >>> [<ffffffff8110abef>] user_path_at_empty+0x9f/0xd0 >>> [<ffffffff81100678>] vfs_fstatat+0x48/0xa0 >>> [<ffffffff8110081f>] SyS_newstat+0x1f/0x50 >>> [<ffffffff81358d42>] system_call_fastpath+0x16/0x1b >>> [<00007f141f4f0d35>] 0x7f141f4f0d34 >>> XFS (sda1): Corruption detected. Unmount and run xfs_repair >>> >>> after a crash with a 4.11-based kernel. >> >> Oh, hm. What caused that crash, do you have logs prior to it? > > Nothing at all in the logs; the crashes were hypervisor ones in > both instances. ok, so this guest was fine, other than getting taken out by the hypervisor? >>> I didn't try xfs_repair-ing >>> the volume in this second instance, as the result from doing so in >>> the first instance was only permanent re-occurrence (and >>> apparently spreading) of the problem. It may be interesting that >>> xfs_check finds only this one corrupted inode, while the kernel >>> also finds at least one more: >> >> xfs_repair -n would be safe, what does it say? (mount/unmount >> first, to clear the log) > > So are you saying "xfs_repair -n" is not the same as "xfs_check"? Different body of code, even though they perform similar functions. >>> In any event I think there are two problems: The corruption itself >>> (possibly an issue with recent enough upstream kernels only) and >>> the fact that running xfs_repair doesn't help in these cases. I'm >>> attaching xfs_check and xfs_metadump warning output for both >>> affected volumes in this second instance. The full files >>> xfs_metadump -agow produced can be provided upon request >>> (500Mb and 80Mb uncompressed respectively). >> >> can you provide one or both compressed xfs_metadumps offline? >> (No need to post URL to the list) > > Well, I have no idea where to upload it to, all the sites I know of > only accept text kind of data, and I don't think that would be > suitable here. (I'm sorry for my ignorance, but that's not something > I ever had a need to do.) How small does the 80mb one compress with xz? You may be able to just mail it my way. If not I'll find another option. -Eric ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-11 15:16 ` Eric Sandeen @ 2017-05-11 15:40 ` Jan Beulich 0 siblings, 0 replies; 18+ messages in thread From: Jan Beulich @ 2017-05-11 15:40 UTC (permalink / raw) To: Eric Sandeen; +Cc: linux-xfs [-- Attachment #1: Type: text/plain, Size: 2155 bytes --] >>> On 11.05.17 at 17:16, <sandeen@sandeen.net> wrote: > On 5/11/17 10:12 AM, Jan Beulich wrote: >>>>> On 11.05.17 at 16:58, <sandeen@sandeen.net> wrote: >>> Oh, hm. What caused that crash, do you have logs prior to it? >> >> Nothing at all in the logs; the crashes were hypervisor ones in >> both instances. > > ok, so this guest was fine, other than getting taken out by the > hypervisor? Well, it was the host (Xen Dom0) actually. But yes, the Dom0 part of the system was entirely fine. >>>> I didn't try xfs_repair-ing >>>> the volume in this second instance, as the result from doing so in >>>> the first instance was only permanent re-occurrence (and >>>> apparently spreading) of the problem. It may be interesting that >>>> xfs_check finds only this one corrupted inode, while the kernel >>>> also finds at least one more: >>> >>> xfs_repair -n would be safe, what does it say? (mount/unmount >>> first, to clear the log) >> >> So are you saying "xfs_repair -n" is not the same as "xfs_check"? > > Different body of code, even though they perform similar functions. Output for both volumes attached. >>>> In any event I think there are two problems: The corruption itself >>>> (possibly an issue with recent enough upstream kernels only) and >>>> the fact that running xfs_repair doesn't help in these cases. I'm >>>> attaching xfs_check and xfs_metadump warning output for both >>>> affected volumes in this second instance. The full files >>>> xfs_metadump -agow produced can be provided upon request >>>> (500Mb and 80Mb uncompressed respectively). >>> >>> can you provide one or both compressed xfs_metadumps offline? >>> (No need to post URL to the list) >> >> Well, I have no idea where to upload it to, all the sites I know of >> only accept text kind of data, and I don't think that would be >> suitable here. (I'm sorry for my ignorance, but that's not something >> I ever had a need to do.) > > How small does the 80mb one compress with xz? You may be able to just > mail it my way. If not I'll find another option. 4.6 Mb. Is that small enough for your mailbox? Jan [-- Attachment #2: sda1.dry --] [-- Type: application/octet-stream, Size: 2636 bytes --] Phase 1 - find and verify superblock... Phase 2 - using internal log - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 inode 104812066 has RT flag set but there is no RT device inode 104812066 has RT flag set but there is no RT device , would fix bad flags. - agno = 1 - agno = 2 Bad flags set in inode 576254627 Bad flags set in inode 576254636 Bad flags set in inode 576254627 , would fix bad flags. found inode 576254627 claiming to be a real-time file would have cleared inode 576254627 Bad flags set in inode 576254636 , would fix bad flags. - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 3 Bad flags set in inode 576254627 , would fix bad flags. found inode 576254627 claiming to be a real-time file would have cleared inode 576254627 Bad flags set in inode 576254636 , would fix bad flags. entry "main-hvm64.o" at block 0 offset 1400 in directory inode 598448917 references free inode 576254627 would clear inode number in entry at offset 1400... inode 104812066 has RT flag set but there is no RT device , would fix bad flags. No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... corrupt dinode 104812066, has realtime flag set. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to xfs@oss.sgi.com. cache_node_purge: refcount was 1, not zero (node=0x7f9040d20e40) couldn't map inode 104812066, err = 117 entry "main-hvm64.o" in directory inode 598448917 points to free inode 576254627, would junk entry bad hash table for directory inode 598448917 (no data entry): would rebuild - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 9320117, would move to lost+found disconnected inode 9320119, would move to lost+found disconnected inode 9320120, would move to lost+found disconnected inode 9320121, would move to lost+found disconnected inode 9320123, would move to lost+found disconnected inode 104812067, would move to lost+found disconnected inode 104812068, would move to lost+found Phase 7 - verify link counts... would have reset inode 104812066 nlinks from 2 to 1 No modify flag set, skipping filesystem flush and exiting. [-- Attachment #3: sdb8.dry --] [-- Type: application/octet-stream, Size: 2548 bytes --] Phase 1 - find and verify superblock... Phase 2 - using internal log - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 Bad flags set in inode 764 Bad flags set in inode 764 , would fix bad flags. found inode 764 claiming to be a real-time file would have cleared inode 764 inode 2068 has RT flag set but there is no RT device directory flags set on non-directory inode 2068 inode 2068 has RT flag set but there is no RT device directory flags set on non-directory inode 2068 , would fix bad flags. - agno = 1 inode 268448674 has RT flag set but there is no RT device directory flags set on non-directory inode 268448674 inode 268448674 has RT flag set but there is no RT device directory flags set on non-directory inode 268448674 , would fix bad flags. found inode 268448674 claiming to be a real-time file would have cleared inode 268448674 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 Bad flags set in inode 764 , would fix bad flags. found inode 764 claiming to be a real-time file would have cleared inode 764 entry "tdb" in shortform directory 779 references free inode 764 would have junked entry "tdb" in directory inode 779 inode 268448674 has RT flag set but there is no RT device directory flags set on non-directory inode 268448674 , would fix bad flags. found inode 268448674 claiming to be a real-time file would have cleared inode 268448674 entry "0" in shortform directory 268448676 references free inode 268448674 would have junked entry "0" in directory inode 268448676 inode 2068 has RT flag set but there is no RT device directory flags set on non-directory inode 2068 , would fix bad flags. No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "tdb" in shortform directory inode 779 points to free inode 764 would junk entry entry "0" in shortform directory inode 268448676 points to free inode 268448674 would junk entry - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-11 14:39 permanent XFS volume corruption Jan Beulich 2017-05-11 14:58 ` Eric Sandeen @ 2017-05-11 16:39 ` Eric Sandeen 2017-05-12 6:26 ` Jan Beulich 1 sibling, 1 reply; 18+ messages in thread From: Eric Sandeen @ 2017-05-11 16:39 UTC (permalink / raw) To: Jan Beulich, linux-xfs On 5/11/17 9:39 AM, Jan Beulich wrote: > Hello, > > It is now on two systems that I'm getting > > XFS (sda1): corrupt dinode 576254627, has realtime flag set. > ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64 IN.............d > ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ > ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f Y.....|/Y....../ > ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80 Y....../........ ... > XFS (sda1): Corruption detected. Unmount and run xfs_repair > > after a crash with a 4.11-based kernel. I didn't try xfs_repair-ing > the volume in this second instance, as the result from doing so in > the first instance was only permanent re-occurrence (and > apparently spreading) of the problem. It may be interesting that > xfs_check finds only this one corrupted inode, while the kernel > also finds at least one more: > > XFS (sda1): corrupt dinode 104812066, has realtime flag set. > ffff88042e88f200: 49 4e 41 ed 02 01 00 00 00 00 03 e8 00 00 00 64 INA............d > ffff88042e88f210: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 ................ > ffff88042e88f220: 59 01 c7 96 00 16 1e 50 59 14 0e a2 15 60 54 2f Y......PY....`T/ > ffff88042e88f230: 59 14 0e a2 15 60 54 2f 00 00 00 00 00 00 00 8e Y....`T/........ > XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c. Caller xfs_iread+0xea/0x2e0 [xfs] > > The kernel used after the crash doesn't appear to matter, all report > the same issues, but I've experienced similar crashes in the past > without ever having seen such corruption before. > > In any event I think there are two problems: The corruption itself > (possibly an issue with recent enough upstream kernels only) and > the fact that running xfs_repair doesn't help in these cases. FWIW, recent xfs_repair (v4.11.0) finds several bad inodes on the sdb8 metadump you sent, and apparently fixes* them without problems. inode 764 has RT flag set but there is no RT device directory flags set on non-directory inode 764 inode 2068 has RT flag set but there is no RT device directory flags set on non-directory inode 2068 inode 268448674 has RT flag set but there is no RT device directory flags set on non-directory inode 268448674 these are: 764: lib/xenstored/tdb 2068: log/messages 268448674: lib/sudo/jbeulich/0 and after repair, I can read all the inodes on the device w/o further errors (upstream linus kernel) As for the original corruption problem, I'm not sure. *Each of the 3 problematic inodes has an odd assortment of flags set (think chattr type flags) - some have immutable, some have noatime, some have nodump, etc. These are what cause xfs_repair t ocomplain. It seems unlikely that any of these were actually set on your filesystem, as these are the only ones with any flags set. After repair, they're showing: --S---dA----------- mnt/log/messages --S-i--A----------- mnt/lib/sudo/jbeulich/0 --S----A----------- mnt/lib/xenstored/tdb Did you happen to set chattr flags on these files intentionally? Two if the problematic inodes also have a generation number of 0, which also seems a bit odd. -Eric > I'm > attaching xfs_check and xfs_metadump warning output for both > affected volumes in this second instance. The full files > xfs_metadump -agow produced can be provided upon request > (500Mb and 80Mb uncompressed respectively). > > Thanks for any advice / fix, > Jan > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-11 16:39 ` Eric Sandeen @ 2017-05-12 6:26 ` Jan Beulich 2017-05-12 13:56 ` Eric Sandeen 0 siblings, 1 reply; 18+ messages in thread From: Jan Beulich @ 2017-05-12 6:26 UTC (permalink / raw) To: Eric Sandeen; +Cc: linux-xfs >>> On 11.05.17 at 18:39, <sandeen@sandeen.net> wrote: > On 5/11/17 9:39 AM, Jan Beulich wrote: >> In any event I think there are two problems: The corruption itself >> (possibly an issue with recent enough upstream kernels only) and >> the fact that running xfs_repair doesn't help in these cases. > > FWIW, recent xfs_repair (v4.11.0) finds several bad inodes on the > sdb8 metadump you sent, and apparently fixes* them without problems. > > inode 764 has RT flag set but there is no RT device > directory flags set on non-directory inode 764 > inode 2068 has RT flag set but there is no RT device > directory flags set on non-directory inode 2068 > inode 268448674 has RT flag set but there is no RT device > directory flags set on non-directory inode 268448674 > > these are: > > 764: lib/xenstored/tdb > 2068: log/messages > 268448674: lib/sudo/jbeulich/0 This matches then was xfs_repair -n has found here. I'm surprised log/messages is among them, as I didn't have to do anything to it to at least avoid the kernel warnings (for the other two I've simply renamed the containing directories, creating fresh ones instead). What I did get kernel warnings for were one or two files under log/xen/, which I've then similarly renamed and re-created. > and after repair, I can read all the inodes on the device w/o > further errors (upstream linus kernel) So on the earlier instance, where I did run actual repairs (and indeed multiple of them), the problem re-surfaces every time I mount the volume again. Iirc the inode numbers didn't change, but in some cases the associated files did (namely when these weren't ones created very soon after mount, i.e. when the order of things is less predictable - it was in particular /var/run/ which was affected there). That's the reason I've refrained from trying to xfs_repair the issues in this second instance. Now one question obviously is whether the repair strategy changed between the xfs_repair versions I use (3.1.8 on the system that I sent you the meta data dump from, likely the same on the other one): Mine clears the inodes with the bad RT flag, while - namely considering the subsequent "block ... type unknown" reported by xfs_check - it might instead be possible to reconstruct the files and clear the RT flag. But of course I know nothing about XFS internals... > *Each of the 3 problematic inodes has an odd assortment of flags > set (think chattr type flags) - some have immutable, some have > noatime, some have nodump, etc. These are what cause xfs_repair > t ocomplain. It seems unlikely that any of these were actually set > on your filesystem, as these are the only ones with any flags set. > After repair, they're showing: > > --S---dA----------- mnt/log/messages > --S-i--A----------- mnt/lib/sudo/jbeulich/0 > --S----A----------- mnt/lib/xenstored/tdb > > Did you happen to set chattr flags on these files intentionally? I certainly didn't. As I don't know how to produce this form of flags display, I can't (for now) compare with what a healthy system has there. Jan ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-12 6:26 ` Jan Beulich @ 2017-05-12 13:56 ` Eric Sandeen 2017-05-12 14:09 ` Jan Beulich 0 siblings, 1 reply; 18+ messages in thread From: Eric Sandeen @ 2017-05-12 13:56 UTC (permalink / raw) To: Jan Beulich; +Cc: linux-xfs On 5/12/17 1:26 AM, Jan Beulich wrote: >>>> On 11.05.17 at 18:39, <sandeen@sandeen.net> wrote: >> On 5/11/17 9:39 AM, Jan Beulich wrote: >>> In any event I think there are two problems: The corruption itself >>> (possibly an issue with recent enough upstream kernels only) and >>> the fact that running xfs_repair doesn't help in these cases. >> >> FWIW, recent xfs_repair (v4.11.0) finds several bad inodes on the >> sdb8 metadump you sent, and apparently fixes* them without problems. >> >> inode 764 has RT flag set but there is no RT device >> directory flags set on non-directory inode 764 >> inode 2068 has RT flag set but there is no RT device >> directory flags set on non-directory inode 2068 >> inode 268448674 has RT flag set but there is no RT device >> directory flags set on non-directory inode 268448674 >> >> these are: >> >> 764: lib/xenstored/tdb >> 2068: log/messages >> 268448674: lib/sudo/jbeulich/0 > > This matches then was xfs_repair -n has found here. > > I'm surprised log/messages is among them, as I didn't have to do > anything to it to at least avoid the kernel warnings (for the other > two I've simply renamed the containing directories, creating fresh > ones instead). What I did get kernel warnings for were one or > two files under log/xen/, which I've then similarly renamed and > re-created. > >> and after repair, I can read all the inodes on the device w/o >> further errors (upstream linus kernel) > > So on the earlier instance, where I did run actual repairs (and > indeed multiple of them), the problem re-surfaces every time > I mount the volume again. Ok, what is the exact sequence there, from repair to re-corruption? > Iirc the inode numbers didn't change, > but in some cases the associated files did (namely when these > weren't ones created very soon after mount, i.e. when the > order of things is less predictable - it was in particular /var/run/ > which was affected there). That's the reason I've refrained > from trying to xfs_repair the issues in this second instance. > > Now one question obviously is whether the repair strategy > changed between the xfs_repair versions I use (3.1.8 on the That's extremely old - Mar 2012, FWIW. > system that I sent you the meta data dump from, likely the > same on the other one): Mine clears the inodes with the bad > RT flag, while - namely considering the subsequent "block ... > type unknown" reported by xfs_check - it might instead be > possible to reconstruct the files and clear the RT flag. But of > course I know nothing about XFS internals... That is what xfs_repair did here - it cleared the bad flags and left the rest intact. >> *Each of the 3 problematic inodes has an odd assortment of flags >> set (think chattr type flags) - some have immutable, some have >> noatime, some have nodump, etc. These are what cause xfs_repair >> t ocomplain. It seems unlikely that any of these were actually set >> on your filesystem, as these are the only ones with any flags set. >> After repair, they're showing: >> >> --S---dA----------- mnt/log/messages >> --S-i--A----------- mnt/lib/sudo/jbeulich/0 >> --S----A----------- mnt/lib/xenstored/tdb >> >> Did you happen to set chattr flags on these files intentionally? > > I certainly didn't. As I don't know how to produce this form of > flags display, I can't (for now) compare with what a healthy > system has there. The above is from the lsattr display. (I suppose it's plausible that i.e. messages would have "append-only" set by something ...) -Eric > Jan > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-12 13:56 ` Eric Sandeen @ 2017-05-12 14:09 ` Jan Beulich 2017-05-12 15:04 ` Eric Sandeen 0 siblings, 1 reply; 18+ messages in thread From: Jan Beulich @ 2017-05-12 14:09 UTC (permalink / raw) To: Eric Sandeen; +Cc: linux-xfs >>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote: > On 5/12/17 1:26 AM, Jan Beulich wrote: >> So on the earlier instance, where I did run actual repairs (and >> indeed multiple of them), the problem re-surfaces every time >> I mount the volume again. > > Ok, what is the exact sequence there, from repair to re-corruption? Simply mount the volume after repairing (with or without an intermediate reboot) and access respective pieces of the fs again. As said, with /var/run affected on that first occasion, I couldn't even cleanly boot again without seeing the corruption re-surface. >> Iirc the inode numbers didn't change, >> but in some cases the associated files did (namely when these >> weren't ones created very soon after mount, i.e. when the >> order of things is less predictable - it was in particular /var/run/ >> which was affected there). That's the reason I've refrained >> from trying to xfs_repair the issues in this second instance. >> >> Now one question obviously is whether the repair strategy >> changed between the xfs_repair versions I use (3.1.8 on the > > That's extremely old - Mar 2012, FWIW. > >> system that I sent you the meta data dump from, likely the >> same on the other one): Mine clears the inodes with the bad >> RT flag, while - namely considering the subsequent "block ... >> type unknown" reported by xfs_check - it might instead be >> possible to reconstruct the files and clear the RT flag. But of >> course I know nothing about XFS internals... > > That is what xfs_repair did here - it cleared the bad flags and left > the rest intact. For all of the cases? According to my run, "Bad flags set in inode..." would lead to the flags being corrected, but "found inode ... claiming to be a real-time file" would lead to the inode being cleared. But of course I can imagine this being dealt with differently in newer versions of the tool... >>> *Each of the 3 problematic inodes has an odd assortment of flags >>> set (think chattr type flags) - some have immutable, some have >>> noatime, some have nodump, etc. These are what cause xfs_repair >>> t ocomplain. It seems unlikely that any of these were actually set >>> on your filesystem, as these are the only ones with any flags set. >>> After repair, they're showing: >>> >>> --S---dA----------- mnt/log/messages >>> --S-i--A----------- mnt/lib/sudo/jbeulich/0 >>> --S----A----------- mnt/lib/xenstored/tdb >>> >>> Did you happen to set chattr flags on these files intentionally? >> >> I certainly didn't. As I don't know how to produce this form of >> flags display, I can't (for now) compare with what a healthy >> system has there. > > The above is from the lsattr display. (I suppose it's plausible that > i.e. messages would have "append-only" set by something ...) On a healthy system all three of them have no flags set at all. Jan ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-12 14:09 ` Jan Beulich @ 2017-05-12 15:04 ` Eric Sandeen 2017-05-12 15:11 ` Eric Sandeen 2017-05-12 15:19 ` Jan Beulich 0 siblings, 2 replies; 18+ messages in thread From: Eric Sandeen @ 2017-05-12 15:04 UTC (permalink / raw) To: Jan Beulich; +Cc: linux-xfs On 5/12/17 9:09 AM, Jan Beulich wrote: >>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote: >> On 5/12/17 1:26 AM, Jan Beulich wrote: >>> So on the earlier instance, where I did run actual repairs (and >>> indeed multiple of them), the problem re-surfaces every time >>> I mount the volume again. >> Ok, what is the exact sequence there, from repair to re-corruption? > Simply mount the volume after repairing (with or without an > intermediate reboot) and access respective pieces of the fs > again. As said, with /var/run affected on that first occasion, > I couldn't even cleanly boot again without seeing the > corruption re-surface. Mount under what kernel, and access in what way? I'm looking for a recipe to reproduce what you've seen using the metadump you've provided. However: With further testing I see that xfs_repair v3.1.8 /does not/ entirely fix the fs; if I run 3.1.8 and then run upstream repair, it finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8 didn't touch. The verifiers in an upstream kernel may keep tripping over that until newer repair fixes it... Thanks, -Eric ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-12 15:04 ` Eric Sandeen @ 2017-05-12 15:11 ` Eric Sandeen 2017-05-15 9:22 ` Jan Beulich 2017-05-12 15:19 ` Jan Beulich 1 sibling, 1 reply; 18+ messages in thread From: Eric Sandeen @ 2017-05-12 15:11 UTC (permalink / raw) To: Jan Beulich; +Cc: linux-xfs On 5/12/17 10:04 AM, Eric Sandeen wrote: > On 5/12/17 9:09 AM, Jan Beulich wrote: >>>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote: >>> On 5/12/17 1:26 AM, Jan Beulich wrote: >>>> So on the earlier instance, where I did run actual repairs (and >>>> indeed multiple of them), the problem re-surfaces every time >>>> I mount the volume again. >>> Ok, what is the exact sequence there, from repair to re-corruption? >> Simply mount the volume after repairing (with or without an >> intermediate reboot) and access respective pieces of the fs >> again. As said, with /var/run affected on that first occasion, >> I couldn't even cleanly boot again without seeing the >> corruption re-surface. > > Mount under what kernel, and access in what way? I'm looking for a > recipe to reproduce what you've seen using the metadump you've provided. > > However: > > With further testing I see that xfs_repair v3.1.8 /does not/ > entirely fix the fs; if I run 3.1.8 and then run upstream repair, it > finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8 > didn't touch. The verifiers in an upstream kernel may keep tripping > over that until newer repair fixes it... (Indeed just running xfs_repair 3.1.8 finds the same corruption over and over) Please try a newer xfs_repair, and see if it resolves your problem. Thanks, -Eric > Thanks, > -Eric > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-12 15:11 ` Eric Sandeen @ 2017-05-15 9:22 ` Jan Beulich 2017-05-15 16:52 ` Eric Sandeen 0 siblings, 1 reply; 18+ messages in thread From: Jan Beulich @ 2017-05-15 9:22 UTC (permalink / raw) To: Eric Sandeen; +Cc: linux-xfs >>> On 12.05.17 at 17:11, <sandeen@sandeen.net> wrote: > > On 5/12/17 10:04 AM, Eric Sandeen wrote: >> On 5/12/17 9:09 AM, Jan Beulich wrote: >>>>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote: >>>> On 5/12/17 1:26 AM, Jan Beulich wrote: >>>>> So on the earlier instance, where I did run actual repairs (and >>>>> indeed multiple of them), the problem re-surfaces every time >>>>> I mount the volume again. >>>> Ok, what is the exact sequence there, from repair to re-corruption? >>> Simply mount the volume after repairing (with or without an >>> intermediate reboot) and access respective pieces of the fs >>> again. As said, with /var/run affected on that first occasion, >>> I couldn't even cleanly boot again without seeing the >>> corruption re-surface. >> >> Mount under what kernel, and access in what way? I'm looking for a >> recipe to reproduce what you've seen using the metadump you've provided. >> >> However: >> >> With further testing I see that xfs_repair v3.1.8 /does not/ >> entirely fix the fs; if I run 3.1.8 and then run upstream repair, it >> finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8 >> didn't touch. The verifiers in an upstream kernel may keep tripping >> over that until newer repair fixes it... > > (Indeed just running xfs_repair 3.1.8 finds the same corruption over and > over) > > Please try a newer xfs_repair, and see if it resolves your problem. It seems to have improved the situation (on the first system I had the issue on), but leaves me with at least "Operation not permitted" upon init scripts (or me manually) rm-ing (or mv-ing) /var/run/*.pid (or mv-ing even /var/run itself). I'm not sure how worried I need to be, but this surely doesn't look overly healthy yet. The kernel warnings are all gone, though. Jan ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-15 9:22 ` Jan Beulich @ 2017-05-15 16:52 ` Eric Sandeen 2017-05-16 10:06 ` Jan Beulich 0 siblings, 1 reply; 18+ messages in thread From: Eric Sandeen @ 2017-05-15 16:52 UTC (permalink / raw) To: Jan Beulich; +Cc: linux-xfs On 5/15/17 4:22 AM, Jan Beulich wrote: >>>> On 12.05.17 at 17:11, <sandeen@sandeen.net> wrote: > >> >> On 5/12/17 10:04 AM, Eric Sandeen wrote: >>> On 5/12/17 9:09 AM, Jan Beulich wrote: >>>>>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote: >>>>> On 5/12/17 1:26 AM, Jan Beulich wrote: >>>>>> So on the earlier instance, where I did run actual repairs (and >>>>>> indeed multiple of them), the problem re-surfaces every time >>>>>> I mount the volume again. >>>>> Ok, what is the exact sequence there, from repair to re-corruption? >>>> Simply mount the volume after repairing (with or without an >>>> intermediate reboot) and access respective pieces of the fs >>>> again. As said, with /var/run affected on that first occasion, >>>> I couldn't even cleanly boot again without seeing the >>>> corruption re-surface. >>> >>> Mount under what kernel, and access in what way? I'm looking for a >>> recipe to reproduce what you've seen using the metadump you've provided. >>> >>> However: >>> >>> With further testing I see that xfs_repair v3.1.8 /does not/ >>> entirely fix the fs; if I run 3.1.8 and then run upstream repair, it >>> finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8 >>> didn't touch. The verifiers in an upstream kernel may keep tripping >>> over that until newer repair fixes it... >> >> (Indeed just running xfs_repair 3.1.8 finds the same corruption over and >> over) >> >> Please try a newer xfs_repair, and see if it resolves your problem. > > It seems to have improved the situation (on the first system I had > the issue on), but leaves me with at least "Operation not permitted" > upon init scripts (or me manually) rm-ing (or mv-ing) /var/run/*.pid > (or mv-ing even /var/run itself). I'm not sure how worried I need to > be, but this surely doesn't look overly healthy yet. The kernel > warnings are all gone, though. xfs_repair simply makes the filesystem consistent, it doesn't perform any other magic. :) The corruption we saw was related to incorrect flags set on an inode - in some cases flags like immutable which can affect access to the file. I'm not sure we've made much progress on the root cause of whatever set those extra flags*, but all repair will do is make them sane from a filesystem consistency POV, not from an OS operation POV. Check the files in question with lsattr, and see if there are unexpected flags still set. -Eric * but backing up towards root cause, you said this all started when a 4.11 kernel crashed, and the log replayed? What kind of crash, what caused it, what were the messages? > Jan ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-15 16:52 ` Eric Sandeen @ 2017-05-16 10:06 ` Jan Beulich 2017-05-16 17:38 ` Eric Sandeen 0 siblings, 1 reply; 18+ messages in thread From: Jan Beulich @ 2017-05-16 10:06 UTC (permalink / raw) To: Eric Sandeen; +Cc: linux-xfs >>> On 15.05.17 at 18:52, <sandeen@sandeen.net> wrote: > On 5/15/17 4:22 AM, Jan Beulich wrote: >> It seems to have improved the situation (on the first system I had >> the issue on), but leaves me with at least "Operation not permitted" >> upon init scripts (or me manually) rm-ing (or mv-ing) /var/run/*.pid >> (or mv-ing even /var/run itself). I'm not sure how worried I need to >> be, but this surely doesn't look overly healthy yet. The kernel >> warnings are all gone, though. > > xfs_repair simply makes the filesystem consistent, it doesn't perform > any other magic. :) The corruption we saw was related to incorrect > flags set on an inode - in some cases flags like immutable which can > affect access to the file. > > I'm not sure we've made much progress on the root cause of whatever set > those extra flags*, Indeed, and that's the primary aspect that worries me, since with working on the hypervisor or kernel it is going to be unavoidable for a crash to happen now and then. While I realize chances are low to find out any useful information for the two past cases of corruption, do you have any advice on how to collect / preserve necessary information on a sooner or later to be expected next instance? Isn't the most likely explanation that the log replay upon next mount has gone wrong (or the data in the log itself was bogus)? > but all repair will do is make them sane from a > filesystem consistency POV, not from an OS operation POV. Understood. > Check the files in question with lsattr, and see if there are unexpected > flags still set. Indeed I had done this (and the resulting cleanup) before replying. And yes, some of the initial issues went away with clearing stray i or a flags. Yet I wasn't able to tie the /var/run problem to any flag settings. I hope to find time to debug where exactly these errors are being generated, ideally allowing me to understand how to correct their cause. Jan ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-16 10:06 ` Jan Beulich @ 2017-05-16 17:38 ` Eric Sandeen 2017-05-17 5:27 ` Jan Beulich 0 siblings, 1 reply; 18+ messages in thread From: Eric Sandeen @ 2017-05-16 17:38 UTC (permalink / raw) To: Jan Beulich; +Cc: linux-xfs On 5/16/17 5:06 AM, Jan Beulich wrote: >> I'm not sure we've made much progress on the root cause of whatever set >> those extra flags*, > Indeed, and that's the primary aspect that worries me, since with > working on the hypervisor or kernel it is going to be unavoidable for > a crash to happen now and then. While I realize chances are low to > find out any useful information for the two past cases of corruption, > do you have any advice on how to collect / preserve necessary > information on a sooner or later to be expected next instance? Isn't > the most likely explanation that the log replay upon next mount has > gone wrong (or the data in the log itself was bogus)? About all I can suggest is to get an xfs_metadump as soon as any new problem shows up, if it does. Your first report seems to indicate that a 4.11 kernel crashed, and the resulting dirty log was replayed by a 3.12-era distro kernel. Is that the correct sequence of events? -Eric ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-16 17:38 ` Eric Sandeen @ 2017-05-17 5:27 ` Jan Beulich 0 siblings, 0 replies; 18+ messages in thread From: Jan Beulich @ 2017-05-17 5:27 UTC (permalink / raw) To: sandeen; +Cc: linux-xfs >>> Eric Sandeen <sandeen@sandeen.net> 05/16/17 7:38 PM >>> >Your first report seems to indicate that a 4.11 kernel crashed, and the >resulting dirty log was replayed by a 3.12-era distro kernel. Is that >the correct sequence of events? That may have been the case in the second instance of the issue, but definitely not the first - there I'm sure I did re-run with a 4.11 kernel right away. But you asking this, yes, the replaying kernel possibly not mattering may suggest it's rather the log writing than the replaying which has an issue. Jan ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-12 15:04 ` Eric Sandeen 2017-05-12 15:11 ` Eric Sandeen @ 2017-05-12 15:19 ` Jan Beulich 2017-05-12 16:23 ` Hans-Peter Jansen 1 sibling, 1 reply; 18+ messages in thread From: Jan Beulich @ 2017-05-12 15:19 UTC (permalink / raw) To: Eric Sandeen; +Cc: linux-xfs >>> On 12.05.17 at 17:04, <sandeen@sandeen.net> wrote: > On 5/12/17 9:09 AM, Jan Beulich wrote: >>>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote: >>> On 5/12/17 1:26 AM, Jan Beulich wrote: >>>> So on the earlier instance, where I did run actual repairs (and >>>> indeed multiple of them), the problem re-surfaces every time >>>> I mount the volume again. >>> Ok, what is the exact sequence there, from repair to re-corruption? >> Simply mount the volume after repairing (with or without an >> intermediate reboot) and access respective pieces of the fs >> again. As said, with /var/run affected on that first occasion, >> I couldn't even cleanly boot again without seeing the >> corruption re-surface. > > Mount under what kernel, and access in what way? I'm looking for a > recipe to reproduce what you've seen using the metadump you've provided. So that's where the problem is: As said, on this occasion I didn't try to run xfs_repair at all. What I'm describing is the situation I've ended in on the earlier occasion, which I didn't send you any data on (if you think it's worth it despite the several rounds of repairs I've run there, I could certainly do so, as I didn't decide yet what to do with that system in order to get it back into working state). In any event, the same re-occurrence of the problem was observed with 3.0, 3.12, 4.4, and 4.11 based kernels. And the accesses were whatever the system does during boot (from the names I presume mostly creating /var/run/*.pid files). But simple directory listings suffice to trigger the kernel warnings (can't tell whether they also suffice to re-introduce the issues). I've even tried mounting the volume ro after repairing, but - not entirely unexpected - the system didn't really like /var being read-only, so I couldn't check whether the corruption in that case would not have been flagged again. > However: > > With further testing I see that xfs_repair v3.1.8 /does not/ > entirely fix the fs; if I run 3.1.8 and then run upstream repair, it > finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8 > didn't touch. The verifiers in an upstream kernel may keep tripping > over that until newer repair fixes it... Well, I can see if I can build those newer tools for myself (would largely depend on how easy/difficult they are to configure/make, and whether there's a testsuite that I can run them over before allowing them to touch live data); I don't expect newer tools to be available for the distro I'm running. Jan ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: permanent XFS volume corruption 2017-05-12 15:19 ` Jan Beulich @ 2017-05-12 16:23 ` Hans-Peter Jansen 0 siblings, 0 replies; 18+ messages in thread From: Hans-Peter Jansen @ 2017-05-12 16:23 UTC (permalink / raw) To: Jan Beulich; +Cc: Eric Sandeen, linux-xfs On Freitag, 12. Mai 2017 09:19:35 Jan Beulich wrote: > >>> On 12.05.17 at 17:04, <sandeen@sandeen.net> wrote: > > However: > > > > With further testing I see that xfs_repair v3.1.8 /does not/ > > entirely fix the fs; if I run 3.1.8 and then run upstream repair, it > > finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8 > > didn't touch. The verifiers in an upstream kernel may keep tripping > > over that until newer repair fixes it... > > Well, I can see if I can build those newer tools for myself (would > largely depend on how easy/difficult they are to configure/make, > and whether there's a testsuite that I can run them over before > allowing them to touch live data); I don't expect newer tools to > be available for the distro I'm running. Pretty sure, you're running some SuSE derivative. Building the xfstools usually is pretty simple with OBS, I do that all the time for unsupported distros: https://build.opensuse.org/project/show/home:frispete:tools If BS wouldn't be down at the moment, I could look, if xfstests run as well, but since you have a backup of course, the risk of damaging your system with newer xfstools is pretty low. It saved my ass a couple of times... Good luck, Pete who notoriously runs outphased SuSE versions (e.g. 9.3 on my PBX) ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2017-05-17 5:27 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-05-11 14:39 permanent XFS volume corruption Jan Beulich 2017-05-11 14:58 ` Eric Sandeen 2017-05-11 15:12 ` Jan Beulich 2017-05-11 15:16 ` Eric Sandeen 2017-05-11 15:40 ` Jan Beulich 2017-05-11 16:39 ` Eric Sandeen 2017-05-12 6:26 ` Jan Beulich 2017-05-12 13:56 ` Eric Sandeen 2017-05-12 14:09 ` Jan Beulich 2017-05-12 15:04 ` Eric Sandeen 2017-05-12 15:11 ` Eric Sandeen 2017-05-15 9:22 ` Jan Beulich 2017-05-15 16:52 ` Eric Sandeen 2017-05-16 10:06 ` Jan Beulich 2017-05-16 17:38 ` Eric Sandeen 2017-05-17 5:27 ` Jan Beulich 2017-05-12 15:19 ` Jan Beulich 2017-05-12 16:23 ` Hans-Peter Jansen
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.