permanent XFS volume corruption

permanent XFS volume corruption

[Jan Beulich]

[-- Attachment #1: Type: text/plain, Size: 3546 bytes --]

Hello,

It is now on two systems that I'm getting

XFS (sda1): corrupt dinode 576254627, has realtime flag set.
ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64  IN.............d
ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f  Y.....|/Y....../
ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80  Y....../........
XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c.  Caller xfs_iread+0xea/0x2e0 [xfs]
CPU: 10 PID: 4418 Comm: smbd Not tainted 3.12.73-sp1-2017-04-26-jb #2
Hardware name: AMD Dinar/Dinar, BIOS RDN1506A 08/31/2014
 0000000000000001 ffffffff81354083 ffffffffa03ea40a ffffffffa03a0952
 0000000000000000 0000000000000075 ffff88042ea63300 ffff88042f508000
 ffff88022efe7000 ffff88042f508028 0000000000000000 ffffffffa03e9b06
Call Trace:
 [<ffffffff81004e3b>] dump_trace+0x7b/0x310
 [<ffffffff81004ad6>] show_stack_log_lvl+0xe6/0x150
 [<ffffffff81005ddc>] show_stack+0x1c/0x50
 [<ffffffff81354083>] dump_stack+0x6f/0x84
 [<ffffffffa03a0952>] xfs_corruption_error+0x62/0xa0 [xfs]
 [<ffffffffa03e9b06>] xfs_iformat_fork+0x3b6/0x530 [xfs]
 [<ffffffffa03ea40a>] xfs_iread+0xea/0x2e0 [xfs]
 [<ffffffffa03a6538>] xfs_iget_cache_miss+0x58/0x1d0 [xfs]
 [<ffffffffa03a67c3>] xfs_iget+0x113/0x190 [xfs]
 [<ffffffffa03e5be8>] xfs_lookup+0xb8/0xd0 [xfs]
 [<ffffffffa03aaddd>] xfs_vn_lookup+0x4d/0x90 [xfs]
 [<ffffffff8110539d>] lookup_real+0x1d/0x60
 [<ffffffff811064d2>] __lookup_hash+0x32/0x50
 [<ffffffff8110a2a4>] path_lookupat+0x7f4/0x8b0
 [<ffffffff8110a38e>] filename_lookup+0x2e/0x90
 [<ffffffff8110abef>] user_path_at_empty+0x9f/0xd0
 [<ffffffff81100678>] vfs_fstatat+0x48/0xa0
 [<ffffffff8110081f>] SyS_newstat+0x1f/0x50
 [<ffffffff81358d42>] system_call_fastpath+0x16/0x1b
 [<00007f141f4f0d35>] 0x7f141f4f0d34
XFS (sda1): Corruption detected. Unmount and run xfs_repair

after a crash with a 4.11-based kernel. I didn't try xfs_repair-ing
the volume in this second instance, as the result from doing so in
the first instance was only permanent re-occurrence (and
apparently spreading) of the problem. It may be interesting that
xfs_check finds only this one corrupted inode, while the kernel
also finds at least one more:

XFS (sda1): corrupt dinode 104812066, has realtime flag set.
ffff88042e88f200: 49 4e 41 ed 02 01 00 00 00 00 03 e8 00 00 00 64  INA............d
ffff88042e88f210: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff88042e88f220: 59 01 c7 96 00 16 1e 50 59 14 0e a2 15 60 54 2f  Y......PY....`T/
ffff88042e88f230: 59 14 0e a2 15 60 54 2f 00 00 00 00 00 00 00 8e  Y....`T/........
XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c.  Caller xfs_iread+0xea/0x2e0 [xfs]

The kernel used after the crash doesn't appear to matter, all report
the same issues, but I've experienced similar crashes in the past
without ever having seen such corruption before.

In any event I think there are two problems: The corruption itself
(possibly an issue with recent enough upstream kernels only) and
the fact that running xfs_repair doesn't help in these cases. I'm
attaching xfs_check and xfs_metadump warning output for both
affected volumes in this second instance. The full files
xfs_metadump -agow produced can be provided upon request
(500Mb and 80Mb uncompressed respectively).

Thanks for any advice / fix,
Jan


[-- Attachment #2: sda1.chk --]
[-- Type: application/octet-stream, Size: 478 bytes --]

inode 576254627 bad rt block number 36019429, offset 0
bad nblocks 9 for inode 576254627, counted 0
block 2/2464997 type unknown not expected
block 2/2464998 type unknown not expected
block 2/2464999 type unknown not expected
block 2/2465000 type unknown not expected
block 2/2465001 type unknown not expected
block 2/2465002 type unknown not expected
block 2/2465003 type unknown not expected
block 2/2465004 type unknown not expected
block 2/2465005 type unknown not expected

[-- Attachment #3: sda1.warn --]
[-- Type: application/octet-stream, Size: 93 bytes --]


xfs_metadump: invalid dqblk inode number (-1)
xfs_metadump: invalid dqblk inode number (-1)

[-- Attachment #4: sdb8.chk --]
[-- Type: application/octet-stream, Size: 679 bytes --]

inode 764 bad rt block number 683, offset 0
bad nblocks 12 for inode 764, counted 0
inode 268448674 bad rt block number 16779005, offset 0
bad nblocks 1 for inode 268448674, counted 0
block 0/683 type unknown not expected
block 0/684 type unknown not expected
block 0/685 type unknown not expected
block 0/686 type unknown not expected
block 0/687 type unknown not expected
block 0/688 type unknown not expected
block 0/689 type unknown not expected
block 0/690 type unknown not expected
block 0/691 type unknown not expected
block 0/692 type unknown not expected
block 0/693 type unknown not expected
block 0/694 type unknown not expected
block 1/1789 type unknown not expected

[-- Attachment #5: sdb8.warn --]
[-- Type: application/octet-stream, Size: 93 bytes --]


xfs_metadump: invalid dqblk inode number (-1)
xfs_metadump: invalid dqblk inode number (-1)

Re: permanent XFS volume corruption

[Eric Sandeen]

On 5/11/17 9:39 AM, Jan Beulich wrote:
> Hello,
> 
> It is now on two systems that I'm getting
> 
> XFS (sda1): corrupt dinode 576254627, has realtime flag set.
> ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64  IN.............d
> ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
> ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f  Y.....|/Y....../
> ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80  Y....../........
> XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c.  Caller xfs_iread+0xea/0x2e0 [xfs]
> CPU: 10 PID: 4418 Comm: smbd Not tainted 3.12.73-sp1-2017-04-26-jb #2

Well, pretty old... oh, ok but you think it came about after a
4.11 crash?

> Hardware name: AMD Dinar/Dinar, BIOS RDN1506A 08/31/2014
>  0000000000000001 ffffffff81354083 ffffffffa03ea40a ffffffffa03a0952
>  0000000000000000 0000000000000075 ffff88042ea63300 ffff88042f508000
>  ffff88022efe7000 ffff88042f508028 0000000000000000 ffffffffa03e9b06
> Call Trace:
>  [<ffffffff81004e3b>] dump_trace+0x7b/0x310
>  [<ffffffff81004ad6>] show_stack_log_lvl+0xe6/0x150
>  [<ffffffff81005ddc>] show_stack+0x1c/0x50
>  [<ffffffff81354083>] dump_stack+0x6f/0x84
>  [<ffffffffa03a0952>] xfs_corruption_error+0x62/0xa0 [xfs]
>  [<ffffffffa03e9b06>] xfs_iformat_fork+0x3b6/0x530 [xfs]
>  [<ffffffffa03ea40a>] xfs_iread+0xea/0x2e0 [xfs]
>  [<ffffffffa03a6538>] xfs_iget_cache_miss+0x58/0x1d0 [xfs]
>  [<ffffffffa03a67c3>] xfs_iget+0x113/0x190 [xfs]
>  [<ffffffffa03e5be8>] xfs_lookup+0xb8/0xd0 [xfs]
>  [<ffffffffa03aaddd>] xfs_vn_lookup+0x4d/0x90 [xfs]
>  [<ffffffff8110539d>] lookup_real+0x1d/0x60
>  [<ffffffff811064d2>] __lookup_hash+0x32/0x50
>  [<ffffffff8110a2a4>] path_lookupat+0x7f4/0x8b0
>  [<ffffffff8110a38e>] filename_lookup+0x2e/0x90
>  [<ffffffff8110abef>] user_path_at_empty+0x9f/0xd0
>  [<ffffffff81100678>] vfs_fstatat+0x48/0xa0
>  [<ffffffff8110081f>] SyS_newstat+0x1f/0x50
>  [<ffffffff81358d42>] system_call_fastpath+0x16/0x1b
>  [<00007f141f4f0d35>] 0x7f141f4f0d34
> XFS (sda1): Corruption detected. Unmount and run xfs_repair
> 
> after a crash with a 4.11-based kernel.

Oh, hm.  What caused that crash, do you have logs prior to it?

> I didn't try xfs_repair-ing
> the volume in this second instance, as the result from doing so in
> the first instance was only permanent re-occurrence (and
> apparently spreading) of the problem. It may be interesting that
> xfs_check finds only this one corrupted inode, while the kernel
> also finds at least one more:

xfs_repair -n would be safe, what does it say?  (mount/unmount
first, to clear the log)

> XFS (sda1): corrupt dinode 104812066, has realtime flag set.
> ffff88042e88f200: 49 4e 41 ed 02 01 00 00 00 00 03 e8 00 00 00 64  INA............d
> ffff88042e88f210: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00  ................
> ffff88042e88f220: 59 01 c7 96 00 16 1e 50 59 14 0e a2 15 60 54 2f  Y......PY....`T/
> ffff88042e88f230: 59 14 0e a2 15 60 54 2f 00 00 00 00 00 00 00 8e  Y....`T/........
> XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c.  Caller xfs_iread+0xea/0x2e0 [xfs]
> 
> The kernel used after the crash doesn't appear to matter, all report
> the same issues, but I've experienced similar crashes in the past
> without ever having seen such corruption before.
> 
> In any event I think there are two problems: The corruption itself
> (possibly an issue with recent enough upstream kernels only) and
> the fact that running xfs_repair doesn't help in these cases. I'm
> attaching xfs_check and xfs_metadump warning output for both
> affected volumes in this second instance. The full files
> xfs_metadump -agow produced can be provided upon request
> (500Mb and 80Mb uncompressed respectively).

can you provide one or both compressed xfs_metadumps offline?
(No need to post URL to the list)

Thanks,
-Eric

> Thanks for any advice / fix,
> Jan
> 

Re: permanent XFS volume corruption

[Jan Beulich]

>>> On 11.05.17 at 16:58, <sandeen@sandeen.net> wrote:
> On 5/11/17 9:39 AM, Jan Beulich wrote:
>> It is now on two systems that I'm getting
>> 
>> XFS (sda1): corrupt dinode 576254627, has realtime flag set.
>> ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64  IN.............d
>> ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
>> ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f  Y.....|/Y....../
>> ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80  Y....../........
>> XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file 
> .../fs/xfs/xfs_inode_fork.c.  Caller xfs_iread+0xea/0x2e0 [xfs]
>> CPU: 10 PID: 4418 Comm: smbd Not tainted 3.12.73-sp1-2017-04-26-jb #2
> 
> Well, pretty old... oh, ok but you think it came about after a
> 4.11 crash?

As said elsewhere, similar messages appear with 4.11 or other kernel
versions I have installed on that box.

>> Hardware name: AMD Dinar/Dinar, BIOS RDN1506A 08/31/2014
>>  0000000000000001 ffffffff81354083 ffffffffa03ea40a ffffffffa03a0952
>>  0000000000000000 0000000000000075 ffff88042ea63300 ffff88042f508000
>>  ffff88022efe7000 ffff88042f508028 0000000000000000 ffffffffa03e9b06
>> Call Trace:
>>  [<ffffffff81004e3b>] dump_trace+0x7b/0x310
>>  [<ffffffff81004ad6>] show_stack_log_lvl+0xe6/0x150
>>  [<ffffffff81005ddc>] show_stack+0x1c/0x50
>>  [<ffffffff81354083>] dump_stack+0x6f/0x84
>>  [<ffffffffa03a0952>] xfs_corruption_error+0x62/0xa0 [xfs]
>>  [<ffffffffa03e9b06>] xfs_iformat_fork+0x3b6/0x530 [xfs]
>>  [<ffffffffa03ea40a>] xfs_iread+0xea/0x2e0 [xfs]
>>  [<ffffffffa03a6538>] xfs_iget_cache_miss+0x58/0x1d0 [xfs]
>>  [<ffffffffa03a67c3>] xfs_iget+0x113/0x190 [xfs]
>>  [<ffffffffa03e5be8>] xfs_lookup+0xb8/0xd0 [xfs]
>>  [<ffffffffa03aaddd>] xfs_vn_lookup+0x4d/0x90 [xfs]
>>  [<ffffffff8110539d>] lookup_real+0x1d/0x60
>>  [<ffffffff811064d2>] __lookup_hash+0x32/0x50
>>  [<ffffffff8110a2a4>] path_lookupat+0x7f4/0x8b0
>>  [<ffffffff8110a38e>] filename_lookup+0x2e/0x90
>>  [<ffffffff8110abef>] user_path_at_empty+0x9f/0xd0
>>  [<ffffffff81100678>] vfs_fstatat+0x48/0xa0
>>  [<ffffffff8110081f>] SyS_newstat+0x1f/0x50
>>  [<ffffffff81358d42>] system_call_fastpath+0x16/0x1b
>>  [<00007f141f4f0d35>] 0x7f141f4f0d34
>> XFS (sda1): Corruption detected. Unmount and run xfs_repair
>> 
>> after a crash with a 4.11-based kernel.
> 
> Oh, hm.  What caused that crash, do you have logs prior to it?

Nothing at all in the logs; the crashes were hypervisor ones in
both instances.

>> I didn't try xfs_repair-ing
>> the volume in this second instance, as the result from doing so in
>> the first instance was only permanent re-occurrence (and
>> apparently spreading) of the problem. It may be interesting that
>> xfs_check finds only this one corrupted inode, while the kernel
>> also finds at least one more:
> 
> xfs_repair -n would be safe, what does it say?  (mount/unmount
> first, to clear the log)

So are you saying "xfs_repair -n" is not the same as "xfs_check"?

>> In any event I think there are two problems: The corruption itself
>> (possibly an issue with recent enough upstream kernels only) and
>> the fact that running xfs_repair doesn't help in these cases. I'm
>> attaching xfs_check and xfs_metadump warning output for both
>> affected volumes in this second instance. The full files
>> xfs_metadump -agow produced can be provided upon request
>> (500Mb and 80Mb uncompressed respectively).
> 
> can you provide one or both compressed xfs_metadumps offline?
> (No need to post URL to the list)

Well, I have no idea where to upload it to, all the sites I know of
only accept text kind of data, and I don't think that would be
suitable here. (I'm sorry for my ignorance, but that's not something
I ever had a need to do.)

Jan

Re: permanent XFS volume corruption

[Eric Sandeen]

On 5/11/17 10:12 AM, Jan Beulich wrote:
>>>> On 11.05.17 at 16:58, <sandeen@sandeen.net> wrote:
>> On 5/11/17 9:39 AM, Jan Beulich wrote:
>>> It is now on two systems that I'm getting
>>>
>>> XFS (sda1): corrupt dinode 576254627, has realtime flag set.
>>> ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64  IN.............d
>>> ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
>>> ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f  Y.....|/Y....../
>>> ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80  Y....../........
>>> XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file 
>> .../fs/xfs/xfs_inode_fork.c.  Caller xfs_iread+0xea/0x2e0 [xfs]
>>> CPU: 10 PID: 4418 Comm: smbd Not tainted 3.12.73-sp1-2017-04-26-jb #2
>>
>> Well, pretty old... oh, ok but you think it came about after a
>> 4.11 crash?
> 
> As said elsewhere, similar messages appear with 4.11 or other kernel
> versions I have installed on that box.
> 
>>> Hardware name: AMD Dinar/Dinar, BIOS RDN1506A 08/31/2014
>>>  0000000000000001 ffffffff81354083 ffffffffa03ea40a ffffffffa03a0952
>>>  0000000000000000 0000000000000075 ffff88042ea63300 ffff88042f508000
>>>  ffff88022efe7000 ffff88042f508028 0000000000000000 ffffffffa03e9b06
>>> Call Trace:
>>>  [<ffffffff81004e3b>] dump_trace+0x7b/0x310
>>>  [<ffffffff81004ad6>] show_stack_log_lvl+0xe6/0x150
>>>  [<ffffffff81005ddc>] show_stack+0x1c/0x50
>>>  [<ffffffff81354083>] dump_stack+0x6f/0x84
>>>  [<ffffffffa03a0952>] xfs_corruption_error+0x62/0xa0 [xfs]
>>>  [<ffffffffa03e9b06>] xfs_iformat_fork+0x3b6/0x530 [xfs]
>>>  [<ffffffffa03ea40a>] xfs_iread+0xea/0x2e0 [xfs]
>>>  [<ffffffffa03a6538>] xfs_iget_cache_miss+0x58/0x1d0 [xfs]
>>>  [<ffffffffa03a67c3>] xfs_iget+0x113/0x190 [xfs]
>>>  [<ffffffffa03e5be8>] xfs_lookup+0xb8/0xd0 [xfs]
>>>  [<ffffffffa03aaddd>] xfs_vn_lookup+0x4d/0x90 [xfs]
>>>  [<ffffffff8110539d>] lookup_real+0x1d/0x60
>>>  [<ffffffff811064d2>] __lookup_hash+0x32/0x50
>>>  [<ffffffff8110a2a4>] path_lookupat+0x7f4/0x8b0
>>>  [<ffffffff8110a38e>] filename_lookup+0x2e/0x90
>>>  [<ffffffff8110abef>] user_path_at_empty+0x9f/0xd0
>>>  [<ffffffff81100678>] vfs_fstatat+0x48/0xa0
>>>  [<ffffffff8110081f>] SyS_newstat+0x1f/0x50
>>>  [<ffffffff81358d42>] system_call_fastpath+0x16/0x1b
>>>  [<00007f141f4f0d35>] 0x7f141f4f0d34
>>> XFS (sda1): Corruption detected. Unmount and run xfs_repair
>>>
>>> after a crash with a 4.11-based kernel.
>>
>> Oh, hm.  What caused that crash, do you have logs prior to it?
> 
> Nothing at all in the logs; the crashes were hypervisor ones in
> both instances.

ok, so this guest was fine, other than getting taken out by the
hypervisor?

>>> I didn't try xfs_repair-ing
>>> the volume in this second instance, as the result from doing so in
>>> the first instance was only permanent re-occurrence (and
>>> apparently spreading) of the problem. It may be interesting that
>>> xfs_check finds only this one corrupted inode, while the kernel
>>> also finds at least one more:
>>
>> xfs_repair -n would be safe, what does it say?  (mount/unmount
>> first, to clear the log)
> 
> So are you saying "xfs_repair -n" is not the same as "xfs_check"?

Different body of code, even though they perform similar functions.

>>> In any event I think there are two problems: The corruption itself
>>> (possibly an issue with recent enough upstream kernels only) and
>>> the fact that running xfs_repair doesn't help in these cases. I'm
>>> attaching xfs_check and xfs_metadump warning output for both
>>> affected volumes in this second instance. The full files
>>> xfs_metadump -agow produced can be provided upon request
>>> (500Mb and 80Mb uncompressed respectively).
>>
>> can you provide one or both compressed xfs_metadumps offline?
>> (No need to post URL to the list)
> 
> Well, I have no idea where to upload it to, all the sites I know of
> only accept text kind of data, and I don't think that would be
> suitable here. (I'm sorry for my ignorance, but that's not something
> I ever had a need to do.)

How small does the 80mb one compress with xz?  You may be able to just
mail it my way.  If not I'll find another option.

-Eric

Re: permanent XFS volume corruption

[Jan Beulich]

[-- Attachment #1: Type: text/plain, Size: 2155 bytes --]

>>> On 11.05.17 at 17:16, <sandeen@sandeen.net> wrote:
> On 5/11/17 10:12 AM, Jan Beulich wrote:
>>>>> On 11.05.17 at 16:58, <sandeen@sandeen.net> wrote:
>>> Oh, hm.  What caused that crash, do you have logs prior to it?
>> 
>> Nothing at all in the logs; the crashes were hypervisor ones in
>> both instances.
> 
> ok, so this guest was fine, other than getting taken out by the
> hypervisor?

Well, it was the host (Xen Dom0) actually. But yes, the Dom0 part
of the system was entirely fine.

>>>> I didn't try xfs_repair-ing
>>>> the volume in this second instance, as the result from doing so in
>>>> the first instance was only permanent re-occurrence (and
>>>> apparently spreading) of the problem. It may be interesting that
>>>> xfs_check finds only this one corrupted inode, while the kernel
>>>> also finds at least one more:
>>>
>>> xfs_repair -n would be safe, what does it say?  (mount/unmount
>>> first, to clear the log)
>> 
>> So are you saying "xfs_repair -n" is not the same as "xfs_check"?
> 
> Different body of code, even though they perform similar functions.

Output for both volumes attached.

>>>> In any event I think there are two problems: The corruption itself
>>>> (possibly an issue with recent enough upstream kernels only) and
>>>> the fact that running xfs_repair doesn't help in these cases. I'm
>>>> attaching xfs_check and xfs_metadump warning output for both
>>>> affected volumes in this second instance. The full files
>>>> xfs_metadump -agow produced can be provided upon request
>>>> (500Mb and 80Mb uncompressed respectively).
>>>
>>> can you provide one or both compressed xfs_metadumps offline?
>>> (No need to post URL to the list)
>> 
>> Well, I have no idea where to upload it to, all the sites I know of
>> only accept text kind of data, and I don't think that would be
>> suitable here. (I'm sorry for my ignorance, but that's not something
>> I ever had a need to do.)
> 
> How small does the 80mb one compress with xz?  You may be able to just
> mail it my way.  If not I'll find another option.

4.6 Mb. Is that small enough for your mailbox?

Jan


[-- Attachment #2: sda1.dry --]
[-- Type: application/octet-stream, Size: 2636 bytes --]

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
inode 104812066 has RT flag set but there is no RT device
inode 104812066 has RT flag set but there is no RT device
, would fix bad flags.
        - agno = 1
        - agno = 2
Bad flags set in inode 576254627
Bad flags set in inode 576254636
Bad flags set in inode 576254627
, would fix bad flags.
found inode 576254627 claiming to be a real-time file
would have cleared inode 576254627
Bad flags set in inode 576254636
, would fix bad flags.
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 2
        - agno = 1
        - agno = 3
Bad flags set in inode 576254627
, would fix bad flags.
found inode 576254627 claiming to be a real-time file
would have cleared inode 576254627
Bad flags set in inode 576254636
, would fix bad flags.
entry "main-hvm64.o" at block 0 offset 1400 in directory inode 598448917 references free inode 576254627
	would clear inode number in entry at offset 1400...
inode 104812066 has RT flag set but there is no RT device
, would fix bad flags.
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
corrupt dinode 104812066, has realtime flag set.  This is a bug.
Please capture the filesystem metadata with xfs_metadump and
report it to xfs@oss.sgi.com.
cache_node_purge: refcount was 1, not zero (node=0x7f9040d20e40)
couldn't map inode 104812066, err = 117
entry "main-hvm64.o" in directory inode 598448917 points to free inode 576254627, would junk entry
bad hash table for directory inode 598448917 (no data entry): would rebuild
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 9320117, would move to lost+found
disconnected inode 9320119, would move to lost+found
disconnected inode 9320120, would move to lost+found
disconnected inode 9320121, would move to lost+found
disconnected inode 9320123, would move to lost+found
disconnected inode 104812067, would move to lost+found
disconnected inode 104812068, would move to lost+found
Phase 7 - verify link counts...
would have reset inode 104812066 nlinks from 2 to 1
No modify flag set, skipping filesystem flush and exiting.

[-- Attachment #3: sdb8.dry --]
[-- Type: application/octet-stream, Size: 2548 bytes --]

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
Bad flags set in inode 764
Bad flags set in inode 764
, would fix bad flags.
found inode 764 claiming to be a real-time file
would have cleared inode 764
inode 2068 has RT flag set but there is no RT device
directory flags set on non-directory inode 2068
inode 2068 has RT flag set but there is no RT device
directory flags set on non-directory inode 2068
, would fix bad flags.
        - agno = 1
inode 268448674 has RT flag set but there is no RT device
directory flags set on non-directory inode 268448674
inode 268448674 has RT flag set but there is no RT device
directory flags set on non-directory inode 268448674
, would fix bad flags.
found inode 268448674 claiming to be a real-time file
would have cleared inode 268448674
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
Bad flags set in inode 764
, would fix bad flags.
found inode 764 claiming to be a real-time file
would have cleared inode 764
entry "tdb" in shortform directory 779 references free inode 764
would have junked entry "tdb" in directory inode 779
inode 268448674 has RT flag set but there is no RT device
directory flags set on non-directory inode 268448674
, would fix bad flags.
found inode 268448674 claiming to be a real-time file
would have cleared inode 268448674
entry "0" in shortform directory 268448676 references free inode 268448674
would have junked entry "0" in directory inode 268448676
inode 2068 has RT flag set but there is no RT device
directory flags set on non-directory inode 2068
, would fix bad flags.
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
entry "tdb" in shortform directory inode 779 points to free inode 764
would junk entry
entry "0" in shortform directory inode 268448676 points to free inode 268448674
would junk entry
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

Re: permanent XFS volume corruption

[Eric Sandeen]

On 5/11/17 9:39 AM, Jan Beulich wrote:
> Hello,
> 
> It is now on two systems that I'm getting
> 
> XFS (sda1): corrupt dinode 576254627, has realtime flag set.
> ffff88042ea63300: 49 4e 81 a4 02 02 00 00 00 00 03 e8 00 00 00 64  IN.............d
> ffff88042ea63310: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
> ffff88042ea63320: 59 14 0e 9f 0f a7 7c 2f 59 14 0e 9f 18 f3 db 2f  Y.....|/Y....../
> ffff88042ea63330: 59 14 0e 9f 18 f3 db 2f 00 00 00 00 00 00 80 80  Y....../........

...

> XFS (sda1): Corruption detected. Unmount and run xfs_repair
> 
> after a crash with a 4.11-based kernel. I didn't try xfs_repair-ing
> the volume in this second instance, as the result from doing so in
> the first instance was only permanent re-occurrence (and
> apparently spreading) of the problem. It may be interesting that
> xfs_check finds only this one corrupted inode, while the kernel
> also finds at least one more:
> 
> XFS (sda1): corrupt dinode 104812066, has realtime flag set.
> ffff88042e88f200: 49 4e 41 ed 02 01 00 00 00 00 03 e8 00 00 00 64  INA............d
> ffff88042e88f210: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00  ................
> ffff88042e88f220: 59 01 c7 96 00 16 1e 50 59 14 0e a2 15 60 54 2f  Y......PY....`T/
> ffff88042e88f230: 59 14 0e a2 15 60 54 2f 00 00 00 00 00 00 00 8e  Y....`T/........
> XFS (sda1): Internal error xfs_iformat(realtime) at line 133 of file .../fs/xfs/xfs_inode_fork.c.  Caller xfs_iread+0xea/0x2e0 [xfs]
> 
> The kernel used after the crash doesn't appear to matter, all report
> the same issues, but I've experienced similar crashes in the past
> without ever having seen such corruption before.
> 
> In any event I think there are two problems: The corruption itself
> (possibly an issue with recent enough upstream kernels only) and
> the fact that running xfs_repair doesn't help in these cases.

FWIW, recent xfs_repair (v4.11.0) finds several bad inodes on the
sdb8 metadump you sent, and apparently fixes* them without problems.

inode 764 has RT flag set but there is no RT device
directory flags set on non-directory inode 764
inode 2068 has RT flag set but there is no RT device
directory flags set on non-directory inode 2068
inode 268448674 has RT flag set but there is no RT device
directory flags set on non-directory inode 268448674

these are:

764: lib/xenstored/tdb
2068: log/messages
268448674: lib/sudo/jbeulich/0

and after repair, I can read all the inodes on the device w/o
further errors (upstream linus kernel)

As for the original corruption problem, I'm not sure.

*Each of the 3 problematic inodes has an odd assortment of flags
set (think chattr type flags) - some have immutable, some have
noatime, some have nodump, etc.  These are what cause xfs_repair
t ocomplain.  It seems unlikely that any of these were actually set
on your filesystem, as these are the only ones with any flags set.
After repair, they're showing:

--S---dA----------- mnt/log/messages
--S-i--A----------- mnt/lib/sudo/jbeulich/0
--S----A----------- mnt/lib/xenstored/tdb

Did you happen to set chattr flags on these files intentionally?

Two if the problematic inodes also have a generation number
of 0, which also seems a bit odd.

-Eric

> I'm
> attaching xfs_check and xfs_metadump warning output for both
> affected volumes in this second instance. The full files
> xfs_metadump -agow produced can be provided upon request
> (500Mb and 80Mb uncompressed respectively).
> 
> Thanks for any advice / fix,
> Jan
> 

Re: permanent XFS volume corruption

[Jan Beulich]

>>> On 11.05.17 at 18:39, <sandeen@sandeen.net> wrote:
> On 5/11/17 9:39 AM, Jan Beulich wrote:
>> In any event I think there are two problems: The corruption itself
>> (possibly an issue with recent enough upstream kernels only) and
>> the fact that running xfs_repair doesn't help in these cases.
> 
> FWIW, recent xfs_repair (v4.11.0) finds several bad inodes on the
> sdb8 metadump you sent, and apparently fixes* them without problems.
> 
> inode 764 has RT flag set but there is no RT device
> directory flags set on non-directory inode 764
> inode 2068 has RT flag set but there is no RT device
> directory flags set on non-directory inode 2068
> inode 268448674 has RT flag set but there is no RT device
> directory flags set on non-directory inode 268448674
> 
> these are:
> 
> 764: lib/xenstored/tdb
> 2068: log/messages
> 268448674: lib/sudo/jbeulich/0

This matches then was xfs_repair -n has found here.

I'm surprised log/messages is among them, as I didn't have to do
anything to it to at least avoid the kernel warnings (for the other
two I've simply renamed the containing directories, creating fresh
ones instead). What I did get kernel warnings for were one or
two files under log/xen/, which I've then similarly renamed and
re-created.

> and after repair, I can read all the inodes on the device w/o
> further errors (upstream linus kernel)

So on the earlier instance, where I did run actual repairs (and
indeed multiple of them), the problem re-surfaces every time
I mount the volume again. Iirc the inode numbers didn't change,
but in some cases the associated files did (namely when these
weren't ones created very soon after mount, i.e. when the
order of things is less predictable - it was in particular /var/run/
which was affected there). That's the reason I've refrained
from trying to xfs_repair the issues in this second instance.

Now one question obviously is whether the repair strategy
changed between the xfs_repair versions I use (3.1.8 on the
system that I sent you the meta data dump from, likely the
same on the other one): Mine clears the inodes with the bad
RT flag, while - namely considering the subsequent "block ...
type unknown" reported by xfs_check - it might instead be
possible to reconstruct the files and clear the RT flag. But of
course I know nothing about XFS internals...

> *Each of the 3 problematic inodes has an odd assortment of flags
> set (think chattr type flags) - some have immutable, some have
> noatime, some have nodump, etc.  These are what cause xfs_repair
> t ocomplain.  It seems unlikely that any of these were actually set
> on your filesystem, as these are the only ones with any flags set.
> After repair, they're showing:
> 
> --S---dA----------- mnt/log/messages
> --S-i--A----------- mnt/lib/sudo/jbeulich/0
> --S----A----------- mnt/lib/xenstored/tdb
> 
> Did you happen to set chattr flags on these files intentionally?

I certainly didn't. As I don't know how to produce this form of
flags display, I can't (for now) compare with what a healthy
system has there.

Jan

Re: permanent XFS volume corruption

[Eric Sandeen]

On 5/12/17 1:26 AM, Jan Beulich wrote:
>>>> On 11.05.17 at 18:39, <sandeen@sandeen.net> wrote:
>> On 5/11/17 9:39 AM, Jan Beulich wrote:
>>> In any event I think there are two problems: The corruption itself
>>> (possibly an issue with recent enough upstream kernels only) and
>>> the fact that running xfs_repair doesn't help in these cases.
>>
>> FWIW, recent xfs_repair (v4.11.0) finds several bad inodes on the
>> sdb8 metadump you sent, and apparently fixes* them without problems.
>>
>> inode 764 has RT flag set but there is no RT device
>> directory flags set on non-directory inode 764
>> inode 2068 has RT flag set but there is no RT device
>> directory flags set on non-directory inode 2068
>> inode 268448674 has RT flag set but there is no RT device
>> directory flags set on non-directory inode 268448674
>>
>> these are:
>>
>> 764: lib/xenstored/tdb
>> 2068: log/messages
>> 268448674: lib/sudo/jbeulich/0
> 
> This matches then was xfs_repair -n has found here.
> 
> I'm surprised log/messages is among them, as I didn't have to do
> anything to it to at least avoid the kernel warnings (for the other
> two I've simply renamed the containing directories, creating fresh
> ones instead). What I did get kernel warnings for were one or
> two files under log/xen/, which I've then similarly renamed and
> re-created.
> 
>> and after repair, I can read all the inodes on the device w/o
>> further errors (upstream linus kernel)
> 
> So on the earlier instance, where I did run actual repairs (and
> indeed multiple of them), the problem re-surfaces every time
> I mount the volume again.

Ok, what is the exact sequence there, from repair to re-corruption?

> Iirc the inode numbers didn't change,
> but in some cases the associated files did (namely when these
> weren't ones created very soon after mount, i.e. when the
> order of things is less predictable - it was in particular /var/run/
> which was affected there). That's the reason I've refrained
> from trying to xfs_repair the issues in this second instance.
> 
> Now one question obviously is whether the repair strategy
> changed between the xfs_repair versions I use (3.1.8 on the

That's extremely old - Mar 2012, FWIW.

> system that I sent you the meta data dump from, likely the
> same on the other one): Mine clears the inodes with the bad
> RT flag, while - namely considering the subsequent "block ...
> type unknown" reported by xfs_check - it might instead be
> possible to reconstruct the files and clear the RT flag. But of
> course I know nothing about XFS internals...

That is what xfs_repair did here - it cleared the bad flags and left
the rest intact.

>> *Each of the 3 problematic inodes has an odd assortment of flags
>> set (think chattr type flags) - some have immutable, some have
>> noatime, some have nodump, etc.  These are what cause xfs_repair
>> t ocomplain.  It seems unlikely that any of these were actually set
>> on your filesystem, as these are the only ones with any flags set.
>> After repair, they're showing:
>>
>> --S---dA----------- mnt/log/messages
>> --S-i--A----------- mnt/lib/sudo/jbeulich/0
>> --S----A----------- mnt/lib/xenstored/tdb
>>
>> Did you happen to set chattr flags on these files intentionally?
> 
> I certainly didn't. As I don't know how to produce this form of
> flags display, I can't (for now) compare with what a healthy
> system has there.

The above is from the lsattr display.  (I suppose it's plausible that
i.e. messages would have "append-only" set by something ...)

-Eric

> Jan
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: permanent XFS volume corruption

[Jan Beulich]

>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote:
> On 5/12/17 1:26 AM, Jan Beulich wrote:
>> So on the earlier instance, where I did run actual repairs (and
>> indeed multiple of them), the problem re-surfaces every time
>> I mount the volume again.
> 
> Ok, what is the exact sequence there, from repair to re-corruption?

Simply mount the volume after repairing (with or without an
intermediate reboot) and access respective pieces of the fs
again. As said, with /var/run affected on that first occasion,
I couldn't even cleanly boot again without seeing the
corruption re-surface.

>> Iirc the inode numbers didn't change,
>> but in some cases the associated files did (namely when these
>> weren't ones created very soon after mount, i.e. when the
>> order of things is less predictable - it was in particular /var/run/
>> which was affected there). That's the reason I've refrained
>> from trying to xfs_repair the issues in this second instance.
>> 
>> Now one question obviously is whether the repair strategy
>> changed between the xfs_repair versions I use (3.1.8 on the
> 
> That's extremely old - Mar 2012, FWIW.
> 
>> system that I sent you the meta data dump from, likely the
>> same on the other one): Mine clears the inodes with the bad
>> RT flag, while - namely considering the subsequent "block ...
>> type unknown" reported by xfs_check - it might instead be
>> possible to reconstruct the files and clear the RT flag. But of
>> course I know nothing about XFS internals...
> 
> That is what xfs_repair did here - it cleared the bad flags and left
> the rest intact.

For all of the cases? According to my run, "Bad flags set in inode..."
would lead to the flags being corrected, but "found inode ... claiming
to be a real-time file" would lead to the inode being cleared. But of
course I can imagine this being dealt with differently in newer
versions of the tool...

>>> *Each of the 3 problematic inodes has an odd assortment of flags
>>> set (think chattr type flags) - some have immutable, some have
>>> noatime, some have nodump, etc.  These are what cause xfs_repair
>>> t ocomplain.  It seems unlikely that any of these were actually set
>>> on your filesystem, as these are the only ones with any flags set.
>>> After repair, they're showing:
>>>
>>> --S---dA----------- mnt/log/messages
>>> --S-i--A----------- mnt/lib/sudo/jbeulich/0
>>> --S----A----------- mnt/lib/xenstored/tdb
>>>
>>> Did you happen to set chattr flags on these files intentionally?
>> 
>> I certainly didn't. As I don't know how to produce this form of
>> flags display, I can't (for now) compare with what a healthy
>> system has there.
> 
> The above is from the lsattr display.  (I suppose it's plausible that
> i.e. messages would have "append-only" set by something ...)

On a healthy system all three of them have no flags set at all.

Jan

Re: permanent XFS volume corruption

[Eric Sandeen]

On 5/12/17 9:09 AM, Jan Beulich wrote:
>>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote:
>> On 5/12/17 1:26 AM, Jan Beulich wrote:
>>> So on the earlier instance, where I did run actual repairs (and
>>> indeed multiple of them), the problem re-surfaces every time
>>> I mount the volume again.
>> Ok, what is the exact sequence there, from repair to re-corruption?
> Simply mount the volume after repairing (with or without an
> intermediate reboot) and access respective pieces of the fs
> again. As said, with /var/run affected on that first occasion,
> I couldn't even cleanly boot again without seeing the
> corruption re-surface.

Mount under what kernel, and access in what way?  I'm looking for a
recipe to reproduce what you've seen using the metadump you've provided.

However:

With further testing I see that xfs_repair v3.1.8 /does not/
entirely fix the fs; if I run 3.1.8 and then run upstream repair, it
finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8
didn't touch.  The verifiers in an upstream kernel may keep tripping
over that until newer repair fixes it...

Thanks,
-Eric

Re: permanent XFS volume corruption

[Eric Sandeen]

On 5/12/17 10:04 AM, Eric Sandeen wrote:
> On 5/12/17 9:09 AM, Jan Beulich wrote:
>>>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote:
>>> On 5/12/17 1:26 AM, Jan Beulich wrote:
>>>> So on the earlier instance, where I did run actual repairs (and
>>>> indeed multiple of them), the problem re-surfaces every time
>>>> I mount the volume again.
>>> Ok, what is the exact sequence there, from repair to re-corruption?
>> Simply mount the volume after repairing (with or without an
>> intermediate reboot) and access respective pieces of the fs
>> again. As said, with /var/run affected on that first occasion,
>> I couldn't even cleanly boot again without seeing the
>> corruption re-surface.
> 
> Mount under what kernel, and access in what way?  I'm looking for a
> recipe to reproduce what you've seen using the metadump you've provided.
> 
> However:
> 
> With further testing I see that xfs_repair v3.1.8 /does not/
> entirely fix the fs; if I run 3.1.8 and then run upstream repair, it
> finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8
> didn't touch.  The verifiers in an upstream kernel may keep tripping
> over that until newer repair fixes it...

(Indeed just running xfs_repair 3.1.8 finds the same corruption over and over)

Please try a newer xfs_repair, and see if it resolves your problem.

Thanks,
-Eric

> Thanks,
> -Eric
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: permanent XFS volume corruption

[Jan Beulich]

>>> On 12.05.17 at 17:04, <sandeen@sandeen.net> wrote:
> On 5/12/17 9:09 AM, Jan Beulich wrote:
>>>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote:
>>> On 5/12/17 1:26 AM, Jan Beulich wrote:
>>>> So on the earlier instance, where I did run actual repairs (and
>>>> indeed multiple of them), the problem re-surfaces every time
>>>> I mount the volume again.
>>> Ok, what is the exact sequence there, from repair to re-corruption?
>> Simply mount the volume after repairing (with or without an
>> intermediate reboot) and access respective pieces of the fs
>> again. As said, with /var/run affected on that first occasion,
>> I couldn't even cleanly boot again without seeing the
>> corruption re-surface.
> 
> Mount under what kernel, and access in what way?  I'm looking for a
> recipe to reproduce what you've seen using the metadump you've provided.

So that's where the problem is: As said, on this occasion I didn't
try to run xfs_repair at all. What I'm describing is the situation
I've ended in on the earlier occasion, which I didn't send you
any data on (if you think it's worth it despite the several rounds
of repairs I've run there, I could certainly do so, as I didn't
decide yet what to do with that system in order to get it back
into working state).

In any event, the same re-occurrence of the problem was
observed with 3.0, 3.12, 4.4, and 4.11 based kernels. And the
accesses were whatever the system does during boot (from
the names I presume mostly creating /var/run/*.pid files). But
simple directory listings suffice to trigger the kernel warnings
(can't tell whether they also suffice to re-introduce the issues).
I've even tried mounting the volume ro after repairing, but -
not entirely unexpected - the system didn't really like /var
being read-only, so I couldn't check whether the corruption in
that case would not have been flagged again.

> However:
> 
> With further testing I see that xfs_repair v3.1.8 /does not/
> entirely fix the fs; if I run 3.1.8 and then run upstream repair, it
> finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8
> didn't touch.  The verifiers in an upstream kernel may keep tripping
> over that until newer repair fixes it...

Well, I can see if I can build those newer tools for myself (would
largely depend on how easy/difficult they are to configure/make,
and whether there's a testsuite that I can run them over before
allowing them to touch live data); I don't expect newer tools to
be available for the distro I'm running.

Jan

Re: permanent XFS volume corruption

[Hans-Peter Jansen]

On Freitag, 12. Mai 2017 09:19:35 Jan Beulich wrote:
> >>> On 12.05.17 at 17:04, <sandeen@sandeen.net> wrote:
> > However:
> > 
> > With further testing I see that xfs_repair v3.1.8 /does not/
> > entirely fix the fs; if I run 3.1.8 and then run upstream repair, it
> > finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8
> > didn't touch.  The verifiers in an upstream kernel may keep tripping
> > over that until newer repair fixes it...
> 
> Well, I can see if I can build those newer tools for myself (would
> largely depend on how easy/difficult they are to configure/make,
> and whether there's a testsuite that I can run them over before
> allowing them to touch live data); I don't expect newer tools to
> be available for the distro I'm running.

Pretty sure, you're running some SuSE derivative. Building the xfstools 
usually is pretty simple with OBS, I do that all the time for unsupported 
distros:

	https://build.opensuse.org/project/show/home:frispete:tools

If BS wouldn't be down at the moment, I could look, if xfstests run as well, 
but since you have a backup of course, the risk of damaging your system with  
newer xfstools is pretty low. It saved my ass a couple of times...

Good luck,
Pete

who notoriously runs outphased SuSE versions (e.g. 9.3 on my PBX)

Re: permanent XFS volume corruption

[Jan Beulich]

>>> On 12.05.17 at 17:11, <sandeen@sandeen.net> wrote:

> 
> On 5/12/17 10:04 AM, Eric Sandeen wrote:
>> On 5/12/17 9:09 AM, Jan Beulich wrote:
>>>>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote:
>>>> On 5/12/17 1:26 AM, Jan Beulich wrote:
>>>>> So on the earlier instance, where I did run actual repairs (and
>>>>> indeed multiple of them), the problem re-surfaces every time
>>>>> I mount the volume again.
>>>> Ok, what is the exact sequence there, from repair to re-corruption?
>>> Simply mount the volume after repairing (with or without an
>>> intermediate reboot) and access respective pieces of the fs
>>> again. As said, with /var/run affected on that first occasion,
>>> I couldn't even cleanly boot again without seeing the
>>> corruption re-surface.
>> 
>> Mount under what kernel, and access in what way?  I'm looking for a
>> recipe to reproduce what you've seen using the metadump you've provided.
>> 
>> However:
>> 
>> With further testing I see that xfs_repair v3.1.8 /does not/
>> entirely fix the fs; if I run 3.1.8 and then run upstream repair, it
>> finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8
>> didn't touch.  The verifiers in an upstream kernel may keep tripping
>> over that until newer repair fixes it...
> 
> (Indeed just running xfs_repair 3.1.8 finds the same corruption over and 
> over)
> 
> Please try a newer xfs_repair, and see if it resolves your problem.

It seems to have improved the situation (on the first system I had
the issue on), but leaves me with at least "Operation not permitted"
upon init scripts (or me manually) rm-ing (or mv-ing) /var/run/*.pid
(or mv-ing even /var/run itself). I'm not sure how worried I need to
be, but this surely doesn't look overly healthy yet. The kernel
warnings are all gone, though.

Jan

Re: permanent XFS volume corruption

[Eric Sandeen]

On 5/15/17 4:22 AM, Jan Beulich wrote:
>>>> On 12.05.17 at 17:11, <sandeen@sandeen.net> wrote:
> 
>>
>> On 5/12/17 10:04 AM, Eric Sandeen wrote:
>>> On 5/12/17 9:09 AM, Jan Beulich wrote:
>>>>>>> On 12.05.17 at 15:56, <sandeen@sandeen.net> wrote:
>>>>> On 5/12/17 1:26 AM, Jan Beulich wrote:
>>>>>> So on the earlier instance, where I did run actual repairs (and
>>>>>> indeed multiple of them), the problem re-surfaces every time
>>>>>> I mount the volume again.
>>>>> Ok, what is the exact sequence there, from repair to re-corruption?
>>>> Simply mount the volume after repairing (with or without an
>>>> intermediate reboot) and access respective pieces of the fs
>>>> again. As said, with /var/run affected on that first occasion,
>>>> I couldn't even cleanly boot again without seeing the
>>>> corruption re-surface.
>>>
>>> Mount under what kernel, and access in what way?  I'm looking for a
>>> recipe to reproduce what you've seen using the metadump you've provided.
>>>
>>> However:
>>>
>>> With further testing I see that xfs_repair v3.1.8 /does not/
>>> entirely fix the fs; if I run 3.1.8 and then run upstream repair, it
>>> finds and fixes more bad flags on inode 764 (lib/xenstored/tdb) that 3.1.8
>>> didn't touch.  The verifiers in an upstream kernel may keep tripping
>>> over that until newer repair fixes it...
>>
>> (Indeed just running xfs_repair 3.1.8 finds the same corruption over and 
>> over)
>>
>> Please try a newer xfs_repair, and see if it resolves your problem.
> 
> It seems to have improved the situation (on the first system I had
> the issue on), but leaves me with at least "Operation not permitted"
> upon init scripts (or me manually) rm-ing (or mv-ing) /var/run/*.pid
> (or mv-ing even /var/run itself). I'm not sure how worried I need to
> be, but this surely doesn't look overly healthy yet. The kernel
> warnings are all gone, though.

xfs_repair simply makes the filesystem consistent, it doesn't perform
any other magic.  :)  The corruption we saw was related to incorrect
flags set on an inode - in some cases flags like immutable which can
affect access to the file.

I'm not sure we've made much progress on the root cause of whatever set
those extra flags*, but all repair will do is make them sane from a
filesystem consistency POV, not from an OS operation POV.

Check the files in question with lsattr, and see if there are unexpected
flags still set.

-Eric

* but backing up towards root cause, you said this all started when a 4.11
kernel crashed, and the log replayed?  What kind of crash, what caused it,
what were the messages?

> Jan

Re: permanent XFS volume corruption

[Jan Beulich]

>>> On 15.05.17 at 18:52, <sandeen@sandeen.net> wrote:
> On 5/15/17 4:22 AM, Jan Beulich wrote:
>> It seems to have improved the situation (on the first system I had
>> the issue on), but leaves me with at least "Operation not permitted"
>> upon init scripts (or me manually) rm-ing (or mv-ing) /var/run/*.pid
>> (or mv-ing even /var/run itself). I'm not sure how worried I need to
>> be, but this surely doesn't look overly healthy yet. The kernel
>> warnings are all gone, though.
> 
> xfs_repair simply makes the filesystem consistent, it doesn't perform
> any other magic.  :)  The corruption we saw was related to incorrect
> flags set on an inode - in some cases flags like immutable which can
> affect access to the file.
> 
> I'm not sure we've made much progress on the root cause of whatever set
> those extra flags*,

Indeed, and that's the primary aspect that worries me, since with
working on the hypervisor or kernel it is going to be unavoidable for
a crash to happen now and then. While I realize chances are low to
find out any useful information for the two past cases of corruption,
do you have any advice on how to collect / preserve necessary
information on a sooner or later to be expected next instance? Isn't
the most likely explanation that the log replay upon next mount has
gone wrong (or the data in the log itself was bogus)?

> but all repair will do is make them sane from a
> filesystem consistency POV, not from an OS operation POV.

Understood.

> Check the files in question with lsattr, and see if there are unexpected
> flags still set.

Indeed I had done this (and the resulting cleanup) before replying.
And yes, some of the initial issues went away with clearing stray i or
a flags. Yet I wasn't able to tie the /var/run problem to any flag
settings. I hope to find time to debug where exactly these errors
are being generated, ideally allowing me to understand how to
correct their cause.

Jan

Re: permanent XFS volume corruption

[Eric Sandeen]

On 5/16/17 5:06 AM, Jan Beulich wrote:
>> I'm not sure we've made much progress on the root cause of whatever set
>> those extra flags*,

> Indeed, and that's the primary aspect that worries me, since with
> working on the hypervisor or kernel it is going to be unavoidable for
> a crash to happen now and then. While I realize chances are low to
> find out any useful information for the two past cases of corruption,
> do you have any advice on how to collect / preserve necessary
> information on a sooner or later to be expected next instance? Isn't
> the most likely explanation that the log replay upon next mount has
> gone wrong (or the data in the log itself was bogus)?

About all I can suggest is to get an xfs_metadump as soon as any new
problem shows up, if it does.


Your first report seems to indicate that a 4.11 kernel crashed, and the
resulting dirty log was replayed by a 3.12-era distro kernel.  Is that
the correct sequence of events?

-Eric

Re: permanent XFS volume corruption

[Jan Beulich]

>>> Eric Sandeen <sandeen@sandeen.net> 05/16/17 7:38 PM >>>
>Your first report seems to indicate that a 4.11 kernel crashed, and the
>resulting dirty log was replayed by a 3.12-era distro kernel.  Is that
>the correct sequence of events?

That may have been the case in the second instance of the issue, but
definitely not the first - there I'm sure I did re-run with a 4.11 kernel
right away. But you asking this, yes, the replaying kernel possibly not
mattering may suggest it's rather the log writing than the replaying
which has an issue.

Jan