[PATCH] xfs_repair: junk leaf attribute if count == 0

[PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

We have recently seen a case where, during log replay, the
attr3 leaf verifier reported corruption when encountering a
leaf attribute with a count of 0 in the header.

We chalked this up to a transient state when a shortform leaf
was created, the attribute didn't fit, and we promoted the
(empty) attribute to the larger leaf form.

I've recently been given a metadump of unknown provenance which actually
contains a leaf attribute with count 0 on disk.  This causes the
verifier to fire every time xfs_repair is run:

 Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000

If this 0-count state is detected, we should just junk the leaf, same
as we would do if the count was too high.  With this change, we now
remedy the problem:

 Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
 bad attribute count 0 in attr block 0, inode 12587828
 problem with attribute contents in inode 12587828
 clearing inode 12587828 attributes
 correcting nblocks for inode 12587828, was 2 - counted 1

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
---

diff --git a/repair/attr_repair.c b/repair/attr_repair.c
index 40cb5f7..b855a10 100644
--- a/repair/attr_repair.c
+++ b/repair/attr_repair.c
@@ -593,7 +593,8 @@ process_leaf_attr_block(
 	stop = xfs_attr3_leaf_hdr_size(leaf);
 
 	/* does the count look sorta valid? */
-	if (leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
+	if (!leafhdr.count ||
+	    leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
 						mp->m_sb.sb_blocksize) {
 		do_warn(
 	_("bad attribute count %d in attr block %u, inode %" PRIu64 "\n"),

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Brian Foster]

On Thu, Dec 08, 2016 at 12:06:03PM -0600, Eric Sandeen wrote:
> We have recently seen a case where, during log replay, the
> attr3 leaf verifier reported corruption when encountering a
> leaf attribute with a count of 0 in the header.
> 
> We chalked this up to a transient state when a shortform leaf
> was created, the attribute didn't fit, and we promoted the
> (empty) attribute to the larger leaf form.
> 
> I've recently been given a metadump of unknown provenance which actually
> contains a leaf attribute with count 0 on disk.  This causes the
> verifier to fire every time xfs_repair is run:
> 
>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
> 
> If this 0-count state is detected, we should just junk the leaf, same
> as we would do if the count was too high.  With this change, we now
> remedy the problem:
> 
>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
>  bad attribute count 0 in attr block 0, inode 12587828
>  problem with attribute contents in inode 12587828
>  clearing inode 12587828 attributes
>  correcting nblocks for inode 12587828, was 2 - counted 1
> 
> Signed-off-by: Eric Sandeen <sandeen@redhat.com>
> ---

Reviewed-by: Brian Foster <bfoster@redhat.com>

> 
> diff --git a/repair/attr_repair.c b/repair/attr_repair.c
> index 40cb5f7..b855a10 100644
> --- a/repair/attr_repair.c
> +++ b/repair/attr_repair.c
> @@ -593,7 +593,8 @@ process_leaf_attr_block(
>  	stop = xfs_attr3_leaf_hdr_size(leaf);
>  
>  	/* does the count look sorta valid? */
> -	if (leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
> +	if (!leafhdr.count ||
> +	    leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
>  						mp->m_sb.sb_blocksize) {
>  		do_warn(
>  	_("bad attribute count %d in attr block %u, inode %" PRIu64 "\n"),
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,
should this patch possibly fix errors i reported in this thread?
https://www.spinics.net/lists/linux-xfs/msg01728.html

Is it safe to test it? (i do have backups)

Thanks,
Libor

On čtvrtek 8. prosince 2016 12:06:03 CET Eric Sandeen wrote:
> We have recently seen a case where, during log replay, the
> attr3 leaf verifier reported corruption when encountering a
> leaf attribute with a count of 0 in the header.
> 
> We chalked this up to a transient state when a shortform leaf
> was created, the attribute didn't fit, and we promoted the
> (empty) attribute to the larger leaf form.
> 
> I've recently been given a metadump of unknown provenance which actually
> contains a leaf attribute with count 0 on disk.  This causes the
> verifier to fire every time xfs_repair is run:
> 
>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
> 
> If this 0-count state is detected, we should just junk the leaf, same
> as we would do if the count was too high.  With this change, we now
> remedy the problem:
> 
>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
>  bad attribute count 0 in attr block 0, inode 12587828
>  problem with attribute contents in inode 12587828
>  clearing inode 12587828 attributes
>  correcting nblocks for inode 12587828, was 2 - counted 1
> 
> Signed-off-by: Eric Sandeen <sandeen@redhat.com>
> ---
> 
> diff --git a/repair/attr_repair.c b/repair/attr_repair.c
> index 40cb5f7..b855a10 100644
> --- a/repair/attr_repair.c
> +++ b/repair/attr_repair.c
> @@ -593,7 +593,8 @@ process_leaf_attr_block(
>  	stop = xfs_attr3_leaf_hdr_size(leaf);
>  
>  	/* does the count look sorta valid? */
> -	if (leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
> +	if (!leafhdr.count ||
> +	    leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
>  						mp->m_sb.sb_blocksize) {
>  		do_warn(
>  	_("bad attribute count %d in attr block %u, inode %" PRIu64 "\n"),
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

On 12/13/16 4:52 AM, Libor Klepáč wrote:
> Hello,
> should this patch possibly fix errors i reported in this thread?
> https://www.spinics.net/lists/linux-xfs/msg01728.html
> 
> Is it safe to test it? (i do have backups)

It should be safe, yes.

(you could always run xfs_repair -n first to be extra careful).

Were those errors during mount/replay, though, or was it when the
filesystem was up and running?

I ask because I also sent a patch to ignore these empty attributes
in the verifier during log replay.

FWIW, Only some of your reported buffers look "empty" though, the one
at 514098.682726 may have had something else wrong.

Anyway, yes, probably worth checking with xfs_repair (-n) with this
patch added.  Let us know what you find! :)

-Eric

 
> Thanks,
> Libor
> 
> On čtvrtek 8. prosince 2016 12:06:03 CET Eric Sandeen wrote:
>> We have recently seen a case where, during log replay, the
>> attr3 leaf verifier reported corruption when encountering a
>> leaf attribute with a count of 0 in the header.
>>
>> We chalked this up to a transient state when a shortform leaf
>> was created, the attribute didn't fit, and we promoted the
>> (empty) attribute to the larger leaf form.
>>
>> I've recently been given a metadump of unknown provenance which actually
>> contains a leaf attribute with count 0 on disk.  This causes the
>> verifier to fire every time xfs_repair is run:
>>
>>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
>>
>> If this 0-count state is detected, we should just junk the leaf, same
>> as we would do if the count was too high.  With this change, we now
>> remedy the problem:
>>
>>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
>>  bad attribute count 0 in attr block 0, inode 12587828
>>  problem with attribute contents in inode 12587828
>>  clearing inode 12587828 attributes
>>  correcting nblocks for inode 12587828, was 2 - counted 1
>>
>> Signed-off-by: Eric Sandeen <sandeen@redhat.com>
>> ---
>>
>> diff --git a/repair/attr_repair.c b/repair/attr_repair.c
>> index 40cb5f7..b855a10 100644
>> --- a/repair/attr_repair.c
>> +++ b/repair/attr_repair.c
>> @@ -593,7 +593,8 @@ process_leaf_attr_block(
>>  	stop = xfs_attr3_leaf_hdr_size(leaf);
>>  
>>  	/* does the count look sorta valid? */
>> -	if (leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
>> +	if (!leafhdr.count ||
>> +	    leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
>>  						mp->m_sb.sb_blocksize) {
>>  		do_warn(
>>  	_("bad attribute count %d in attr block %u, inode %" PRIu64 "\n"),
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,

On úterý 13. prosince 2016 10:04:27 CET Eric Sandeen wrote:
> On 12/13/16 4:52 AM, Libor Klep�? wrote:
> > Hello,
> > should this patch possibly fix errors i reported in this thread?
> > https://www.spinics.net/lists/linux-xfs/msg01728.html
> > 
> > Is it safe to test it? (i do have backups)
> 
> It should be safe, yes.
>
ok,  will try it soon(ish) with machines named vps1 and vps3 in my mails. vps2 
is problematic to shutdown and test.
 
> (you could always run xfs_repair -n first to be extra careful).
> 
> Were those errors during mount/replay, though, or was it when the
> filesystem was up and running?
> 
It was always when running, some errors were just logged, some caused fs 
shutdown

> I ask because I also sent a patch to ignore these empty attributes
> in the verifier during log replay.
> 

> FWIW, Only some of your reported buffers look "empty" though, the one
> at 514098.682726 may have had something else wrong.
> 
Ok, i see, it has some "data" in it

> Anyway, yes, probably worth checking with xfs_repair (-n) with this
> patch added.  Let us know what you find! :)
> 
> -Eric
> 

Libor

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,

On úterý 13. prosince 2016 10:04:27 CET Eric Sandeen wrote:
> On 12/13/16 4:52 AM, Libor Klep�? wrote:
> > Hello,
> > should this patch possibly fix errors i reported in this thread?
> > https://www.spinics.net/lists/linux-xfs/msg01728.html
> > 
> > Is it safe to test it? (i do have backups)
> 
> It should be safe, yes.
> 
> (you could always run xfs_repair -n first to be extra careful).
> 
> Were those errors during mount/replay, though, or was it when the
> filesystem was up and running?
> 
> I ask because I also sent a patch to ignore these empty attributes
> in the verifier during log replay.

Is that patch in 4.8.11 kernel?
I ask, because i rebooted machine from this email
https://www.spinics.net/lists/linux-xfs/msg02672.html

with kernel 4.8.11-1~bpo8+1 (from debian)
mount was clean 
[    3.220692] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[    3.222135] XFS (dm-2): Mounting V4 Filesystem
[    3.284697] XFS (dm-2): Ending clean mount


and i ran xfs_repair -n -v (xfsprogs 4.8.0 without your patch) and it came clean I think
---------------------------------
#xfs_repair -n -v /dev/dm-2
Phase 1 - find and verify superblock...
        - block cache size set to 758064 entries
Phase 2 - using internal log
        - zero log...
zero_log: head block 5035 tail block 5035
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

        XFS_REPAIR Summary    Tue Dec 20 06:01:14 2016

Phase           Start           End             Duration
Phase 1:        12/20 05:49:02  12/20 05:49:02
Phase 2:        12/20 05:49:02  12/20 05:49:02
Phase 3:        12/20 05:49:02  12/20 05:59:24  10 minutes, 22 seconds
Phase 4:        12/20 05:59:24  12/20 06:00:17  53 seconds
Phase 5:        Skipped
Phase 6:        12/20 06:00:17  12/20 06:01:13  56 seconds
Phase 7:        12/20 06:01:13  12/20 06:01:14  1 second

Total run time: 12 minutes, 12 seconds
---------------------------------

then i ran metadump and then i ran out of time of my repair window ;)



Libor


> 
> FWIW, Only some of your reported buffers look "empty" though, the one
> at 514098.682726 may have had something else wrong.
> 
> Anyway, yes, probably worth checking with xfs_repair (-n) with this
> patch added.  Let us know what you find! :)
> 
> -Eric
> 
>  
> > Thanks,
> > Libor
> > 
> > On ?tvrtek 8. prosince 2016 12:06:03 CET Eric Sandeen wrote:
> >> We have recently seen a case where, during log replay, the
> >> attr3 leaf verifier reported corruption when encountering a
> >> leaf attribute with a count of 0 in the header.
> >>
> >> We chalked this up to a transient state when a shortform leaf
> >> was created, the attribute didn't fit, and we promoted the
> >> (empty) attribute to the larger leaf form.
> >>
> >> I've recently been given a metadump of unknown provenance which actually
> >> contains a leaf attribute with count 0 on disk.  This causes the
> >> verifier to fire every time xfs_repair is run:
> >>
> >>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
> >>
> >> If this 0-count state is detected, we should just junk the leaf, same
> >> as we would do if the count was too high.  With this change, we now
> >> remedy the problem:
> >>
> >>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
> >>  bad attribute count 0 in attr block 0, inode 12587828
> >>  problem with attribute contents in inode 12587828
> >>  clearing inode 12587828 attributes
> >>  correcting nblocks for inode 12587828, was 2 - counted 1
> >>
> >> Signed-off-by: Eric Sandeen <sandeen@redhat.com>
> >> ---
> >>
> >> diff --git a/repair/attr_repair.c b/repair/attr_repair.c
> >> index 40cb5f7..b855a10 100644
> >> --- a/repair/attr_repair.c
> >> +++ b/repair/attr_repair.c
> >> @@ -593,7 +593,8 @@ process_leaf_attr_block(
> >>  	stop = xfs_attr3_leaf_hdr_size(leaf);
> >>  
> >>  	/* does the count look sorta valid? */
> >> -	if (leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
> >> +	if (!leafhdr.count ||
> >> +	    leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
> >>  						mp->m_sb.sb_blocksize) {
> >>  		do_warn(
> >>  	_("bad attribute count %d in attr block %u, inode %" PRIu64 "\n"),
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 


--------
[1] mailto:libor.klepac@bcom.cz
[2] tel:+420377457676
[3] http://www.bcom.cz

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

On 12/21/16 2:25 AM, Libor Klepáč wrote:
> Hello,
> 
> On úterý 13. prosince 2016 10:04:27 CET Eric Sandeen wrote:
>> On 12/13/16 4:52 AM, Libor Klep�? wrote:
>>> Hello,
>>> should this patch possibly fix errors i reported in this thread?
>>> https://www.spinics.net/lists/linux-xfs/msg01728.html
>>>
>>> Is it safe to test it? (i do have backups)
>>
>> It should be safe, yes.
>>
>> (you could always run xfs_repair -n first to be extra careful).
>>
>> Were those errors during mount/replay, though, or was it when the
>> filesystem was up and running?
>>
>> I ask because I also sent a patch to ignore these empty attributes
>> in the verifier during log replay.
> 
> Is that patch in 4.8.11 kernel?

nope

> I ask, because i rebooted machine from this email
> https://www.spinics.net/lists/linux-xfs/msg02672.html
> 
> with kernel 4.8.11-1~bpo8+1 (from debian)
> mount was clean 
> [    3.220692] SGI XFS with ACLs, security attributes, realtime, no debug enabled
> [    3.222135] XFS (dm-2): Mounting V4 Filesystem
> [    3.284697] XFS (dm-2): Ending clean mount
> 
> 
> and i ran xfs_repair -n -v (xfsprogs 4.8.0 without your patch) and it came clean I think
> ---------------------------------

older repair did not detect the problem, that was the point
of the patch.

-Eric

> #xfs_repair -n -v /dev/dm-2
> Phase 1 - find and verify superblock...
>         - block cache size set to 758064 entries
> Phase 2 - using internal log
>         - zero log...
> zero_log: head block 5035 tail block 5035
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan (but don't clear) agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - agno = 15
>         - agno = 16
>         - agno = 17
>         - agno = 18
>         - agno = 19
>         - agno = 20
>         - agno = 21
>         - agno = 22
>         - agno = 23
>         - agno = 24
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - agno = 15
>         - agno = 16
>         - agno = 17
>         - agno = 18
>         - agno = 19
>         - agno = 20
>         - agno = 21
>         - agno = 22
>         - agno = 23
>         - agno = 24
> No modify flag set, skipping phase 5
> Phase 6 - check inode connectivity...
>         - traversing filesystem ...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - agno = 15
>         - agno = 16
>         - agno = 17
>         - agno = 18
>         - agno = 19
>         - agno = 20
>         - agno = 21
>         - agno = 22
>         - agno = 23
>         - agno = 24
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify link counts...
> No modify flag set, skipping filesystem flush and exiting.
> 
>         XFS_REPAIR Summary    Tue Dec 20 06:01:14 2016
> 
> Phase           Start           End             Duration
> Phase 1:        12/20 05:49:02  12/20 05:49:02
> Phase 2:        12/20 05:49:02  12/20 05:49:02
> Phase 3:        12/20 05:49:02  12/20 05:59:24  10 minutes, 22 seconds
> Phase 4:        12/20 05:59:24  12/20 06:00:17  53 seconds
> Phase 5:        Skipped
> Phase 6:        12/20 06:00:17  12/20 06:01:13  56 seconds
> Phase 7:        12/20 06:01:13  12/20 06:01:14  1 second
> 
> Total run time: 12 minutes, 12 seconds
> ---------------------------------
> 
> then i ran metadump and then i ran out of time of my repair window ;)
> 
> 
> 
> Libor
> 
> 
>>
>> FWIW, Only some of your reported buffers look "empty" though, the one
>> at 514098.682726 may have had something else wrong.
>>
>> Anyway, yes, probably worth checking with xfs_repair (-n) with this
>> patch added.  Let us know what you find! :)
>>
>> -Eric
>>
>>  
>>> Thanks,
>>> Libor
>>>
>>> On ?tvrtek 8. prosince 2016 12:06:03 CET Eric Sandeen wrote:
>>>> We have recently seen a case where, during log replay, the
>>>> attr3 leaf verifier reported corruption when encountering a
>>>> leaf attribute with a count of 0 in the header.
>>>>
>>>> We chalked this up to a transient state when a shortform leaf
>>>> was created, the attribute didn't fit, and we promoted the
>>>> (empty) attribute to the larger leaf form.
>>>>
>>>> I've recently been given a metadump of unknown provenance which actually
>>>> contains a leaf attribute with count 0 on disk.  This causes the
>>>> verifier to fire every time xfs_repair is run:
>>>>
>>>>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
>>>>
>>>> If this 0-count state is detected, we should just junk the leaf, same
>>>> as we would do if the count was too high.  With this change, we now
>>>> remedy the problem:
>>>>
>>>>  Metadata corruption detected at xfs_attr3_leaf block 0x480988/0x1000
>>>>  bad attribute count 0 in attr block 0, inode 12587828
>>>>  problem with attribute contents in inode 12587828
>>>>  clearing inode 12587828 attributes
>>>>  correcting nblocks for inode 12587828, was 2 - counted 1
>>>>
>>>> Signed-off-by: Eric Sandeen <sandeen@redhat.com>
>>>> ---
>>>>
>>>> diff --git a/repair/attr_repair.c b/repair/attr_repair.c
>>>> index 40cb5f7..b855a10 100644
>>>> --- a/repair/attr_repair.c
>>>> +++ b/repair/attr_repair.c
>>>> @@ -593,7 +593,8 @@ process_leaf_attr_block(
>>>>  	stop = xfs_attr3_leaf_hdr_size(leaf);
>>>>  
>>>>  	/* does the count look sorta valid? */
>>>> -	if (leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
>>>> +	if (!leafhdr.count ||
>>>> +	    leafhdr.count * sizeof(xfs_attr_leaf_entry_t) + stop >
>>>>  						mp->m_sb.sb_blocksize) {
>>>>  		do_warn(
>>>>  	_("bad attribute count %d in attr block %u, inode %" PRIu64 "\n"),
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
> 
> 
> --------
> [1] mailto:libor.klepac@bcom.cz
> [2] tel:+420377457676
> [3] http://www.bcom.cz
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,
sorry for late reply. It didn't crash since than and i forgot and moved on to another tasks.

Yesterday it crashed on one of machines (running 4.8.11)
-------------------------
Jan 30 07:18:13 vps2 kernel: [5881831.379547] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x12f63f40
Jan 30 07:18:13 vps2 kernel: [5881831.381721] XFS (dm-2): Unmount and run xfs_repair
Jan 30 07:18:13 vps2 kernel: [5881831.382750] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Jan 30 07:18:13 vps2 kernel: [5881831.387810] XFS (dm-2): metadata I/O error: block 0x12f63f40 ("xfs_trans_read_buf_map") error 117 numblks 8
Jan 30 07:26:02 vps2 kernel: [5882300.524528] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x12645ef8
Jan 30 07:26:02 vps2 kernel: [5882300.525993] XFS (dm-2): Unmount and run xfs_repair
Jan 30 07:26:02 vps2 kernel: [5882300.526539] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Jan 30 07:26:02 vps2 kernel: [5882300.529224] XFS (dm-2): metadata I/O error: block 0x12645ef8 ("xfs_trans_read_buf_map") error 117 numblks 8
Jan 30 10:00:27 vps2 kernel: [5891564.682483] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x127b5578
Jan 30 10:00:27 vps2 kernel: [5891564.683962] XFS (dm-2): Unmount and run xfs_repair
Jan 30 10:00:27 vps2 kernel: [5891564.684536] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Jan 30 10:00:27 vps2 kernel: [5891564.687223] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1250 of file /build/linux-lVEVrl/linux-4.7.8/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc06747f2
Jan 30 10:00:27 vps2 kernel: [5891564.687230] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
Jan 30 10:00:27 vps2 kernel: [5891564.687778] XFS (dm-2): Please umount the filesystem and rectify the problem(s)

and later 
Jan 30 21:10:31 vps2 kernel: [39747.917831] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24c17ba8
Jan 30 21:10:31 vps2 kernel: [39747.918130] XFS (dm-2): metadata I/O error: block 0x24c17ba8 ("xfs_trans_read_buf_map") error 117 numblks 8
-------------------------

I have scheduled repair on today, all these blocks were repaired using xfsprogs 4.9.0
Kernel is now 4.8.15

-------------------------
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
Metadata corruption detected at xfs_attr3_leaf block 0x12645ef8/0x1000
bad attribute count 0 in attr block 0, inode 1074268922
problem with attribute contents in inode 1074268922
clearing inode 1074268922 attributes
correcting nblocks for inode 1074268922, was 1 - counted 0
Metadata corruption detected at xfs_attr3_leaf block 0x127b5578/0x1000
bad attribute count 0 in attr block 0, inode 1077334032
problem with attribute contents in inode 1077334032
clearing inode 1077334032 attributes
correcting nblocks for inode 1077334032, was 1 - counted 0
Metadata corruption detected at xfs_attr3_leaf block 0x12f63f40/0x1000
bad attribute count 0 in attr block 0, inode 1093437859
problem with attribute contents in inode 1093437859
clearing inode 1093437859 attributes
correcting nblocks for inode 1093437859, was 1 - counted 0
        - agno = 2
Metadata corruption detected at xfs_attr3_leaf block 0x24c17ba8/0x1000
bad attribute count 0 in attr block 0, inode 2147673775
problem with attribute contents in inode 2147673775
clearing inode 2147673775 attributes
correcting nblocks for inode 2147673775, was 1 - counted 0
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
bad attribute format 1 in inode 1074268922, resetting value
bad attribute format 1 in inode 1077334032, resetting value
bad attribute format 1 in inode 1093437859, resetting value
        - agno = 2
bad attribute format 1 in inode 2147673775, resetting value
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
-------------------------

Thank you very much for patch, it has done it's work

Libor

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,
problem with this host again, after running uninterrupted from last email/repair on kernel 4.8.15. (so since 31. January)

Today, metadata corruption occured again.
Mar 13 11:16:31 vps2 kernel: [3563991.623260] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
Mar 13 11:16:31 vps2 kernel: [3563991.624321] XFS (dm-2): Unmount and run xfs_repair
Mar 13 11:16:31 vps2 kernel: [3563991.624696] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Mar 13 11:16:31 vps2 kernel: [3563991.625085] ffff994543410000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
Mar 13 11:16:31 vps2 kernel: [3563991.625511] ffff994543410010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
Mar 13 11:16:31 vps2 kernel: [3563991.625983] ffff994543410020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 13 11:16:31 vps2 kernel: [3563991.626398] ffff994543410030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 13 11:16:31 vps2 kernel: [3563991.626829] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1322 of file /build/linux-aPrr8L/linux-4.8.15/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc08295c4
Mar 13 11:16:31 vps2 kernel: [3563991.627210] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Mar 13 11:16:31 vps2 kernel: [3563991.627212] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
Mar 13 11:16:31 vps2 kernel: [3563991.627215] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
Mar 13 11:16:31 vps2 kernel: [3563991.628752] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3420 of file /build/linux-aPrr8L/linux-4.8.15/fs/xfs/xfs_inode.c.  Return address = 0xffffffffc083fc1e
Mar 13 11:16:48 vps2 kernel: [3564008.557340] XFS (dm-2): xfs_log_force: error -5 returned.

After reboot, sometimes it logs
Mar 13 12:51:10 vps2 kernel: [ 5283.025665] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
Mar 13 12:51:10 vps2 kernel: [ 5283.026879] XFS (dm-2): Unmount and run xfs_repair
Mar 13 12:51:10 vps2 kernel: [ 5283.027471] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Mar 13 12:51:10 vps2 kernel: [ 5283.028074] ffff933f16f8c000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
Mar 13 12:51:10 vps2 kernel: [ 5283.028669] ffff933f16f8c010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
Mar 13 12:51:10 vps2 kernel: [ 5283.029240] ffff933f16f8c020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 13 12:51:10 vps2 kernel: [ 5283.029814] ffff933f16f8c030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 13 12:51:10 vps2 kernel: [ 5283.030428] XFS (dm-2): metadata I/O error: block 0x24e70268 ("xfs_trans_read_buf_map") error 117 numblks 8
Mar 13 12:51:10 vps2 kernel: [ 5283.036222] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
Mar 13 12:51:10 vps2 kernel: [ 5283.037443] XFS (dm-2): Unmount and run xfs_repair
Mar 13 12:51:10 vps2 kernel: [ 5283.038049] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Mar 13 12:51:10 vps2 kernel: [ 5283.038644] ffff933f16f8c000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
Mar 13 12:51:10 vps2 kernel: [ 5283.039257] ffff933f16f8c010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
Mar 13 12:51:10 vps2 kernel: [ 5283.039838] ffff933f16f8c020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 13 12:51:10 vps2 kernel: [ 5283.040397] ffff933f16f8c030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 13 12:51:10 vps2 kernel: [ 5283.041482] XFS (dm-2): metadata I/O error: block 0x24e70268 ("xfs_trans_read_buf_map") error 117 numblks 8

I have installed kernel 4.9.13 from backports and installed tools 4.10.0.
Collegue will reboot yesterday and do the repair.

It seems to be the same pattern again. Do you have any clue where it comes from? How can we prevent it from happening?

Thanks,
Libor

On úterý 31. ledna 2017 9:03:02 CET Libor Klepáč wrote:
> 
> Hello,
> sorry for late reply. It didn't crash since than and i forgot and moved on to another tasks.
> 
> Yesterday it crashed on one of machines (running 4.8.11)
> -------------------------
> Jan 30 07:18:13 vps2 kernel: [5881831.379547] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x12f63f40
> Jan 30 07:18:13 vps2 kernel: [5881831.381721] XFS (dm-2): Unmount and run xfs_repair
> Jan 30 07:18:13 vps2 kernel: [5881831.382750] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> Jan 30 07:18:13 vps2 kernel: [5881831.387810] XFS (dm-2): metadata I/O error: block 0x12f63f40 ("xfs_trans_read_buf_map") error 117 numblks 8
> Jan 30 07:26:02 vps2 kernel: [5882300.524528] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x12645ef8
> Jan 30 07:26:02 vps2 kernel: [5882300.525993] XFS (dm-2): Unmount and run xfs_repair
> Jan 30 07:26:02 vps2 kernel: [5882300.526539] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> Jan 30 07:26:02 vps2 kernel: [5882300.529224] XFS (dm-2): metadata I/O error: block 0x12645ef8 ("xfs_trans_read_buf_map") error 117 numblks 8
> Jan 30 10:00:27 vps2 kernel: [5891564.682483] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x127b5578
> Jan 30 10:00:27 vps2 kernel: [5891564.683962] XFS (dm-2): Unmount and run xfs_repair
> Jan 30 10:00:27 vps2 kernel: [5891564.684536] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> Jan 30 10:00:27 vps2 kernel: [5891564.687223] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1250 of file /build/linux-lVEVrl/linux-4.7.8/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc06747f2
> Jan 30 10:00:27 vps2 kernel: [5891564.687230] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
> Jan 30 10:00:27 vps2 kernel: [5891564.687778] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
> 
> and later
> Jan 30 21:10:31 vps2 kernel: [39747.917831] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24c17ba8
> Jan 30 21:10:31 vps2 kernel: [39747.918130] XFS (dm-2): metadata I/O error: block 0x24c17ba8 ("xfs_trans_read_buf_map") error 117 numblks 8
> -------------------------
> 
> I have scheduled repair on today, all these blocks were repaired using xfsprogs 4.9.0
> Kernel is now 4.8.15
> 
> -------------------------
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan and clear agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
> Metadata corruption detected at xfs_attr3_leaf block 0x12645ef8/0x1000
> bad attribute count 0 in attr block 0, inode 1074268922
> problem with attribute contents in inode 1074268922
> clearing inode 1074268922 attributes
> correcting nblocks for inode 1074268922, was 1 - counted 0
> Metadata corruption detected at xfs_attr3_leaf block 0x127b5578/0x1000
> bad attribute count 0 in attr block 0, inode 1077334032
> problem with attribute contents in inode 1077334032
> clearing inode 1077334032 attributes
> correcting nblocks for inode 1077334032, was 1 - counted 0
> Metadata corruption detected at xfs_attr3_leaf block 0x12f63f40/0x1000
> bad attribute count 0 in attr block 0, inode 1093437859
> problem with attribute contents in inode 1093437859
> clearing inode 1093437859 attributes
> correcting nblocks for inode 1093437859, was 1 - counted 0
>         - agno = 2
> Metadata corruption detected at xfs_attr3_leaf block 0x24c17ba8/0x1000
> bad attribute count 0 in attr block 0, inode 2147673775
> problem with attribute contents in inode 2147673775
> clearing inode 2147673775 attributes
> correcting nblocks for inode 2147673775, was 1 - counted 0
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
> bad attribute format 1 in inode 1074268922, resetting value
> bad attribute format 1 in inode 1077334032, resetting value
> bad attribute format 1 in inode 1093437859, resetting value
>         - agno = 2
> bad attribute format 1 in inode 2147673775, resetting value
> Phase 5 - rebuild AG headers and trees...
>         - reset superblock...
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify and correct link counts...
> done
> -------------------------
> 
> Thank you very much for patch, it has done it's work
> 
> Libor
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

On 3/13/17 8:48 AM, Libor Klepáč wrote:
> Hello,
> problem with this host again, after running uninterrupted from last email/repair on kernel 4.8.15. (so since 31. January)
> 
> Today, metadata corruption occured again.
> Mar 13 11:16:31 vps2 kernel: [3563991.623260] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
> Mar 13 11:16:31 vps2 kernel: [3563991.624321] XFS (dm-2): Unmount and run xfs_repair

Ok, interesting that you hit this when writing an attr.

Can you turn the logging level way up:
# echo 11 > /proc/sys/fs/xfs/error_level

and then things like the force shutdown and the metadata will give you a backtrace, which might be useful (noisy, but useful)...

-Eric

> Mar 13 11:16:31 vps2 kernel: [3563991.624696] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> Mar 13 11:16:31 vps2 kernel: [3563991.625085] ffff994543410000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> Mar 13 11:16:31 vps2 kernel: [3563991.625511] ffff994543410010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> Mar 13 11:16:31 vps2 kernel: [3563991.625983] ffff994543410020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Mar 13 11:16:31 vps2 kernel: [3563991.626398] ffff994543410030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Mar 13 11:16:31 vps2 kernel: [3563991.626829] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1322 of file /build/linux-aPrr8L/linux-4.8.15/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc08295c4
> Mar 13 11:16:31 vps2 kernel: [3563991.627210] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
> Mar 13 11:16:31 vps2 kernel: [3563991.627212] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
> Mar 13 11:16:31 vps2 kernel: [3563991.627215] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
> Mar 13 11:16:31 vps2 kernel: [3563991.628752] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3420 of file /build/linux-aPrr8L/linux-4.8.15/fs/xfs/xfs_inode.c.  Return address = 0xffffffffc083fc1e
> Mar 13 11:16:48 vps2 kernel: [3564008.557340] XFS (dm-2): xfs_log_force: error -5 returned.
> 
> After reboot, sometimes it logs
> Mar 13 12:51:10 vps2 kernel: [ 5283.025665] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
> Mar 13 12:51:10 vps2 kernel: [ 5283.026879] XFS (dm-2): Unmount and run xfs_repair
> Mar 13 12:51:10 vps2 kernel: [ 5283.027471] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> Mar 13 12:51:10 vps2 kernel: [ 5283.028074] ffff933f16f8c000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> Mar 13 12:51:10 vps2 kernel: [ 5283.028669] ffff933f16f8c010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> Mar 13 12:51:10 vps2 kernel: [ 5283.029240] ffff933f16f8c020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Mar 13 12:51:10 vps2 kernel: [ 5283.029814] ffff933f16f8c030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Mar 13 12:51:10 vps2 kernel: [ 5283.030428] XFS (dm-2): metadata I/O error: block 0x24e70268 ("xfs_trans_read_buf_map") error 117 numblks 8
> Mar 13 12:51:10 vps2 kernel: [ 5283.036222] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
> Mar 13 12:51:10 vps2 kernel: [ 5283.037443] XFS (dm-2): Unmount and run xfs_repair
> Mar 13 12:51:10 vps2 kernel: [ 5283.038049] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> Mar 13 12:51:10 vps2 kernel: [ 5283.038644] ffff933f16f8c000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> Mar 13 12:51:10 vps2 kernel: [ 5283.039257] ffff933f16f8c010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> Mar 13 12:51:10 vps2 kernel: [ 5283.039838] ffff933f16f8c020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Mar 13 12:51:10 vps2 kernel: [ 5283.040397] ffff933f16f8c030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Mar 13 12:51:10 vps2 kernel: [ 5283.041482] XFS (dm-2): metadata I/O error: block 0x24e70268 ("xfs_trans_read_buf_map") error 117 numblks 8
> 
> I have installed kernel 4.9.13 from backports and installed tools 4.10.0.
> Collegue will reboot yesterday and do the repair.
> 
> It seems to be the same pattern again. Do you have any clue where it comes from? How can we prevent it from happening?
> 
> Thanks,
> Libor
> 
> On úterý 31. ledna 2017 9:03:02 CET Libor Klepáč wrote:
>>
>> Hello,
>> sorry for late reply. It didn't crash since than and i forgot and moved on to another tasks.
>>
>> Yesterday it crashed on one of machines (running 4.8.11)
>> -------------------------
>> Jan 30 07:18:13 vps2 kernel: [5881831.379547] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x12f63f40
>> Jan 30 07:18:13 vps2 kernel: [5881831.381721] XFS (dm-2): Unmount and run xfs_repair
>> Jan 30 07:18:13 vps2 kernel: [5881831.382750] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
>> Jan 30 07:18:13 vps2 kernel: [5881831.387810] XFS (dm-2): metadata I/O error: block 0x12f63f40 ("xfs_trans_read_buf_map") error 117 numblks 8
>> Jan 30 07:26:02 vps2 kernel: [5882300.524528] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x12645ef8
>> Jan 30 07:26:02 vps2 kernel: [5882300.525993] XFS (dm-2): Unmount and run xfs_repair
>> Jan 30 07:26:02 vps2 kernel: [5882300.526539] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
>> Jan 30 07:26:02 vps2 kernel: [5882300.529224] XFS (dm-2): metadata I/O error: block 0x12645ef8 ("xfs_trans_read_buf_map") error 117 numblks 8
>> Jan 30 10:00:27 vps2 kernel: [5891564.682483] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x127b5578
>> Jan 30 10:00:27 vps2 kernel: [5891564.683962] XFS (dm-2): Unmount and run xfs_repair
>> Jan 30 10:00:27 vps2 kernel: [5891564.684536] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
>> Jan 30 10:00:27 vps2 kernel: [5891564.687223] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1250 of file /build/linux-lVEVrl/linux-4.7.8/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc06747f2
>> Jan 30 10:00:27 vps2 kernel: [5891564.687230] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
>> Jan 30 10:00:27 vps2 kernel: [5891564.687778] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
>>
>> and later
>> Jan 30 21:10:31 vps2 kernel: [39747.917831] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24c17ba8
>> Jan 30 21:10:31 vps2 kernel: [39747.918130] XFS (dm-2): metadata I/O error: block 0x24c17ba8 ("xfs_trans_read_buf_map") error 117 numblks 8
>> -------------------------
>>
>> I have scheduled repair on today, all these blocks were repaired using xfsprogs 4.9.0
>> Kernel is now 4.8.15
>>
>> -------------------------
>> Phase 1 - find and verify superblock...
>> Phase 2 - using internal log
>>         - zero log...
>>         - scan filesystem freespace and inode maps...
>>         - found root inode chunk
>> Phase 3 - for each AG...
>>         - scan and clear agi unlinked lists...
>>         - process known inodes and perform inode discovery...
>>         - agno = 0
>>         - agno = 1
>> Metadata corruption detected at xfs_attr3_leaf block 0x12645ef8/0x1000
>> bad attribute count 0 in attr block 0, inode 1074268922
>> problem with attribute contents in inode 1074268922
>> clearing inode 1074268922 attributes
>> correcting nblocks for inode 1074268922, was 1 - counted 0
>> Metadata corruption detected at xfs_attr3_leaf block 0x127b5578/0x1000
>> bad attribute count 0 in attr block 0, inode 1077334032
>> problem with attribute contents in inode 1077334032
>> clearing inode 1077334032 attributes
>> correcting nblocks for inode 1077334032, was 1 - counted 0
>> Metadata corruption detected at xfs_attr3_leaf block 0x12f63f40/0x1000
>> bad attribute count 0 in attr block 0, inode 1093437859
>> problem with attribute contents in inode 1093437859
>> clearing inode 1093437859 attributes
>> correcting nblocks for inode 1093437859, was 1 - counted 0
>>         - agno = 2
>> Metadata corruption detected at xfs_attr3_leaf block 0x24c17ba8/0x1000
>> bad attribute count 0 in attr block 0, inode 2147673775
>> problem with attribute contents in inode 2147673775
>> clearing inode 2147673775 attributes
>> correcting nblocks for inode 2147673775, was 1 - counted 0
>>         - process newly discovered inodes...
>> Phase 4 - check for duplicate blocks...
>>         - setting up duplicate extent list...
>>         - check for inodes claiming duplicate blocks...
>>         - agno = 0
>>         - agno = 1
>> bad attribute format 1 in inode 1074268922, resetting value
>> bad attribute format 1 in inode 1077334032, resetting value
>> bad attribute format 1 in inode 1093437859, resetting value
>>         - agno = 2
>> bad attribute format 1 in inode 2147673775, resetting value
>> Phase 5 - rebuild AG headers and trees...
>>         - reset superblock...
>> Phase 6 - check inode connectivity...
>>         - resetting contents of realtime bitmap and summary inodes
>>         - traversing filesystem ...
>>         - traversal finished ...
>>         - moving disconnected inodes to lost+found ...
>> Phase 7 - verify and correct link counts...
>> done
>> -------------------------
>>
>> Thank you very much for patch, it has done it's work
>>
>> Libor
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,
i have this during night with error_level = 11, no force shutdown
(kernel 4.8.15)
Mar 14 02:36:29 vps2 kernel: [54799.061956] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
Mar 14 02:36:29 vps2 kernel: [54799.063194] XFS (dm-2): Unmount and run xfs_repair
Mar 14 02:36:29 vps2 kernel: [54799.063786] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Mar 14 02:36:29 vps2 kernel: [54799.064377] ffff933db1988000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
Mar 14 02:36:29 vps2 kernel: [54799.064972] ffff933db1988010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
Mar 14 02:36:29 vps2 kernel: [54799.065569] ffff933db1988020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 14 02:36:29 vps2 kernel: [54799.066141] ffff933db1988030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 14 02:36:29 vps2 kernel: [54799.066683] CPU: 1 PID: 22609 Comm: kworker/1:30 Tainted: G            E   4.8.0-0.bpo.2-amd64 #1 Debian 4.8.15-2~bpo8+2
Mar 14 02:36:29 vps2 kernel: [54799.066684] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
Mar 14 02:36:29 vps2 kernel: [54799.066710] Workqueue: xfs-buf/dm-2 xfs_buf_ioend_work [xfs]
Mar 14 02:36:29 vps2 kernel: [54799.066712]  0000000000000286 00000000940e3248 ffffffff92322b05 ffff933daff0e300
Mar 14 02:36:29 vps2 kernel: [54799.066714]  ffff934016e70480 ffffffffc073895a ffff9341bfc98700 00000000940e3248
Mar 14 02:36:29 vps2 kernel: [54799.066715]  ffff933daff0e300 ffff934016e70480 0000000000000000 ffffffffc0779a34
Mar 14 02:36:29 vps2 kernel: [54799.066716] Call Trace:
Mar 14 02:36:29 vps2 kernel: [54799.066736]  [<ffffffff92322b05>] ? dump_stack+0x5c/0x77
Mar 14 02:36:29 vps2 kernel: [54799.066751]  [<ffffffffc073895a>] ? xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs]
Mar 14 02:36:29 vps2 kernel: [54799.066768]  [<ffffffffc0779a34>] ? xfs_buf_ioend+0x54/0x1d0 [xfs]
Mar 14 02:36:29 vps2 kernel: [54799.066776]  [<ffffffff9209000b>] ? process_one_work+0x14b/0x410
Mar 14 02:36:29 vps2 kernel: [54799.066777]  [<ffffffff92090ac5>] ? worker_thread+0x65/0x4a0
Mar 14 02:36:29 vps2 kernel: [54799.066778]  [<ffffffff92090a60>] ? rescuer_thread+0x340/0x340
Mar 14 02:36:29 vps2 kernel: [54799.066780]  [<ffffffff92095d5f>] ? kthread+0xdf/0x100
Mar 14 02:36:29 vps2 kernel: [54799.066805]  [<ffffffff9202478b>] ? __switch_to+0x2bb/0x710
Mar 14 02:36:29 vps2 kernel: [54799.066809]  [<ffffffff925ecf2f>] ? ret_from_fork+0x1f/0x40
Mar 14 02:36:29 vps2 kernel: [54799.066810]  [<ffffffff92095c80>] ? kthread_park+0x50/0x50
Mar 14 02:36:29 vps2 kernel: [54799.067562] XFS (dm-2): metadata I/O error: block 0x24e70268 ("xfs_trans_read_buf_map") error 117 numblks 8

This one is from now, when i tried to find inode=2152616264 which corresponds to block 0x24e70268                                                                                                                   
(kernel 4.9.13)
Mar 14 09:01:31 vps2 kernel: [10644.526839] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268                                                                                                                   
Mar 14 09:01:31 vps2 kernel: [10644.526896] XFS (dm-2): Unmount and run xfs_repair                                                                                                                                                                                                     
Mar 14 09:01:31 vps2 kernel: [10644.526917] XFS (dm-2): First 64 bytes of corrupted metadata buffer:                                                                                                                                                                                   
Mar 14 09:01:31 vps2 kernel: [10644.526944] ffff9f0b5b5d2000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................                                                                                                                                                        
Mar 14 09:01:31 vps2 kernel: [10644.526980] ffff9f0b5b5d2010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........                                                                                                                                                        
Mar 14 09:01:31 vps2 kernel: [10644.527015] ffff9f0b5b5d2020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................                                                                                                                                                        
Mar 14 09:01:31 vps2 kernel: [10644.527049] ffff9f0b5b5d2030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................                                                                                                                                                        
Mar 14 09:01:31 vps2 kernel: [10644.527086] CPU: 2 PID: 30689 Comm: kworker/2:3 Tainted: G            E   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.13-1~bpo8+1                                                                                                                                
Mar 14 09:01:31 vps2 kernel: [10644.527087] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
Mar 14 09:01:31 vps2 kernel: [10644.527119] Workqueue: xfs-buf/dm-2 xfs_buf_ioend_work [xfs]
Mar 14 09:01:31 vps2 kernel: [10644.527121]  0000000000000000 ffffffff9cf29cd5 ffff9f0b91105b00 ffff9f0b6acd2900
Mar 14 09:01:31 vps2 kernel: [10644.527123]  ffffffffc0749aca ffff9f0e7fd18700 000000006076a0c0 ffff9f0b91105b00
Mar 14 09:01:31 vps2 kernel: [10644.527124]  ffff9f0b6acd2900 0000000000000000 ffffffffc0794414 ffff9f0b91105ba8
Mar 14 09:01:31 vps2 kernel: [10644.527126] Call Trace:
Mar 14 09:01:31 vps2 kernel: [10644.527132]  [<ffffffff9cf29cd5>] ? dump_stack+0x5c/0x77
Mar 14 09:01:31 vps2 kernel: [10644.527155]  [<ffffffffc0749aca>] ? xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs]
Mar 14 09:01:31 vps2 kernel: [10644.527184]  [<ffffffffc0794414>] ? xfs_buf_ioend+0x54/0x1d0 [xfs]
Mar 14 09:01:31 vps2 kernel: [10644.527187]  [<ffffffff9cc9171b>] ? process_one_work+0x14b/0x410
Mar 14 09:01:31 vps2 kernel: [10644.527189]  [<ffffffff9cc921d5>] ? worker_thread+0x65/0x4a0
Mar 14 09:01:31 vps2 kernel: [10644.527190]  [<ffffffff9cc92170>] ? rescuer_thread+0x340/0x340
Mar 14 09:01:31 vps2 kernel: [10644.527191]  [<ffffffff9cc92170>] ? rescuer_thread+0x340/0x340
Mar 14 09:01:31 vps2 kernel: [10644.527194]  [<ffffffff9cc03b81>] ? do_syscall_64+0x81/0x190
Mar 14 09:01:31 vps2 kernel: [10644.527197]  [<ffffffff9cc7c730>] ? SyS_exit_group+0x10/0x10
Mar 14 09:01:31 vps2 kernel: [10644.527198]  [<ffffffff9cc974c0>] ? kthread+0xe0/0x100
Mar 14 09:01:31 vps2 kernel: [10644.527200]  [<ffffffff9cc2476b>] ? __switch_to+0x2bb/0x700
Mar 14 09:01:31 vps2 kernel: [10644.527202]  [<ffffffff9cc973e0>] ? kthread_park+0x60/0x60
Mar 14 09:01:31 vps2 kernel: [10644.527205]  [<ffffffff9d1fb675>] ? ret_from_fork+0x25/0x30
Mar 14 09:01:31 vps2 kernel: [10644.527213] XFS (dm-2): metadata I/O error: block 0x24e70268 ("xfs_trans_read_buf_map") error 117 numblks 8


---------------------------------------------------------
There are logs from my collegue, he wasn't able to do repair, device was busy, probably something forgoten to stop (container with private mounts?)

# xfs_repair -n /dev/mapper/vgDisk-lvData
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
sb_fdblocks 2245059, counted 2253251
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
bad attribute count 0 in attr block 0, inode 2152616264
problem with attribute contents in inode 2152616264
would clear attr fork
bad nblocks 1 for inode 2152616264, would reset to 0
bad anextents 1 for inode 2152616264, would reset to 0
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
 
# xfs_db -r /dev/mapper/vgDisk2-lvData
xfs_db> inode 2152616264
xfs_db> print
core.magic = 0x494e
core.mode = 0100664
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 1
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 20045
core.gid = 20045
core.flushiter = 0
core.atime.sec = Mon Mar 13 11:16:31 2017
core.atime.nsec = 815645008
core.mtime.sec = Mon Mar 13 11:16:31 2017
core.mtime.nsec = 815645008
core.ctime.sec = Mon Mar 13 11:16:31 2017
core.ctime.nsec = 815645008
core.size = 0
core.nblocks = 1
core.extsize = 0
core.nextents = 0
core.naextents = 1
core.forkoff = 15
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 3279391285
next_unlinked = null
u = (empty)
a.bmx[0] = [startoff,startblock,blockcount,extentflag]
0:[0,134538317,1,0]
 
# xfs_repair /dev/mapper/vgDisk2-lvData
xfs_repair: cannot open /dev/mapper/vgDisk2-lvData: Device or resource busy



Libor



On pondělí 13. března 2017 9:14:53 CET Eric Sandeen wrote:
> On 3/13/17 8:48 AM, Libor Klepáč wrote:
> > Hello,
> > problem with this host again, after running uninterrupted from last email/repair on kernel 4.8.15. (so since 31. January)
> > 
> > Today, metadata corruption occured again.
> > Mar 13 11:16:31 vps2 kernel: [3563991.623260] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
> > Mar 13 11:16:31 vps2 kernel: [3563991.624321] XFS (dm-2): Unmount and run xfs_repair
> 
> Ok, interesting that you hit this when writing an attr.
> 
> Can you turn the logging level way up:
> # echo 11 > /proc/sys/fs/xfs/error_level
> 
> and then things like the force shutdown and the metadata will give you a backtrace, which might be useful (noisy, but useful)...
> 
> -Eric
> 
> > Mar 13 11:16:31 vps2 kernel: [3563991.624696] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> > Mar 13 11:16:31 vps2 kernel: [3563991.625085] ffff994543410000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> > Mar 13 11:16:31 vps2 kernel: [3563991.625511] ffff994543410010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> > Mar 13 11:16:31 vps2 kernel: [3563991.625983] ffff994543410020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Mar 13 11:16:31 vps2 kernel: [3563991.626398] ffff994543410030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Mar 13 11:16:31 vps2 kernel: [3563991.626829] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1322 of file /build/linux-aPrr8L/linux-4.8.15/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc08295c4
> > Mar 13 11:16:31 vps2 kernel: [3563991.627210] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
> > Mar 13 11:16:31 vps2 kernel: [3563991.627212] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
> > Mar 13 11:16:31 vps2 kernel: [3563991.627215] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
> > Mar 13 11:16:31 vps2 kernel: [3563991.628752] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3420 of file /build/linux-aPrr8L/linux-4.8.15/fs/xfs/xfs_inode.c.  Return address = 0xffffffffc083fc1e
> > Mar 13 11:16:48 vps2 kernel: [3564008.557340] XFS (dm-2): xfs_log_force: error -5 returned.
> > 
> > After reboot, sometimes it logs
> > Mar 13 12:51:10 vps2 kernel: [ 5283.025665] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
> > Mar 13 12:51:10 vps2 kernel: [ 5283.026879] XFS (dm-2): Unmount and run xfs_repair
> > Mar 13 12:51:10 vps2 kernel: [ 5283.027471] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> > Mar 13 12:51:10 vps2 kernel: [ 5283.028074] ffff933f16f8c000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> > Mar 13 12:51:10 vps2 kernel: [ 5283.028669] ffff933f16f8c010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> > Mar 13 12:51:10 vps2 kernel: [ 5283.029240] ffff933f16f8c020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Mar 13 12:51:10 vps2 kernel: [ 5283.029814] ffff933f16f8c030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Mar 13 12:51:10 vps2 kernel: [ 5283.030428] XFS (dm-2): metadata I/O error: block 0x24e70268 ("xfs_trans_read_buf_map") error 117 numblks 8
> > Mar 13 12:51:10 vps2 kernel: [ 5283.036222] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
> > Mar 13 12:51:10 vps2 kernel: [ 5283.037443] XFS (dm-2): Unmount and run xfs_repair
> > Mar 13 12:51:10 vps2 kernel: [ 5283.038049] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> > Mar 13 12:51:10 vps2 kernel: [ 5283.038644] ffff933f16f8c000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> > Mar 13 12:51:10 vps2 kernel: [ 5283.039257] ffff933f16f8c010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> > Mar 13 12:51:10 vps2 kernel: [ 5283.039838] ffff933f16f8c020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Mar 13 12:51:10 vps2 kernel: [ 5283.040397] ffff933f16f8c030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Mar 13 12:51:10 vps2 kernel: [ 5283.041482] XFS (dm-2): metadata I/O error: block 0x24e70268 ("xfs_trans_read_buf_map") error 117 numblks 8
> > 
> > I have installed kernel 4.9.13 from backports and installed tools 4.10.0.
> > Collegue will reboot yesterday and do the repair.
> > 
> > It seems to be the same pattern again. Do you have any clue where it comes from? How can we prevent it from happening?
> > 
> > Thanks,
> > Libor
> > 
> > On úterý 31. ledna 2017 9:03:02 CET Libor Klepáč wrote:
> >>
> >> Hello,
> >> sorry for late reply. It didn't crash since than and i forgot and moved on to another tasks.
> >>
> >> Yesterday it crashed on one of machines (running 4.8.11)
> >> -------------------------
> >> Jan 30 07:18:13 vps2 kernel: [5881831.379547] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x12f63f40
> >> Jan 30 07:18:13 vps2 kernel: [5881831.381721] XFS (dm-2): Unmount and run xfs_repair
> >> Jan 30 07:18:13 vps2 kernel: [5881831.382750] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> >> Jan 30 07:18:13 vps2 kernel: [5881831.387810] XFS (dm-2): metadata I/O error: block 0x12f63f40 ("xfs_trans_read_buf_map") error 117 numblks 8
> >> Jan 30 07:26:02 vps2 kernel: [5882300.524528] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x12645ef8
> >> Jan 30 07:26:02 vps2 kernel: [5882300.525993] XFS (dm-2): Unmount and run xfs_repair
> >> Jan 30 07:26:02 vps2 kernel: [5882300.526539] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> >> Jan 30 07:26:02 vps2 kernel: [5882300.529224] XFS (dm-2): metadata I/O error: block 0x12645ef8 ("xfs_trans_read_buf_map") error 117 numblks 8
> >> Jan 30 10:00:27 vps2 kernel: [5891564.682483] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x127b5578
> >> Jan 30 10:00:27 vps2 kernel: [5891564.683962] XFS (dm-2): Unmount and run xfs_repair
> >> Jan 30 10:00:27 vps2 kernel: [5891564.684536] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> >> Jan 30 10:00:27 vps2 kernel: [5891564.687223] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1250 of file /build/linux-lVEVrl/linux-4.7.8/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc06747f2
> >> Jan 30 10:00:27 vps2 kernel: [5891564.687230] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
> >> Jan 30 10:00:27 vps2 kernel: [5891564.687778] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
> >>
> >> and later
> >> Jan 30 21:10:31 vps2 kernel: [39747.917831] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24c17ba8
> >> Jan 30 21:10:31 vps2 kernel: [39747.918130] XFS (dm-2): metadata I/O error: block 0x24c17ba8 ("xfs_trans_read_buf_map") error 117 numblks 8
> >> -------------------------
> >>
> >> I have scheduled repair on today, all these blocks were repaired using xfsprogs 4.9.0
> >> Kernel is now 4.8.15
> >>
> >> -------------------------
> >> Phase 1 - find and verify superblock...
> >> Phase 2 - using internal log
> >>         - zero log...
> >>         - scan filesystem freespace and inode maps...
> >>         - found root inode chunk
> >> Phase 3 - for each AG...
> >>         - scan and clear agi unlinked lists...
> >>         - process known inodes and perform inode discovery...
> >>         - agno = 0
> >>         - agno = 1
> >> Metadata corruption detected at xfs_attr3_leaf block 0x12645ef8/0x1000
> >> bad attribute count 0 in attr block 0, inode 1074268922
> >> problem with attribute contents in inode 1074268922
> >> clearing inode 1074268922 attributes
> >> correcting nblocks for inode 1074268922, was 1 - counted 0
> >> Metadata corruption detected at xfs_attr3_leaf block 0x127b5578/0x1000
> >> bad attribute count 0 in attr block 0, inode 1077334032
> >> problem with attribute contents in inode 1077334032
> >> clearing inode 1077334032 attributes
> >> correcting nblocks for inode 1077334032, was 1 - counted 0
> >> Metadata corruption detected at xfs_attr3_leaf block 0x12f63f40/0x1000
> >> bad attribute count 0 in attr block 0, inode 1093437859
> >> problem with attribute contents in inode 1093437859
> >> clearing inode 1093437859 attributes
> >> correcting nblocks for inode 1093437859, was 1 - counted 0
> >>         - agno = 2
> >> Metadata corruption detected at xfs_attr3_leaf block 0x24c17ba8/0x1000
> >> bad attribute count 0 in attr block 0, inode 2147673775
> >> problem with attribute contents in inode 2147673775
> >> clearing inode 2147673775 attributes
> >> correcting nblocks for inode 2147673775, was 1 - counted 0
> >>         - process newly discovered inodes...
> >> Phase 4 - check for duplicate blocks...
> >>         - setting up duplicate extent list...
> >>         - check for inodes claiming duplicate blocks...
> >>         - agno = 0
> >>         - agno = 1
> >> bad attribute format 1 in inode 1074268922, resetting value
> >> bad attribute format 1 in inode 1077334032, resetting value
> >> bad attribute format 1 in inode 1093437859, resetting value
> >>         - agno = 2
> >> bad attribute format 1 in inode 2147673775, resetting value
> >> Phase 5 - rebuild AG headers and trees...
> >>         - reset superblock...
> >> Phase 6 - check inode connectivity...
> >>         - resetting contents of realtime bitmap and summary inodes
> >>         - traversing filesystem ...
> >>         - traversal finished ...
> >>         - moving disconnected inodes to lost+found ...
> >> Phase 7 - verify and correct link counts...
> >> done
> >> -------------------------
> >>
> >> Thank you very much for patch, it has done it's work
> >>
> >> Libor
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


--------
[1] mailto:libor.klepac@bcom.cz
[2] tel:+420377457676
[3] http://www.bcom.cz

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

On 3/14/17 3:15 AM, Libor Klepáč wrote:
> Hello,
> i have this during night with error_level = 11, no force shutdown

Just to double check - are these all getting freshly created?  I.e.
you repair and fix the filesystem each time you see this, but they
keep re-appearing?

> (kernel 4.8.15)
> Mar 14 02:36:29 vps2 kernel: [54799.061956] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268

Unfortunately the read path is a bit less interesting.  We found something
on disk, but we're not sure how it got there.

If we could catch a write verifier failing that /might/ be a little more
useful.

I'm at a loss as to why you seem to be uniquely able to hit this problem.  :(

-Eric

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

On 3/14/17 11:54 AM, Eric Sandeen wrote:
> On 3/14/17 3:15 AM, Libor Klepáč wrote:
>> Hello,
>> i have this during night with error_level = 11, no force shutdown
> 
> Just to double check - are these all getting freshly created?  I.e.
> you repair and fix the filesystem each time you see this, but they
> keep re-appearing?

And is anything else happening in between - unclean shutdowns,
crashes, storage problems etc or are you just riding along, and
they show up out of nowhere on an otherwise normal box?

(sorry if that's elsewhere in this long thread, but a summary
of the problem at this point might help).

-Eric

>> (kernel 4.8.15)
>> Mar 14 02:36:29 vps2 kernel: [54799.061956] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
> 
> Unfortunately the read path is a bit less interesting.  We found something
> on disk, but we're not sure how it got there.
> 
> If we could catch a write verifier failing that /might/ be a little more
> useful.
> 
> I'm at a loss as to why you seem to be uniquely able to hit this problem.  :(
> 
> -Eric
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,

all VMs are running smoothly between these problems with XFS, no HW problems, no crashes, no force reboots.

On úterý 14. března 2017 11:54:37 CET Eric Sandeen wrote:
> On 3/14/17 3:15 AM, Libor Klepáč wrote:
> > Hello,
> > i have this during night with error_level = 11, no force shutdown
> 
> Just to double check - are these all getting freshly created?  I.e.
> you repair and fix the filesystem each time you see this, but they
> keep re-appearing?

I have looked to old emails and every xfs_repair fixes problem and next time different block/inode pops out.
I'm trying to lookup files using inode numbers from old emails.

If i remember correctly, files/dirs are usually created in some website cache directory. (ie. generated from template by php and saved to disk for later use).
So my lookup will probably come to nothing. Last reported force shutdown was on cache file.

> 
> > (kernel 4.8.15)
> > Mar 14 02:36:29 vps2 kernel: [54799.061956] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0x24e70268
> 
> Unfortunately the read path is a bit less interesting.  We found something
> on disk, but we're not sure how it got there.
> If we could catch a write verifier failing that /might/ be a little more
> useful.

I'm prepared to run all affected hosts with error_level=11 , if it doesn't mean performance hit.

> 
> I'm at a loss as to why you seem to be uniquely able to hit this problem.  :(

And I was questioning my googling skills ;)

> 
> -Eric
> 

Libor

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

On 3/15/17 5:07 AM, Libor Klepáč wrote:
> Hello,

...

>> Unfortunately the read path is a bit less interesting.  We found something
>> on disk, but we're not sure how it got there.
>> If we could catch a write verifier failing that /might/ be a little more
>> useful.
> 
> I'm prepared to run all affected hosts with error_level=11 , if it doesn't mean performance hit.

It shouldn't.  It only changes logging behavior on error.  The printks to the
console probably take a bit of extra time but at that point you've already lost,
right?

-Eric

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hi,
On středa 15. března 2017 10:22:05 CET Eric Sandeen wrote:
> On 3/15/17 5:07 AM, Libor Klepáč wrote:
> > Hello,
> 
> ...
> 
> >> Unfortunately the read path is a bit less interesting.  We found something
> >> on disk, but we're not sure how it got there.
> >> If we could catch a write verifier failing that /might/ be a little more
> >> useful.
> > 
> > I'm prepared to run all affected hosts with error_level=11 , if it doesn't mean performance hit.
> 
> It shouldn't.  It only changes logging behavior on error.  The printks to the
> console probably take a bit of extra time but at that point you've already lost,
> right?

Ok, thanks for clarification, I was wondering whether higher error_level isn't triggering some extra code paths during normal operation.
I will leave it on 11 on all machines with problems.

Btw. i naively created 
/etc/sysctl.d/fs_xfs_error_level.conf
with 
fs.xfs.error_level=11
inside.
But it's not set after reboot. May it be because XFS module (not root filesystem) is loaded after sysctl is set?

Bellow is result of repair.
I was expecting 
bad attribute count 0 in attr block 0, inode 2152616264
to be
bad attribute count 0 in attr block 0x24e70268, inode 2152616264

Function process_leaf_attr_block should be called with non-zero da_bno in process_leaf_attr_level , right?
But it's called with da_bno = 0 in process_longform_attr .

Of course I don't know the code.

Libor

# xfs_repair  /dev/mapper/vgDisk2-lvData
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
bad attribute count 0 in attr block 0, inode 2152616264
problem with attribute contents in inode 2152616264
clearing inode 2152616264 attributes
correcting nblocks for inode 2152616264, was 1 - counted 0
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
bad attribute format 1 in inode 2152616264, resetting value
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Done

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

On 3/16/17 1:58 AM, Libor Klepáč wrote:
> Hi,
> On středa 15. března 2017 10:22:05 CET Eric Sandeen wrote:
>> On 3/15/17 5:07 AM, Libor Klepáč wrote:
>>> Hello,
>>
>> ...
>>
>>>> Unfortunately the read path is a bit less interesting.  We found something
>>>> on disk, but we're not sure how it got there.
>>>> If we could catch a write verifier failing that /might/ be a little more
>>>> useful.
>>>
>>> I'm prepared to run all affected hosts with error_level=11 , if it doesn't mean performance hit.
>>
>> It shouldn't.  It only changes logging behavior on error.  The printks to the
>> console probably take a bit of extra time but at that point you've already lost,
>> right?
> 
> Ok, thanks for clarification, I was wondering whether higher error_level isn't triggering some extra code paths during normal operation.
> I will leave it on 11 on all machines with problems.

Understood.  Yup, it's really not, it only changes reporting post-error.

> Btw. i naively created 
> /etc/sysctl.d/fs_xfs_error_level.conf
> with 
> fs.xfs.error_level=11
> inside.
> But it's not set after reboot. May it be because XFS module (not root filesystem) is loaded after sysctl is set?

Must be, the systctl file will not appear until xfs is loaded.
 
> Bellow is result of repair.
> I was expecting 
> bad attribute count 0 in attr block 0, inode 2152616264
> to be
> bad attribute count 0 in attr block 0x24e70268, inode 2152616264
> 
> Function process_leaf_attr_block should be called with non-zero da_bno in process_leaf_attr_level , right?
> But it's called with da_bno = 0 in process_longform_attr .

typedef __uint32_t      xfs_dablk_t;    /* dir/attr block number (in file) */

It's the relative block number in the attribute, i.e. block 0 (first block)

> Of course I don't know the code.
> 
> Libor
> 
> # xfs_repair  /dev/mapper/vgDisk2-lvData
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan and clear agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
> bad attribute count 0 in attr block 0, inode 2152616264
> problem with attribute contents in inode 2152616264
> clearing inode 2152616264 attributes
> correcting nblocks for inode 2152616264, was 1 - counted 0
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
> bad attribute format 1 in inode 2152616264, resetting value
> Phase 5 - rebuild AG headers and trees...
>         - reset superblock...
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify and correct link counts...
> Done
> 
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,
so here is trace from today.
Hope it helps you

Libor

Mar 29 15:05:12 vps2 kernel: [1154612.424431] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x24dd5310
Mar 29 15:05:12 vps2 kernel: [1154612.424882] XFS (dm-2): Unmount and run xfs_repair
Mar 29 15:05:12 vps2 kernel: [1154612.425072] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Mar 29 15:05:12 vps2 kernel: [1154612.425262] ffff979235612000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
Mar 29 15:05:12 vps2 kernel: [1154612.425478] ffff979235612010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
Mar 29 15:05:12 vps2 kernel: [1154612.425708] ffff979235612020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 29 15:05:12 vps2 kernel: [1154612.425960] ffff979235612030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 29 15:05:12 vps2 kernel: [1154612.426259] CPU: 2 PID: 453 Comm: xfsaild/dm-2 Tainted: G            E   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.13-1~bpo8+1
Mar 29 15:05:12 vps2 kernel: [1154612.426260] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
Mar 29 15:05:12 vps2 kernel: [1154612.426262]  0000000000000000 ffffffffb1b29cd5 ffff9794ac29c980 ffff9791295d83c0
Mar 29 15:05:12 vps2 kernel: [1154612.426264]  ffffffffc064aa58 0000000000000000 0000000001d9a9ff 0000000000000000
Mar 29 15:05:12 vps2 kernel: [1154612.426266]  ffffbd1ac1fb3d40 ffffffffc0695997 ffff9792b0dd60c0 ffffffffc0693cc4
Mar 29 15:05:12 vps2 kernel: [1154612.426268] Call Trace:
Mar 29 15:05:12 vps2 kernel: [1154612.426277]  [<ffffffffb1b29cd5>] ? dump_stack+0x5c/0x77
Mar 29 15:05:12 vps2 kernel: [1154612.426339]  [<ffffffffc064aa58>] ? xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426361]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426384]  [<ffffffffc0693cc4>] ? _xfs_buf_ioapply+0x84/0x450 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426389]  [<ffffffffb18a2b30>] ? wake_up_q+0x60/0x60
Mar 29 15:05:12 vps2 kernel: [1154612.426411]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426446]  [<ffffffffc06956d7>] ? xfs_buf_submit+0x67/0x1f0 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426473]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426505]  [<ffffffffc06c777b>] ? xfsaild+0x2cb/0x7b0 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426531]  [<ffffffffc06bf2f9>] ? xfs_inode_item_push+0x99/0x140 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426558]  [<ffffffffc06c777b>] ? xfsaild+0x2cb/0x7b0 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426585]  [<ffffffffc06c74b0>] ? xfs_trans_ail_cursor_first+0x80/0x80 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.426589]  [<ffffffffb18974c0>] ? kthread+0xe0/0x100
Mar 29 15:05:12 vps2 kernel: [1154612.426592]  [<ffffffffb182476b>] ? __switch_to+0x2bb/0x700
Mar 29 15:05:12 vps2 kernel: [1154612.426594]  [<ffffffffb18973e0>] ? kthread_park+0x60/0x60
Mar 29 15:05:12 vps2 kernel: [1154612.426597]  [<ffffffffb1dfb675>] ? ret_from_fork+0x25/0x30
Mar 29 15:05:12 vps2 kernel: [1154612.426602] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1323 of file /home/zumbi/linux-4.9.13/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc0694084
Mar 29 15:05:12 vps2 kernel: [1154612.427033] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
Mar 29 15:05:12 vps2 kernel: [1154612.427323] CPU: 2 PID: 453 Comm: xfsaild/dm-2 Tainted: G            E   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.13-1~bpo8+1
Mar 29 15:05:12 vps2 kernel: [1154612.427323] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
Mar 29 15:05:12 vps2 kernel: [1154612.427324]  0000000000000000 ffffffffb1b29cd5 0000000000000008 ffff979528c42000
Mar 29 15:05:12 vps2 kernel: [1154612.427326]  ffffffffc069c1fa ffffffffc0694084 0000000000000000 ffffbd1ac1fb3d40
Mar 29 15:05:12 vps2 kernel: [1154612.427328]  ffffffffc0695997 ffff9792b0dd60c0 ffffffffc0694084 0000000000000000
Mar 29 15:05:12 vps2 kernel: [1154612.427329] Call Trace:
Mar 29 15:05:12 vps2 kernel: [1154612.427335]  [<ffffffffb1b29cd5>] ? dump_stack+0x5c/0x77
Mar 29 15:05:12 vps2 kernel: [1154612.427380]  [<ffffffffc069c1fa>] ? xfs_do_force_shutdown+0x13a/0x140 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427408]  [<ffffffffc0694084>] ? _xfs_buf_ioapply+0x444/0x450 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427432]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427458]  [<ffffffffc0694084>] ? _xfs_buf_ioapply+0x444/0x450 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427462]  [<ffffffffb18a2b30>] ? wake_up_q+0x60/0x60
Mar 29 15:05:12 vps2 kernel: [1154612.427485]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427508]  [<ffffffffc06956d7>] ? xfs_buf_submit+0x67/0x1f0 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427530]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427562]  [<ffffffffc06c777b>] ? xfsaild+0x2cb/0x7b0 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427596]  [<ffffffffc06bf2f9>] ? xfs_inode_item_push+0x99/0x140 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427625]  [<ffffffffc06c777b>] ? xfsaild+0x2cb/0x7b0 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427652]  [<ffffffffc06c74b0>] ? xfs_trans_ail_cursor_first+0x80/0x80 [xfs]
Mar 29 15:05:12 vps2 kernel: [1154612.427654]  [<ffffffffb18974c0>] ? kthread+0xe0/0x100
Mar 29 15:05:12 vps2 kernel: [1154612.427656]  [<ffffffffb182476b>] ? __switch_to+0x2bb/0x700
Mar 29 15:05:12 vps2 kernel: [1154612.427657]  [<ffffffffb18973e0>] ? kthread_park+0x60/0x60
Mar 29 15:05:12 vps2 kernel: [1154612.427659]  [<ffffffffb1dfb675>] ? ret_from_fork+0x25/0x30
Mar 29 15:05:12 vps2 kernel: [1154612.427671] XFS (dm-2): Please umount the filesystem and rectify the problem(s)

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,
did you get anything useful from this last trace?
It happened again today, sadly I don't have trace, it seems, I forgot to raise error level last time.
PS: I don't know, if I mentioned it somewhere - it's XFSv4 filesystem

Thanks,
Libor

On středa 29. března 2017 15:33:00 CEST Libor Klepáč wrote:
> [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
> 
> Hello,
> so here is trace from today.
> Hope it helps you
> 
> Libor
> 
> Mar 29 15:05:12 vps2 kernel: [1154612.424431] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x24dd5310
> Mar 29 15:05:12 vps2 kernel: [1154612.424882] XFS (dm-2): Unmount and run xfs_repair
> Mar 29 15:05:12 vps2 kernel: [1154612.425072] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> Mar 29 15:05:12 vps2 kernel: [1154612.425262] ffff979235612000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> Mar 29 15:05:12 vps2 kernel: [1154612.425478] ffff979235612010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> Mar 29 15:05:12 vps2 kernel: [1154612.425708] ffff979235612020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Mar 29 15:05:12 vps2 kernel: [1154612.425960] ffff979235612030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Mar 29 15:05:12 vps2 kernel: [1154612.426259] CPU: 2 PID: 453 Comm: xfsaild/dm-2 Tainted: G            E   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.13-1~bpo8+1
> Mar 29 15:05:12 vps2 kernel: [1154612.426260] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
> Mar 29 15:05:12 vps2 kernel: [1154612.426262]  0000000000000000 ffffffffb1b29cd5 ffff9794ac29c980 ffff9791295d83c0
> Mar 29 15:05:12 vps2 kernel: [1154612.426264]  ffffffffc064aa58 0000000000000000 0000000001d9a9ff 0000000000000000
> Mar 29 15:05:12 vps2 kernel: [1154612.426266]  ffffbd1ac1fb3d40 ffffffffc0695997 ffff9792b0dd60c0 ffffffffc0693cc4
> Mar 29 15:05:12 vps2 kernel: [1154612.426268] Call Trace:
> Mar 29 15:05:12 vps2 kernel: [1154612.426277]  [<ffffffffb1b29cd5>] ? dump_stack+0x5c/0x77
> Mar 29 15:05:12 vps2 kernel: [1154612.426339]  [<ffffffffc064aa58>] ? xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426361]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426384]  [<ffffffffc0693cc4>] ? _xfs_buf_ioapply+0x84/0x450 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426389]  [<ffffffffb18a2b30>] ? wake_up_q+0x60/0x60
> Mar 29 15:05:12 vps2 kernel: [1154612.426411]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426446]  [<ffffffffc06956d7>] ? xfs_buf_submit+0x67/0x1f0 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426473]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426505]  [<ffffffffc06c777b>] ? xfsaild+0x2cb/0x7b0 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426531]  [<ffffffffc06bf2f9>] ? xfs_inode_item_push+0x99/0x140 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426558]  [<ffffffffc06c777b>] ? xfsaild+0x2cb/0x7b0 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426585]  [<ffffffffc06c74b0>] ? xfs_trans_ail_cursor_first+0x80/0x80 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.426589]  [<ffffffffb18974c0>] ? kthread+0xe0/0x100
> Mar 29 15:05:12 vps2 kernel: [1154612.426592]  [<ffffffffb182476b>] ? __switch_to+0x2bb/0x700
> Mar 29 15:05:12 vps2 kernel: [1154612.426594]  [<ffffffffb18973e0>] ? kthread_park+0x60/0x60
> Mar 29 15:05:12 vps2 kernel: [1154612.426597]  [<ffffffffb1dfb675>] ? ret_from_fork+0x25/0x30
> Mar 29 15:05:12 vps2 kernel: [1154612.426602] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1323 of file /home/zumbi/linux-4.9.13/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc0694084
> Mar 29 15:05:12 vps2 kernel: [1154612.427033] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
> Mar 29 15:05:12 vps2 kernel: [1154612.427323] CPU: 2 PID: 453 Comm: xfsaild/dm-2 Tainted: G            E   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.13-1~bpo8+1
> Mar 29 15:05:12 vps2 kernel: [1154612.427323] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
> Mar 29 15:05:12 vps2 kernel: [1154612.427324]  0000000000000000 ffffffffb1b29cd5 0000000000000008 ffff979528c42000
> Mar 29 15:05:12 vps2 kernel: [1154612.427326]  ffffffffc069c1fa ffffffffc0694084 0000000000000000 ffffbd1ac1fb3d40
> Mar 29 15:05:12 vps2 kernel: [1154612.427328]  ffffffffc0695997 ffff9792b0dd60c0 ffffffffc0694084 0000000000000000
> Mar 29 15:05:12 vps2 kernel: [1154612.427329] Call Trace:
> Mar 29 15:05:12 vps2 kernel: [1154612.427335]  [<ffffffffb1b29cd5>] ? dump_stack+0x5c/0x77
> Mar 29 15:05:12 vps2 kernel: [1154612.427380]  [<ffffffffc069c1fa>] ? xfs_do_force_shutdown+0x13a/0x140 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427408]  [<ffffffffc0694084>] ? _xfs_buf_ioapply+0x444/0x450 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427432]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427458]  [<ffffffffc0694084>] ? _xfs_buf_ioapply+0x444/0x450 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427462]  [<ffffffffb18a2b30>] ? wake_up_q+0x60/0x60
> Mar 29 15:05:12 vps2 kernel: [1154612.427485]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427508]  [<ffffffffc06956d7>] ? xfs_buf_submit+0x67/0x1f0 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427530]  [<ffffffffc0695997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427562]  [<ffffffffc06c777b>] ? xfsaild+0x2cb/0x7b0 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427596]  [<ffffffffc06bf2f9>] ? xfs_inode_item_push+0x99/0x140 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427625]  [<ffffffffc06c777b>] ? xfsaild+0x2cb/0x7b0 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427652]  [<ffffffffc06c74b0>] ? xfs_trans_ail_cursor_first+0x80/0x80 [xfs]
> Mar 29 15:05:12 vps2 kernel: [1154612.427654]  [<ffffffffb18974c0>] ? kthread+0xe0/0x100
> Mar 29 15:05:12 vps2 kernel: [1154612.427656]  [<ffffffffb182476b>] ? __switch_to+0x2bb/0x700
> Mar 29 15:05:12 vps2 kernel: [1154612.427657]  [<ffffffffb18973e0>] ? kthread_park+0x60/0x60
> Mar 29 15:05:12 vps2 kernel: [1154612.427659]  [<ffffffffb1dfb675>] ? ret_from_fork+0x25/0x30
> Mar 29 15:05:12 vps2 kernel: [1154612.427671] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hi,
after quiet two months , another crash, trace is same as before.
There are no XFS lines before this, kern.log goes back to 23. april.

May 24 12:05:12 vps2 kernel: [3729926.243353] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0x171c60c0
May 24 12:05:12 vps2 kernel: [3729926.247586] XFS (dm-2): Unmount and run xfs_repair
May 24 12:05:12 vps2 kernel: [3729926.249022] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
May 24 12:05:12 vps2 kernel: [3729926.250424] ffff9fbb1d73a000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
May 24 12:05:12 vps2 kernel: [3729926.251870] ffff9fbb1d73a010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
May 24 12:05:12 vps2 kernel: [3729926.253412] ffff9fbb1d73a020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
May 24 12:05:12 vps2 kernel: [3729926.254941] ffff9fbb1d73a030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
May 24 12:05:12 vps2 kernel: [3729926.256478] CPU: 1 PID: 454 Comm: xfsaild/dm-2 Tainted: G            E   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.13-1~bpo8+1
May 24 12:05:12 vps2 kernel: [3729926.256480] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
May 24 12:05:12 vps2 kernel: [3729926.256483]  0000000000000000 ffffffffb1329cd5 ffff9fba7f543ac0 ffff9fb9cb6ca5a0
May 24 12:05:12 vps2 kernel: [3729926.256488]  ffffffffc06b4a58 0000000000000000 0000000004772296 0000000000000000
May 24 12:05:12 vps2 kernel: [3729926.256492]  ffffab7701f7bd40 ffffffffc06ff997 ffff9fbb830dbac0 ffffffffc06fdcc4
May 24 12:05:12 vps2 kernel: [3729926.256495] Call Trace:
May 24 12:05:12 vps2 kernel: [3729926.256505]  [<ffffffffb1329cd5>] ? dump_stack+0x5c/0x77
May 24 12:05:12 vps2 kernel: [3729926.256585]  [<ffffffffc06b4a58>] ? xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.256645]  [<ffffffffc06ff997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.256712]  [<ffffffffc06fdcc4>] ? _xfs_buf_ioapply+0x84/0x450 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.256719]  [<ffffffffb10a2b30>] ? wake_up_q+0x60/0x60
May 24 12:05:12 vps2 kernel: [3729926.256781]  [<ffffffffc06ff997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.256845]  [<ffffffffc06ff6d7>] ? xfs_buf_submit+0x67/0x1f0 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.256911]  [<ffffffffc06ff997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.256975]  [<ffffffffc073177b>] ? xfsaild+0x2cb/0x7b0 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.257038]  [<ffffffffc06ff365>] ? xfs_buf_unlock+0x15/0x70 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.257105]  [<ffffffffc073177b>] ? xfsaild+0x2cb/0x7b0 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.257174]  [<ffffffffc07314b0>] ? xfs_trans_ail_cursor_first+0x80/0x80 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.257182]  [<ffffffffb10974c0>] ? kthread+0xe0/0x100
May 24 12:05:12 vps2 kernel: [3729926.257186]  [<ffffffffb102476b>] ? __switch_to+0x2bb/0x700
May 24 12:05:12 vps2 kernel: [3729926.257189]  [<ffffffffb10973e0>] ? kthread_park+0x60/0x60
May 24 12:05:12 vps2 kernel: [3729926.257194]  [<ffffffffb15fb675>] ? ret_from_fork+0x25/0x30
May 24 12:05:12 vps2 kernel: [3729926.257201] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1323 of file /home/zumbi/linux-4.9.13/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc06fe084
May 24 12:05:12 vps2 kernel: [3729926.260093] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
May 24 12:05:12 vps2 kernel: [3729926.262115] CPU: 1 PID: 454 Comm: xfsaild/dm-2 Tainted: G            E   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.13-1~bpo8+1
May 24 12:05:12 vps2 kernel: [3729926.262117] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
May 24 12:05:12 vps2 kernel: [3729926.262120]  0000000000000000 ffffffffb1329cd5 0000000000000008 ffff9fbc275f3000
May 24 12:05:12 vps2 kernel: [3729926.262125]  ffffffffc07061fa ffffffffc06fe084 0000000000000000 ffffab7701f7bd40
May 24 12:05:12 vps2 kernel: [3729926.262129]  ffffffffc06ff997 ffff9fbb830dbac0 ffffffffc06fe084 ffff9fb934f86080
May 24 12:05:12 vps2 kernel: [3729926.262133] Call Trace:
May 24 12:05:12 vps2 kernel: [3729926.262141]  [<ffffffffb1329cd5>] ? dump_stack+0x5c/0x77
May 24 12:05:12 vps2 kernel: [3729926.262228]  [<ffffffffc07061fa>] ? xfs_do_force_shutdown+0x13a/0x140 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262294]  [<ffffffffc06fe084>] ? _xfs_buf_ioapply+0x444/0x450 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262361]  [<ffffffffc06ff997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262427]  [<ffffffffc06fe084>] ? _xfs_buf_ioapply+0x444/0x450 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262433]  [<ffffffffb10a2b30>] ? wake_up_q+0x60/0x60
May 24 12:05:12 vps2 kernel: [3729926.262503]  [<ffffffffc06ff997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262573]  [<ffffffffc06ff6d7>] ? xfs_buf_submit+0x67/0x1f0 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262643]  [<ffffffffc06ff997>] ? xfs_buf_delwri_submit_buffers+0x137/0x260 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262717]  [<ffffffffc073177b>] ? xfsaild+0x2cb/0x7b0 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262789]  [<ffffffffc06ff365>] ? xfs_buf_unlock+0x15/0x70 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262864]  [<ffffffffc073177b>] ? xfsaild+0x2cb/0x7b0 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262939]  [<ffffffffc07314b0>] ? xfs_trans_ail_cursor_first+0x80/0x80 [xfs]
May 24 12:05:12 vps2 kernel: [3729926.262944]  [<ffffffffb10974c0>] ? kthread+0xe0/0x100
May 24 12:05:12 vps2 kernel: [3729926.262948]  [<ffffffffb102476b>] ? __switch_to+0x2bb/0x700
May 24 12:05:12 vps2 kernel: [3729926.262950]  [<ffffffffb10973e0>] ? kthread_park+0x60/0x60
May 24 12:05:12 vps2 kernel: [3729926.262955]  [<ffffffffb15fb675>] ? ret_from_fork+0x25/0x30
May 24 12:05:12 vps2 kernel: [3729926.263078] XFS (dm-2): Please umount the filesystem and rectify the problem(s)


After filesystem shutdown, before reboot i have upgraded kernel to 4.9.25-1~bpo8+1

Customer was calling us, just when nagios alerted us on host problem, that he was copying some files between accounts (which means changing acls), i have asked him for more information.

Libor

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,

On středa 24. května 2017 13:18:47 CEST Libor Klepáč wrote:
> Hi,
> after quiet two months , another crash, trace is same as before.
> There are no XFS lines before this, kern.log goes back to 23. april.
> 
---snip trace----
> 
> After filesystem shutdown, before reboot i have upgraded kernel to 
4.9.25-1~bpo8+1
> 
> Customer was calling us, just when nagios alerted us on host problem, that 
he was copying some files between accounts (which means changing acls), i have 
asked him for more information.

So according to customer, he was on ssh and was copying files from one 
customer account to another dedicated for development, using midnight 
commander.
It involves changing file ACLs.

original dir acl

##getfacl .
# file: .
# owner: cust1
# group: cust1
# flags: -s-
user::rwx
user:backup:--x
user:cust1:rwx
group::rwx
group:www-data:r-x
group:cust1:rwx
mask::rwx
other::---
default:user::rwx
default:user:backup:--x
default:user:cust1:rwx
default:group::rwx
default:group:www-data:r-x
default:group:cust1:rwx
default:mask::rwx
default:other::r-x

destination dir acls
# getfacl .
# file: .
# owner: admin-ssh
# group: www-data
user::rwx
user:backup:--x
group::rwx
group:www-data:rwx
mask::rwx
other::r-x
default:user::rwx
default:user:backup:--x
default:group::rwx
default:group:www-data:rwx
default:mask::rwx
default:other::r-x


----

user admin-ssh is member of www-data group so he can copy from customer dir

Libor

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,
we tried also on vps1 reported here in bottom of email
https://www.spinics.net/lists/linux-xfs/msg01728.html
and vps3 from this email
https://www.spinics.net/lists/linux-xfs/msg02672.html

Both came clean. Does it mean, that corruption was really only in memory 
and did not made it to disks?
Both machines are on 4.8.15 and xfsprogs 4.9.0

#root@vps3 # xfs_repair /dev/mapper/vg2Disk2-lvData
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Note - quota info will be regenerated on next quota mount.
done

---------------------------------
#root@vps1:~# xfs_repair /dev/mapper/vgVPS1Disk2-lvData
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
-------------------------

Thanks, 
Libor

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

On 2/1/17 6:48 AM, Libor Klepáč wrote:
> Hello,
> we tried also on vps1 reported here in bottom of email
> https://www.spinics.net/lists/linux-xfs/msg01728.html
> and vps3 from this email
> https://www.spinics.net/lists/linux-xfs/msg02672.html
> 
> Both came clean. Does it mean, that corruption was really only in memory 
> and did not made it to disks?

The point of the xfs_repair patch was to fix the problem
on-disk.

I'm not sure what steps your machines have been through, so I
can't really speak to their situation, but assuming the repairs
below were with the patch in place, it appears that they are
clean now.

-Eric

> Both machines are on 4.8.15 and xfsprogs 4.9.0
> 
> #root@vps3 # xfs_repair /dev/mapper/vg2Disk2-lvData
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan and clear agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - agno = 15
>         - agno = 16
>         - agno = 17
>         - agno = 18
>         - agno = 19
>         - agno = 20
>         - agno = 21
>         - agno = 22
>         - agno = 23
>         - agno = 24
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - agno = 15
>         - agno = 16
>         - agno = 17
>         - agno = 18
>         - agno = 19
>         - agno = 20
>         - agno = 21
>         - agno = 22
>         - agno = 23
>         - agno = 24
> Phase 5 - rebuild AG headers and trees...
>         - reset superblock...
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify and correct link counts...
> Note - quota info will be regenerated on next quota mount.
> done
> 
> ---------------------------------
> #root@vps1:~# xfs_repair /dev/mapper/vgVPS1Disk2-lvData
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan and clear agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
> Phase 5 - rebuild AG headers and trees...
>         - reset superblock...
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify and correct link counts...
> done
> -------------------------
> 
> Thanks, 
> Libor
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hi,

On středa 1. února 2017 16:49:49 CET Eric Sandeen wrote:
> On 2/1/17 6:48 AM, Libor Klepáč wrote:
> > Hello,
> > we tried also on vps1 reported here in bottom of email
> > https://www.spinics.net/lists/linux-xfs/msg01728.html
> > and vps3 from this email
> > https://www.spinics.net/lists/linux-xfs/msg02672.html
> > 
> > Both came clean. Does it mean, that corruption was really only in memory 
> > and did not made it to disks?
> 
> The point of the xfs_repair patch was to fix the problem
> on-disk.
Yes, i understand that

> 
> I'm not sure what steps your machines have been through, so I
> can't really speak to their situation, but assuming the repairs
> below were with the patch in place, it appears that they are
> clean now.
> 
Ok, seems fair ;)

> -Eric

Thanks,
Libor

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hi,
it happened again on one machine vps3 from last mail, which had clean xfs_repair run
It's running kernel 4.9.0-0.bpo.1-amd64 (so it's 4.9.2) since 6. Feb. It was upgraded from 4.8.15.

Error was
Feb 22 11:04:21 vps3 kernel: [1316281.466922] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0xa000718
Feb 22 11:04:21 vps3 kernel: [1316281.468665] XFS (dm-2): Unmount and run xfs_repair
Feb 22 11:04:21 vps3 kernel: [1316281.469440] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Feb 22 11:04:21 vps3 kernel: [1316281.470212] ffffa06e686ac000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
Feb 22 11:04:21 vps3 kernel: [1316281.470964] ffffa06e686ac010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
Feb 22 11:04:21 vps3 kernel: [1316281.471691] ffffa06e686ac020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Feb 22 11:04:21 vps3 kernel: [1316281.472431] ffffa06e686ac030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Feb 22 11:04:21 vps3 kernel: [1316281.473129] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1322 of file /home/zumbi/linux-4.9.2/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc05e0dc4
Feb 22 11:04:21 vps3 kernel: [1316281.473685] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
Feb 22 11:04:21 vps3 kernel: [1316281.474402] XFS (dm-2): Please umount the filesystem and rectify the problem(s)

After reboot, there was once this:
Feb 22 11:46:41 vps3 kernel: [ 2440.571092] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0xa000718
Feb 22 11:46:41 vps3 kernel: [ 2440.571160] XFS (dm-2): Unmount and run xfs_repair
Feb 22 11:46:41 vps3 kernel: [ 2440.571177] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
Feb 22 11:46:41 vps3 kernel: [ 2440.571198] ffff8c46fdbe5000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
Feb 22 11:46:41 vps3 kernel: [ 2440.571225] ffff8c46fdbe5010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
Feb 22 11:46:41 vps3 kernel: [ 2440.571252] ffff8c46fdbe5020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Feb 22 11:46:41 vps3 kernel: [ 2440.571278] ffff8c46fdbe5030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Feb 22 11:46:41 vps3 kernel: [ 2440.571313] XFS (dm-2): metadata I/O error: block 0xa000718 ("xfs_trans_read_buf_map") error 117 numblks 8

We will run repair tomorrow. Is it worth upgrading xfsprogs from 4.9.0 to 4.10.0-rc1 before repair?

Thanks,
Libor




On středa 1. února 2017 13:48:57 CET Libor Klepáč wrote:
> [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
> 
> Hello,
> we tried also on vps1 reported here in bottom of email
> https://www.spinics.net/lists/linux-xfs/msg01728.html
> and vps3 from this email
> https://www.spinics.net/lists/linux-xfs/msg02672.html
> 
> Both came clean. Does it mean, that corruption was really only in memory
> and did not made it to disks?
> Both machines are on 4.8.15 and xfsprogs 4.9.0
> 
> #root@vps3 # xfs_repair /dev/mapper/vg2Disk2-lvData
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan and clear agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - agno = 15
>         - agno = 16
>         - agno = 17
>         - agno = 18
>         - agno = 19
>         - agno = 20
>         - agno = 21
>         - agno = 22
>         - agno = 23
>         - agno = 24
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - agno = 15
>         - agno = 16
>         - agno = 17
>         - agno = 18
>         - agno = 19
>         - agno = 20
>         - agno = 21
>         - agno = 22
>         - agno = 23
>         - agno = 24
> Phase 5 - rebuild AG headers and trees...
>         - reset superblock...
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify and correct link counts...
> Note - quota info will be regenerated on next quota mount.
> done
> 
> ---------------------------------
> #root@vps1:~# xfs_repair /dev/mapper/vgVPS1Disk2-lvData
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan and clear agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
> Phase 5 - rebuild AG headers and trees...
>         - reset superblock...
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify and correct link counts...
> done
> -------------------------
> 
> Thanks,
> Libor
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


--------
[1] mailto:libor.klepac@bcom.cz
[2] tel:+420377457676
[3] http://www.bcom.cz

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Eric Sandeen]

On 2/22/17 5:42 AM, Libor Klepáč wrote:
> Hi,
> it happened again on one machine vps3 from last mail, which had clean xfs_repair run
> It's running kernel 4.9.0-0.bpo.1-amd64 (so it's 4.9.2) since 6. Feb. It was upgraded from 4.8.15.
> 
> Error was
> Feb 22 11:04:21 vps3 kernel: [1316281.466922] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0xa000718
> Feb 22 11:04:21 vps3 kernel: [1316281.468665] XFS (dm-2): Unmount and run xfs_repair
> Feb 22 11:04:21 vps3 kernel: [1316281.469440] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> Feb 22 11:04:21 vps3 kernel: [1316281.470212] ffffa06e686ac000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> Feb 22 11:04:21 vps3 kernel: [1316281.470964] ffffa06e686ac010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> Feb 22 11:04:21 vps3 kernel: [1316281.471691] ffffa06e686ac020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Feb 22 11:04:21 vps3 kernel: [1316281.472431] ffffa06e686ac030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Feb 22 11:04:21 vps3 kernel: [1316281.473129] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1322 of file /home/zumbi/linux-4.9.2/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc05e0dc4
> Feb 22 11:04:21 vps3 kernel: [1316281.473685] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
> Feb 22 11:04:21 vps3 kernel: [1316281.474402] XFS (dm-2): Please umount the filesystem and rectify the problem(s)

Ok, and what happened to this machine in the meantime?
I don't understand why this has been showing up for you; it'd be
nice to know if anything "interesting" happened prior to this -
any other shutdown and log replay, for example?  Or what
is the workload that's leading to this, if you can tell?

If you run repair and it tells you which inode it is, go find
that inode and see if there is anything "interesting" about its
lifetime or attribute use, perhaps?

> After reboot, there was once this:
> Feb 22 11:46:41 vps3 kernel: [ 2440.571092] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0xa000718
> Feb 22 11:46:41 vps3 kernel: [ 2440.571160] XFS (dm-2): Unmount and run xfs_repair
> Feb 22 11:46:41 vps3 kernel: [ 2440.571177] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> Feb 22 11:46:41 vps3 kernel: [ 2440.571198] ffff8c46fdbe5000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> Feb 22 11:46:41 vps3 kernel: [ 2440.571225] ffff8c46fdbe5010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> Feb 22 11:46:41 vps3 kernel: [ 2440.571252] ffff8c46fdbe5020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Feb 22 11:46:41 vps3 kernel: [ 2440.571278] ffff8c46fdbe5030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> Feb 22 11:46:41 vps3 kernel: [ 2440.571313] XFS (dm-2): metadata I/O error: block 0xa000718 ("xfs_trans_read_buf_map") error 117 numblks 8
> 
> We will run repair tomorrow. Is it worth upgrading xfsprogs from 4.9.0 to 4.10.0-rc1 before repair?

Should be no need, though always happy to have testing.  :)

> Thanks,
> Libor
> 
> 
> 
> 
> On středa 1. února 2017 13:48:57 CET Libor Klepáč wrote:
>> [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
>>
>> Hello,
>> we tried also on vps1 reported here in bottom of email
>> https://www.spinics.net/lists/linux-xfs/msg01728.html
>> and vps3 from this email
>> https://www.spinics.net/lists/linux-xfs/msg02672.html
>>
>> Both came clean. Does it mean, that corruption was really only in memory
>> and did not made it to disks?
>> Both machines are on 4.8.15 and xfsprogs 4.9.0
>>
>> #root@vps3 # xfs_repair /dev/mapper/vg2Disk2-lvData
>> Phase 1 - find and verify superblock...
>> Phase 2 - using internal log
>>         - zero log...
>>         - scan filesystem freespace and inode maps...
>>         - found root inode chunk
>> Phase 3 - for each AG...
>>         - scan and clear agi unlinked lists...
>>         - process known inodes and perform inode discovery...
>>         - agno = 0
>>         - agno = 1
>>         - agno = 2
>>         - agno = 3
>>         - agno = 4
>>         - agno = 5
>>         - agno = 6
>>         - agno = 7
>>         - agno = 8
>>         - agno = 9
>>         - agno = 10
>>         - agno = 11
>>         - agno = 12
>>         - agno = 13
>>         - agno = 14
>>         - agno = 15
>>         - agno = 16
>>         - agno = 17
>>         - agno = 18
>>         - agno = 19
>>         - agno = 20
>>         - agno = 21
>>         - agno = 22
>>         - agno = 23
>>         - agno = 24
>>         - process newly discovered inodes...
>> Phase 4 - check for duplicate blocks...
>>         - setting up duplicate extent list...
>>         - check for inodes claiming duplicate blocks...
>>         - agno = 0
>>         - agno = 1
>>         - agno = 2
>>         - agno = 3
>>         - agno = 4
>>         - agno = 5
>>         - agno = 6
>>         - agno = 7
>>         - agno = 8
>>         - agno = 9
>>         - agno = 10
>>         - agno = 11
>>         - agno = 12
>>         - agno = 13
>>         - agno = 14
>>         - agno = 15
>>         - agno = 16
>>         - agno = 17
>>         - agno = 18
>>         - agno = 19
>>         - agno = 20
>>         - agno = 21
>>         - agno = 22
>>         - agno = 23
>>         - agno = 24
>> Phase 5 - rebuild AG headers and trees...
>>         - reset superblock...
>> Phase 6 - check inode connectivity...
>>         - resetting contents of realtime bitmap and summary inodes
>>         - traversing filesystem ...
>>         - traversal finished ...
>>         - moving disconnected inodes to lost+found ...
>> Phase 7 - verify and correct link counts...
>> Note - quota info will be regenerated on next quota mount.
>> done
>>
>> ---------------------------------
>> #root@vps1:~# xfs_repair /dev/mapper/vgVPS1Disk2-lvData
>> Phase 1 - find and verify superblock...
>> Phase 2 - using internal log
>>         - zero log...
>>         - scan filesystem freespace and inode maps...
>>         - found root inode chunk
>> Phase 3 - for each AG...
>>         - scan and clear agi unlinked lists...
>>         - process known inodes and perform inode discovery...
>>         - agno = 0
>>         - agno = 1
>>         - agno = 2
>>         - agno = 3
>>         - process newly discovered inodes...
>> Phase 4 - check for duplicate blocks...
>>         - setting up duplicate extent list...
>>         - check for inodes claiming duplicate blocks...
>>         - agno = 0
>>         - agno = 1
>>         - agno = 2
>>         - agno = 3
>> Phase 5 - rebuild AG headers and trees...
>>         - reset superblock...
>> Phase 6 - check inode connectivity...
>>         - resetting contents of realtime bitmap and summary inodes
>>         - traversing filesystem ...
>>         - traversal finished ...
>>         - moving disconnected inodes to lost+found ...
>> Phase 7 - verify and correct link counts...
>> done
>> -------------------------
>>
>> Thanks,
>> Libor
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> 
> --------
> [1] mailto:libor.klepac@bcom.cz
> [2] tel:+420377457676
> [3] http://www.bcom.cz
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

On středa 22. února 2017 7:45:38 CET Eric Sandeen wrote:
> On 2/22/17 5:42 AM, Libor Klepáč wrote:
> > Hi,
> > it happened again on one machine vps3 from last mail, which had clean xfs_repair run
> > It's running kernel 4.9.0-0.bpo.1-amd64 (so it's 4.9.2) since 6. Feb. It was upgraded from 4.8.15.
> > 
> > Error was
> > Feb 22 11:04:21 vps3 kernel: [1316281.466922] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0xa000718
> > Feb 22 11:04:21 vps3 kernel: [1316281.468665] XFS (dm-2): Unmount and run xfs_repair
> > Feb 22 11:04:21 vps3 kernel: [1316281.469440] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> > Feb 22 11:04:21 vps3 kernel: [1316281.470212] ffffa06e686ac000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> > Feb 22 11:04:21 vps3 kernel: [1316281.470964] ffffa06e686ac010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> > Feb 22 11:04:21 vps3 kernel: [1316281.471691] ffffa06e686ac020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Feb 22 11:04:21 vps3 kernel: [1316281.472431] ffffa06e686ac030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Feb 22 11:04:21 vps3 kernel: [1316281.473129] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 1322 of file /home/zumbi/linux-4.9.2/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc05e0dc4
> > Feb 22 11:04:21 vps3 kernel: [1316281.473685] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
> > Feb 22 11:04:21 vps3 kernel: [1316281.474402] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
> 
> Ok, and what happened to this machine in the meantime?
> I don't understand why this has been showing up for you; it'd be
> nice to know if anything "interesting" happened prior to this -
> any other shutdown and log replay, for example?  Or what
> is the workload that's leading to this, if you can tell?

Machine was running all the time, only rebooted for kernel upgrade (cleanly) to 4.9.2 on 6 Feb. 

It's a web hosting (apache + php + mysql) vps. All data are on XFS partition, meaning web site data, mysql databases, apache access/error logs.
Group quota is enabled, it's used for web site data, not for mysql or apache logs.

Server is not overloaded, serving just few sites.
But before this problem, it received around 10 thousand hits on one website, webshop created with Prestashop.
Load spiked to 100.
But no other error message than reported one appeared in kernel logs.

> 
> If you run repair and it tells you which inode it is, go find
> that inode and see if there is anything "interesting" about its
> lifetime or attribute use, perhaps?

Ok, will do

> 
> > After reboot, there was once this:
> > Feb 22 11:46:41 vps3 kernel: [ 2440.571092] XFS (dm-2): Metadata corruption detected at xfs_attr3_leaf_read_verify+0x5a/0x100 [xfs], xfs_attr3_leaf block 0xa000718
> > Feb 22 11:46:41 vps3 kernel: [ 2440.571160] XFS (dm-2): Unmount and run xfs_repair
> > Feb 22 11:46:41 vps3 kernel: [ 2440.571177] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
> > Feb 22 11:46:41 vps3 kernel: [ 2440.571198] ffff8c46fdbe5000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
> > Feb 22 11:46:41 vps3 kernel: [ 2440.571225] ffff8c46fdbe5010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
> > Feb 22 11:46:41 vps3 kernel: [ 2440.571252] ffff8c46fdbe5020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Feb 22 11:46:41 vps3 kernel: [ 2440.571278] ffff8c46fdbe5030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> > Feb 22 11:46:41 vps3 kernel: [ 2440.571313] XFS (dm-2): metadata I/O error: block 0xa000718 ("xfs_trans_read_buf_map") error 117 numblks 8
> > 
> > We will run repair tomorrow. Is it worth upgrading xfsprogs from 4.9.0 to 4.10.0-rc1 before repair?
> 
> Should be no need, though always happy to have testing.  :)
> 

Ok, i will prepare package for server

Libor

Re: [PATCH] xfs_repair: junk leaf attribute if count == 0

[Libor Klepáč]

Hello,
so repair did something, in log it says "block 0", is that ok?

Inode=335629253 is now in lost+found - empty dir belonging to user cust1.
When looking at 
u.sfdir2.hdr.parent.i4 = 319041478
from xfs_db before repair, it seemed to be in dir, which is owned by cust2, 
totally unrelated to cust1, is that possible? 
>From our view, it's not possible because of FS rights (homedir of each user has rights 0750)

Second inode=1992635 , repaired by xfs_repair, is directory lost+found

Libor

Commands output follows:

Kernel 4.9.2, xfsprogs 4.10.0-rc1
-----------------------------------------------------------------------------
---- check
# xfs_repair -n /dev/mapper/vgDisk2-lvData
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
agi unlinked bucket 5 is 84933 in ag 20 (inode=335629253)
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
bad attribute count 0 in attr block 0, inode 335629253
problem with attribute contents in inode 335629253
would clear attr fork
bad nblocks 1 for inode 335629253, would reset to 0
bad anextents 1 for inode 335629253, would reset to 0
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
       - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected dir inode 335629253, would move to lost+found
Phase 7 - verify link counts...
would have reset inode 335629253 nlinks from 0 to 2
No modify flag set, skipping filesystem flush and exiting.
-----------------------------------------------------------------------------
---- repair
# xfs_repair  /dev/mapper/vgDisk2-lvData
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
agi unlinked bucket 5 is 84933 in ag 20 (inode=335629253)
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
bad attribute count 0 in attr block 0, inode 335629253
problem with attribute contents in inode 335629253
clearing inode 335629253 attributes
correcting nblocks for inode 335629253, was 1 - counted 0
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
bad attribute format 1 in inode 335629253, resetting value
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected dir inode 335629253, moving to lost+found
Phase 7 - verify and correct link counts...
resetting inode 1992635 nlinks from 2 to 3
resetting inode 335629253 nlinks from 0 to 2
Note - quota info will be regenerated on next quota mount.
Done
-----------------------------------------------------------------------------
---- xfs_db before repair
# xfs_db -r /dev/vgDisk2/lvData
xfs_db> inode 335629253
xfs_db> print
core.magic = 0x494e
core.mode = 040775
core.version = 2
core.format = 1 (local)
core.nlinkv2 = 0
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 10106
core.gid = 10106
core.flushiter = 2
core.atime.sec = Wed Feb 22 11:04:21 2017
core.atime.nsec = 464104444
core.mtime.sec = Wed Feb 22 11:46:41 2017
core.mtime.nsec = 548670485
core.ctime.sec = Wed Feb 22 11:46:41 2017
core.ctime.nsec = 548670485
core.size = 6
core.nblocks = 1
core.extsize = 0
core.nextents = 0
core.naextents = 1
core.forkoff = 15
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1322976790
next_unlinked = null
u.sfdir2.hdr.count = 0
u.sfdir2.hdr.i8count = 0
u.sfdir2.hdr.parent.i4 = 319041478
a.bmx[0] = [startoff,startblock,blockcount,extentflag]
0:[0,20976867,1,0]

-----------------------------------------------------------------------------
---- xfs_db after repair
# xfs_db -r /dev/vgDisk2/lvData
xfs_db> inode 335629253
xfs_db> print
core.magic = 0x494e
core.mode = 040775
core.version = 2
core.format = 1 (local)
core.nlinkv2 = 2
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 10106
core.gid = 10106
core.flushiter = 2
core.atime.sec = Wed Feb 22 11:04:21 2017
core.atime.nsec = 464104444
core.mtime.sec = Wed Feb 22 11:46:41 2017
core.mtime.nsec = 548670485
core.ctime.sec = Wed Feb 22 11:46:41 2017
core.ctime.nsec = 548670485
core.size = 6
core.nblocks = 0
core.extsize = 0
core.nextents = 0
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1322976790
next_unlinked = null
u.sfdir2.hdr.count = 0
u.sfdir2.hdr.i8count = 0
u.sfdir2.hdr.parent.i4 = 1992635
xfs_db> quit