Q: xfs_db: invalid numrecs (*) in bmapbtd block

Q: xfs_db: invalid numrecs (*) in bmapbtd block

[Timofey Titovets]

Good time of day list,
i do regular check fragmentation on my servers and xfs_db show strange erros:

# xfs_db -c frag -r /dev/sdc1
invalid numrecs (51914) in bmapbtd block
invalid numrecs (2827) in bmapbtd block
invalid numrecs (65419) in bmapbtd block
invalid numrecs (9047) in bmapbtd block
invalid numrecs (26469) in bmapbtd block
invalid numrecs (17784) in bmapbtd block
invalid numrecs (17784) in bmapbtd block
invalid numrecs (13240) in bmapbtd block
invalid numrecs (60489) in bmapbtd block
invalid numrecs (23478) in bmapbtd block
invalid numrecs (24767) in bmapbtd block
invalid numrecs (33180) in bmapbtd block
invalid numrecs (25257) in bmapbtd block
invalid numrecs (20648) in bmapbtd block
invalid numrecs (23136) in bmapbtd block
invalid numrecs (25282) in bmapbtd block
invalid numrecs (24743) in bmapbtd block
invalid numrecs (56229) in bmapbtd block
invalid numrecs (42902) in bmapbtd block
invalid numrecs (12316) in bmapbtd block
invalid numrecs (1588) in bmapbtd block
invalid numrecs (7065) in bmapbtd block
invalid numrecs (65280) in bmapbtd block
invalid numrecs (3378) in bmapbtd block
invalid numrecs (61507) in bmapbtd block
invalid numrecs (56024) in bmapbtd block
invalid numrecs (25390) in bmapbtd block
invalid numrecs (25942) in bmapbtd block
invalid numrecs (12134) in bmapbtd block
invalid numrecs (26991) in bmapbtd block
invalid numrecs (43528) in bmapbtd block
invalid numrecs (10265) in bmapbtd block
invalid numrecs (44698) in bmapbtd block
invalid numrecs (15937) in bmapbtd block
actual 504454, ideal 126776, fragmentation factor 74.87%

I already google error, and can't find any answer or explanation what
is it, how danger it or something like.
It's a production servers with several TB of data and this output make
me unhappy. =.=
And i can't do fast stop it, unmount and check file system.
(this is nfs server for virtual machines images)

Thanks for any help, anyway.
-- 
Have a nice day,
Timofey.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Q: xfs_db: invalid numrecs (*) in bmapbtd block

[Eric Sandeen]

On 6/28/15 1:44 PM, Timofey Titovets wrote:
> Good time of day list,
> i do regular check fragmentation on my servers and xfs_db show strange erros:

No need, really.

http://xfs.org/index.php/XFS_FAQ#Q:_The_xfs_db_.22frag.22_command_says_I.27m_over_50.25._Is_that_bad.3F

> # xfs_db -c frag -r /dev/sdc1
> invalid numrecs (51914) in bmapbtd block
> invalid numrecs (2827) in bmapbtd block

...

> actual 504454, ideal 126776, fragmentation factor 74.87%


 
> I already google error, and can't find any answer or explanation what
> is it, how danger it or something like.
> It's a production servers with several TB of data and this output make
> me unhappy. =.=

does xfs_repair find any errors?

> And i can't do fast stop it, unmount and check file system.
> (this is nfs server for virtual machines images)

Oh.  So you're doing it with an online/mounted filesystem?

The on-disk filesystem isn't necessarily consistent when mounted,
some of that consistency lives in the log.

You're likely just seeing that inconsitency because you're reading
it while it's mounted; possibly even while it's being modified.

-Eric

> Thanks for any help, anyway.
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Q: xfs_db: invalid numrecs (*) in bmapbtd block

[Timofey Titovets]

2015-06-29 1:12 GMT+03:00 Eric Sandeen <sandeen@sandeen.net>:
> On 6/28/15 1:44 PM, Timofey Titovets wrote:
>> Good time of day list,
>> i do regular check fragmentation on my servers and xfs_db show strange erros:
>
> No need, really.
>
> http://xfs.org/index.php/XFS_FAQ#Q:_The_xfs_db_.22frag.22_command_says_I.27m_over_50.25._Is_that_bad.3F

I just collects metrics for Zabbix monitoring.

>> # xfs_db -c frag -r /dev/sdc1
>> invalid numrecs (51914) in bmapbtd block
>> invalid numrecs (2827) in bmapbtd block
>
> ...
>
>> actual 504454, ideal 126776, fragmentation factor 74.87%
>
>
>
>> I already google error, and can't find any answer or explanation what
>> is it, how danger it or something like.
>> It's a production servers with several TB of data and this output make
>> me unhappy. =.=
>
> does xfs_repair find any errors?

i can't run xfs_repair on mounted fs as you know =)

>> And i can't do fast stop it, unmount and check file system.
>> (this is nfs server for virtual machines images)
>
> Oh.  So you're doing it with an online/mounted filesystem?

Yes, it must be not a problem.
(This error happen only on 2 from 9 servers, after many month of
stable working, others seems in good state)

> The on-disk filesystem isn't necessarily consistent when mounted,
> some of that consistency lives in the log.
>
>
> You're likely just seeing that inconsitency because you're reading
> it while it's mounted; possibly even while it's being modified.
>
> -Eric

I already thinked about this case, and it seems incredible.
1. i can get consistent or very close to consistent fs, by run 'sync'
2. Okay, supposing you are right,
then i must get different errors over time, because fs will change own
state over time (also after every sync call).
But its don't happen, i get same errors over long period of time (1
day+, i think on busy FS server, its enoug to think what data on disk
has changed)
Also, i have several others server, and on it i didn't see this
problem (over 2-3 month).


Then I can conclude this is errors in FS, only what matter to me now,
this type of errors can damage my data and i must move virtual
machines from it ASAP, or i can continue working and this is just
"artefact".

Fix me if i'm wrong.

>> Thanks for any help, anyway.
>>
>

-- 
Have a nice day,
Timofey.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Q: xfs_db: invalid numrecs (*) in bmapbtd block

[Dave Chinner]

On Sun, Jun 28, 2015 at 09:44:20PM +0300, Timofey Titovets wrote:
> Good time of day list,
> i do regular check fragmentation on my servers and xfs_db show strange erros:
> 
> # xfs_db -c frag -r /dev/sdc1
> invalid numrecs (51914) in bmapbtd block

That's expected behaviour if you are running that on a mounted
filesystem. I'm assuming this is the case because you are telling
xfs_db to run in read only mode...

Why? Because an actively modified filesystem is only coherent on
disk when you consider all the modifications staged in the journal,
and xfs_db ignores the journal. Hence what it reads of disk is not
coherent and hence is not guaranteed valid.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Q: xfs_db: invalid numrecs (*) in bmapbtd block

[Timofey Titovets]

2015-06-29 3:24 GMT+03:00 Dave Chinner <david@fromorbit.com>:
> On Sun, Jun 28, 2015 at 09:44:20PM +0300, Timofey Titovets wrote:
>> Good time of day list,
>> i do regular check fragmentation on my servers and xfs_db show strange erros:
>>
>> # xfs_db -c frag -r /dev/sdc1
>> invalid numrecs (51914) in bmapbtd block
>
> That's expected behaviour if you are running that on a mounted
> filesystem. I'm assuming this is the case because you are telling
> xfs_db to run in read only mode...
>
> Why? Because an actively modified filesystem is only coherent on
> disk when you consider all the modifications staged in the journal,
> and xfs_db ignores the journal. Hence what it reads of disk is not
> coherent and hence is not guaranteed valid.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com


Big Thanks Dave for explanation.
=^_^=

-- 
Have a nice day,
Timofey.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Q: xfs_db: invalid numrecs (*) in bmapbtd block

[Eric Sandeen]

On 6/28/15 5:29 PM, Timofey Titovets wrote:
> 2015-06-29 1:12 GMT+03:00 Eric Sandeen <sandeen@sandeen.net>:

...

>> You're likely just seeing that inconsitency because you're reading
>> it while it's mounted; possibly even while it's being modified.
>>
>> -Eric
> 
> I already thinked about this case, and it seems incredible.
> 1. i can get consistent or very close to consistent fs, by run 'sync'
> 2. Okay, supposing you are right,
> then i must get different errors over time, because fs will change own
> state over time (also after every sync call).
> But its don't happen, i get same errors over long period of time (1
> day+, i think on busy FS server, its enoug to think what data on disk
> has changed)
> Also, i have several others server, and on it i didn't see this
> problem (over 2-3 month).
> 
> 
> Then I can conclude this is errors in FS, only what matter to me now,
> this type of errors can damage my data and i must move virtual
> machines from it ASAP, or i can continue working and this is just
> "artefact".
> 
> Fix me if i'm wrong.

Your other option is to unmount and run xfs_repair, and find out if
you actually have on-disk corruption.  It's the only way to know
for sure (or: freeze the fs, snapshot the device, and run repair on that)

-Eric

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs