XFS umount issue

XFS umount issue

[Nuno Subtil]

I have an MD RAID-1 array with two SATA drives, formatted as XFS.
Occasionally, doing an umount followed by a mount causes the mount to
fail with errors that strongly suggest some sort of filesystem
corruption (usually 'bad clientid' with a seemingly arbitrary ID, but
occasionally invalid log errors as well).

The one thing in common among all these failures is that they require
xfs_repair -L to recover from. This has already caused a few
lost+found entries (and data loss on recently written files). I
originally noticed this bug because of mount failures at boot, but
I've managed to repro it reliably with this script:

while true; do
	mount /store
	(cd /store && tar xf test.tar)
	umount /store
	mount /store
	rm -rf /store/test-data
	umount /store
done

test.tar contains around 100 files with various sizes inside
test-data/, ranging from a few hundred KB to around 5-6MB. The failure
triggers within minutes of starting this loop.

I'm not entirely sure that this is XFS-specific, but the same script
does run successfully overnight on the same MD array with ext3 on it.
This is on an ARM system running kernel 2.6.39.

Has something like this been seen before?

Thanks,
Nuno

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Dave Chinner]

On Mon, May 23, 2011 at 02:39:39PM -0700, Nuno Subtil wrote:
> I have an MD RAID-1 array with two SATA drives, formatted as XFS.

Hi Nuno. it is probably best to say this at the start, too:

> This is on an ARM system running kernel 2.6.39.

So we know what platform this is occurring on.

> Occasionally, doing an umount followed by a mount causes the mount to
> fail with errors that strongly suggest some sort of filesystem
> corruption (usually 'bad clientid' with a seemingly arbitrary ID, but
> occasionally invalid log errors as well).

So reading back the journal is getting bad data?
> 
> The one thing in common among all these failures is that they require
> xfs_repair -L to recover from. This has already caused a few
> lost+found entries (and data loss on recently written files). I
> originally noticed this bug because of mount failures at boot, but
> I've managed to repro it reliably with this script:

Yup, that's normal with recovery errors.

> while true; do
> 	mount /store
> 	(cd /store && tar xf test.tar)
> 	umount /store
> 	mount /store
> 	rm -rf /store/test-data
> 	umount /store
> done

Ok, so there's nothing here that actually says it's an unmount
error. More likely it is a vmap problem in log recovery resulting in
aliasing or some other stale data appearing in the buffer pages.

Can you add a 'xfs_logprint -t <device>' after the umount? You
should always see something like this telling you the log is clean:

$ xfs_logprint -t /dev/vdb
xfs_logprint:
    data device: 0xfd10
    log device: 0xfd10 daddr: 11534368 length: 20480

    log tail: 51 head: 51 state: <CLEAN>

If the log is not clean on an unmount, then you may have an unmount
problem. If it is clean when the recovery error occurs, then it's
almost certainly a problem with you platform not implementing vmap
cache flushing correctly, not an XFS problem.

> I'm not entirely sure that this is XFS-specific, but the same script
> does run successfully overnight on the same MD array with ext3 on it.

ext3 doesn't use vmapped buffers at all, so won't show such a
proble,.

> Has something like this been seen before?

Every so often on ARM, MIPS, etc platforms that have virtually
indexed caches.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Nuno Subtil]

Thanks for chiming in. Replies inline below:

On Mon, May 23, 2011 at 17:02, Dave Chinner <david@fromorbit.com> wrote:
> On Mon, May 23, 2011 at 02:39:39PM -0700, Nuno Subtil wrote:
>> I have an MD RAID-1 array with two SATA drives, formatted as XFS.
>
> Hi Nuno. it is probably best to say this at the start, too:
>
>> This is on an ARM system running kernel 2.6.39.
>
> So we know what platform this is occurring on.

Will keep that in mind. Thanks.

>
>> Occasionally, doing an umount followed by a mount causes the mount to
>> fail with errors that strongly suggest some sort of filesystem
>> corruption (usually 'bad clientid' with a seemingly arbitrary ID, but
>> occasionally invalid log errors as well).
>
> So reading back the journal is getting bad data?

I'm not sure. XFS claims it found a bad clientid. I'm not too versed
in filesystems to be able to tell for myself :)

>>
>> The one thing in common among all these failures is that they require
>> xfs_repair -L to recover from. This has already caused a few
>> lost+found entries (and data loss on recently written files). I
>> originally noticed this bug because of mount failures at boot, but
>> I've managed to repro it reliably with this script:
>
> Yup, that's normal with recovery errors.
>
>> while true; do
>>       mount /store
>>       (cd /store && tar xf test.tar)
>>       umount /store
>>       mount /store
>>       rm -rf /store/test-data
>>       umount /store
>> done
>
> Ok, so there's nothing here that actually says it's an unmount
> error. More likely it is a vmap problem in log recovery resulting in
> aliasing or some other stale data appearing in the buffer pages.
>
> Can you add a 'xfs_logprint -t <device>' after the umount? You
> should always see something like this telling you the log is clean:

Well, I just ran into this again even without using the script:

root@howl:/# umount /dev/md5
root@howl:/# xfs_logprint -t /dev/md5
xfs_logprint:
    data device: 0x905
    log device: 0x905 daddr: 488382880 length: 476936

    log tail: 731 head: 859 state: <DIRTY>


LOG REC AT LSN cycle 1 block 731 (0x1, 0x2db)

LOG REC AT LSN cycle 1 block 795 (0x1, 0x31b)

I see nothing in dmesg at umount time. Attempting to mount the device
at this point, I got:

[  764.516319] XFS (md5): Mounting Filesystem
[  764.601082] XFS (md5): Starting recovery (logdev: internal)
[  764.626294] XFS (md5): xlog_recover_process_data: bad clientid 0x0
[  764.632559] XFS (md5): log mount/recovery failed: error 5
[  764.638151] XFS (md5): log mount failed

Based on your description, this would be an unmount problem rather
than a vmap problem?

I've tried adding a sync before each umount, as well as testing on a
plain old disk partition (i.e., without going through MD), but the
problem persists either way.

Thanks,
Nuno

>
> $ xfs_logprint -t /dev/vdb
> xfs_logprint:
>    data device: 0xfd10
>    log device: 0xfd10 daddr: 11534368 length: 20480
>
>    log tail: 51 head: 51 state: <CLEAN>
>
> If the log is not clean on an unmount, then you may have an unmount
> problem. If it is clean when the recovery error occurs, then it's
> almost certainly a problem with you platform not implementing vmap
> cache flushing correctly, not an XFS problem.
>
>> I'm not entirely sure that this is XFS-specific, but the same script
>> does run successfully overnight on the same MD array with ext3 on it.
>
> ext3 doesn't use vmapped buffers at all, so won't show such a
> proble,.
>
>> Has something like this been seen before?
>
> Every so often on ARM, MIPS, etc platforms that have virtually
> indexed caches.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Dave Chinner]

On Mon, May 23, 2011 at 11:29:19PM -0700, Nuno Subtil wrote:
> Thanks for chiming in. Replies inline below:
> 
> On Mon, May 23, 2011 at 17:02, Dave Chinner <david@fromorbit.com> wrote:
> > On Mon, May 23, 2011 at 02:39:39PM -0700, Nuno Subtil wrote:
> >> I have an MD RAID-1 array with two SATA drives, formatted as XFS.
> >
> > Hi Nuno. it is probably best to say this at the start, too:
> >
> >> This is on an ARM system running kernel 2.6.39.
> >
> > So we know what platform this is occurring on.
> 
> Will keep that in mind. Thanks.
> 
> >
> >> Occasionally, doing an umount followed by a mount causes the mount to
> >> fail with errors that strongly suggest some sort of filesystem
> >> corruption (usually 'bad clientid' with a seemingly arbitrary ID, but
> >> occasionally invalid log errors as well).
> >
> > So reading back the journal is getting bad data?
> 
> I'm not sure. XFS claims it found a bad clientid. I'm not too versed
> in filesystems to be able to tell for myself :)
> 
> >>
> >> The one thing in common among all these failures is that they require
> >> xfs_repair -L to recover from. This has already caused a few
> >> lost+found entries (and data loss on recently written files). I
> >> originally noticed this bug because of mount failures at boot, but
> >> I've managed to repro it reliably with this script:
> >
> > Yup, that's normal with recovery errors.
> >
> >> while true; do
> >>       mount /store
> >>       (cd /store && tar xf test.tar)
> >>       umount /store
> >>       mount /store
> >>       rm -rf /store/test-data
> >>       umount /store
> >> done
> >
> > Ok, so there's nothing here that actually says it's an unmount
> > error. More likely it is a vmap problem in log recovery resulting in
> > aliasing or some other stale data appearing in the buffer pages.
> >
> > Can you add a 'xfs_logprint -t <device>' after the umount? You
> > should always see something like this telling you the log is clean:
> 
> Well, I just ran into this again even without using the script:
> 
> root@howl:/# umount /dev/md5
> root@howl:/# xfs_logprint -t /dev/md5
> xfs_logprint:
>     data device: 0x905
>     log device: 0x905 daddr: 488382880 length: 476936
> 
>     log tail: 731 head: 859 state: <DIRTY>
> 
> 
> LOG REC AT LSN cycle 1 block 731 (0x1, 0x2db)
> 
> LOG REC AT LSN cycle 1 block 795 (0x1, 0x31b)

Was there any other output? If there were valid transactions between
the head and tail of the log xfs_logprint should have decoded them.

> I see nothing in dmesg at umount time. Attempting to mount the device
> at this point, I got:
> 
> [  764.516319] XFS (md5): Mounting Filesystem
> [  764.601082] XFS (md5): Starting recovery (logdev: internal)
> [  764.626294] XFS (md5): xlog_recover_process_data: bad clientid 0x0

Yup, that's got bad information in a transaction header.

> [  764.632559] XFS (md5): log mount/recovery failed: error 5
> [  764.638151] XFS (md5): log mount failed
> 
> Based on your description, this would be an unmount problem rather
> than a vmap problem?

Not clear yet. I forgot to mention that you need to do

# echo 3 > /proc/sys/vm/drop_caches

before you run xfs_logprint, otherwise it will see stale cached
pages and give erroneous results..

You might want to find out if your platform needs to (and does)
implement these functions:

flush_kernel_dcache_page()
flush_kernel_vmap_range()
void invalidate_kernel_vmap_range()

as these are what XFS relies on platforms to implement correctly to
avoid cache aliasing issues on CPUs with virtually indexed caches.

> I've tried adding a sync before each umount, as well as testing on a
> plain old disk partition (i.e., without going through MD), but the
> problem persists either way.

The use of sync before unmount implies it is not an unmount problem,
and ruling out MD is also a good thing to know.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Nuno Subtil]

On Tue, May 24, 2011 at 00:54, Dave Chinner <david@fromorbit.com> wrote:

...

>> > Ok, so there's nothing here that actually says it's an unmount
>> > error. More likely it is a vmap problem in log recovery resulting in
>> > aliasing or some other stale data appearing in the buffer pages.
>> >
>> > Can you add a 'xfs_logprint -t <device>' after the umount? You
>> > should always see something like this telling you the log is clean:
>>
>> Well, I just ran into this again even without using the script:
>>
>> root@howl:/# umount /dev/md5
>> root@howl:/# xfs_logprint -t /dev/md5
>> xfs_logprint:
>>     data device: 0x905
>>     log device: 0x905 daddr: 488382880 length: 476936
>>
>>     log tail: 731 head: 859 state: <DIRTY>
>>
>>
>> LOG REC AT LSN cycle 1 block 731 (0x1, 0x2db)
>>
>> LOG REC AT LSN cycle 1 block 795 (0x1, 0x31b)
>
> Was there any other output? If there were valid transactions between
> the head and tail of the log xfs_logprint should have decoded them.

There was no more output here.

>
>> I see nothing in dmesg at umount time. Attempting to mount the device
>> at this point, I got:
>>
>> [  764.516319] XFS (md5): Mounting Filesystem
>> [  764.601082] XFS (md5): Starting recovery (logdev: internal)
>> [  764.626294] XFS (md5): xlog_recover_process_data: bad clientid 0x0
>
> Yup, that's got bad information in a transaction header.
>
>> [  764.632559] XFS (md5): log mount/recovery failed: error 5
>> [  764.638151] XFS (md5): log mount failed
>>
>> Based on your description, this would be an unmount problem rather
>> than a vmap problem?
>
> Not clear yet. I forgot to mention that you need to do
>
> # echo 3 > /proc/sys/vm/drop_caches
>
> before you run xfs_logprint, otherwise it will see stale cached
> pages and give erroneous results..

I added that before each xfs_logprint and ran the script again. Still
the same results:

...
+ mount /store
+ cd /store
+ tar xf test.tar
+ sync
+ umount /store
+ echo 3
+ xfs_logprint -t /dev/sda1
xfs_logprint:
    data device: 0x801
    log device: 0x801 daddr: 488384032 length: 476936

    log tail: 2048 head: 2176 state: <DIRTY>


LOG REC AT LSN cycle 1 block 2048 (0x1, 0x800)

LOG REC AT LSN cycle 1 block 2112 (0x1, 0x840)
+ mount /store
mount: /dev/sda1: can't read superblock

Same messages in dmesg at this point.

> You might want to find out if your platform needs to (and does)
> implement these functions:
>
> flush_kernel_dcache_page()
> flush_kernel_vmap_range()
> void invalidate_kernel_vmap_range()
>
> as these are what XFS relies on platforms to implement correctly to
> avoid cache aliasing issues on CPUs with virtually indexed caches.

Is this what /proc/sys/vm/drop_caches relies on as well?

flush_kernel_dcache_page is empty, the others are not but are
conditionalized on the type of cache that is present. I wonder if that
is somehow not being detected properly. Wouldn't that cause other
areas of the system to misbehave as well?

Nuno

>
>> I've tried adding a sync before each umount, as well as testing on a
>> plain old disk partition (i.e., without going through MD), but the
>> problem persists either way.
>
> The use of sync before unmount implies it is not an unmount problem,
> and ruling out MD is also a good thing to know.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Paul Anderson]

Hi Nuno - can you elaborate on the ARM hardware?  I noticed that my
XFS on ARM was mildly unstable, but felt it wasn't XFS code, but
rather the ARM port of Linux.  My test case is a Seagate Dockstar
hacked to run Linux.

I'll see if I can update to the latest kernel and test this use case
as well - it would be interesting to see how well it works (I'd like
to run my Dockstar as a mythtv server - stability was good enough for
proof of concept, but not longer term use).

Thanks,

Paul

On Mon, May 23, 2011 at 5:39 PM, Nuno Subtil <subtil@gmail.com> wrote:
> I have an MD RAID-1 array with two SATA drives, formatted as XFS.
> Occasionally, doing an umount followed by a mount causes the mount to
> fail with errors that strongly suggest some sort of filesystem
> corruption (usually 'bad clientid' with a seemingly arbitrary ID, but
> occasionally invalid log errors as well).
>
> The one thing in common among all these failures is that they require
> xfs_repair -L to recover from. This has already caused a few
> lost+found entries (and data loss on recently written files). I
> originally noticed this bug because of mount failures at boot, but
> I've managed to repro it reliably with this script:
>
> while true; do
>        mount /store
>        (cd /store && tar xf test.tar)
>        umount /store
>        mount /store
>        rm -rf /store/test-data
>        umount /store
> done
>
> test.tar contains around 100 files with various sizes inside
> test-data/, ranging from a few hundred KB to around 5-6MB. The failure
> triggers within minutes of starting this loop.
>
> I'm not entirely sure that this is XFS-specific, but the same script
> does run successfully overnight on the same MD array with ext3 on it.
> This is on an ARM system running kernel 2.6.39.
>
> Has something like this been seen before?
>
> Thanks,
> Nuno
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Nuno Subtil]

On Tue, May 24, 2011 at 06:33, Paul Anderson <pha@umich.edu> wrote:
> Hi Nuno - can you elaborate on the ARM hardware?  I noticed that my
> XFS on ARM was mildly unstable, but felt it wasn't XFS code, but
> rather the ARM port of Linux.  My test case is a Seagate Dockstar
> hacked to run Linux.

Mine is a Netgear Stora. The interesting bit is that the stock
firmware runs kernel 2.6.22.18 and uses XFS as well, but I don't know
how stable it was to begin with.

>
> I'll see if I can update to the latest kernel and test this use case
> as well - it would be interesting to see how well it works (I'd like
> to run my Dockstar as a mythtv server - stability was good enough for
> proof of concept, but not longer term use).
>
> Thanks,
>
> Paul
>
> On Mon, May 23, 2011 at 5:39 PM, Nuno Subtil <subtil@gmail.com> wrote:
>> I have an MD RAID-1 array with two SATA drives, formatted as XFS.
>> Occasionally, doing an umount followed by a mount causes the mount to
>> fail with errors that strongly suggest some sort of filesystem
>> corruption (usually 'bad clientid' with a seemingly arbitrary ID, but
>> occasionally invalid log errors as well).
>>
>> The one thing in common among all these failures is that they require
>> xfs_repair -L to recover from. This has already caused a few
>> lost+found entries (and data loss on recently written files). I
>> originally noticed this bug because of mount failures at boot, but
>> I've managed to repro it reliably with this script:
>>
>> while true; do
>>        mount /store
>>        (cd /store && tar xf test.tar)
>>        umount /store
>>        mount /store
>>        rm -rf /store/test-data
>>        umount /store
>> done
>>
>> test.tar contains around 100 files with various sizes inside
>> test-data/, ranging from a few hundred KB to around 5-6MB. The failure
>> triggers within minutes of starting this loop.
>>
>> I'm not entirely sure that this is XFS-specific, but the same script
>> does run successfully overnight on the same MD array with ext3 on it.
>> This is on an ARM system running kernel 2.6.39.
>>
>> Has something like this been seen before?
>>
>> Thanks,
>> Nuno
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
>>
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Dave Chinner]

On Tue, May 24, 2011 at 03:18:11AM -0700, Nuno Subtil wrote:
> On Tue, May 24, 2011 at 00:54, Dave Chinner <david@fromorbit.com> wrote:
> 
> ...
> 
> >> > Ok, so there's nothing here that actually says it's an unmount
> >> > error. More likely it is a vmap problem in log recovery resulting in
> >> > aliasing or some other stale data appearing in the buffer pages.
> >> >
> >> > Can you add a 'xfs_logprint -t <device>' after the umount? You
> >> > should always see something like this telling you the log is clean:
> >>
> >> Well, I just ran into this again even without using the script:
> >>
> >> root@howl:/# umount /dev/md5
> >> root@howl:/# xfs_logprint -t /dev/md5
> >> xfs_logprint:
> >>     data device: 0x905
> >>     log device: 0x905 daddr: 488382880 length: 476936
> >>
> >>     log tail: 731 head: 859 state: <DIRTY>
> >>
> >>
> >> LOG REC AT LSN cycle 1 block 731 (0x1, 0x2db)
> >>
> >> LOG REC AT LSN cycle 1 block 795 (0x1, 0x31b)
> >
> > Was there any other output? If there were valid transactions between
> > the head and tail of the log xfs_logprint should have decoded them.
> 
> There was no more output here.

That doesn't seem quite right. Does it always look like sthis, even
if you do a sync before unmount?

> >> I see nothing in dmesg at umount time. Attempting to mount the device
> >> at this point, I got:
> >>
> >> [  764.516319] XFS (md5): Mounting Filesystem
> >> [  764.601082] XFS (md5): Starting recovery (logdev: internal)
> >> [  764.626294] XFS (md5): xlog_recover_process_data: bad clientid 0x0
> >
> > Yup, that's got bad information in a transaction header.
> >
> >> [  764.632559] XFS (md5): log mount/recovery failed: error 5
> >> [  764.638151] XFS (md5): log mount failed
> >>
> >> Based on your description, this would be an unmount problem rather
> >> than a vmap problem?
> >
> > Not clear yet. I forgot to mention that you need to do
> >
> > # echo 3 > /proc/sys/vm/drop_caches
> >
> > before you run xfs_logprint, otherwise it will see stale cached
> > pages and give erroneous results..
> 
> I added that before each xfs_logprint and ran the script again. Still
> the same results:
> 
> ...
> + mount /store
> + cd /store
> + tar xf test.tar
> + sync
> + umount /store
> + echo 3
> + xfs_logprint -t /dev/sda1
> xfs_logprint:
>     data device: 0x801
>     log device: 0x801 daddr: 488384032 length: 476936
> 
>     log tail: 2048 head: 2176 state: <DIRTY>
> 
> 
> LOG REC AT LSN cycle 1 block 2048 (0x1, 0x800)
> 
> LOG REC AT LSN cycle 1 block 2112 (0x1, 0x840)
> + mount /store
> mount: /dev/sda1: can't read superblock
> 
> Same messages in dmesg at this point.
> 
> > You might want to find out if your platform needs to (and does)
> > implement these functions:
> >
> > flush_kernel_dcache_page()
> > flush_kernel_vmap_range()
> > void invalidate_kernel_vmap_range()
> >
> > as these are what XFS relies on platforms to implement correctly to
> > avoid cache aliasing issues on CPUs with virtually indexed caches.
> 
> Is this what /proc/sys/vm/drop_caches relies on as well?

No, drop_caches frees the page cache and slab caches so future reads
need to be looked up from disk.

> flush_kernel_dcache_page is empty, the others are not but are
> conditionalized on the type of cache that is present. I wonder if that
> is somehow not being detected properly. Wouldn't that cause other
> areas of the system to misbehave as well?

vmap is not widely used throughout the kernel, and as a result
people porting linux to a new arch/CPU type often don't realise
there's anything to implement there because their system seems to be
working. That is, of course, until someone tries to use XFS.....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Dave Chinner]

On Tue, May 24, 2011 at 12:10:44PM -0700, Nuno Subtil wrote:
> On Tue, May 24, 2011 at 06:33, Paul Anderson <pha@umich.edu> wrote:
> > Hi Nuno - can you elaborate on the ARM hardware?  I noticed that my
> > XFS on ARM was mildly unstable, but felt it wasn't XFS code, but
> > rather the ARM port of Linux.  My test case is a Seagate Dockstar
> > hacked to run Linux.
> 
> Mine is a Netgear Stora. The interesting bit is that the stock
> firmware runs kernel 2.6.22.18 and uses XFS as well, but I don't know
> how stable it was to begin with.

Have you checked to see whether there are extra patches added to
that kernel by Netgear? It's not uncommon for these embedded systems
to run a kernel that has been patched to fix problems that you are
seeing.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Nuno Subtil]

On Tue, May 24, 2011 at 17:29, Dave Chinner <david@fromorbit.com> wrote:
...
>> Mine is a Netgear Stora. The interesting bit is that the stock
>> firmware runs kernel 2.6.22.18 and uses XFS as well, but I don't know
>> how stable it was to begin with.
>
> Have you checked to see whether there are extra patches added to
> that kernel by Netgear? It's not uncommon for these embedded systems
> to run a kernel that has been patched to fix problems that you are
> seeing.

I looked through it and didn't see anything that stood out, although I
could have easily missed it (the diff is quite noisy, plus it sounds
like the vmap cache invalidation functions have changed names in the
meantime?).

The one interesting bit that I found was that the Netgear kernel
comments out the test that disables write barriers at mount time,
which I found quite odd, but it sounds unrelated to this issue.

Nuno

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS umount issue

[Nuno Subtil]

On Tue, May 24, 2011 at 16:39, Dave Chinner <david@fromorbit.com> wrote:
> On Tue, May 24, 2011 at 03:18:11AM -0700, Nuno Subtil wrote:
>> On Tue, May 24, 2011 at 00:54, Dave Chinner <david@fromorbit.com> wrote:
>>
>> ...
>>
>> >> > Ok, so there's nothing here that actually says it's an unmount
>> >> > error. More likely it is a vmap problem in log recovery resulting in
>> >> > aliasing or some other stale data appearing in the buffer pages.
>> >> >
>> >> > Can you add a 'xfs_logprint -t <device>' after the umount? You
>> >> > should always see something like this telling you the log is clean:
>> >>
>> >> Well, I just ran into this again even without using the script:
>> >>
>> >> root@howl:/# umount /dev/md5
>> >> root@howl:/# xfs_logprint -t /dev/md5
>> >> xfs_logprint:
>> >>     data device: 0x905
>> >>     log device: 0x905 daddr: 488382880 length: 476936
>> >>
>> >>     log tail: 731 head: 859 state: <DIRTY>
>> >>
>> >>
>> >> LOG REC AT LSN cycle 1 block 731 (0x1, 0x2db)
>> >>
>> >> LOG REC AT LSN cycle 1 block 795 (0x1, 0x31b)
>> >
>> > Was there any other output? If there were valid transactions between
>> > the head and tail of the log xfs_logprint should have decoded them.
>>
>> There was no more output here.
>
> That doesn't seem quite right. Does it always look like sthis, even
> if you do a sync before unmount?

Not always, but almost. Sometimes there's a number of transactions in
the log as well, but this is by far the most common output I got. I'll
try to capture the output for that case as well.

>> >> I see nothing in dmesg at umount time. Attempting to mount the device
>> >> at this point, I got:
>> >>
>> >> [  764.516319] XFS (md5): Mounting Filesystem
>> >> [  764.601082] XFS (md5): Starting recovery (logdev: internal)
>> >> [  764.626294] XFS (md5): xlog_recover_process_data: bad clientid 0x0
>> >
>> > Yup, that's got bad information in a transaction header.
>> >
>> >> [  764.632559] XFS (md5): log mount/recovery failed: error 5
>> >> [  764.638151] XFS (md5): log mount failed
>> >>
>> >> Based on your description, this would be an unmount problem rather
>> >> than a vmap problem?
>> >
>> > Not clear yet. I forgot to mention that you need to do
>> >
>> > # echo 3 > /proc/sys/vm/drop_caches
>> >
>> > before you run xfs_logprint, otherwise it will see stale cached
>> > pages and give erroneous results..
>>
>> I added that before each xfs_logprint and ran the script again. Still
>> the same results:
>>
>> ...
>> + mount /store
>> + cd /store
>> + tar xf test.tar
>> + sync
>> + umount /store
>> + echo 3
>> + xfs_logprint -t /dev/sda1
>> xfs_logprint:
>>     data device: 0x801
>>     log device: 0x801 daddr: 488384032 length: 476936
>>
>>     log tail: 2048 head: 2176 state: <DIRTY>
>>
>>
>> LOG REC AT LSN cycle 1 block 2048 (0x1, 0x800)
>>
>> LOG REC AT LSN cycle 1 block 2112 (0x1, 0x840)
>> + mount /store
>> mount: /dev/sda1: can't read superblock
>>
>> Same messages in dmesg at this point.
>>
>> > You might want to find out if your platform needs to (and does)
>> > implement these functions:
>> >
>> > flush_kernel_dcache_page()
>> > flush_kernel_vmap_range()
>> > void invalidate_kernel_vmap_range()
>> >
>> > as these are what XFS relies on platforms to implement correctly to
>> > avoid cache aliasing issues on CPUs with virtually indexed caches.
>>
>> Is this what /proc/sys/vm/drop_caches relies on as well?
>
> No, drop_caches frees the page cache and slab caches so future reads
> need to be looked up from disk.
>
>> flush_kernel_dcache_page is empty, the others are not but are
>> conditionalized on the type of cache that is present. I wonder if that
>> is somehow not being detected properly. Wouldn't that cause other
>> areas of the system to misbehave as well?
>
> vmap is not widely used throughout the kernel, and as a result
> people porting linux to a new arch/CPU type often don't realise
> there's anything to implement there because their system seems to be
> working. That is, of course, until someone tries to use XFS.....
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs