XFS on Fedora i686, armv7hl

XFS on Fedora i686, armv7hl

[Chris Murphy]

Hi,

Fedora is considering XFS as their default file system. They support three primary architectures: x86_64, i686, and armv7hl.  Do XFS devs have any reservations about XFS as a default file system on either i686, or arm?

So far the only thing I've run into with kernel 3.13.4-200.fc20.i686+PAE will not mount an XFS volume larger than 16TB. But I haven't tried filling a < 16TB volume with a significant amount of data while running 32bit, and anyway it's just easier to ask if there are other gotchas, or reservations about this combination.

XFS (sdc): file system too large to be mounted on this system.
[snip]
XFS (sdc): Internal error xfs_sb_read_verify at line 630 of file fs/xfs/xfs_sb.c.  Caller 0xf8b19e1c
[1]


Thanks,


Chris Murphy


[1] The too large warning happens with ext4 at mount time also. A Btrfs volume mounts OK, although I don't know if it actually withstands significant use.
http://paste.fedoraproject.org/80784/93470280/
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS on Fedora i686, armv7hl

[Dave Chinner]

On Wed, Feb 26, 2014 at 10:37:59PM -0700, Chris Murphy wrote:
> Hi,
> 
> Fedora is considering XFS as their default file system. They
> support three primary architectures: x86_64, i686, and armv7hl.
> Do XFS devs have any reservations about XFS as a default file
> system on either i686, or arm?

i686 is regularly tested on upstream dev kernels. ARM is less tested
as it's not the primary development platform for anyone - we tend to
rely on community feedback for arm because the hardware is so wide
and varied and there are some crackpot CPU cache architectures out
there in ARM land that we simply can't test against....

> So far the only thing I've run into with kernel
> 3.13.4-200.fc20.i686+PAE will not mount an XFS volume larger than
> 16TB.

That's not an XFS limit - that's a limit of the block device caused
by the page cache address space being limited to 16TB. Techinically
the XFS kernel doesn't have such a limit because it doesn't use the
block device address space to index or cache metadata, but that
doesn't help anyone if the userspace tools don't work on anything
larger than a 16TB block device.

As it is, you're crazy if you put more than a couple of TB of
storage on a 32 bit system. The machiens simply don't have the
process address space to repair a filesystem larger than a few
terabytes (i.e. 2GB RAM limit). That holds true for any filesystem -
ext3 and ext4 also have the same problems when running e2fsck...

> But I haven't tried filling a < 16TB volume with a
> significant amount of data while running 32bit, and anyway it's
> just easier to ask if there are other gotchas, or reservations
> about this combination.

It'll work just as well as ext3 and ext4 in such situations. That
doesn't mean we recommend that you do it ;)

> XFS (sdc): file system too large to be mounted on this system.
> [snip] XFS (sdc): Internal error xfs_sb_read_verify at line 630 of
> file fs/xfs/xfs_sb.c.  Caller 0xf8b19e1c [1]
> 
> 
> Thanks,
> 
> 
> Chris Murphy
> 
> 
> [1] The too large warning happens with ext4 at mount time also. A

*nod*. For the same reasons as XFS is limited to 16TB.

> Btrfs volume mounts OK, although I don't know if it actually
> withstands significant use.
> http://paste.fedoraproject.org/80784/93470280/

I bet that's because nobody has filled a btrfs filesystem past the
point where it tries to access beyond 16TB on a 32 bit system and so
it's never been reported as a bug... :/

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS on Fedora i686, armv7hl

[Chris Murphy]

On Feb 27, 2014, at 12:21 AM, Dave Chinner <david@fromorbit.com> wrote:

> On Wed, Feb 26, 2014 at 10:37:59PM -0700, Chris Murphy wrote:
>> Hi,
>> 
>> Fedora is considering XFS as their default file system. They
>> support three primary architectures: x86_64, i686, and armv7hl.
>> Do XFS devs have any reservations about XFS as a default file
>> system on either i686, or arm?
> 
> i686 is regularly tested on upstream dev kernels. ARM is less tested
> as it's not the primary development platform for anyone - we tend to
> rely on community feedback for arm because the hardware is so wide
> and varied and there are some crackpot CPU cache architectures out
> there in ARM land that we simply can't test against….

OK good, I'll post the URL for your response to the relevant Fedora lists.

> 
>> So far the only thing I've run into with kernel
>> 3.13.4-200.fc20.i686+PAE will not mount an XFS volume larger than
>> 16TB.
> 
> That's not an XFS limit - that's a limit of the block device caused
> by the page cache address space being limited to 16TB. Techinically
> the XFS kernel doesn't have such a limit because it doesn't use the
> block device address space to index or cache metadata, but that
> doesn't help anyone if the userspace tools don't work on anything
> larger than a 16TB block device.

Are the kernel messages regarding corruption slightly misleading? i.e. the file system on-disk isn't corrupt, but the kernel's view of it is distorted because of the page cache limit? Someone has a weird drunken bet, because I can't think of a good reason why, and they mount a valued 16+TB XFS volume from a 64-bit system on a 32-bit system, they don't really have to run xfs_repair once putting it back on the 64-bit system, do they?

> 
> As it is, you're crazy if you put more than a couple of TB of
> storage on a 32 bit system. The machiens simply don't have the
> process address space to repair a filesystem larger than a few
> terabytes (i.e. 2GB RAM limit). That holds true for any filesystem -
> ext3 and ext4 also have the same problems when running e2fsck…

Right no kidding.

> 
>> But I haven't tried filling a < 16TB volume with a
>> significant amount of data while running 32bit, and anyway it's
>> just easier to ask if there are other gotchas, or reservations
>> about this combination.
> 
> It'll work just as well as ext3 and ext4 in such situations. That
> doesn't mean we recommend that you do it ;)

Sure. It's good to have that feedback.

> 
> I bet that's because nobody has filled a btrfs filesystem past the
> point where it tries to access beyond 16TB on a 32 bit system and so
> it's never been reported as a bug… :/

That makes sense.

Chris Murphy

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS on Fedora i686, armv7hl

[Dave Chinner]

On Thu, Feb 27, 2014 at 01:12:28AM -0700, Chris Murphy wrote:
> On Feb 27, 2014, at 12:21 AM, Dave Chinner <david@fromorbit.com>
> wrote:
> > On Wed, Feb 26, 2014 at 10:37:59PM -0700, Chris Murphy wrote:
> >> Hi,
> >> 
> >> Fedora is considering XFS as their default file system. They
> >> support three primary architectures: x86_64, i686, and armv7hl.
> >> Do XFS devs have any reservations about XFS as a default file
> >> system on either i686, or arm?
> > 
> > i686 is regularly tested on upstream dev kernels. ARM is less
> > tested as it's not the primary development platform for anyone -
> > we tend to rely on community feedback for arm because the
> > hardware is so wide and varied and there are some crackpot CPU
> > cache architectures out there in ARM land that we simply can't
> > test against….
> 
> OK good, I'll post the URL for your response to the relevant
> Fedora lists.
> 
> > 
> >> So far the only thing I've run into with kernel
> >> 3.13.4-200.fc20.i686+PAE will not mount an XFS volume larger
> >> than 16TB.
> > 
> > That's not an XFS limit - that's a limit of the block device
> > caused by the page cache address space being limited to 16TB.
> > Techinically the XFS kernel doesn't have such a limit because it
> > doesn't use the block device address space to index or cache
> > metadata, but that doesn't help anyone if the userspace tools
> > don't work on anything larger than a 16TB block device.
> 
> Are the kernel messages regarding corruption slightly misleading?

Yes.

> i.e. the file system on-disk isn't corrupt, but the kernel's view
> of it is distorted because of the page cache limit? Someone has a
> weird drunken bet, because I can't think of a good reason why, and
> they mount a valued 16+TB XFS volume from a 64-bit system on a
> 32-bit system, they don't really have to run xfs_repair once
> putting it back on the 64-bit system, do they?

No, it's not corrupt. The verifier on the superblock has been a bit
heavy-handed in considering  bad configurations such as this as
corruption. i.e. it's out of range, it must be corrupt! However, the
failure we are returning is not corruption, but EFBIG, and hence
this patch just went into 3.14-rc3:

http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commitdiff;h=5ef11eb0700f806c4671ba33e5befa784a2f70ef

to fix that classification problem - the superblock verifier should
now only be noisy if actual corruption is detected.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS on Fedora i686, armv7hl

[Eric Sandeen]

On 2/26/14, 11:37 PM, Chris Murphy wrote:
> Hi,
> 
> Fedora is considering XFS as their default file system. They support
> three primary architectures: x86_64, i686, and armv7hl.  Do XFS devs
> have any reservations about XFS as a default file system on either
> i686, or arm?

As Dave said, we rely on others to do ARM testing for the most part,
though I've certainly jumped in and debugged some issues from time
to time.

It'd be super if Fedora could run the xfstests test suite on arm
as part of QE.  I'd be more than happy to help get that started
if people are interested.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS on Fedora i686, armv7hl

[Chris Murphy]

On Feb 27, 2014, at 8:06 AM, Eric Sandeen <sandeen@sandeen.net> wrote:

> On 2/26/14, 11:37 PM, Chris Murphy wrote:
>> Hi,
>> 
>> Fedora is considering XFS as their default file system. They support
>> three primary architectures: x86_64, i686, and armv7hl.  Do XFS devs
>> have any reservations about XFS as a default file system on either
>> i686, or arm?
> 
> As Dave said, we rely on others to do ARM testing for the most part,
> though I've certainly jumped in and debugged some issues from time
> to time.
> 
> It'd be super if Fedora could run the xfstests test suite on arm
> as part of QE.  I'd be more than happy to help get that started
> if people are interested.

I don't know that Fedora QA has the resources to do this, but I'll cc the Fedora test@ (QA) arm@ lists. If these are highly automatable tests it might be possible, if they have the hardware. More likely I think it's that we need some ARM community folks to look at splitting up some of this work.

I'm not sure yet what concerns the ARM group might have with XFS either as this hasn't been decided, but the Fedora Server product working group is slightly leaning toward XFS by default. Performance and CPU hit wise on x86_64, XFS seems to match up well with ext4 and maybe even a bit better ratio of throughput/CPUtime for booting workload (systemd is parallel!) so if were the same on ARM XFS could work out slightly better for them.


Chris Murphy
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS on Fedora i686, armv7hl

[Chris Murphy]

On Feb 27, 2014, at 12:21 AM, Dave Chinner <david@fromorbit.com> wrote:

> On Wed, Feb 26, 2014 at 10:37:59PM -0700, Chris Murphy wrote:
> 
>> Btrfs volume mounts OK, although I don't know if it actually
>> withstands significant use.
>> http://paste.fedoraproject.org/80784/93470280/
> 
> I bet that's because nobody has filled a btrfs filesystem past the
> point where it tries to access beyond 16TB on a 32 bit system and so
> it's never been reported as a bug… :/

Guess what? Someone just ran into what sounded like this very problem today on the Btrfs list [1]. So I reported it and referenced this thread, and it looks like you were correct. Josef is on it [2].


Chris Murphy

[1] http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg31856.html
[2] http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg31862.html

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs