xfsprogs 4.16.0 format change, GRUB error unknown filesystem

xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Chris Murphy]

Downstream bug, Fedora Rawhide being a canary in the coal mine!
https://bugzilla.redhat.com/show_bug.cgi?id=1575797

Comment 8 about the kernel warning related to freeze/thaw is curious
but ends up being unrelated to this problem.

The gist is that xfsprogs 4.16.0 formatted volumes aren't recognized
by either GRUB user space tools, or the GRUB core.img (whether of the
embedded BIOS variety or the grubx64.efi variety)

Anyway, open question for XFS devs if this should be fixed in xfsprogs
or GRUB. I haven't tested this with syslinux to know if it has a
similar problem.


-- 
Chris Murphy

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Darrick J. Wong]

On Mon, May 14, 2018 at 11:15:27AM -0600, Chris Murphy wrote:
> Downstream bug, Fedora Rawhide being a canary in the coal mine!
> https://bugzilla.redhat.com/show_bug.cgi?id=1575797
> 
> Comment 8 about the kernel warning related to freeze/thaw is curious
> but ends up being unrelated to this problem.
> 
> The gist is that xfsprogs 4.16.0 formatted volumes aren't recognized
> by either GRUB user space tools, or the GRUB core.img (whether of the
> embedded BIOS variety or the grubx64.efi variety)
> 
> Anyway, open question for XFS devs if this should be fixed in xfsprogs
> or GRUB. I haven't tested this with syslinux to know if it has a
> similar problem.

mkfs.xfs 4.16 enables sparse inodes by default, which is why grub
doesn't recognise the boot fs.  grub needs fixing, or $installer needs
to disable sparse inodes, I think.

--D

> 
> -- 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Chris Murphy]

On Mon, May 14, 2018 at 11:36 AM, Darrick J. Wong
<darrick.wong@oracle.com> wrote:
> On Mon, May 14, 2018 at 11:15:27AM -0600, Chris Murphy wrote:
>> Downstream bug, Fedora Rawhide being a canary in the coal mine!
>> https://bugzilla.redhat.com/show_bug.cgi?id=1575797
>>
>> Comment 8 about the kernel warning related to freeze/thaw is curious
>> but ends up being unrelated to this problem.
>>
>> The gist is that xfsprogs 4.16.0 formatted volumes aren't recognized
>> by either GRUB user space tools, or the GRUB core.img (whether of the
>> embedded BIOS variety or the grubx64.efi variety)
>>
>> Anyway, open question for XFS devs if this should be fixed in xfsprogs
>> or GRUB. I haven't tested this with syslinux to know if it has a
>> similar problem.
>
> mkfs.xfs 4.16 enables sparse inodes by default, which is why grub
> doesn't recognise the boot fs.  grub needs fixing, or $installer needs
> to disable sparse inodes, I think.
>

Yep I just discovered this looking through 4.16.0 change log, and then
tried swapping the feature: xfsprogs 4.15.1 with -i sparse=1 results
in failure; and xfsprogs 4.16.0 with -i sparse=0 works. I updated the
downstream bug. Historically the installer team is very reluctant to
use non-default mkfs, so I suspect there needs to be a GRUB patch, or
chances are the installer team will proscribe /boot on XFS rather than
hold up xfsprogs 4.16.1.

-- 
Chris Murphy

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Chris Murphy]

On Mon, May 14, 2018 at 11:36 AM, Darrick J. Wong
<darrick.wong@oracle.com> wrote:
> On Mon, May 14, 2018 at 11:15:27AM -0600, Chris Murphy wrote:
>> Downstream bug, Fedora Rawhide being a canary in the coal mine!
>> https://bugzilla.redhat.com/show_bug.cgi?id=1575797
>>
>> Comment 8 about the kernel warning related to freeze/thaw is curious
>> but ends up being unrelated to this problem.
>>
>> The gist is that xfsprogs 4.16.0 formatted volumes aren't recognized
>> by either GRUB user space tools, or the GRUB core.img (whether of the
>> embedded BIOS variety or the grubx64.efi variety)
>>
>> Anyway, open question for XFS devs if this should be fixed in xfsprogs
>> or GRUB. I haven't tested this with syslinux to know if it has a
>> similar problem.
>
> mkfs.xfs 4.16 enables sparse inodes by default, which is why grub
> doesn't recognise the boot fs.  grub needs fixing, or $installer needs
> to disable sparse inodes, I think.

Is there a reason why this was made default before GRUB was fixed? Or
would XFS devs prefer to not support /boot on XFS?

-- 
Chris Murphy

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Eric Sandeen]

On 5/14/18 12:52 PM, Chris Murphy wrote:
> On Mon, May 14, 2018 at 11:36 AM, Darrick J. Wong
> <darrick.wong@oracle.com> wrote:
>> On Mon, May 14, 2018 at 11:15:27AM -0600, Chris Murphy wrote:
>>> Downstream bug, Fedora Rawhide being a canary in the coal mine!
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1575797
>>>
>>> Comment 8 about the kernel warning related to freeze/thaw is curious
>>> but ends up being unrelated to this problem.
>>>
>>> The gist is that xfsprogs 4.16.0 formatted volumes aren't recognized
>>> by either GRUB user space tools, or the GRUB core.img (whether of the
>>> embedded BIOS variety or the grubx64.efi variety)
>>>
>>> Anyway, open question for XFS devs if this should be fixed in xfsprogs
>>> or GRUB. I haven't tested this with syslinux to know if it has a
>>> similar problem.
>>
>> mkfs.xfs 4.16 enables sparse inodes by default, which is why grub
>> doesn't recognise the boot fs.  grub needs fixing, or $installer needs
>> to disable sparse inodes, I think.
> 
> Is there a reason why this was made default before GRUB was fixed? Or
> would XFS devs prefer to not support /boot on XFS?

Coordination would have been good, but nobody thought about this issue.
It was a mistake, not a preference.

-Eric

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Chris Murphy]

On Mon, May 14, 2018 at 12:34 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>
>
> On 5/14/18 12:52 PM, Chris Murphy wrote:
>> On Mon, May 14, 2018 at 11:36 AM, Darrick J. Wong
>> <darrick.wong@oracle.com> wrote:
>>> On Mon, May 14, 2018 at 11:15:27AM -0600, Chris Murphy wrote:
>>>> Downstream bug, Fedora Rawhide being a canary in the coal mine!
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1575797
>>>>
>>>> Comment 8 about the kernel warning related to freeze/thaw is curious
>>>> but ends up being unrelated to this problem.
>>>>
>>>> The gist is that xfsprogs 4.16.0 formatted volumes aren't recognized
>>>> by either GRUB user space tools, or the GRUB core.img (whether of the
>>>> embedded BIOS variety or the grubx64.efi variety)
>>>>
>>>> Anyway, open question for XFS devs if this should be fixed in xfsprogs
>>>> or GRUB. I haven't tested this with syslinux to know if it has a
>>>> similar problem.
>>>
>>> mkfs.xfs 4.16 enables sparse inodes by default, which is why grub
>>> doesn't recognise the boot fs.  grub needs fixing, or $installer needs
>>> to disable sparse inodes, I think.
>>
>> Is there a reason why this was made default before GRUB was fixed? Or
>> would XFS devs prefer to not support /boot on XFS?
>
> Coordination would have been good, but nobody thought about this issue.
> It was a mistake, not a preference.
>

Can someone point to documentation for the change to include in a
report for GRUB devs?

I found this:
https://www.spinics.net/lists/xfs/msg36686.html

But all the documentation here seems stale:
http://xfs.org/index.php/XFS_Papers_and_Documentation
http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure/tmp/en-US/html/index.html



-- 
Chris Murphy

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Darrick J. Wong]

On Mon, May 14, 2018 at 01:08:45PM -0600, Chris Murphy wrote:
> On Mon, May 14, 2018 at 12:34 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> >
> >
> > On 5/14/18 12:52 PM, Chris Murphy wrote:
> >> On Mon, May 14, 2018 at 11:36 AM, Darrick J. Wong
> >> <darrick.wong@oracle.com> wrote:
> >>> On Mon, May 14, 2018 at 11:15:27AM -0600, Chris Murphy wrote:
> >>>> Downstream bug, Fedora Rawhide being a canary in the coal mine!
> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1575797
> >>>>
> >>>> Comment 8 about the kernel warning related to freeze/thaw is curious
> >>>> but ends up being unrelated to this problem.
> >>>>
> >>>> The gist is that xfsprogs 4.16.0 formatted volumes aren't recognized
> >>>> by either GRUB user space tools, or the GRUB core.img (whether of the
> >>>> embedded BIOS variety or the grubx64.efi variety)
> >>>>
> >>>> Anyway, open question for XFS devs if this should be fixed in xfsprogs
> >>>> or GRUB. I haven't tested this with syslinux to know if it has a
> >>>> similar problem.
> >>>
> >>> mkfs.xfs 4.16 enables sparse inodes by default, which is why grub
> >>> doesn't recognise the boot fs.  grub needs fixing, or $installer needs
> >>> to disable sparse inodes, I think.
> >>
> >> Is there a reason why this was made default before GRUB was fixed? Or
> >> would XFS devs prefer to not support /boot on XFS?
> >
> > Coordination would have been good, but nobody thought about this issue.
> > It was a mistake, not a preference.
> >
> 
> Can someone point to documentation for the change to include in a
> report for GRUB devs?
> 
> I found this:
> https://www.spinics.net/lists/xfs/msg36686.html
> 
> But all the documentation here seems stale:
> http://xfs.org/index.php/XFS_Papers_and_Documentation
> http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure/tmp/en-US/html/index.html

https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf

We're working on moving all that to Documentation/, time permitting.

--D
> 
> 
> 
> -- 
> Chris Murphy

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Chris Murphy]

On Mon, May 14, 2018 at 1:11 PM, Darrick J. Wong
<darrick.wong@oracle.com> wrote:
> On Mon, May 14, 2018 at 01:08:45PM -0600, Chris Murphy wrote:
>> On Mon, May 14, 2018 at 12:34 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>> >
>> >
>> > On 5/14/18 12:52 PM, Chris Murphy wrote:
>> >> On Mon, May 14, 2018 at 11:36 AM, Darrick J. Wong
>> >> <darrick.wong@oracle.com> wrote:
>> >>> On Mon, May 14, 2018 at 11:15:27AM -0600, Chris Murphy wrote:
>> >>>> Downstream bug, Fedora Rawhide being a canary in the coal mine!
>> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1575797
>> >>>>
>> >>>> Comment 8 about the kernel warning related to freeze/thaw is curious
>> >>>> but ends up being unrelated to this problem.
>> >>>>
>> >>>> The gist is that xfsprogs 4.16.0 formatted volumes aren't recognized
>> >>>> by either GRUB user space tools, or the GRUB core.img (whether of the
>> >>>> embedded BIOS variety or the grubx64.efi variety)
>> >>>>
>> >>>> Anyway, open question for XFS devs if this should be fixed in xfsprogs
>> >>>> or GRUB. I haven't tested this with syslinux to know if it has a
>> >>>> similar problem.
>> >>>
>> >>> mkfs.xfs 4.16 enables sparse inodes by default, which is why grub
>> >>> doesn't recognise the boot fs.  grub needs fixing, or $installer needs
>> >>> to disable sparse inodes, I think.
>> >>
>> >> Is there a reason why this was made default before GRUB was fixed? Or
>> >> would XFS devs prefer to not support /boot on XFS?
>> >
>> > Coordination would have been good, but nobody thought about this issue.
>> > It was a mistake, not a preference.
>> >
>>
>> Can someone point to documentation for the change to include in a
>> report for GRUB devs?
>>
>> I found this:
>> https://www.spinics.net/lists/xfs/msg36686.html
>>
>> But all the documentation here seems stale:
>> http://xfs.org/index.php/XFS_Papers_and_Documentation
>> http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure/tmp/en-US/html/index.html
>
> https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf
>
> We're working on moving all that to Documentation/, time permitting.


OK thanks. I've emailed grub-devel about the issue.

-- 
Chris Murphy

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Christoph Hellwig]

On Mon, May 14, 2018 at 12:11:14PM -0700, Darrick J. Wong wrote:
> We're working on moving all that to Documentation/, time permitting.

Of the kernel tree?  Of xfsprogs?  Why?

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Darrick J. Wong]

On Tue, May 15, 2018 at 05:20:15AM -0700, Christoph Hellwig wrote:
> On Mon, May 14, 2018 at 12:11:14PM -0700, Darrick J. Wong wrote:
> > We're working on moving all that to Documentation/, time permitting.
> 
> Of the kernel tree?  Of xfsprogs?  Why?

The kernel Documentation/ directory.  It's fewer trees for us/me to
maintain, and we can take advantage of the kernel's builtin
html/pdf/epub generator.  That said, the whole thing could get bogged
down in conversion hell, but for now I'm starting by incorporating
algorithms into the existing book and fixing the errors & other
weirdness in the existing text.

--D

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Christoph Hellwig]

On Mon, May 14, 2018 at 11:52:18AM -0600, Chris Murphy wrote:
> Is there a reason why this was made default before GRUB was fixed? Or
> would XFS devs prefer to not support /boot on XFS?

A bootloader that reads any non-trivial file system (and that includes
XFS for sure) is a bad idea.  So yes, if distros would not do that by
default it would help a lot.

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Chris Murphy]

On Tue, May 15, 2018 at 1:50 AM, Christoph Hellwig <hch@infradead.org> wrote:
> On Mon, May 14, 2018 at 11:52:18AM -0600, Chris Murphy wrote:
>> Is there a reason why this was made default before GRUB was fixed? Or
>> would XFS devs prefer to not support /boot on XFS?
>
> A bootloader that reads any non-trivial file system (and that includes
> XFS for sure) is a bad idea.

What constitutes non-trivial? It seems like this ship sailed a very
long time ago. The Windows bootloader's first job is to teach the
computer how to read NTFS, that's been true for over 20 years. Apple's
done the same with journaled HFS+ and most recently with their switch
to APFS. GRUB and syslinux have read Btrfs for around 8 years.

> So yes, if distros would not do that by
> default it would help a lot.

Fedora does not do this by default, but it doesn't disallow it. The
problem was discovered with a custom installation where /boot is a
dedicated XFS volume; but it's also reproducible in the likely more
common case where /boot is a directory on an XFS root fs.

Solutions are:
1. Update GRUB to support sparse inodes.
2. If /boot is on XFS, installer uses '-i sparse=0' at mkfs time.
3. Installer proscribes /boot on XFS (for any installation).

Fedora's release criteria requires that the system actually be able to
boot following an installation, so it's not OK to just pancake and let
the poor hapless user go find documentation or a bug report. Option 2
gets a bit complicated to account for the /boot dir on XFS case. GRUB
folks say they will eventually get to it, but have no time now or a
time estimate, so off hand I'd say that's check back in ~6 months.

I'm thinking it'll end up needing to be option 3, in the short term at least.


-- 
Chris Murphy

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Eric Sandeen]

On 5/16/18 1:19 AM, Chris Murphy wrote:
> On Tue, May 15, 2018 at 1:50 AM, Christoph Hellwig <hch@infradead.org> wrote:
>> On Mon, May 14, 2018 at 11:52:18AM -0600, Chris Murphy wrote:
>>> Is there a reason why this was made default before GRUB was fixed? Or
>>> would XFS devs prefer to not support /boot on XFS?
>>
>> A bootloader that reads any non-trivial file system (and that includes
>> XFS for sure) is a bad idea.
> 
> What constitutes non-trivial? It seems like this ship sailed a very
> long time ago. The Windows bootloader's first job is to teach the
> computer how to read NTFS, that's been true for over 20 years. Apple's
> done the same with journaled HFS+ and most recently with their switch
> to APFS. GRUB and syslinux have read Btrfs for around 8 years.
> 
>> So yes, if distros would not do that by
>> default it would help a lot.
> 
> Fedora does not do this by default, but it doesn't disallow it. The
> problem was discovered with a custom installation where /boot is a
> dedicated XFS volume; but it's also reproducible in the likely more
> common case where /boot is a directory on an XFS root fs.
> 
> Solutions are:
> 1. Update GRUB to support sparse inodes.
> 2. If /boot is on XFS, installer uses '-i sparse=0' at mkfs time.
> 3. Installer proscribes /boot on XFS (for any installation).
> 
> Fedora's release criteria requires that the system actually be able to
> boot following an installation, so it's not OK to just pancake and let
> the poor hapless user go find documentation or a bug report. Option 2
> gets a bit complicated to account for the /boot dir on XFS case. GRUB
> folks say they will eventually get to it, but have no time now or a
> time estimate, so off hand I'd say that's check back in ~6 months.
> 
> I'm thinking it'll end up needing to be option 3, in the short term at least.

while y'all were scurrying around hacking up fedora bits, I sent a patch
to fix grub, FWIW.  Hopefully Fedora will pick that up and move on.

-Eric

xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Chris Murphy]

Hi,

xfsprogs 4.16.0 defaults to enabling "sparse inode" which is not
compatible with GRUB.

Downstream bug in Fedora, grub-probe fails so grub-install can't work;
and on systems with an already properly installed core.img with XFS
support, it can't read an XFS file system with this feature set.
https://bugzilla.redhat.com/show_bug.cgi?id=1575797

Make them default
https://www.spinics.net/lists/linux-xfs/msg16598.html

Change description, page 48
https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf

It'd be helpful to know the approximate time frame for getting this
fixed, to know whether for Fedora 29 /boot on XFS should be inhibited
until a proper tested patch for GRUB exists.

Thanks,


-- 
Chris Murphy

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Daniel Kiper]

Hi,

On Mon, May 14, 2018 at 01:12:27PM -0600, Chris Murphy wrote:
> Hi,
>
> xfsprogs 4.16.0 defaults to enabling "sparse inode" which is not
> compatible with GRUB.
>
> Downstream bug in Fedora, grub-probe fails so grub-install can't work;
> and on systems with an already properly installed core.img with XFS
> support, it can't read an XFS file system with this feature set.
> https://bugzilla.redhat.com/show_bug.cgi?id=1575797
>
> Make them default
> https://www.spinics.net/lists/linux-xfs/msg16598.html
>
> Change description, page 48
> https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf
>
> It'd be helpful to know the approximate time frame for getting this
> fixed, to know whether for Fedora 29 /boot on XFS should be inhibited
> until a proper tested patch for GRUB exists.

Sorry, I do not have resources right now to do that. However,
if somebody will post the patches I am happy to review them.

Daniel

Re: xfsprogs 4.16.0 format change, GRUB error unknown filesystem

[Chris Murphy]

On Tue, May 15, 2018 at 6:33 AM, Daniel Kiper <dkiper@net-space.pl> wrote:
> Hi,
>
> On Mon, May 14, 2018 at 01:12:27PM -0600, Chris Murphy wrote:
>> Hi,
>>
>> xfsprogs 4.16.0 defaults to enabling "sparse inode" which is not
>> compatible with GRUB.
>>
>> Downstream bug in Fedora, grub-probe fails so grub-install can't work;
>> and on systems with an already properly installed core.img with XFS
>> support, it can't read an XFS file system with this feature set.
>> https://bugzilla.redhat.com/show_bug.cgi?id=1575797
>>
>> Make them default
>> https://www.spinics.net/lists/linux-xfs/msg16598.html
>>
>> Change description, page 48
>> https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf
>>
>> It'd be helpful to know the approximate time frame for getting this
>> fixed, to know whether for Fedora 29 /boot on XFS should be inhibited
>> until a proper tested patch for GRUB exists.
>
> Sorry, I do not have resources right now to do that. However,
> if somebody will post the patches I am happy to review them.

OK thanks for the update.

We appear to have a near term work around for the installer: if /boot
is XFS, then format with '-i sparse=0' which will avoid the problem.
Meanwhile / or /home still use the new default of sparse=1.


-- 
Chris Murphy