Small project: Make it easier to upgrade root fs (i.e. to bigtime)

Small project: Make it easier to upgrade root fs (i.e. to bigtime)

[Eric Sandeen]

Darrick taught xfs_admin to upgrade filesystems to bigtime and inobtcount, which is
nice! But it operates via xfs_repair on an unmounted filesystem, so it's a bit tricky
to do for the root fs.

It occurs to me that with the /forcefsck and /fsckoptions files[1], we might be able
to make this a bit easier. i.e. touch /forcefsck and add "-c bigtime=1" to /fsckoptions,
and then the initrd/initramfs should run xfs_repair with -c bigtime=1 and do the upgrade.

However ... fsck.xfs doesn't accept that option, and doesn't pass it on to repair, so
that doesn't work.

It seems reasonable to me for fsck.xfs, when it gets the "-f" option via init, and
the special handling we do already to actually Do Something(tm)[2], we could then also
pass on any additional options we got via the /fsckoptions method.

Does anyone see a problem with this?
If not, would anyone like to take this on as a small project?

Thanks,
-Eric


[1] this works at least on some OSes, I don't know if it works on all.

[2]
if $FORCE; then
         xfs_repair -e $DEV
         repair2fsck_code $?
         exit $?
fi

Re: Small project: Make it easier to upgrade root fs (i.e. to bigtime)

[Dave Chinner]

On Tue, Oct 19, 2021 at 10:18:31AM -0500, Eric Sandeen wrote:
> Darrick taught xfs_admin to upgrade filesystems to bigtime and inobtcount, which is
> nice! But it operates via xfs_repair on an unmounted filesystem, so it's a bit tricky
> to do for the root fs.
> 
> It occurs to me that with the /forcefsck and /fsckoptions files[1], we might be able
> to make this a bit easier. i.e. touch /forcefsck and add "-c bigtime=1" to /fsckoptions,
> and then the initrd/initramfs should run xfs_repair with -c bigtime=1 and do the upgrade.

Does that happen before/after swap is enabled?

What problems can arise from a failed repair here?

Also, ISTR historical problems with doing initrd based root fs
operations because it's not uncommon for the root filesystem to fail
to cleanly unmount on shutdown.  i.e. it can end up not having the
unmount record written because shutdown finishes with the superblock
still referenced. Hence the filesystem has to be mounted and the log
replayed before repair can be run on it....

> However ... fsck.xfs doesn't accept that option, and doesn't pass
> it on to repair, so that doesn't work.
> 
> It seems reasonable to me for fsck.xfs, when it gets the "-f"
> option via init, and the special handling we do already to
> actually Do Something(tm)[2], we could then also pass on any
> additional options we got via the /fsckoptions method.
> 
> Does anyone see a problem with this?  If not, would anyone like to
> take this on as a small project?

As long as it doesn't result in an unbootable system on failure, it
sounds like a good idea to me.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Small project: Make it easier to upgrade root fs (i.e. to bigtime)

[Eric Sandeen]

On 10/19/21 3:44 PM, Dave Chinner wrote:
> On Tue, Oct 19, 2021 at 10:18:31AM -0500, Eric Sandeen wrote:
>> Darrick taught xfs_admin to upgrade filesystems to bigtime and inobtcount, which is
>> nice! But it operates via xfs_repair on an unmounted filesystem, so it's a bit tricky
>> to do for the root fs.
>>
>> It occurs to me that with the /forcefsck and /fsckoptions files[1], we might be able
>> to make this a bit easier. i.e. touch /forcefsck and add "-c bigtime=1" to /fsckoptions,
>> and then the initrd/initramfs should run xfs_repair with -c bigtime=1 and do the upgrade.
> 
> Does that happen before/after swap is enabled?

Good question, and I don't know.

> What problems can arise from a failed repair here?

In general, we've always said that an aborted repair should leave things
in a not-worse state, I think? As in "repair is safe to cancel?"

I don't know if that holds true on an upgrade path though...

> Also, ISTR historical problems with doing initrd based root fs
> operations because it's not uncommon for the root filesystem to fail
> to cleanly unmount on shutdown.  i.e. it can end up not having the
> unmount record written because shutdown finishes with the superblock
> still referenced. Hence the filesystem has to be mounted and the log
> replayed before repair can be run on it....
> 
>> However ... fsck.xfs doesn't accept that option, and doesn't pass
>> it on to repair, so that doesn't work.
>>
>> It seems reasonable to me for fsck.xfs, when it gets the "-f"
>> option via init, and the special handling we do already to
>> actually Do Something(tm)[2], we could then also pass on any
>> additional options we got via the /fsckoptions method.
>>
>> Does anyone see a problem with this?  If not, would anyone like to
>> take this on as a small project?
> 
> As long as it doesn't result in an unbootable system on failure, it
> sounds like a good idea to me.

The /forcefsck is already honored, so I think the only additional possible
pitfalls would be specifically related to the upgrade path.

But, that upgrade path would encourage more people to make use of it, so...

I guess in general, if we honor forcefsck and translate that into a real
xfs_repair run, it probably makes sense to honor command line options in
general.

-Eric

Re: Small project: Make it easier to upgrade root fs (i.e. to bigtime)

[Carlos Maiolino]

On Tue, Oct 19, 2021 at 04:04:14PM -0500, Eric Sandeen wrote:
> On 10/19/21 3:44 PM, Dave Chinner wrote:
> > On Tue, Oct 19, 2021 at 10:18:31AM -0500, Eric Sandeen wrote:
> > > Darrick taught xfs_admin to upgrade filesystems to bigtime and inobtcount, which is
> > > nice! But it operates via xfs_repair on an unmounted filesystem, so it's a bit tricky
> > > to do for the root fs.
> > > 
> > > It occurs to me that with the /forcefsck and /fsckoptions files[1], we might be able
> > > to make this a bit easier. i.e. touch /forcefsck and add "-c bigtime=1" to /fsckoptions,
> > > and then the initrd/initramfs should run xfs_repair with -c bigtime=1 and do the upgrade.
> > 
> > Does that happen before/after swap is enabled?

IIRC in general, it follows the /etc/fstab mount order, and to access that,
rootfs should be mounted, and, (also IIRC), the rootfs is mounted RO, and then
remounted RW once the boot pre-reqs are read, but I can confirm that.

> 
> > Also, ISTR historical problems with doing initrd based root fs
> > operations because it's not uncommon for the root filesystem to fail
> > to cleanly unmount on shutdown.  i.e. it can end up not having the
> > unmount record written because shutdown finishes with the superblock
> > still referenced. Hence the filesystem has to be mounted and the log
> > replayed before repair can be run on it....
> > 

I suppose this is already true nowadays? If /forcefsck exists, we are already
running fsck the on the rootfs, so, I wonder what happens nowadays, as I haven't
tried to use /forcefsck. But anyway, I don't think the behavior will be much
different from the current one. I should check what happens today..

> > > Does anyone see a problem with this?  If not, would anyone like to
> > > take this on as a small project?

If nobody has any objections, I'll be happy to work on this :)

Cheers
-- 
Carlos

Re: Small project: Make it easier to upgrade root fs (i.e. to bigtime)

[Eric Sandeen]

On 10/20/21 6:36 AM, Carlos Maiolino wrote:
> On Tue, Oct 19, 2021 at 04:04:14PM -0500, Eric Sandeen wrote:
>> On 10/19/21 3:44 PM, Dave Chinner wrote:
>>> On Tue, Oct 19, 2021 at 10:18:31AM -0500, Eric Sandeen wrote:
>>>> Darrick taught xfs_admin to upgrade filesystems to bigtime and inobtcount, which is
>>>> nice! But it operates via xfs_repair on an unmounted filesystem, so it's a bit tricky
>>>> to do for the root fs.
>>>>
>>>> It occurs to me that with the /forcefsck and /fsckoptions files[1], we might be able
>>>> to make this a bit easier. i.e. touch /forcefsck and add "-c bigtime=1" to /fsckoptions,
>>>> and then the initrd/initramfs should run xfs_repair with -c bigtime=1 and do the upgrade.
>>>
>>> Does that happen before/after swap is enabled?
> 
> IIRC in general, it follows the /etc/fstab mount order, and to access that,
> rootfs should be mounted, and, (also IIRC), the rootfs is mounted RO, and then
> remounted RW once the boot pre-reqs are read, but I can confirm that.
> 
>>
>>> Also, ISTR historical problems with doing initrd based root fs
>>> operations because it's not uncommon for the root filesystem to fail
>>> to cleanly unmount on shutdown.  i.e. it can end up not having the
>>> unmount record written because shutdown finishes with the superblock
>>> still referenced. Hence the filesystem has to be mounted and the log
>>> replayed before repair can be run on it....
>>>
> 
> I suppose this is already true nowadays? If /forcefsck exists, we are already
> running fsck the on the rootfs, so, I wonder what happens nowadays, as I haven't
> tried to use /forcefsck. But anyway, I don't think the behavior will be much
> different from the current one. I should check what happens today..
> 
>>>> Does anyone see a problem with this?  If not, would anyone like to
>>>> take this on as a small project?
> 
> If nobody has any objections, I'll be happy to work on this :)

Sure, go for it. Talking to Darrick yesterday, we were wondering how swap is
handled in the initramfs.  If you haven't mounted root, you don't have fstab
to find swap. But blkid *could* identify all swap partitions anyway. I doubt
it does that.

The other thing I wondered about was checking system memory, and limiting
repair to use that much with -m. If it's not available, would it fail more
gracefully than OOMing halfway through?

if this all feels to crazy, I'm ok with not going forward but it seems worth
investigating, because operations like this on the root fs can be a real pain.

-Eric