linux-unionfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* failed to verify upper - halts boot
@ 2021-12-05  1:04 Carl Karsten
  2021-12-05  1:13 ` Carl Karsten
  2021-12-07  5:55 ` Amir Goldstein
  0 siblings, 2 replies; 5+ messages in thread
From: Carl Karsten @ 2021-12-05  1:04 UTC (permalink / raw)
  To: overlayfs

I powercyced my box, I'm assuming I did a sudo poweroff or halt or
something, but maybe not.

It got stuck, (enter rescue mode, root account disabled, hit enter to
enter rescue mode... )  so I killed power.

I edited fstab, changed auto to noauto, booted.

overlay   /srv/nfs/rpi/bullseye/boot/merged    overlay
noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on
   0   0

# mount /srv/nfs/rpi/bullseye/boot/merged
mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed:
Stale file handle.

dmesg shows:

[   45.941350] overlayfs: failed to verify upper (boot/play,
ino=379405, err=-116)
[   45.941369] overlayfs: failed to verify index dir 'upper' xattr
[   45.941379] overlayfs: try deleting index dir or mounting with '-o
index=off' to disable inodes index.

It is a bit concerning that I need to consider a power cycle may
require local console use.  I plan on managing this remotely.  For now
I'll leave it noauto and remotely mount when needed.  so this isn't a
critical blocker to me.

What would happen:  errors=remount-ro

errors={continue|remount-ro|panic}
    Define the behavior  when  an  error  is  encountered.  The
default is set in the filesystem superblock,  ...     and can be
changed using tune2fs(8).

In normal operation I want the mount ro anyway.
continue unmounted is also an option.  which would be better than
noauto if I am assured it won't get stuck at boot.

-- 
Carl K

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: failed to verify upper - halts boot
  2021-12-05  1:04 failed to verify upper - halts boot Carl Karsten
@ 2021-12-05  1:13 ` Carl Karsten
  2021-12-07  5:55 ` Amir Goldstein
  1 sibling, 0 replies; 5+ messages in thread
From: Carl Karsten @ 2021-12-05  1:13 UTC (permalink / raw)
  To: overlayfs

On Sat, Dec 4, 2021 at 5:04 PM Carl Karsten <carl@nextdayvideo.com> wrote:
>
> I powercyced my box, I'm assuming I did a sudo poweroff or halt or
> something, but maybe not.
>
> It got stuck, (enter rescue mode, root account disabled, hit enter to
> enter rescue mode... )  so I killed power.
>
> I edited fstab, changed auto to noauto, booted.
>
> overlay   /srv/nfs/rpi/bullseye/boot/merged    overlay
> noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on
>    0   0
>
> # mount /srv/nfs/rpi/bullseye/boot/merged
> mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed:
> Stale file handle.
>
> dmesg shows:
>
> [   45.941350] overlayfs: failed to verify upper (boot/play,
> ino=379405, err=-116)
> [   45.941369] overlayfs: failed to verify index dir 'upper' xattr
> [   45.941379] overlayfs: try deleting index dir or mounting with '-o
> index=off' to disable inodes index.
>
> It is a bit concerning that I need to consider a power cycle may
> require local console use.  I plan on managing this remotely.  For now
> I'll leave it noauto and remotely mount when needed.  so this isn't a
> critical blocker to me.
>
> What would happen:  errors=remount-ro
>
> errors={continue|remount-ro|panic}
>     Define the behavior  when  an  error  is  encountered.  The
> default is set in the filesystem superblock,  ...     and can be
> changed using tune2fs(8).
>
> In normal operation I want the mount ro anyway.
> continue unmounted is also an option.  which would be better than
> noauto if I am assured it won't get stuck at boot.

welp...
noauto,defaults,ro,errors=continue
and remount-ro

[ 1157.781768] overlayfs: unrecognized mount option "errors=continue"
or missing value
[ 1565.415281] overlayfs: unrecognized mount option
"errors=remount-ro" or missing value


>
> --
> Carl K



-- 
Carl K

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: failed to verify upper - halts boot
  2021-12-05  1:04 failed to verify upper - halts boot Carl Karsten
  2021-12-05  1:13 ` Carl Karsten
@ 2021-12-07  5:55 ` Amir Goldstein
  2021-12-07 21:17   ` Carl Karsten
  1 sibling, 1 reply; 5+ messages in thread
From: Amir Goldstein @ 2021-12-07  5:55 UTC (permalink / raw)
  To: Carl Karsten; +Cc: overlayfs

On Tue, Dec 7, 2021 at 4:38 AM Carl Karsten <carl@nextdayvideo.com> wrote:
>
> I powercyced my box, I'm assuming I did a sudo poweroff or halt or
> something, but maybe not.
>
> It got stuck, (enter rescue mode, root account disabled, hit enter to
> enter rescue mode... )  so I killed power.
>
> I edited fstab, changed auto to noauto, booted.
>
> overlay   /srv/nfs/rpi/bullseye/boot/merged    overlay
> noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on
>    0   0
>
> # mount /srv/nfs/rpi/bullseye/boot/merged
> mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed:
> Stale file handle.
>
> dmesg shows:
>
> [   45.941350] overlayfs: failed to verify upper (boot/play,
> ino=379405, err=-116)
> [   45.941369] overlayfs: failed to verify index dir 'upper' xattr
> [   45.941379] overlayfs: try deleting index dir or mounting with '-o
> index=off' to disable inodes index.
>

Did you by any change re-create boot/play dir without re-creating
boot/work dir? because that is what that message means.
I doubt it has to do with the power cycle.

> It is a bit concerning that I need to consider a power cycle may
> require local console use.  I plan on managing this remotely.  For now
> I'll leave it noauto and remotely mount when needed.  so this isn't a
> critical blocker to me.
>
> What would happen:  errors=remount-ro
>
> errors={continue|remount-ro|panic}
>     Define the behavior  when  an  error  is  encountered.  The
> default is set in the filesystem superblock,  ...     and can be
> changed using tune2fs(8).
>

This looks like a quote from the man page section about ext2 mount options.
Those are not options supported for overlayfs.

Anyway, with that error, as the kernel log says, you could have mounted
overlayfs ro with -o index=off and we could have added a mount option
to fallback to ro mount with index=off, but that means no nfs_export,
so not sure if that is what you wanted.

Thanks,
Amir.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: failed to verify upper - halts boot
  2021-12-07  5:55 ` Amir Goldstein
@ 2021-12-07 21:17   ` Carl Karsten
  2021-12-08  7:01     ` Amir Goldstein
  0 siblings, 1 reply; 5+ messages in thread
From: Carl Karsten @ 2021-12-07 21:17 UTC (permalink / raw)
  To: Amir Goldstein; +Cc: overlayfs

On Mon, Dec 6, 2021 at 9:55 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Tue, Dec 7, 2021 at 4:38 AM Carl Karsten <carl@nextdayvideo.com> wrote:
> >
> > I powercyced my box, I'm assuming I did a sudo poweroff or halt or
> > something, but maybe not.
> >
> > It got stuck, (enter rescue mode, root account disabled, hit enter to
> > enter rescue mode... )  so I killed power.
> >
> > I edited fstab, changed auto to noauto, booted.
> >
> > overlay   /srv/nfs/rpi/bullseye/boot/merged    overlay
> > noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on
> >    0   0
> >
> > # mount /srv/nfs/rpi/bullseye/boot/merged
> > mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed:
> > Stale file handle.
> >
> > dmesg shows:
> >
> > [   45.941350] overlayfs: failed to verify upper (boot/play,
> > ino=379405, err=-116)
> > [   45.941369] overlayfs: failed to verify index dir 'upper' xattr
> > [   45.941379] overlayfs: try deleting index dir or mounting with '-o
> > index=off' to disable inodes index.
> >
>
> Did you by any change re-create boot/play dir without re-creating
> boot/work dir? because that is what that message means.
> I doubt it has to do with the power cycle.

Maybe.

would that include:
mount: lower=base,setup upper= updates
unmount
mount: lower=base,setup,updates upper=play
?

That might have happened.

>
> > It is a bit concerning that I need to consider a power cycle may
> > require local console use.  I plan on managing this remotely.  For now
> > I'll leave it noauto and remotely mount when needed.  so this isn't a
> > critical blocker to me.
> >
> > What would happen:  errors=remount-ro
> >
> > errors={continue|remount-ro|panic}
> >     Define the behavior  when  an  error  is  encountered.  The
> > default is set in the filesystem superblock,  ...     and can be
> > changed using tune2fs(8).
> >
>
> This looks like a quote from the man page section about ext2 mount options.
> Those are not options supported for overlayfs.
>
> Anyway, with that error, as the kernel log says, you could have mounted
> overlayfs ro with -o index=off and we could have added a mount option
> to fallback to ro mount with index=off, but that means no nfs_export,
> so not sure if that is what you wanted.

That would be an improvement, as would errors=continue.

Either of these would let a healthy system come up and be usable, and
errors would let the system come up (finish booting) but the
nfs/overlay part needs attention.

Currently I have noauto so any reboot requires attention.
If I set auto, an error will cause the boot to stop and I don't have a
way to fix it remotely. I don't want to risk that, so noauto.

I think I would prefer errors=continue as It would be more obvious
that the overlayfs mount has issues.

>
> Thanks,
> Amir.



-- 
Carl K

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: failed to verify upper - halts boot
  2021-12-07 21:17   ` Carl Karsten
@ 2021-12-08  7:01     ` Amir Goldstein
  0 siblings, 0 replies; 5+ messages in thread
From: Amir Goldstein @ 2021-12-08  7:01 UTC (permalink / raw)
  To: Carl Karsten; +Cc: overlayfs

On Tue, Dec 7, 2021 at 11:17 PM Carl Karsten <carl@nextdayvideo.com> wrote:
>
> On Mon, Dec 6, 2021 at 9:55 PM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > On Tue, Dec 7, 2021 at 4:38 AM Carl Karsten <carl@nextdayvideo.com> wrote:
> > >
> > > I powercyced my box, I'm assuming I did a sudo poweroff or halt or
> > > something, but maybe not.
> > >
> > > It got stuck, (enter rescue mode, root account disabled, hit enter to
> > > enter rescue mode... )  so I killed power.
> > >
> > > I edited fstab, changed auto to noauto, booted.
> > >
> > > overlay   /srv/nfs/rpi/bullseye/boot/merged    overlay
> > > noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on
> > >    0   0
> > >
> > > # mount /srv/nfs/rpi/bullseye/boot/merged
> > > mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed:
> > > Stale file handle.
> > >
> > > dmesg shows:
> > >
> > > [   45.941350] overlayfs: failed to verify upper (boot/play,
> > > ino=379405, err=-116)
> > > [   45.941369] overlayfs: failed to verify index dir 'upper' xattr
> > > [   45.941379] overlayfs: try deleting index dir or mounting with '-o
> > > index=off' to disable inodes index.
> > >
> >
> > Did you by any change re-create boot/play dir without re-creating
> > boot/work dir? because that is what that message means.
> > I doubt it has to do with the power cycle.
>
> Maybe.
>
> would that include:
> mount: lower=base,setup upper= updates
> unmount
> mount: lower=base,setup,updates upper=play
> ?
>
> That might have happened.

Yes, that would do it.

>
> >
> > > It is a bit concerning that I need to consider a power cycle may
> > > require local console use.  I plan on managing this remotely.  For now
> > > I'll leave it noauto and remotely mount when needed.  so this isn't a
> > > critical blocker to me.
> > >
> > > What would happen:  errors=remount-ro
> > >
> > > errors={continue|remount-ro|panic}
> > >     Define the behavior  when  an  error  is  encountered.  The
> > > default is set in the filesystem superblock,  ...     and can be
> > > changed using tune2fs(8).
> > >
> >
> > This looks like a quote from the man page section about ext2 mount options.
> > Those are not options supported for overlayfs.
> >
> > Anyway, with that error, as the kernel log says, you could have mounted
> > overlayfs ro with -o index=off and we could have added a mount option
> > to fallback to ro mount with index=off, but that means no nfs_export,
> > so not sure if that is what you wanted.
>
> That would be an improvement, as would errors=continue.
>
> Either of these would let a healthy system come up and be usable, and
> errors would let the system come up (finish booting) but the
> nfs/overlay part needs attention.
>
> Currently I have noauto so any reboot requires attention.
> If I set auto, an error will cause the boot to stop and I don't have a
> way to fix it remotely. I don't want to risk that, so noauto.
>

There is no need for a kernel change.
You can use boot scripts or mount helper to implement the mount options
fallback that suits your system needs.

> I think I would prefer errors=continue as It would be more obvious
> that the overlayfs mount has issues.
>

That is not likely to happen and FYI errors=continue does not even
mean what you think for ext4 where it is supported.
It does not continue on failure to mount, it continues on IO and
fs corruption errors detected during operations on a mounted fs.

Thanks,
Amir.

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-12-08  7:01 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-05  1:04 failed to verify upper - halts boot Carl Karsten
2021-12-05  1:13 ` Carl Karsten
2021-12-07  5:55 ` Amir Goldstein
2021-12-07 21:17   ` Carl Karsten
2021-12-08  7:01     ` Amir Goldstein

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).