Re: failed to verify upper - halts boot

[Carl Karsten <carl@nextdayvideo.com>]

On Mon, Dec 6, 2021 at 9:55 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Tue, Dec 7, 2021 at 4:38 AM Carl Karsten <carl@nextdayvideo.com> wrote:
> >
> > I powercyced my box, I'm assuming I did a sudo poweroff or halt or
> > something, but maybe not.
> >
> > It got stuck, (enter rescue mode, root account disabled, hit enter to
> > enter rescue mode... )  so I killed power.
> >
> > I edited fstab, changed auto to noauto, booted.
> >
> > overlay   /srv/nfs/rpi/bullseye/boot/merged    overlay
> > noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on
> >    0   0
> >
> > # mount /srv/nfs/rpi/bullseye/boot/merged
> > mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed:
> > Stale file handle.
> >
> > dmesg shows:
> >
> > [   45.941350] overlayfs: failed to verify upper (boot/play,
> > ino=379405, err=-116)
> > [   45.941369] overlayfs: failed to verify index dir 'upper' xattr
> > [   45.941379] overlayfs: try deleting index dir or mounting with '-o
> > index=off' to disable inodes index.
> >
>
> Did you by any change re-create boot/play dir without re-creating
> boot/work dir? because that is what that message means.
> I doubt it has to do with the power cycle.

Maybe.

would that include:
mount: lower=base,setup upper= updates
unmount
mount: lower=base,setup,updates upper=play
?

That might have happened.

>
> > It is a bit concerning that I need to consider a power cycle may
> > require local console use.  I plan on managing this remotely.  For now
> > I'll leave it noauto and remotely mount when needed.  so this isn't a
> > critical blocker to me.
> >
> > What would happen:  errors=remount-ro
> >
> > errors={continue|remount-ro|panic}
> >     Define the behavior  when  an  error  is  encountered.  The
> > default is set in the filesystem superblock,  ...     and can be
> > changed using tune2fs(8).
> >
>
> This looks like a quote from the man page section about ext2 mount options.
> Those are not options supported for overlayfs.
>
> Anyway, with that error, as the kernel log says, you could have mounted
> overlayfs ro with -o index=off and we could have added a mount option
> to fallback to ro mount with index=off, but that means no nfs_export,
> so not sure if that is what you wanted.

That would be an improvement, as would errors=continue.

Either of these would let a healthy system come up and be usable, and
errors would let the system come up (finish booting) but the
nfs/overlay part needs attention.

Currently I have noauto so any reboot requires attention.
If I set auto, an error will cause the boot to stop and I don't have a
way to fix it remotely. I don't want to risk that, so noauto.

I think I would prefer errors=continue as It would be more obvious
that the overlayfs mount has issues.

>
> Thanks,
> Amir.



-- 
Carl K