* failed to verify upper - halts boot
@ 2021-12-05 1:04 Carl Karsten
2021-12-05 1:13 ` Carl Karsten
2021-12-07 5:55 ` Amir Goldstein
0 siblings, 2 replies; 5+ messages in thread
From: Carl Karsten @ 2021-12-05 1:04 UTC (permalink / raw)
To: overlayfs
I powercyced my box, I'm assuming I did a sudo poweroff or halt or
something, but maybe not.
It got stuck, (enter rescue mode, root account disabled, hit enter to
enter rescue mode... ) so I killed power.
I edited fstab, changed auto to noauto, booted.
overlay /srv/nfs/rpi/bullseye/boot/merged overlay
noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on
0 0
# mount /srv/nfs/rpi/bullseye/boot/merged
mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed:
Stale file handle.
dmesg shows:
[ 45.941350] overlayfs: failed to verify upper (boot/play,
ino=379405, err=-116)
[ 45.941369] overlayfs: failed to verify index dir 'upper' xattr
[ 45.941379] overlayfs: try deleting index dir or mounting with '-o
index=off' to disable inodes index.
It is a bit concerning that I need to consider a power cycle may
require local console use. I plan on managing this remotely. For now
I'll leave it noauto and remotely mount when needed. so this isn't a
critical blocker to me.
What would happen: errors=remount-ro
errors={continue|remount-ro|panic}
Define the behavior when an error is encountered. The
default is set in the filesystem superblock, ... and can be
changed using tune2fs(8).
In normal operation I want the mount ro anyway.
continue unmounted is also an option. which would be better than
noauto if I am assured it won't get stuck at boot.
--
Carl K
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: failed to verify upper - halts boot 2021-12-05 1:04 failed to verify upper - halts boot Carl Karsten @ 2021-12-05 1:13 ` Carl Karsten 2021-12-07 5:55 ` Amir Goldstein 1 sibling, 0 replies; 5+ messages in thread From: Carl Karsten @ 2021-12-05 1:13 UTC (permalink / raw) To: overlayfs On Sat, Dec 4, 2021 at 5:04 PM Carl Karsten <carl@nextdayvideo.com> wrote: > > I powercyced my box, I'm assuming I did a sudo poweroff or halt or > something, but maybe not. > > It got stuck, (enter rescue mode, root account disabled, hit enter to > enter rescue mode... ) so I killed power. > > I edited fstab, changed auto to noauto, booted. > > overlay /srv/nfs/rpi/bullseye/boot/merged overlay > noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on > 0 0 > > # mount /srv/nfs/rpi/bullseye/boot/merged > mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed: > Stale file handle. > > dmesg shows: > > [ 45.941350] overlayfs: failed to verify upper (boot/play, > ino=379405, err=-116) > [ 45.941369] overlayfs: failed to verify index dir 'upper' xattr > [ 45.941379] overlayfs: try deleting index dir or mounting with '-o > index=off' to disable inodes index. > > It is a bit concerning that I need to consider a power cycle may > require local console use. I plan on managing this remotely. For now > I'll leave it noauto and remotely mount when needed. so this isn't a > critical blocker to me. > > What would happen: errors=remount-ro > > errors={continue|remount-ro|panic} > Define the behavior when an error is encountered. The > default is set in the filesystem superblock, ... and can be > changed using tune2fs(8). > > In normal operation I want the mount ro anyway. > continue unmounted is also an option. which would be better than > noauto if I am assured it won't get stuck at boot. welp... noauto,defaults,ro,errors=continue and remount-ro [ 1157.781768] overlayfs: unrecognized mount option "errors=continue" or missing value [ 1565.415281] overlayfs: unrecognized mount option "errors=remount-ro" or missing value > > -- > Carl K -- Carl K ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: failed to verify upper - halts boot 2021-12-05 1:04 failed to verify upper - halts boot Carl Karsten 2021-12-05 1:13 ` Carl Karsten @ 2021-12-07 5:55 ` Amir Goldstein 2021-12-07 21:17 ` Carl Karsten 1 sibling, 1 reply; 5+ messages in thread From: Amir Goldstein @ 2021-12-07 5:55 UTC (permalink / raw) To: Carl Karsten; +Cc: overlayfs On Tue, Dec 7, 2021 at 4:38 AM Carl Karsten <carl@nextdayvideo.com> wrote: > > I powercyced my box, I'm assuming I did a sudo poweroff or halt or > something, but maybe not. > > It got stuck, (enter rescue mode, root account disabled, hit enter to > enter rescue mode... ) so I killed power. > > I edited fstab, changed auto to noauto, booted. > > overlay /srv/nfs/rpi/bullseye/boot/merged overlay > noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on > 0 0 > > # mount /srv/nfs/rpi/bullseye/boot/merged > mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed: > Stale file handle. > > dmesg shows: > > [ 45.941350] overlayfs: failed to verify upper (boot/play, > ino=379405, err=-116) > [ 45.941369] overlayfs: failed to verify index dir 'upper' xattr > [ 45.941379] overlayfs: try deleting index dir or mounting with '-o > index=off' to disable inodes index. > Did you by any change re-create boot/play dir without re-creating boot/work dir? because that is what that message means. I doubt it has to do with the power cycle. > It is a bit concerning that I need to consider a power cycle may > require local console use. I plan on managing this remotely. For now > I'll leave it noauto and remotely mount when needed. so this isn't a > critical blocker to me. > > What would happen: errors=remount-ro > > errors={continue|remount-ro|panic} > Define the behavior when an error is encountered. The > default is set in the filesystem superblock, ... and can be > changed using tune2fs(8). > This looks like a quote from the man page section about ext2 mount options. Those are not options supported for overlayfs. Anyway, with that error, as the kernel log says, you could have mounted overlayfs ro with -o index=off and we could have added a mount option to fallback to ro mount with index=off, but that means no nfs_export, so not sure if that is what you wanted. Thanks, Amir. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: failed to verify upper - halts boot 2021-12-07 5:55 ` Amir Goldstein @ 2021-12-07 21:17 ` Carl Karsten 2021-12-08 7:01 ` Amir Goldstein 0 siblings, 1 reply; 5+ messages in thread From: Carl Karsten @ 2021-12-07 21:17 UTC (permalink / raw) To: Amir Goldstein; +Cc: overlayfs On Mon, Dec 6, 2021 at 9:55 PM Amir Goldstein <amir73il@gmail.com> wrote: > > On Tue, Dec 7, 2021 at 4:38 AM Carl Karsten <carl@nextdayvideo.com> wrote: > > > > I powercyced my box, I'm assuming I did a sudo poweroff or halt or > > something, but maybe not. > > > > It got stuck, (enter rescue mode, root account disabled, hit enter to > > enter rescue mode... ) so I killed power. > > > > I edited fstab, changed auto to noauto, booted. > > > > overlay /srv/nfs/rpi/bullseye/boot/merged overlay > > noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on > > 0 0 > > > > # mount /srv/nfs/rpi/bullseye/boot/merged > > mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed: > > Stale file handle. > > > > dmesg shows: > > > > [ 45.941350] overlayfs: failed to verify upper (boot/play, > > ino=379405, err=-116) > > [ 45.941369] overlayfs: failed to verify index dir 'upper' xattr > > [ 45.941379] overlayfs: try deleting index dir or mounting with '-o > > index=off' to disable inodes index. > > > > Did you by any change re-create boot/play dir without re-creating > boot/work dir? because that is what that message means. > I doubt it has to do with the power cycle. Maybe. would that include: mount: lower=base,setup upper= updates unmount mount: lower=base,setup,updates upper=play ? That might have happened. > > > It is a bit concerning that I need to consider a power cycle may > > require local console use. I plan on managing this remotely. For now > > I'll leave it noauto and remotely mount when needed. so this isn't a > > critical blocker to me. > > > > What would happen: errors=remount-ro > > > > errors={continue|remount-ro|panic} > > Define the behavior when an error is encountered. The > > default is set in the filesystem superblock, ... and can be > > changed using tune2fs(8). > > > > This looks like a quote from the man page section about ext2 mount options. > Those are not options supported for overlayfs. > > Anyway, with that error, as the kernel log says, you could have mounted > overlayfs ro with -o index=off and we could have added a mount option > to fallback to ro mount with index=off, but that means no nfs_export, > so not sure if that is what you wanted. That would be an improvement, as would errors=continue. Either of these would let a healthy system come up and be usable, and errors would let the system come up (finish booting) but the nfs/overlay part needs attention. Currently I have noauto so any reboot requires attention. If I set auto, an error will cause the boot to stop and I don't have a way to fix it remotely. I don't want to risk that, so noauto. I think I would prefer errors=continue as It would be more obvious that the overlayfs mount has issues. > > Thanks, > Amir. -- Carl K ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: failed to verify upper - halts boot 2021-12-07 21:17 ` Carl Karsten @ 2021-12-08 7:01 ` Amir Goldstein 0 siblings, 0 replies; 5+ messages in thread From: Amir Goldstein @ 2021-12-08 7:01 UTC (permalink / raw) To: Carl Karsten; +Cc: overlayfs On Tue, Dec 7, 2021 at 11:17 PM Carl Karsten <carl@nextdayvideo.com> wrote: > > On Mon, Dec 6, 2021 at 9:55 PM Amir Goldstein <amir73il@gmail.com> wrote: > > > > On Tue, Dec 7, 2021 at 4:38 AM Carl Karsten <carl@nextdayvideo.com> wrote: > > > > > > I powercyced my box, I'm assuming I did a sudo poweroff or halt or > > > something, but maybe not. > > > > > > It got stuck, (enter rescue mode, root account disabled, hit enter to > > > enter rescue mode... ) so I killed power. > > > > > > I edited fstab, changed auto to noauto, booted. > > > > > > overlay /srv/nfs/rpi/bullseye/boot/merged overlay > > > noauto,defaults,ro,lowerdir=/srv/nfs/rpi/bullseye/boot/updates:/srv/nfs/rpi/bullseye/boot/setup:/srv/nfs/rpi/bullseye/boot/base,upperdir=/srv/nfs/rpi/bullseye/boot/play,workdir=/srv/nfs/rpi/bullseye/boot/work,nfs_export=on > > > 0 0 > > > > > > # mount /srv/nfs/rpi/bullseye/boot/merged > > > mount: /srv/nfs/rpi/bullseye/boot/merged: mount(2) system call failed: > > > Stale file handle. > > > > > > dmesg shows: > > > > > > [ 45.941350] overlayfs: failed to verify upper (boot/play, > > > ino=379405, err=-116) > > > [ 45.941369] overlayfs: failed to verify index dir 'upper' xattr > > > [ 45.941379] overlayfs: try deleting index dir or mounting with '-o > > > index=off' to disable inodes index. > > > > > > > Did you by any change re-create boot/play dir without re-creating > > boot/work dir? because that is what that message means. > > I doubt it has to do with the power cycle. > > Maybe. > > would that include: > mount: lower=base,setup upper= updates > unmount > mount: lower=base,setup,updates upper=play > ? > > That might have happened. Yes, that would do it. > > > > > > It is a bit concerning that I need to consider a power cycle may > > > require local console use. I plan on managing this remotely. For now > > > I'll leave it noauto and remotely mount when needed. so this isn't a > > > critical blocker to me. > > > > > > What would happen: errors=remount-ro > > > > > > errors={continue|remount-ro|panic} > > > Define the behavior when an error is encountered. The > > > default is set in the filesystem superblock, ... and can be > > > changed using tune2fs(8). > > > > > > > This looks like a quote from the man page section about ext2 mount options. > > Those are not options supported for overlayfs. > > > > Anyway, with that error, as the kernel log says, you could have mounted > > overlayfs ro with -o index=off and we could have added a mount option > > to fallback to ro mount with index=off, but that means no nfs_export, > > so not sure if that is what you wanted. > > That would be an improvement, as would errors=continue. > > Either of these would let a healthy system come up and be usable, and > errors would let the system come up (finish booting) but the > nfs/overlay part needs attention. > > Currently I have noauto so any reboot requires attention. > If I set auto, an error will cause the boot to stop and I don't have a > way to fix it remotely. I don't want to risk that, so noauto. > There is no need for a kernel change. You can use boot scripts or mount helper to implement the mount options fallback that suits your system needs. > I think I would prefer errors=continue as It would be more obvious > that the overlayfs mount has issues. > That is not likely to happen and FYI errors=continue does not even mean what you think for ext4 where it is supported. It does not continue on failure to mount, it continues on IO and fs corruption errors detected during operations on a mounted fs. Thanks, Amir. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-12-08 7:01 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-12-05 1:04 failed to verify upper - halts boot Carl Karsten 2021-12-05 1:13 ` Carl Karsten 2021-12-07 5:55 ` Amir Goldstein 2021-12-07 21:17 ` Carl Karsten 2021-12-08 7:01 ` Amir Goldstein
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).