nfs_export Stale file handle

nfs_export Stale file handle

[Carl Karsten]

I am sure this worked once, then I rebooted and now something ....

root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=on
overlay -olowerdir=img,upperdir=upper,workdir=work merged
mount: /srv/nfs/rpi/root/merged: mount(2) system call failed: Stale file handle.

# this works:
root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=off
overlay -olowerdir=img,upperdir=upper,workdir=work merged
mount: overlay mounted on /srv/nfs/rpi/root/merged.
root@negk:/srv/nfs/rpi/root# umount merged

# syslog:

[   80.317697] overlayfs: failed to verify origin (root/img,
ino=2374476, err=-116)
[   80.317703] overlayfs: failed to verify upper root origin

# no help:
systemctl disable nfs-server.service
reboot
root@negk:/srv/nfs/rpi/root# systemctl status nfs-server.service
● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; disabled;
vendor preset: enabled)
     Active: inactive (dead)





-- 
Carl K

Re: nfs_export Stale file handle

[Amir Goldstein]

On Tue, Oct 19, 2021 at 10:49 PM Carl Karsten <carl@nextdayvideo.com> wrote:
>
> I am sure this worked once, then I rebooted and now something ....

Are you saying it is a regression? From which kernel?

>
> root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=on
> overlay -olowerdir=img,upperdir=upper,workdir=work merged
> mount: /srv/nfs/rpi/root/merged: mount(2) system call failed: Stale file handle.
>

What are lower and upper?
cat /proc/self/mountinfo please
If you happen to be using squashfs for lower fs, there were a bunch
of changes and fixes in recent kernels.

> # this works:
> root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=off
> overlay -olowerdir=img,upperdir=upper,workdir=work merged
> mount: overlay mounted on /srv/nfs/rpi/root/merged.
> root@negk:/srv/nfs/rpi/root# umount merged
>
> # syslog:
>
> [   80.317697] overlayfs: failed to verify origin (root/img,
> ino=2374476, err=-116)
> [   80.317703] overlayfs: failed to verify upper root origin
>
> # no help:
> systemctl disable nfs-server.service
> reboot
> root@negk:/srv/nfs/rpi/root# systemctl status nfs-server.service
> ● nfs-server.service - NFS server and services
>      Loaded: loaded (/lib/systemd/system/nfs-server.service; disabled;
> vendor preset: enabled)
>      Active: inactive (dead)
>

The error has nothing to do with NFS.
The staleness is that of the root/img directory.
Overlayfs believes that someone has replaced the original
root/img directory with another directory on the same name
but a different file handle.

Did you re-create the lower image filesystem/subtree?
See Documentation clarification commit
13c6ad0f45fd ("ovl: document lower modification caveats")

Thanks,
Amir.

Re: nfs_export Stale file handle

[Carl Karsten]

On Wed, Oct 20, 2021 at 12:42 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Tue, Oct 19, 2021 at 10:49 PM Carl Karsten <carl@nextdayvideo.com> wrote:
> >
> > I am sure this worked once, then I rebooted and now something ....
>
> Are you saying it is a regression? From which kernel?

Not a regression - just something that I am sure was OK on my box,
then 20 min later the 'same thing' is causing problems.  We all know
it isn't the same, but this is what has me scratching my head.

>
> >
> > root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=on
> > overlay -olowerdir=img,upperdir=upper,workdir=work merged
> > mount: /srv/nfs/rpi/root/merged: mount(2) system call failed: Stale file handle.
> >
>
> What are lower and upper?

img was
/dev/mapper/loop2p2 on /srv/nfs/rpi/buster/root/img type ext4 (rw,relatime)

> cat /proc/self/mountinfo please

sorry, I've reinstalled the os and I'm not having this problem any more.

> If you happen to be using squashfs for lower fs, there were a bunch
> of changes and fixes in recent kernels.

no squashfs

>
> > # this works:
> > root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=off
> > overlay -olowerdir=img,upperdir=upper,workdir=work merged
> > mount: overlay mounted on /srv/nfs/rpi/root/merged.
> > root@negk:/srv/nfs/rpi/root# umount merged
> >
> > # syslog:
> >
> > [   80.317697] overlayfs: failed to verify origin (root/img,
> > ino=2374476, err=-116)
> > [   80.317703] overlayfs: failed to verify upper root origin
> >
> > # no help:
> > systemctl disable nfs-server.service
> > reboot
> > root@negk:/srv/nfs/rpi/root# systemctl status nfs-server.service
> > ● nfs-server.service - NFS server and services
> >      Loaded: loaded (/lib/systemd/system/nfs-server.service; disabled;
> > vendor preset: enabled)
> >      Active: inactive (dead)
> >
>
> The error has nothing to do with NFS.
> The staleness is that of the root/img directory.
> Overlayfs believes that someone has replaced the original
> root/img directory with another directory on the same name
> but a different file handle.
>

the only difference between error and no error was

nfs_export=off / on

Which is mostly why I posted - this little detail seemed odd.

> Did you re-create the lower image filesystem/subtree?

maybe.  I'm futzing around with this pi image file trying to figure
out how to manage various flavors of modifications. .
I was hoping the lowest level would be the fs in the .img file, but
there isn't a nice way to deal with the loop devices, so I'll copy the
files onto the base ext5 fs and get on with things.

At the moment my little sysadmin project has lots of moving parts that
are not at all reliable, it's hard to tell what is and isn't working.
robust is the goal, I'm getting there.


> See Documentation clarification commit
> 13c6ad0f45fd ("ovl: document lower modification caveats")
>
> Thanks,
> Amir.



-- 
Carl K

Re: nfs_export Stale file handle

[Amir Goldstein]

On Wed, Oct 20, 2021 at 9:49 AM Carl Karsten <carl@nextdayvideo.com> wrote:
>
> On Wed, Oct 20, 2021 at 12:42 AM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > On Tue, Oct 19, 2021 at 10:49 PM Carl Karsten <carl@nextdayvideo.com> wrote:
> > >
> > > I am sure this worked once, then I rebooted and now something ....
> >
> > Are you saying it is a regression? From which kernel?
>
> Not a regression - just something that I am sure was OK on my box,
> then 20 min later the 'same thing' is causing problems.  We all know
> it isn't the same, but this is what has me scratching my head.
>
> >
> > >
> > > root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=on
> > > overlay -olowerdir=img,upperdir=upper,workdir=work merged
> > > mount: /srv/nfs/rpi/root/merged: mount(2) system call failed: Stale file handle.
> > >
> >
> > What are lower and upper?
>
> img was
> /dev/mapper/loop2p2 on /srv/nfs/rpi/buster/root/img type ext4 (rw,relatime)
>
> > cat /proc/self/mountinfo please
>
> sorry, I've reinstalled the os and I'm not having this problem any more.
>
> > If you happen to be using squashfs for lower fs, there were a bunch
> > of changes and fixes in recent kernels.
>
> no squashfs
>
> >
> > > # this works:
> > > root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=off
> > > overlay -olowerdir=img,upperdir=upper,workdir=work merged
> > > mount: overlay mounted on /srv/nfs/rpi/root/merged.
> > > root@negk:/srv/nfs/rpi/root# umount merged
> > >
> > > # syslog:
> > >
> > > [   80.317697] overlayfs: failed to verify origin (root/img,
> > > ino=2374476, err=-116)
> > > [   80.317703] overlayfs: failed to verify upper root origin
> > >
> > > # no help:
> > > systemctl disable nfs-server.service
> > > reboot
> > > root@negk:/srv/nfs/rpi/root# systemctl status nfs-server.service
> > > ● nfs-server.service - NFS server and services
> > >      Loaded: loaded (/lib/systemd/system/nfs-server.service; disabled;
> > > vendor preset: enabled)
> > >      Active: inactive (dead)
> > >
> >
> > The error has nothing to do with NFS.
> > The staleness is that of the root/img directory.
> > Overlayfs believes that someone has replaced the original
> > root/img directory with another directory on the same name
> > but a different file handle.
> >
>
> the only difference between error and no error was
>
> nfs_export=off / on
>
> Which is mostly why I posted - this little detail seemed odd.

This is intentional as the documentation commit clarifies.
You are allowed to re-create lower dir when not enabling
index or nfs_export which implies index=on.
You are not allowed to re-create lower dir when using new features.
Specifically, index and nfs_export features simply cannot work
with re-created lower. Read about this in overlay documentation.

>
> > Did you re-create the lower image filesystem/subtree?
>
> maybe.  I'm futzing around with this pi image file trying to figure
> out how to manage various flavors of modifications. .
> I was hoping the lowest level would be the fs in the .img file, but
> there isn't a nice way to deal with the loop devices, so I'll copy the
> files onto the base ext5 fs and get on with things.
>

When you copy lower layer, you need to re-create an empty upper dir
you cannot mount an old upper dir with new lowerdir when nfs_export is
enabled.

> At the moment my little sysadmin project has lots of moving parts that
> are not at all reliable, it's hard to tell what is and isn't working.
> robust is the goal, I'm getting there.
>

Good luck.
Thanks,
Amir.

Re: nfs_export Stale file handle

[Carl Karsten]

 bumping into this again, and now I have more stable .. thing.

my stack of dirs:
base - copy of files from raspios-buster-armhf-lite.img
setup - config files my setup adds
updates - apt update/upgrade
play - random things I am testing

work - overlayfs working dir
merged - overlayfs mount point

I cp files in base and setup,
mount base setup updates play on mounted
write some files into mounted

Now I want to undo:
rm play work merged
mkdir play work merged
mount...
error Stale file handle.

juser@negk:~$ sudo ./test.sh
+ rm -rf /srv/nfs/rpi/buster/boot/play /srv/nfs/rpi/buster/boot/work
/srv/nfs/rpi/buster/boot/merged
+ mkdir /srv/nfs/rpi/buster/boot/play /srv/nfs/rpi/buster/boot/work
/srv/nfs/rpi/buster/boot/merged
+ mount -t overlay overlay -o
nfs_export=on,lowerdir=/srv/nfs/rpi/buster/boot/base:/srv/nfs/rpi/buster/boot/setup,upperdir=/srv/nfs/rpi/buster/boot/updates,workdir=/srv/nfs/rpi/buster/boot/work
/srv/nfs/rpi/buster/boot/merged
mount: /srv/nfs/rpi/buster/boot/merged: mount(2) system call failed:
Stale file handle.
[ 1602.271239] overlayfs: failed to verify origin (boot/base,
ino=4194652, err=-116)
[ 1602.271244] overlayfs: failed to verify upper root origin

On Wed, Oct 20, 2021 at 2:47 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Wed, Oct 20, 2021 at 9:49 AM Carl Karsten <carl@nextdayvideo.com> wrote:
> >
> > On Wed, Oct 20, 2021 at 12:42 AM Amir Goldstein <amir73il@gmail.com> wrote:
> > >
> > > On Tue, Oct 19, 2021 at 10:49 PM Carl Karsten <carl@nextdayvideo.com> wrote:
> > > >
> > > > I am sure this worked once, then I rebooted and now something ....
> > >
> > > Are you saying it is a regression? From which kernel?
> >
> > Not a regression - just something that I am sure was OK on my box,
> > then 20 min later the 'same thing' is causing problems.  We all know
> > it isn't the same, but this is what has me scratching my head.
> >
> > >
> > > >
> > > > root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=on
> > > > overlay -olowerdir=img,upperdir=upper,workdir=work merged
> > > > mount: /srv/nfs/rpi/root/merged: mount(2) system call failed: Stale file handle.
> > > >
> > >
> > > What are lower and upper?
> >
> > img was
> > /dev/mapper/loop2p2 on /srv/nfs/rpi/buster/root/img type ext4 (rw,relatime)
> >
> > > cat /proc/self/mountinfo please
> >
> > sorry, I've reinstalled the os and I'm not having this problem any more.
> >
> > > If you happen to be using squashfs for lower fs, there were a bunch
> > > of changes and fixes in recent kernels.
> >
> > no squashfs
> >
> > >
> > > > # this works:
> > > > root@negk:/srv/nfs/rpi/root# mount -v -t overlay -o nfs_export=off
> > > > overlay -olowerdir=img,upperdir=upper,workdir=work merged
> > > > mount: overlay mounted on /srv/nfs/rpi/root/merged.
> > > > root@negk:/srv/nfs/rpi/root# umount merged
> > > >
> > > > # syslog:
> > > >
> > > > [   80.317697] overlayfs: failed to verify origin (root/img,
> > > > ino=2374476, err=-116)
> > > > [   80.317703] overlayfs: failed to verify upper root origin
> > > >
> > > > # no help:
> > > > systemctl disable nfs-server.service
> > > > reboot
> > > > root@negk:/srv/nfs/rpi/root# systemctl status nfs-server.service
> > > > ● nfs-server.service - NFS server and services
> > > >      Loaded: loaded (/lib/systemd/system/nfs-server.service; disabled;
> > > > vendor preset: enabled)
> > > >      Active: inactive (dead)
> > > >
> > >
> > > The error has nothing to do with NFS.
> > > The staleness is that of the root/img directory.
> > > Overlayfs believes that someone has replaced the original
> > > root/img directory with another directory on the same name
> > > but a different file handle.
> > >
> >
> > the only difference between error and no error was
> >
> > nfs_export=off / on
> >
> > Which is mostly why I posted - this little detail seemed odd.
>
> This is intentional as the documentation commit clarifies.
> You are allowed to re-create lower dir when not enabling
> index or nfs_export which implies index=on.
> You are not allowed to re-create lower dir when using new features.
> Specifically, index and nfs_export features simply cannot work
> with re-created lower. Read about this in overlay documentation.
>
> >
> > > Did you re-create the lower image filesystem/subtree?
> >
> > maybe.  I'm futzing around with this pi image file trying to figure
> > out how to manage various flavors of modifications. .
> > I was hoping the lowest level would be the fs in the .img file, but
> > there isn't a nice way to deal with the loop devices, so I'll copy the
> > files onto the base ext5 fs and get on with things.
> >
>
> When you copy lower layer, you need to re-create an empty upper dir
> you cannot mount an old upper dir with new lowerdir when nfs_export is
> enabled.
>
> > At the moment my little sysadmin project has lots of moving parts that
> > are not at all reliable, it's hard to tell what is and isn't working.
> > robust is the goal, I'm getting there.
> >
>
> Good luck.
> Thanks,
> Amir.



-- 
Carl K

Re: nfs_export Stale file handle

[Amir Goldstein]

On Thu, Oct 28, 2021 at 9:44 PM Carl Karsten <carl@nextdayvideo.com> wrote:
>
>  bumping into this again, and now I have more stable .. thing.
>
> my stack of dirs:
> base - copy of files from raspios-buster-armhf-lite.img
> setup - config files my setup adds
> updates - apt update/upgrade
> play - random things I am testing
>
> work - overlayfs working dir
> merged - overlayfs mount point
>
> I cp files in base and setup,
> mount base setup updates play on mounted
> write some files into mounted
>
> Now I want to undo:
> rm play work merged
> mkdir play work merged
> mount...
> error Stale file handle.

You are not allowed to re-create lower dir when using nfs_expoprt.
If you re-create/replace/rearrange any of the lower dirs, you need
to also re-create the upperdir.

>
> juser@negk:~$ sudo ./test.sh
> + rm -rf /srv/nfs/rpi/buster/boot/play /srv/nfs/rpi/buster/boot/work
> /srv/nfs/rpi/buster/boot/merged
> + mkdir /srv/nfs/rpi/buster/boot/play /srv/nfs/rpi/buster/boot/work
> /srv/nfs/rpi/buster/boot/merged
> + mount -t overlay overlay -o
> nfs_export=on,lowerdir=/srv/nfs/rpi/buster/boot/base:/srv/nfs/rpi/buster/boot/setup,upperdir=/srv/nfs/rpi/buster/boot/updates,workdir=/srv/nfs/rpi/buster/boot/work

is ../boot/base/ supposed to be the uppermost lower layer.
It's name and you description above sounds like it should be the
bottom most layer,
but you stacked ../boot/setup below it.

> /srv/nfs/rpi/buster/boot/merged
> mount: /srv/nfs/rpi/buster/boot/merged: mount(2) system call failed:
> Stale file handle.
> [ 1602.271239] overlayfs: failed to verify origin (boot/base,
> ino=4194652, err=-116)
> [ 1602.271244] overlayfs: failed to verify upper root origin
>

translation: boot/base is not the directory that used to be right below
boot/updates. I guess you meant boot/setup to be on top of  boot/base?

Thanks,
Amir.

Re: nfs_export Stale file handle

[Carl Karsten]

On Thu, Oct 28, 2021 at 5:47 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Thu, Oct 28, 2021 at 9:44 PM Carl Karsten <carl@nextdayvideo.com> wrote:
> >
> >  bumping into this again, and now I have more stable .. thing.
> >
> > my stack of dirs:
> > base - copy of files from raspios-buster-armhf-lite.img
> > setup - config files my setup adds
> > updates - apt update/upgrade
> > play - random things I am testing
> >
> > work - overlayfs working dir
> > merged - overlayfs mount point
> >
> > I cp files in base and setup,
> > mount base setup updates play on mounted
> > write some files into mounted
> >
> > Now I want to undo:
> > rm play work merged
> > mkdir play work merged
> > mount...
> > error Stale file handle.
>
> You are not allowed to re-create lower dir when using nfs_expoprt.
> If you re-create/replace/rearrange any of the lower dirs, you need
> to also re-create the upperdir.
>
> >
> > juser@negk:~$ sudo ./test.sh
> > + rm -rf /srv/nfs/rpi/buster/boot/play /srv/nfs/rpi/buster/boot/work
> > /srv/nfs/rpi/buster/boot/merged
> > + mkdir /srv/nfs/rpi/buster/boot/play /srv/nfs/rpi/buster/boot/work
> > /srv/nfs/rpi/buster/boot/merged
> > + mount -t overlay overlay -o
> > nfs_export=on,lowerdir=/srv/nfs/rpi/buster/boot/base:/srv/nfs/rpi/buster/boot/setup,upperdir=/srv/nfs/rpi/buster/boot/updates,workdir=/srv/nfs/rpi/buster/boot/work
>
> is ../boot/base/ supposed to be the uppermost lower layer.
> It's name and you description above sounds like it should be the
> bottom most layer,
> but you stacked ../boot/setup below it.
>
> > /srv/nfs/rpi/buster/boot/merged
> > mount: /srv/nfs/rpi/buster/boot/merged: mount(2) system call failed:
> > Stale file handle.
> > [ 1602.271239] overlayfs: failed to verify origin (boot/base,
> > ino=4194652, err=-116)
> > [ 1602.271244] overlayfs: failed to verify upper root origin
> >
>
> translation: boot/base is not the directory that used to be right below
> boot/updates. I guess you meant boot/setup to be on top of  boot/base?
>

whoops.

> Thanks,
> Amir.

no no, thank you ;)


-- 
Carl K