"rmdir_path: lstat of <path> failed" issues, despite updated autofs RPM

"rmdir_path: lstat of <path> failed" issues, despite updated autofs RPM

[Greg Earle]

Ian et al.:

We just upgraded a production server from RHEL 5.4 to 5.10.  (Yeah, yeah, I
know - I work at a place that has things called "flight project freezes".)

Ever since then, we keep getting these mysterious autofs messages:

[root@mipldiv log]# grep rmdir_path messages | wc -l
100

[root@mipldiv log]# grep rmdir_path messages | tail -5
Nov  6 08:30:05 mipldiv automount[31419]: rmdir_path: lstat of /home/objects failed
Nov  6 09:30:05 mipldiv automount[31419]: rmdir_path: lstat of /home/objects failed
Nov  6 11:30:05 mipldiv automount[31419]: rmdir_path: lstat of /home/refs failed
Nov  6 11:45:05 mipldiv automount[31419]: rmdir_path: lstat of /home/objects failed
Nov  6 12:30:05 mipldiv automount[31419]: rmdir_path: lstat of /home/refs failed

[root@mipldiv log]# ypcat -k auto.home | grep objects
[root@mipldiv log]# ypcat -k auto.home | grep refs
[root@mipldiv log]# 

(I have no idea where these "/home/objects" or "/home/refs" paths come from;
we do not have any accounts with those names, obviously.)

We're running the latest RHEL 5.x autofs version as far as I can tell:

[root@mipldiv log]# rpm -qa autofs*
autofs-5.0.1-0.rc2.184.el5

I have this problem on several machines; including - bizarrely enough -
3 machines on one flight project that get a rash of them all at exactly
the same time (usually 8:45 PM).  Those systems are currently frozen but
may also be upgraded to 5.10 soon.

Bug 336961 implies that this was addressed in an Errata (RHBA-2008-0354),
but we are a full superseded Errata revision (RHBA-2014:1240-2) beyond that
one already, and yet I'm still seeing these messages.

https://bugzilla.redhat.com/show_bug.cgi?id=336961

Given the age of RHEL 5.x am I better off trying to track down the cause
of these phantom "/home" paths so the messages won't come out at all?
Because I really don't have a clue where these "objects" & "refs" references
are coming from.

	- Greg

--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: "rmdir_path: lstat of <path> failed" issues, despite updated autofs RPM

[Ian Kent]

On Fri, 2015-11-06 at 13:03 -0800, Greg Earle wrote:
> Ian et al.:
> 
> We just upgraded a production server from RHEL 5.4 to 5.10.  (Yeah,
> yeah, I
> know - I work at a place that has things called "flight project
> freezes".)
> 
> Ever since then, we keep getting these mysterious autofs messages:
> 
> [root@mipldiv log]# grep rmdir_path messages | wc -l
> 100
> 
> [root@mipldiv log]# grep rmdir_path messages | tail -5
> Nov  6 08:30:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/objects failed
> Nov  6 09:30:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/objects failed
> Nov  6 11:30:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/refs failed
> Nov  6 11:45:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/objects failed
> Nov  6 12:30:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/refs failed
> 
> [root@mipldiv log]# ypcat -k auto.home | grep objects
> [root@mipldiv log]# ypcat -k auto.home | grep refs
> [root@mipldiv log]# 
> 
> (I have no idea where these "/home/objects" or "/home/refs" paths
> come from;
> we do not have any accounts with those names, obviously.)
> 
> We're running the latest RHEL 5.x autofs version as far as I can
> tell:
> 
> [root@mipldiv log]# rpm -qa autofs*
> autofs-5.0.1-0.rc2.184.el5
> 
> I have this problem on several machines; including - bizarrely enough
> -
> 3 machines on one flight project that get a rash of them all at
> exactly
> the same time (usually 8:45 PM).  Those systems are currently frozen
> but
> may also be upgraded to 5.10 soon.
> 
> Bug 336961 implies that this was addressed in an Errata (RHBA-2008
> -0354),
> but we are a full superseded Errata revision (RHBA-2014:1240-2)
> beyond that
> one already, and yet I'm still seeing these messages.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=336961
> 
> Given the age of RHEL 5.x am I better off trying to track down the
> cause
> of these phantom "/home" paths so the messages won't come out at all?
> Because I really don't have a clue where these "objects" & "refs"
> references
> are coming from.

Not really sure.
That is kinda old!

Do those directories actually exist within /home?
Has the machine been up for a long time?
Are they actually causing a problem, other than annoyance value?

They could be left over from when they where keys in a map but not
removed on cleanup for some reason or they could be the result of a
failed or partial lookup and also not cleaned up on error.

You could try scheduling a reboot at some opportune time as that will
result in a clean autofs directory at system startup since the autofs
pseudo file system is ram based. Then if the messages continue it must
be some process attempting access of a path starting with /home/<dir>.

Ian
--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: "rmdir_path: lstat of <path> failed" issues, despite updated autofs RPM

[Greg Earle]

> On Nov 7, 2015, at 10:33 PM, Ian Kent <raven@themaw.net> wrote:
> 
> Do those directories actually exist within /home?

Thanks for responding, Ian.

"objects" exists (apparently due to tickling by the automounter) but "refs"
does not:

[root@mipldiv log]# ls -ltF /home | head -2
total 40
dr-xr-xr-x  2 root   root        0 Nov  8 00:15 objects/

[root@mipldiv log]# ls -ltF /home/objects
ls: /home/objects: No such file or directory

> Has the machine been up for a long time?

Ironically, they started happening shortly AFTER the machine was upgraded
from RHEL 5.4 to 5.10 (with several reboots).  (It's been up for just over
4 days now, and it's still doing it every 15 or 30 minutes.)

[root@mipldiv log]# grep failed messages | grep -v bin/nrpe | tail -5
Nov  7 22:00:05 mipldiv automount[31419]: rmdir_path: lstat of /home/refs failed
Nov  7 22:15:05 mipldiv automount[31419]: rmdir_path: lstat of /home/objects failed
Nov  7 23:00:05 mipldiv automount[31419]: rmdir_path: lstat of /home/refs failed
Nov  7 23:30:05 mipldiv automount[31419]: rmdir_path: lstat of /home/refs failed
Nov  8 00:00:06 mipldiv automount[31419]: rmdir_path: lstat of /home/objects failed

> Are they actually causing a problem, other than annoyance value?

Just an annoyance as far as I can tell - our Xymon monitoring system
triggers on "failed" so we get red alerts from the system(s) whenever
this happens.  My officemate isn't happy  :-)

> They could be left over from when they where keys in a map but not
> removed on cleanup for some reason, or they could be the result of a
> failed or partial lookup and also not cleaned up on error.

The keys were never in a map.  That is what is so weird.  I went to our
Solaris NIS master and there is no reference to "objects" or "refs" in
/etc/auto_home or /etc/auto_*.  I can't imagine what could be running
that is trying to access those two phantom "/home" paths.

> You could try scheduling a reboot at some opportune time as that will
> result in a clean autofs directory at system startup since the autofs
> pseudo file system is RAM based.  Then if the messages continue, it must
> be some process attempting access of a path starting with /home/<dir>.

Well, given that the behavior really kicked in after the last post-install
reboot, I'm guessing your latter explanation is far more likely.  :-)

I've just turned process accounting on to see what is running on those
15-minute intervals that might be triggering these.  Already checked the
"cron" jobs and there aren't any on that frequency.  Most curious ...

	- Greg

--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: "rmdir_path: lstat of <path> failed" issues, despite updated autofs RPM

[Ian Kent]

On Sun, 2015-11-08 at 00:22 -0800, Greg Earle wrote:
> > On Nov 7, 2015, at 10:33 PM, Ian Kent <raven@themaw.net> wrote:
> > 
> > Do those directories actually exist within /home?
> 
> Thanks for responding, Ian.
> 
> "objects" exists (apparently due to tickling by the automounter) but
> "refs"
> does not:
> 
> [root@mipldiv log]# ls -ltF /home | head -2
> total 40
> dr-xr-xr-x  2 root   root        0 Nov  8 00:15 objects/
> 
> [root@mipldiv log]# ls -ltF /home/objects
> ls: /home/objects: No such file or directory
> 
> > Has the machine been up for a long time?
> 
> Ironically, they started happening shortly AFTER the machine was
> upgraded
> from RHEL 5.4 to 5.10 (with several reboots).  (It's been up for just
> over
> 4 days now, and it's still doing it every 15 or 30 minutes.)
> 
> [root@mipldiv log]# grep failed messages | grep -v bin/nrpe | tail -5
> Nov  7 22:00:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/refs failed
> Nov  7 22:15:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/objects failed
> Nov  7 23:00:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/refs failed
> Nov  7 23:30:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/refs failed
> Nov  8 00:00:06 mipldiv automount[31419]: rmdir_path: lstat of
> /home/objects failed
> 
> > Are they actually causing a problem, other than annoyance value?
> 
> Just an annoyance as far as I can tell - our Xymon monitoring system
> triggers on "failed" so we get red alerts from the system(s) whenever
> this happens.  My officemate isn't happy  :-)
> 
> > They could be left over from when they where keys in a map but not
> > removed on cleanup for some reason, or they could be the result of
> > a
> > failed or partial lookup and also not cleaned up on error.
> 
> The keys were never in a map.  That is what is so weird.  I went to
> our
> Solaris NIS master and there is no reference to "objects" or "refs"
> in
> /etc/auto_home or /etc/auto_*.  I can't imagine what could be running
> that is trying to access those two phantom "/home" paths.

Yeah, that function is called from a few places.

At map read time, at autofs fs mount time and on HUP signal or if a
lookup makes autofs think the map has been modified and at expire when
attempting to remove a path on failed mount attempts for some fs types,
notably NFS.

The read map is driven by what's in the internal map entry cache
so they could be negative entries added due to a failed lookup.

The problem might be that the function could get called multiple times
and fails on the later but that would imply bad path lookups occurring
fairly regularly.

Puzzling thing is I haven't had other reports of it.

> 
> > You could try scheduling a reboot at some opportune time as that
> > will
> > result in a clean autofs directory at system startup since the
> > autofs
> > pseudo file system is RAM based.  Then if the messages continue, it
> > must
> > be some process attempting access of a path starting with
> > /home/<dir>.
> 
> Well, given that the behavior really kicked in after the last post
> -install
> reboot, I'm guessing your latter explanation is far more likely.  :-)
> 
> I've just turned process accounting on to see what is running on
> those
> 15-minute intervals that might be triggering these.  Already checked
> the
> "cron" jobs and there aren't any on that frequency.  Most curious ...
> 
> 	- Greg
> 
> --
> To unsubscribe from this list: send the line "unsubscribe autofs" in
--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: "rmdir_path: lstat of <path> failed" issues, despite updated autofs RPM

[Greg Earle]

> On Nov 8, 2015, at 3:47 PM, Ian Kent <raven@themaw.net> wrote:
> 
>> The keys were never in a map.  That is what is so weird.  I went to
>> our
>> Solaris NIS master and there is no reference to "objects" or "refs"
>> in
>> /etc/auto_home or /etc/auto_*.  I can't imagine what could be running
>> that is trying to access those two phantom "/home" paths.
> 
> Yeah, that function is called from a few places.
> 
> At map read time, at autofs fs mount time and on HUP signal or if a
> lookup makes autofs think the map has been modified and at expire when
> attempting to remove a path on failed mount attempts for some fs types,
> notably NFS.

I checked and of course it was still doing it:

Nov  9 12:30:05 mipldiv automount[31419]: rmdir_path: lstat of /home/objects \
failed

So I SIGHUP'ed the daemon and immediately got another one:

Nov  9 12:35:36 mipldiv automount[31419]: rmdir_path: lstat of /home/objects \
failed

I can't stop & restart the autofs service because it's a production server
for a flight project.

Seems like there's some sort of saved state that it reloads when HUP'ed,
that brings this message out?

	- Greg

--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: "rmdir_path: lstat of <path> failed" issues, despite updated autofs RPM

[Ian Kent]

On Mon, 2015-11-09 at 12:42 -0800, Greg Earle wrote:
> > On Nov 8, 2015, at 3:47 PM, Ian Kent <raven@themaw.net> wrote:
> > 
> > > The keys were never in a map.  That is what is so weird.  I went
> > > to
> > > our
> > > Solaris NIS master and there is no reference to "objects" or
> > > "refs"
> > > in
> > > /etc/auto_home or /etc/auto_*.  I can't imagine what could be
> > > running
> > > that is trying to access those two phantom "/home" paths.
> > 
> > Yeah, that function is called from a few places.
> > 
> > At map read time, at autofs fs mount time and on HUP signal or if a
> > lookup makes autofs think the map has been modified and at expire
> > when
> > attempting to remove a path on failed mount attempts for some fs
> > types,
> > notably NFS.
> 
> I checked and of course it was still doing it:
> 
> Nov  9 12:30:05 mipldiv automount[31419]: rmdir_path: lstat of
> /home/objects \
> failed
> 
> So I SIGHUP'ed the daemon and immediately got another one:
> 
> Nov  9 12:35:36 mipldiv automount[31419]: rmdir_path: lstat of
> /home/objects \
> failed

So it sounds like it's occurring in the map entry cache prune function.

The puzzle is why the lstat() failing since that directory does exist
according to what we have above.

There's also the question of why it continues to occur.

I looked briefly at the RHEL code (actually rev 184 was 5.11) for the
prune function, which is driven by what's in the map entry cache, and
it looks like the map entry cache entry is removed just before the
rmdir_path() call. So it shouldn't keep coming back if there are no
further accesses.

I probably didn't look closely enough though.

> 
> I can't stop & restart the autofs service because it's a production
> server
> for a flight project.

Even though you should be able to perform a restart without disrupting
autofs users it's not something I'd recommend on a production machine.

There is a small window of time when autofs won't respond to requests
and if requests arrive within that window they could hang forever. Not
sure how to mitigate against that either.

But that probably wouldn't be enough anyway.

If there are directories that are possibly broken in some way then to
clear them up the autofs managed mount point must be umounted.

But if any mounts are busy at shutdown they are left mounted, and the
autofs mount itself obviously must be left mounted too, then autofs re
-connects to the mounts when it starts again.

That complexity is the reason I usually just recommend a re-boot be
scheduled, plus if there is some sort of brokenness within the mounted
autofs file system there's no knowing if there were side effects when
it happened.

Sorry, I'm not really much help with this.
Ian

--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: "rmdir_path: lstat of <path> failed" issues, despite updated autofs RPM

[Greg Earle]

> On Nov 9, 2015, at 9:48 PM, Ian Kent <raven@themaw.net> wrote:
> 
> If there are directories that are possibly broken in some way then to
> clear them up the autofs managed mount point must be umounted.
> 
> But if any mounts are busy at shutdown they are left mounted, and the
> autofs mount itself obviously must be left mounted too, then autofs re
> -connects to the mounts when it starts again.

That's interesting, never knew that before.  So maybe it thinks these
phantom mounts should still be "mounted", and keeps trying to do it.


> That complexity is the reason I usually just recommend a re-boot be
> scheduled, plus if there is some sort of brokenness within the mounted
> autofs file system there's no knowing if there were side effects when
> it happened.
> 
> Sorry, I'm not really much help with this.

Well, I've already told Xymon to ignore these (at my peril!), so as long
as they are just harmless messages, I'll live  :-)

Thanks for looking at the code just in case.  Was hoping it might be
something simple; obviously it wasn't.

	- Greg

--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: "rmdir_path: lstat of <path> failed" issues, despite updated autofs RPM

[Ian Kent]

On Mon, 2015-11-09 at 23:18 -0800, Greg Earle wrote:
> > On Nov 9, 2015, at 9:48 PM, Ian Kent <raven@themaw.net> wrote:
> > 
> > If there are directories that are possibly broken in some way then
> > to
> > clear them up the autofs managed mount point must be umounted.
> > 
> > But if any mounts are busy at shutdown they are left mounted, and
> > the
> > autofs mount itself obviously must be left mounted too, then autofs
> > re
> > -connects to the mounts when it starts again.
> 
> That's interesting, never knew that before.  So maybe it thinks these
> phantom mounts should still be "mounted", and keeps trying to do it.
> 
> 
> > That complexity is the reason I usually just recommend a re-boot be
> > scheduled, plus if there is some sort of brokenness within the
> > mounted
> > autofs file system there's no knowing if there were side effects
> > when
> > it happened.
> > 
> > Sorry, I'm not really much help with this.
> 
> Well, I've already told Xymon to ignore these (at my peril!), so as
> long
> as they are just harmless messages, I'll live  :-)

It's hard to know for sure if these indicate you're in for further
problems but there doesn't seem to be much you can do about it for the
time being.

As long as the system continues to provide the expected service then I
have to say it should be fine to ignore them.

But I always get a bit nervous when I see unexpected things like this
so I still recommend scheduling a re-boot. As long as you aren't seeing
other bad things happen the urgency of doing so isn't high.

So as and when the opportunity arises would be the thing to do. At
least get the request on the operations radar.

> 
> Thanks for looking at the code just in case.  Was hoping it might be
> something simple; obviously it wasn't.
> 
> 	- Greg
> 
> --
> To unsubscribe from this list: send the line "unsubscribe autofs" in
--
To unsubscribe from this list: send the line "unsubscribe autofs" in