Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Stephen Harris]

A lot of talk here has been about syslog and DNS blocking, but the
original message mentioned:

> If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3,
> kernel 2.2.x), within a few minutes you will find your entire machine
> grinds to a halt.  For example, nobody can log in.

Has this been addressed (and I missed it) or did the question get
diverted?  If it has been addressed, just please mail me personally to
save traffic on the list.

Thanks!
-- 
                                 Stephen Harris
                 sweh@spuddy.mew.co.uk   http://www.spuddy.org/
      The truth is the truth, and opinion just opinion.  But what is what?
       My employer pays to ignore my opinions; you get to do it for free.      
  * Meeeeow ! Call  Spud the Cat on > 01708 442043 < for free Usenet access *
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Horst von Brand]

sweh@spuddy.mew.co.uk (Stephen Harris) said:
> A lot of talk here has been about syslog and DNS blocking, but the
> original message mentioned:
> 
> > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3,
> > kernel 2.2.x), within a few minutes you will find your entire machine
> > grinds to a halt.  For example, nobody can log in.

Great! Yet another way in which root can get the rope to shoot herself in
the foot. Anything _really_ new?
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Stephen Harris]

Horst von Brand wrote:

> > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3,
> > > kernel 2.2.x), within a few minutes you will find your entire machine
> > > grinds to a halt.  For example, nobody can log in.
> 
> Great! Yet another way in which root can get the rope to shoot herself in
> the foot. Anything _really_ new?

OK, let's go a step further - what if syslog dies or breaks in some way
shape or form so that the syslog() function blocks...?

My worry is the one that was originally raised but ignored:  syslog() should
not BLOCK regardless of whether it's local or remote.  syslog is not a
reliable mechanism and many programs have been written assuming they can
fire off syslog() calls without worry.

                                 Stephen Harris
                 sweh@spuddy.mew.co.uk   http://www.spuddy.org/
      The truth is the truth, and opinion just opinion.  But what is what?
       My employer pays to ignore my opinions; you get to do it for free.      
  * Meeeeow ! Call  Spud the Cat on > 01708 442043 < for free Usenet access *
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Jesse Pollard]

On Sun, 29 Oct 2000, Stephen Harris wrote:
>Horst von Brand wrote:
>
>> > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3,
>> > > kernel 2.2.x), within a few minutes you will find your entire machine
>> > > grinds to a halt.  For example, nobody can log in.
>> 
>> Great! Yet another way in which root can get the rope to shoot herself in
>> the foot. Anything _really_ new?
>
>OK, let's go a step further - what if syslog dies or breaks in some way
>shape or form so that the syslog() function blocks...?
>
>My worry is the one that was originally raised but ignored:  syslog() should
>not BLOCK regardless of whether it's local or remote.  syslog is not a
>reliable mechanism and many programs have been written assuming they can
>fire off syslog() calls without worry.

It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a
few seconds (depending on the log rate...).

I do believe that restarting syslog should be possible... Perhaps syslog
should be started by inetd at the very beginning. Then it could be restarted
after an exit/abort.

This can STILL fail if the syslog.conf is completely invalid - but then the
system SHOULD be stopped pending the investigation of why the file has been
corrupted, or syslogd falls back on a default configuration (record everything
in the syslog file).
-- 
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@cats-chateau.net

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Stephen Harris]

> It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a

Huh?  "SHOULD"?   Why?  If syslog dies for any reason (bug, DOS, hack,
admin stupidity) then I sure don't want the system freezing up.

( heh...  at work on Solaris I monitor 300+ systems, and it's not unusual
to find 1 box a week with syslog not running for some reason or another.
I can't decide whether it's admin stupidity or bugs in Solaris syslog - of
which there are many :-(( )

syslog is not meant to be a secure audit system.  Messages can be
legitimately dropped.   Applications have been coded assuming that they
will not be frozen in syslog().  Linux should not be different in this
respect.   Hmm... it might be nice to be this a system tunable parameter
but I'm not sure the best way of doing that (glibc maybe?)

                                 Stephen Harris
                 sweh@spuddy.mew.co.uk   http://www.spuddy.org/
      The truth is the truth, and opinion just opinion.  But what is what?
       My employer pays to ignore my opinions; you get to do it for free.      
  * Meeeeow ! Call  Spud the Cat on > 01708 442043 < for free Usenet access *
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[James Sutherland]

On Sun, 29 Oct 2000, Jesse Pollard wrote:

> On Sun, 29 Oct 2000, Stephen Harris wrote:
> >Horst von Brand wrote:
> >
> >> > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3,
> >> > > kernel 2.2.x), within a few minutes you will find your entire machine
> >> > > grinds to a halt.  For example, nobody can log in.
> >> 
> >> Great! Yet another way in which root can get the rope to shoot herself in
> >> the foot. Anything _really_ new?
> >
> >OK, let's go a step further - what if syslog dies or breaks in some way
> >shape or form so that the syslog() function blocks...?
> >
> >My worry is the one that was originally raised but ignored:  syslog() should
> >not BLOCK regardless of whether it's local or remote.  syslog is not a
> >reliable mechanism and many programs have been written assuming they can
> >fire off syslog() calls without worry.
> 
> It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a
> few seconds (depending on the log rate...).

No. You should have the OPTION of stopping the machine, if accurate syslog
information is more important to you than system functionality. This
should also be an OS function, rather than just freezing processes as and
when they call syslog()!

> I do believe that restarting syslog should be possible... Perhaps syslog
> should be started by inetd at the very beginning. Then it could be restarted
> after an exit/abort.

inetd? I'd have thought init would make a more suitable choice, perhaps
with a monitor to lock the machine up or reboot if syslog fails and can't
be restored.

> This can STILL fail if the syslog.conf is completely invalid - but then the
> system SHOULD be stopped pending the investigation of why the file has been
> corrupted, or syslogd falls back on a default configuration (record everything
> in the syslog file).

Again, that might be useful in some cases, but certainly shouldn't be the
only, or even default, mode.


James.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Igmar Palsenberg]

> > It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a
> 
> Huh?  "SHOULD"?   Why?  If syslog dies for any reason (bug, DOS, hack,
> admin stupidity) then I sure don't want the system freezing up.

In some cases, I find the syslog messages of more importance then a
working system. I like to know what's going on on my machines.

> ( heh...  at work on Solaris I monitor 300+ systems, and it's not unusual
> to find 1 box a week with syslog not running for some reason or another.
> I can't decide whether it's admin stupidity or bugs in Solaris syslog - of
> which there are many :-(( )
> 
> syslog is not meant to be a secure audit system.  Messages can be
> legitimately dropped.

I find dropping messages unacceptable.

>   Applications have been coded assuming that they
> will not be frozen in syslog().  Linux should not be different in this
> respect.   Hmm... it might be nice to be this a system tunable parameter
> but I'm not sure the best way of doing that (glibc maybe?)

I needs to be in glibc, yes.

> 
>                                  Stephen Harris


	Igmar

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Jesse Pollard]

---------  Received message begins Here  ---------

Jesse Pollard <pollard@cats-chateau.net>:
> On Sun, 29 Oct 2000, Stephen Harris wrote:
> >Horst von Brand wrote:
> >
> >> > > If you send SIGSTOP to syslogd on a Red Hat 6.2 system (glibc 2.1.3,
> >> > > kernel 2.2.x), within a few minutes you will find your entire machine
> >> > > grinds to a halt.  For example, nobody can log in.
> >> 
> >> Great! Yet another way in which root can get the rope to shoot herself in
> >> the foot. Anything _really_ new?
> >
> >OK, let's go a step further - what if syslog dies or breaks in some way
> >shape or form so that the syslog() function blocks...?
> >
> >My worry is the one that was originally raised but ignored:  syslog() should
> >not BLOCK regardless of whether it's local or remote.  syslog is not a
> >reliable mechanism and many programs have been written assuming they can
> >fire off syslog() calls without worry.
> 
> It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a
> few seconds (depending on the log rate...).
> 
> I do believe that restarting syslog should be possible... Perhaps syslog
> should be started by inetd at the very beginning. Then it could be restarted
> after an exit/abort.
> 
> This can STILL fail if the syslog.conf is completely invalid - but then the
> system SHOULD be stopped pending the investigation of why the file has been
> corrupted, or syslogd falls back on a default configuration (record everything
> in the syslog file).

As was pointed out in a separate E-mail: I ment to say "init" instead of
"inetd", since inetd can generate syslog messages...

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Jesse Pollard]

Stephen Harris <sweh@spuddy.mew.co.uk>:
> > It was NOT ignored. If syslogd dies, then the system SHOULD stop, after a
> 
> Huh?  "SHOULD"?   Why?  If syslog dies for any reason (bug, DOS, hack,
> admin stupidity) then I sure don't want the system freezing up.

Should because this is the only audit logging facility presently available
to Linux.

> ( heh...  at work on Solaris I monitor 300+ systems, and it's not unusual
> to find 1 box a week with syslog not running for some reason or another.
> I can't decide whether it's admin stupidity or bugs in Solaris syslog - of
> which there are many :-(( )

On these boxes you should be running the audit log. Which has the property
of shutting the system down when it is aborted...

> syslog is not meant to be a secure audit system.  Messages can be
> legitimately dropped.   Applications have been coded assuming that they
> will not be frozen in syslog().  Linux should not be different in this
> respect.   Hmm... it might be nice to be this a system tunable parameter
> but I'm not sure the best way of doing that (glibc maybe?)

The best way would be to have a true audit daemon that has the property of
hanging/shutdown of the system. I would prefer a shutdown to single user
mode than a hang. That way I would get a chance to examine the log/restart
the daemon and examine the log. Even better would be a way to
suspend/checkpoint all processing, switch to a "audit emergency" mode with
no network activity allowed, and then examine things. It would provide an
option to clean the system and reboot, or restart the audit daemon and
resume multiuser mode (resuming all suspended/checkpointed processes).

Once there is an audit daemon then the security messages/alerts, and only
those messages, would be sent to it.

That way syslogd is available for non-security related events, and these
could be dropped when necessary.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Igmar Palsenberg]

> Sadly, you WILL still lose entries if the system crashes before fs metadata
> has been flushed to disk.  Unless the inode has the correct size stored, the
> crap fsync()ed to disk doesn't make much difference.

Yep. I can't really think of a case where you wouldn't lose data in case
of for example a hard lock.

> (This is amplified by dcache.)

I'm not that familiar with kernel internals..

To bring up that point : Anyone had a good advice on a good OS internals
book ?? I'm gonna read the 2.2.x kernel sources, so I could use a good
background.


	Regards,

		Igmar

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: syslog() blocks on glibc 2.1.3 with kernel 2.2.x

[Igmar Palsenberg]

> This is not an option for us, unfortunately.  Many of our IP addresses
> are dynamically assigned, with the DNS tables dynamically updated.

Not an option in that case.
 
> Thank you for the patch to syslogd, though!  Can you try to get your
> "-x" option into the standard distributions of syslogd, or should I
> work up a bug report / feature request for Red Hat myself?

I have no idea who officially maintains it.. putting a bug report with RH
should be a good idea. The patch is untested, so someone needs to verify
that remote logging indeed nog is IP only. 

>  - Pat
> 

	
	Igmar

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/