OpenSM 3.3.16 at 100% CPU load, "console off"

OpenSM 3.3.16 at 100% CPU load, "console off"

[Sebastian Riemer]

Hi Hal,

we've encountered an issue with OpenSM 3.3.16 and the config option
"console off".
OpenSM processes are at 100% CPU load.

>From strace:
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0

As far as I've seen in the code, the function osm_console() from
opensm/osm_console.c is the only function which uses poll().

Is this issue already known or perhaps already fixed?

Thanks,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM 3.3.16 at 100% CPU load, "console off"

[Hal Rosenstock]

Hi Sebastian,

On 10/9/2013 7:10 AM, Sebastian Riemer wrote:
> Hi Hal,
> 
> we've encountered an issue with OpenSM 3.3.16 and the config option
> "console off".
> OpenSM processes are at 100% CPU load.
> 
>>From strace:
> poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
> read(0, "", 4096)                       = 0
> poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
> read(0, "", 4096)                       = 0
> poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
> read(0, "", 4096)                       = 0

So this doesn't block for 1 second and that's why the CPU is 100% ?

> As far as I've seen in the code, the function osm_console() from
> opensm/osm_console.c is the only function which uses poll().

osm_vendor_ibumad has a receiver thread polling umad under the covers of
umad_recv but I think that uses infinite rather than 1 second.

> Is this issue already known or perhaps already fixed?

This area of the console code has not changed in quite a while.
Any idea if this works with older versions of OpenSM ?

This is first I've heard of this issue.

-- Hal

> Thanks,
> Sebastian
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM 3.3.16 at 100% CPU load, "console off"

[David Dillow]

On Wed, 2013-10-09 at 09:28 -0400, Hal Rosenstock wrote:
> >>From strace:
> > poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
> > read(0, "", 4096)                       = 0
> > poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
> > read(0, "", 4096)                       = 0
> > poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
> > read(0, "", 4096)                       = 0
> 
> So this doesn't block for 1 second and that's why the CPU is 100% ?

Looks like it is spinning on a closed socket (or stdin) -- calling
poll() on such will return immediately...

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM 3.3.16 at 100% CPU load, "console off"

[Sebastian Riemer]

On 09.10.2013 15:30, David Dillow wrote:
> On Wed, 2013-10-09 at 09:28 -0400, Hal Rosenstock wrote:
>>> >From strace:
>>> poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
>>> read(0, "", 4096)                       = 0
>>> poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
>>> read(0, "", 4096)                       = 0
>>> poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
>>> read(0, "", 4096)                       = 0
>>
>> So this doesn't block for 1 second and that's why the CPU is 100% ?
> 
> Looks like it is spinning on a closed socket (or stdin) -- calling
> poll() on such will return immediately...
> 

Thanks for the responses!

I've seen in the code that the local console is initialized but is not
released correctly. Should be done in osm_console_exit().

Something like this:

       if (p_oct->in_fd >= 0) {
               p_oct->in = NULL;
               p_oct->out = NULL;
               p_oct->in_fd = -1;
               p_oct->out_fd = -1;
       }

I guess what happened was that "console local" was set, changed in the
config to "console off" and the service has been restarted. Restarting
the service again didn't help.

It is strange that the console_init_flag is still set. The function
osm_console() returns 0 if poll() fails. If it would return something
else, then the console_init_flag would be set to 0 again and there would
be no issue anymore I suppose.

Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM 3.3.16 at 100% CPU load, "console off"

[Hal Rosenstock]

Hi Sebastian,

On 10/9/2013 9:28 AM, Hal Rosenstock wrote:
> Hi Sebastian,
> 
> On 10/9/2013 7:10 AM, Sebastian Riemer wrote:
>> Hi Hal,
>>
>> we've encountered an issue with OpenSM 3.3.16 and the config option
>> "console off".
>> OpenSM processes are at 100% CPU load.
>>
>> >From strace:
>> poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
>> read(0, "", 4096)                       = 0
>> poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
>> read(0, "", 4096)                       = 0
>> poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
>> read(0, "", 4096)                       = 0
> 
> So this doesn't block for 1 second and that's why the CPU is 100% ?
> 
>> As far as I've seen in the code, the function osm_console() from
>> opensm/osm_console.c is the only function which uses poll().
> 
> osm_vendor_ibumad has a receiver thread polling umad under the covers of
> umad_recv but I think that uses infinite rather than 1 second.
> 
>> Is this issue already known or perhaps already fixed?
> 
> This area of the console code has not changed in quite a while.
> Any idea if this works with older versions of OpenSM ?
> 
> This is first I've heard of this issue.

Do you recall the sequence to get to this ?

Was console option changed to off and then OpenSM SIGHUP'd ? Something
else ?

Is this reproducible ?

-- Hal

> -- Hal
> 
>> Thanks,
>> Sebastian
>>
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM 3.3.16 at 100% CPU load, "console off"

[Sebastian Riemer]

On 09.10.2013 16:00, Hal Rosenstock wrote:

> Do you recall the sequence to get to this ?
> 
> Was console option changed to off and then OpenSM SIGHUP'd ? Something
> else ?
> 
> Is this reproducible ?

Yes, now I can reproduce it. The opensm has been initially started with
"console off" and I activate "console local" and restart the service.
CPU load is at 100% immediately. I set "console off" again and restart
the service and CPU load is low again.

I did this three times in a row, now. And the third time it even
remained at 100% load in the "off" state. I've set "local" and "off"
again and CPU load was low again.

Cheers,
Sebastian

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM 3.3.16 at 100% CPU load, "console off"

[Hal Rosenstock]

On 10/9/2013 10:45 AM, Sebastian Riemer wrote:
> On 09.10.2013 16:00, Hal Rosenstock wrote:
> 
>> Do you recall the sequence to get to this ?
>>
>> Was console option changed to off and then OpenSM SIGHUP'd ? Something
>> else ?
>>
>> Is this reproducible ?
> 
> Yes, now I can reproduce it. The opensm has been initially started with
> "console off" and I activate "console local" and restart the service.
> CPU load is at 100% immediately. I set "console off" again and restart
> the service and CPU load is low again.
> 
> I did this three times in a row, now. And the third time it even
> remained at 100% load in the "off" state. I've set "local" and "off"
> again and CPU load was low again.

What does service restart do in terms of OpenSM ?

Note that the console parameter is _not_ changeable "on the fly" right
now so if OpenSM is being SIGHUP'd by service restart then this is a
current limitation (and is clearly not detected/protected against in the
current code base). It sounds like that may be what is going on.

-- Hal

> Cheers,
> Sebastian
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM 3.3.16 at 100% CPU load, "console off"

[Sebastian Riemer]

On 09.10.2013 17:15, Hal Rosenstock wrote:
> What does service restart do in terms of OpenSM ?
> 
> Note that the console parameter is _not_ changeable "on the fly" right
> now so if OpenSM is being SIGHUP'd by service restart then this is a
> current limitation (and is clearly not detected/protected against in the
> current code base). It sounds like that may be what is going on.

Yes, it emits SIGHUP. Thanks for the information! The opensm is a
critical component. So IMHO it needs to be fixed in a way that it either
protects itself against such changes by ignoring them on the fly or it
needs to support these changes.

The current situation is not really acceptable and the opensm stability
is crucial. So I'll think about fixing it.
Are you interested in patches in this regard?

Cheers,
Sebastian


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM 3.3.16 at 100% CPU load, "console off"

[Hal Rosenstock]

On 10/9/2013 11:52 AM, Sebastian Riemer wrote:
> On 09.10.2013 17:15, Hal Rosenstock wrote:
>> What does service restart do in terms of OpenSM ?
>>
>> Note that the console parameter is _not_ changeable "on the fly" right
>> now so if OpenSM is being SIGHUP'd by service restart then this is a
>> current limitation (and is clearly not detected/protected against in the
>> current code base). It sounds like that may be what is going on.
> 
> Yes, it emits SIGHUP. Thanks for the information! The opensm is a
> critical component. So IMHO it needs to be fixed in a way that it either
> protects itself against such changes by ignoring them on the fly or it
> needs to support these changes.
> 
> The current situation is not really acceptable and the opensm stability
> is crucial. So I'll think about fixing it.
> Are you interested in patches in this regard?

Yes; such patches are always welcome! Thanks.

-- Hal

> 
> Cheers,
> Sebastian
> 
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html