Unresponsive BMC after booting into Ubuntu

Unresponsive BMC after booting into Ubuntu

[Patrick Voelker]

I have a situation where, while booting the host into Ubuntu desktop 18.04, my OpenBMC ast2500 BMC stops responding on all interfaces including its debug serial console.  It doesn't do this when the host boots into Windows.

Nothing came up when I did an initial google search.  Also, nothing unusual jumps out at me if I watch the BMC's 'journal -f' prior to the issue.  No kernel panic or anything like that.

Are there any known issues like this?  If not, are there some pointers to relevant debug strategies for when BMC is not available?  I'd like to avoid JTAG debugging the kernel if possible.  :-)

Re: Unresponsive BMC after booting into Ubuntu

[Andrei Kartashev]

Hi,
I had similar issue, but it was just "not loading some pages in webui
and doesn't response on ipmi commands", ssh and serial console used to
work. And yes, it appears after Linux boots on the host, but not with
Windows.
The root problem in my case was in PECI driver and power saving CPU
modes.
Here was the patch which solves the problem for me:
https://patchwork.ozlabs.org/project/openbmc/patch/e6f4ab29854ac58a878189c1edece4e8eed2e7fd.camel@yadro.com/

I'm not sure if it is your problem also, if you use dbus-sensors you
can check by stopping CPUsensor: 
	systemctl stop xyz.openbmc_project.cpusensor.service
If the problem gone after this, you got it =)


On Mon, 2020-11-23 at 22:49 +0000, Patrick Voelker wrote:
> I have a situation where, while booting the host into Ubuntu desktop
> 18.04, my OpenBMC ast2500 BMC stops responding on all interfaces
> including its debug serial console.  It doesn't do this when the host
> boots into Windows.
> 
> Nothing came up when I did an initial google search.  Also, nothing
> unusual jumps out at me if I watch the BMC's 'journal -f' prior to
> the issue.  No kernel panic or anything like that.
> 
> Are there any known issues like this?  If not, are there some
> pointers to relevant debug strategies for when BMC is not
> available?  I'd like to avoid JTAG debugging the kernel if
> possible.  :-)
> 
-- 
Best regards,
Andrei Kartashev

RE: Unresponsive BMC after booting into Ubuntu

[Patrick Voelker]

Hm.  Sounds like a different issue. I don't think this solution will help me since I can't access the serial console.

> -----Original Message-----
> From: Andrei Kartashev [mailto:a.kartashev@yadro.com]
> Sent: Monday, November 23, 2020 9:18 PM
> To: Patrick Voelker; OpenBMC (openbmc@lists.ozlabs.org)
> Subject: Re: Unresponsive BMC after booting into Ubuntu
> 
> Hi,
> I had similar issue, but it was just "not loading some pages in webui
> and doesn't response on ipmi commands", ssh and serial console used to
> work. And yes, it appears after Linux boots on the host, but not with
> Windows.
> The root problem in my case was in PECI driver and power saving CPU
> modes.
> Here was the patch which solves the problem for me:
> https://patchwork.ozlabs.org/project/openbmc/patch/e6f4ab29854ac58a87
> 8189c1edece4e8eed2e7fd.camel@yadro.com/
> 
> I'm not sure if it is your problem also, if you use dbus-sensors you
> can check by stopping CPUsensor:
> 	systemctl stop xyz.openbmc_project.cpusensor.service
> If the problem gone after this, you got it =)
> 
> 
> On Mon, 2020-11-23 at 22:49 +0000, Patrick Voelker wrote:
> > I have a situation where, while booting the host into Ubuntu desktop
> > 18.04, my OpenBMC ast2500 BMC stops responding on all interfaces
> > including its debug serial console.  It doesn't do this when the host
> > boots into Windows.
> >
> > Nothing came up when I did an initial google search.  Also, nothing
> > unusual jumps out at me if I watch the BMC's 'journal -f' prior to
> > the issue.  No kernel panic or anything like that.
> >
> > Are there any known issues like this?  If not, are there some
> > pointers to relevant debug strategies for when BMC is not
> > available?  I'd like to avoid JTAG debugging the kernel if
> > possible.  :-)
> >
> --
> Best regards,
> Andrei Kartashev
> 

Re: Unresponsive BMC after booting into Ubuntu

[Andrew Jeffery]

On Tue, 24 Nov 2020, at 18:05, Patrick Voelker wrote:
> Hm.  Sounds like a different issue. I don't think this solution will 
> help me since I can't access the serial console.

I'm not Andrei, but my understanding was that he wasn't suggesting you try to 
apply the change after the lockup occurred, but rather before. And then if you 
don't see lockups with the change applied, then it might be the same problem.
It wasn't about recovering the BMC from the lockup context.

But yeah, hopefully I'm not adding to the confusion here!

Andrew

RE: Unresponsive BMC after booting into Ubuntu

[Patrick Voelker]

I gave disabling the cpusensor service a try and it didn't make a difference.  The BMC still goes out to lunch.  Thanks for the suggestion.

I tried disabling all the IPMI handlers but that didn't seem to help either:
systemctl stop phosphor-ipmi-kcs@ipmi_kcs3
systemctl stop phosphor-ipmi-kcs@ipmi_kcs4
systemctl stop phosphor-ipmi-net@eth0.socket
systemctl stop phosphor-ipmi-net@eth1.socket
systemctl stop phosphor-ipmi-host

I think my next step will be to try disabling kernel config options that are related to things that can be influenced by the host.

> -----Original Message-----
> From: openbmc [mailto:openbmc-
> bounces+patrick_voelker=phoenix.com@lists.ozlabs.org] On Behalf Of
> Andrew Jeffery
> Sent: Monday, November 30, 2020 4:47 PM
> To: openbmc@lists.ozlabs.org
> Subject: Re: Unresponsive BMC after booting into Ubuntu
> 
> 
> 
> On Tue, 24 Nov 2020, at 18:05, Patrick Voelker wrote:
> > Hm.  Sounds like a different issue. I don't think this solution will
> > help me since I can't access the serial console.
> 
> I'm not Andrei, but my understanding was that he wasn't suggesting you try
> to
> apply the change after the lockup occurred, but rather before. And then if
> you
> don't see lockups with the change applied, then it might be the same
> problem.
> It wasn't about recovering the BMC from the lockup context.
> 
> But yeah, hopefully I'm not adding to the confusion here!
> 
> Andrew

Re: Unresponsive BMC after booting into Ubuntu

[Konstantin Klubnichkin]

[-- Attachment #1: Type: text/html, Size: 3388 bytes --]