anyone interested in chip register error diagnostics?

anyone interested in chip register error diagnostics?

[zshelle]

On POWER, I work on a component that listens for hardware errors 
reported by registers in the system chips (processors, memory buffers, 
I/O chips, etc.) and performs service actions based on those errors. I 
have been working on porting some of this code to the BMC for system 
fatal error analysis (see my work-in-progress proposals: 
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and 
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part 
of the new design, we are building a generic, data-driven register error 
isolator, which will be used by several applications within POWER. 
However, it has the potential to be useful on other architectures as 
well. I am curious if anyone in the community is interested in this.

Re: anyone interested in chip register error diagnostics?

[Brad Bishop]

oops…I did not mean to take the list off copy.  Adding it back on…

> On Mar 4, 2019, at 3:56 PM, Supreeth Venkatesh <Supreeth.Venkatesh@arm.com> wrote:
> 
> 
> Thanks Brad.
> 
> Hi Zane/Brad,
> 
> On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.
> The format of the errors are defined in Appendix N in UEFI specification
> http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf
> 
> I have not read the proposal in its entirety, but this seems similar to Reliability, Availability, Serviceability (RAS) feature using
> System Management Mode/Management Mode, but on the BMC side.
> 
> I will take a look at the reviews posted and provide more feedback.
> 
> If this is something similar to RAS feature, I have in fact proposed in DMTF PMCI WG to include CPER formats to be added to one of
> the PLDM specifications.
> 
> Arm would be interested in the design of this component, if it can accommodate the above error formats and component can be designed in
> an architecture agnostic way.
> 
> Thanks,
> Supreeth
> 
> -----Original Message-----
> From: Brad Bishop <bradleyb@fuzziesquirrel.com>
> Sent: Monday, March 4, 2019 2:38 PM
> To: zshelle <zshelle@linux.vnet.ibm.com>; Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>; ed.tanous@intel.com
> Subject: Re: anyone interested in chip register error diagnostics?
> 
> On Mon, Mar 04, 2019 at 02:22:45PM -0600, zshelle wrote:
>> On POWER, I work on a component that listens for hardware errors
>> reported by registers in the system chips (processors, memory buffers,
>> I/O chips, etc.) and performs service actions based on those errors. I
>> have been working on porting some of this code to the BMC for system
>> fatal error analysis (see my work-in-progress proposals:
>> https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and
>> https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part
>> of the new design, we are building a generic, data-driven register
>> error isolator, which will be used by several applications within
>> POWER. However, it has the potential to be useful on other
>> architectures as well. I am curious if anyone in the community is
>> interested in this.
> 
> Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one.  Ed, Supreeth - do you understand the function being proposed here?  How does this work on x86 and arm servers?
> 
> thx - brad
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

RE: anyone interested in chip register error diagnostics?

[zshelle]

On 2019-03-04 14:56, Supreeth Venkatesh wrote:
> Thanks Brad.
> 
> Hi Zane/Brad,
> 
> On Arm Platforms, We use Common Platform Error Record (CPER) to report
> these kinds of hardware errors.
> The format of the errors are defined in Appendix N in UEFI 
> specification
> http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf
> 
> I have not read the proposal in its entirety, but this seems similar
> to Reliability, Availability, Serviceability (RAS) feature using
> System Management Mode/Management Mode, but on the BMC side.
> 
> I will take a look at the reviews posted and provide more feedback.
> 
> If this is something similar to RAS feature, I have in fact proposed
> in DMTF PMCI WG to include CPER formats to be added to one of
> the PLDM specifications.
> 
> Arm would be interested in the design of this component, if it can
> accommodate the above error formats and component can be designed in
> an architecture agnostic way.
> 
> Thanks,
> Supreeth
> 
> -----Original Message-----
> From: Brad Bishop <bradleyb@fuzziesquirrel.com>
> Sent: Monday, March 4, 2019 2:38 PM
> To: zshelle <zshelle@linux.vnet.ibm.com>; Supreeth Venkatesh
> <Supreeth.Venkatesh@arm.com>; ed.tanous@intel.com
> Subject: Re: anyone interested in chip register error diagnostics?
> 
> On Mon, Mar 04, 2019 at 02:22:45PM -0600, zshelle wrote:
>> On POWER, I work on a component that listens for hardware errors
>> reported by registers in the system chips (processors, memory buffers,
>> I/O chips, etc.) and performs service actions based on those errors. I
>> have been working on porting some of this code to the BMC for system
>> fatal error analysis (see my work-in-progress proposals:
>> https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and
>> https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part
>> of the new design, we are building a generic, data-driven register
>> error isolator, which will be used by several applications within
>> POWER. However, it has the potential to be useful on other
>> architectures as well. I am curious if anyone in the community is
>> interested in this.
> 
> Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one.  Ed,
> Supreeth - do you understand the function being proposed here?  How
> does this work on x86 and arm servers?
> 
> thx - brad
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose
> the contents to any other person, use it for any purpose, or store or
> copy the information in any medium. Thank you.

Looks like CPER is similar to IBM's Platform Error Log (PEL). At this 
time, I am not really focused on the log format at the moment, but it is 
on my list of things to investigate. I heard a rumor that there may be a 
standard "OpenBMC" logging mechanism in the works. With those, you could 
convert them into the CPER or PEL format if needed.

My proposals are more focused on reading the registers from hardware and 
determining what caused the error. On POWER, we have hundreds of what we 
call fault isolation registers (FIRs), where each bit within those 
registers can signify a different hardware error event. It is also 
possible that there may be several active bits on at the same time. So 
my component will sort through all of those registers, find the active 
bits, and determine what is the root cause of the failure versus 
side-effect errors. Once root cause is determined, we then perform any 
services actions defined by our RAS team (and commit a log). Is there 
anything like this on ARM?

Re: anyone interested in chip register error diagnostics?

[Supreeth Venkatesh]

On Mon, 2019-03-04 at 16:44 -0600, zshelle wrote:
> On 2019-03-04 14:56, Supreeth Venkatesh wrote:
> > Thanks Brad.
> > 
> > Hi Zane/Brad,
> > 
> > On Arm Platforms, We use Common Platform Error Record (CPER) to
> > report
> > these kinds of hardware errors.
> > The format of the errors are defined in Appendix N in UEFI 
> > specification
> > 
http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf
> > 
> > I have not read the proposal in its entirety, but this seems
> > similar
> > to Reliability, Availability, Serviceability (RAS) feature using
> > System Management Mode/Management Mode, but on the BMC side.
> > 
> > I will take a look at the reviews posted and provide more feedback.
> > 
> > If this is something similar to RAS feature, I have in fact
> > proposed
> > in DMTF PMCI WG to include CPER formats to be added to one of
> > the PLDM specifications.
> > 
> > Arm would be interested in the design of this component, if it can
> > accommodate the above error formats and component can be designed
> > in
> > an architecture agnostic way.
> > 
> > Thanks,
> > Supreeth
> > 
> > -----Original Message-----
> > From: Brad Bishop <bradleyb@fuzziesquirrel.com>
> > Sent: Monday, March 4, 2019 2:38 PM
> > To: zshelle <zshelle@linux.vnet.ibm.com>; Supreeth Venkatesh
> > <Supreeth.Venkatesh@arm.com>; ed.tanous@intel.com
> > Subject: Re: anyone interested in chip register error diagnostics?
> > 
> > On Mon, Mar 04, 2019 at 02:22:45PM -0600, zshelle wrote:
> > > On POWER, I work on a component that listens for hardware errors
> > > reported by registers in the system chips (processors, memory
> > > buffers,
> > > I/O chips, etc.) and performs service actions based on those
> > > errors. I
> > > have been working on porting some of this code to the BMC for
> > > system
> > > fatal error analysis (see my work-in-progress proposals:
> > > https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and
> > > https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As
> > > part
> > > of the new design, we are building a generic, data-driven
> > > register
> > > error isolator, which will be used by several applications within
> > > POWER. However, it has the potential to be useful on other
> > > architectures as well. I am curious if anyone in the community is
> > > interested in this.
> > 
> > Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one.  Ed,
> > Supreeth - do you understand the function being proposed here?  How
> > does this work on x86 and arm servers?
> > 
> > thx - brad
> > IMPORTANT NOTICE: The contents of this email and any attachments
> > are
> > confidential and may also be privileged. If you are not the
> > intended
> > recipient, please notify the sender immediately and do not disclose
> > the contents to any other person, use it for any purpose, or store
> > or
> > copy the information in any medium. Thank you.
> 
> Looks like CPER is similar to IBM's Platform Error Log (PEL). At
> this 
> time, I am not really focused on the log format at the moment, but it
> is 
> on my list of things to investigate. I heard a rumor that there may
> be a 
> standard "OpenBMC" logging mechanism in the works. With those, you
> could 
> convert them into the CPER or PEL format if needed.
Ok. You are not currently focussed on format of the log itself but
rather on the error syndrome/fault registers. Right?
However, at present, error syndrome information gathered from the fault
registers are converted to CPER/PEL format for logging purposes in the
host firmware/microcontroller firmware.

> 
> My proposals are more focused on reading the registers from hardware
> and 
> determining what caused the error. On POWER, we have hundreds of what
> we 
> call fault isolation registers (FIRs), where each bit within those 
> registers can signify a different hardware error event. It is also 
> possible that there may be several active bits on at the same time.
> So 
> my component will sort through all of those registers, find the
> active 
> bits, and determine what is the root cause of the failure versus 
> side-effect errors. Once root cause is determined, we then perform
> any 
> services actions defined by our RAS team (and commit a log). Is
> there 
> anything like this on ARM?
On Arm platforms also, There are several error syndrome registers which
are read. 
This information leads to contruction of CPER record which will be used
by OS to take service actions as per RAS policy.

After reading 18831, it looks like you want to move error data
collection to BMC from host firmware and for that you collect all fault
isolation registers.
Is there a security implication here?

Thank you for the proposal, I will read 18591 thoroughly to understand,
whether we can reuse this on arm architecture.

Re: anyone interested in chip register error diagnostics?

[zshelle]

On 2019-03-04 17:53, Supreeth Venkatesh wrote:
> On Mon, 2019-03-04 at 16:44 -0600, zshelle wrote:
>> On 2019-03-04 14:56, Supreeth Venkatesh wrote:
>> > Thanks Brad.
>> >
>> > Hi Zane/Brad,
>> >
>> > On Arm Platforms, We use Common Platform Error Record (CPER) to
>> > report
>> > these kinds of hardware errors.
>> > The format of the errors are defined in Appendix N in UEFI
>> > specification
>> >
> http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf
>> >
>> > I have not read the proposal in its entirety, but this seems
>> > similar
>> > to Reliability, Availability, Serviceability (RAS) feature using
>> > System Management Mode/Management Mode, but on the BMC side.
>> >
>> > I will take a look at the reviews posted and provide more feedback.
>> >
>> > If this is something similar to RAS feature, I have in fact
>> > proposed
>> > in DMTF PMCI WG to include CPER formats to be added to one of
>> > the PLDM specifications.
>> >
>> > Arm would be interested in the design of this component, if it can
>> > accommodate the above error formats and component can be designed
>> > in
>> > an architecture agnostic way.
>> >
>> > Thanks,
>> > Supreeth
>> >
>> > -----Original Message-----
>> > From: Brad Bishop <bradleyb@fuzziesquirrel.com>
>> > Sent: Monday, March 4, 2019 2:38 PM
>> > To: zshelle <zshelle@linux.vnet.ibm.com>; Supreeth Venkatesh
>> > <Supreeth.Venkatesh@arm.com>; ed.tanous@intel.com
>> > Subject: Re: anyone interested in chip register error diagnostics?
>> >
>> > On Mon, Mar 04, 2019 at 02:22:45PM -0600, zshelle wrote:
>> > > On POWER, I work on a component that listens for hardware errors
>> > > reported by registers in the system chips (processors, memory
>> > > buffers,
>> > > I/O chips, etc.) and performs service actions based on those
>> > > errors. I
>> > > have been working on porting some of this code to the BMC for
>> > > system
>> > > fatal error analysis (see my work-in-progress proposals:
>> > > https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and
>> > > https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As
>> > > part
>> > > of the new design, we are building a generic, data-driven
>> > > register
>> > > error isolator, which will be used by several applications within
>> > > POWER. However, it has the potential to be useful on other
>> > > architectures as well. I am curious if anyone in the community is
>> > > interested in this.
>> >
>> > Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one.  Ed,
>> > Supreeth - do you understand the function being proposed here?  How
>> > does this work on x86 and arm servers?
>> >
>> > thx - brad
>> > IMPORTANT NOTICE: The contents of this email and any attachments
>> > are
>> > confidential and may also be privileged. If you are not the
>> > intended
>> > recipient, please notify the sender immediately and do not disclose
>> > the contents to any other person, use it for any purpose, or store
>> > or
>> > copy the information in any medium. Thank you.
>> 
>> Looks like CPER is similar to IBM's Platform Error Log (PEL). At
>> this
>> time, I am not really focused on the log format at the moment, but it
>> is
>> on my list of things to investigate. I heard a rumor that there may
>> be a
>> standard "OpenBMC" logging mechanism in the works. With those, you
>> could
>> convert them into the CPER or PEL format if needed.
> Ok. You are not currently focussed on format of the log itself but
> rather on the error syndrome/fault registers. Right?
> However, at present, error syndrome information gathered from the fault
> registers are converted to CPER/PEL format for logging purposes in the
> host firmware/microcontroller firmware.
> 
>> 
>> My proposals are more focused on reading the registers from hardware
>> and
>> determining what caused the error. On POWER, we have hundreds of what
>> we
>> call fault isolation registers (FIRs), where each bit within those
>> registers can signify a different hardware error event. It is also
>> possible that there may be several active bits on at the same time.
>> So
>> my component will sort through all of those registers, find the
>> active
>> bits, and determine what is the root cause of the failure versus
>> side-effect errors. Once root cause is determined, we then perform
>> any
>> services actions defined by our RAS team (and commit a log). Is
>> there
>> anything like this on ARM?
> On Arm platforms also, There are several error syndrome registers which
> are read.
> This information leads to contruction of CPER record which will be used
> by OS to take service actions as per RAS policy.
> 
> After reading 18831, it looks like you want to move error data
> collection to BMC from host firmware and for that you collect all fault
> isolation registers.
> Is there a security implication here?
> 
> Thank you for the proposal, I will read 18591 thoroughly to understand,
> whether we can reuse this on arm architecture.
On POWER, reading the FIR data has no security concerns. Writing data to 
those registers does have security concerns, but that should not be 
allowed from the BMC.

Re: anyone interested in chip register error diagnostics?

[Brad Bishop]

On Mon, Mar 04, 2019 at 05:53:45PM -0600, Supreeth Venkatesh wrote:
>On Mon, 2019-03-04 at 16:44 -0600, zshelle wrote:

>On Arm platforms also, There are several error syndrome registers which
>are read.
>This information leads to contruction of CPER record which will be used
>by OS to take service actions as per RAS policy.
>
>After reading 18831, it looks like you want to move error data
>collection to BMC from host firmware and for that you collect all fault
>isolation registers.

It isn't really moving - we run it on the host as well as the BMC, for
situations where the POWER processor is not able to execute instructions
due to a malfunction somewhere.

>Is there a security implication here?

Are you able to expand on this?

>
>Thank you for the proposal, I will read 18591 thoroughly to understand,
>whether we can reuse this on arm architecture.

Thanks Supreeth!

Re: anyone interested in chip register error diagnostics?

[Brad Bishop]

On Mon, Mar 04, 2019 at 04:01:16PM -0500, Brad Bishop wrote:
>oops…I did not mean to take the list off copy.  Adding it back on…
>
>> On Mar 4, 2019, at 3:56 PM, Supreeth Venkatesh <Supreeth.Venkatesh@arm.com> wrote:
>>
>>
>> Thanks Brad.
>>
>> Hi Zane/Brad,
>>
>> On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.

Supreeth - Is the code that generates these open source?

Re: anyone interested in chip register error diagnostics?

[Supreeth Venkatesh]

On Tue, 2019-03-05 at 12:39 -0500, Brad Bishop wrote:
> On Mon, Mar 04, 2019 at 05:53:45PM -0600, Supreeth Venkatesh wrote:
> > On Mon, 2019-03-04 at 16:44 -0600, zshelle wrote:
> > On Arm platforms also, There are several error syndrome registers
> > which
> > are read.
> > This information leads to contruction of CPER record which will be
> > used
> > by OS to take service actions as per RAS policy.
> > 
> > After reading 18831, it looks like you want to move error data
> > collection to BMC from host firmware and for that you collect all
> > fault
> > isolation registers.
> 
> It isn't really moving - we run it on the host as well as the BMC,
> for
> situations where the POWER processor is not able to execute
> instructions
> due to a malfunction somewhere.
Thanks for the clarification Brad.

> 
> > Is there a security implication here?
> 
> Are you able to expand on this?
Yes. I meant if this involves direct host/target register access from
BMC, then there may be a security issue. 
However, from Zane's reply, it does not seem to be the case.    


> 
> > 
> > Thank you for the proposal, I will read 18591 thoroughly to
> > understand,
> > whether we can reuse this on arm architecture.
> 
> Thanks Supreeth!

Re: anyone interested in chip register error diagnostics?

[Supreeth Venkatesh]

On Tue, 2019-03-05 at 12:41 -0500, Brad Bishop wrote:
> On Mon, Mar 04, 2019 at 04:01:16PM -0500, Brad Bishop wrote:
> > oops…I did not mean to take the list off copy.  Adding it back on…
> > 
> > > On Mar 4, 2019, at 3:56 PM, Supreeth Venkatesh <
> > > Supreeth.Venkatesh@arm.com> wrote:
> > > 
> > > 
> > > Thanks Brad.
> > > 
> > > Hi Zane/Brad,
> > > 
> > > On Arm Platforms, We use Common Platform Error Record (CPER) to
> > > report these kinds of hardware errors.
> 
> Supreeth - Is the code that generates these open source?
Yes. 
arm-tf: https://github.com/ARM-software/arm-trusted-firmware
edk2: https://github.com/tianocore/edk2/tree/master/StandaloneMmPkg

Re: anyone interested in chip register error diagnostics?

[Brad Bishop]

On Wed, Mar 06, 2019 at 12:31:23PM -0600, Supreeth Venkatesh wrote:
>On Tue, 2019-03-05 at 12:41 -0500, Brad Bishop wrote:
>> On Mon, Mar 04, 2019 at 04:01:16PM -0500, Brad Bishop wrote:
>> > oops…I did not mean to take the list off copy.  Adding it back on…
>> >
>> > > On Mar 4, 2019, at 3:56 PM, Supreeth Venkatesh <
>> > > Supreeth.Venkatesh@arm.com> wrote:
>> > >
>> > >
>> > > Thanks Brad.
>> > >
>> > > Hi Zane/Brad,
>> > >
>> > > On Arm Platforms, We use Common Platform Error Record (CPER) to
>> > > report these kinds of hardware errors.
>>
>> Supreeth - Is the code that generates these open source?
>Yes.
>arm-tf: https://github.com/ARM-software/arm-trusted-firmware
>edk2: https://github.com/tianocore/edk2/tree/master/StandaloneMmPkg

Thanks for the info Supreeth!

-brad