Bios upgrade from BMC

Bios upgrade from BMC

[Vijay Khemka]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

Hi,
I am writing an application for Bios upgrade. Currently I have created a hook to our bmc updater which expects a systemd unit file to be run at the time of activating update. But I am thinking of adding some of this unit file functionality in updater itself and wanted to run by you all if it is common procedure used by every platform. Below are the process we use in our platform as a part of bios upgrade.


  1.  Power off host server.
  2.  Set ME/NM (Management engine or Node manager in x86) to recovery mode
  3.  Flip GPIO to access SPI flash used by host.
  4.  Bind spi driver to access flash
  5.  Flashcp image to device.
  6.  Unbind spi driver
  7.  Flip GPIO back for host to access SPI flash
  8.  Set ME/NM to operational mode
  9.  Power on server.

I can have some flexibility in this sequence based on each platform configuration. Looking forward to your suggestions.

Regards
-Vijay

[-- Attachment #2: Type: text/html, Size: 5713 bytes --]

RE: Bios upgrade from BMC

[CS20 CTCchien]

[-- Attachment #1: Type: text/plain, Size: 2065 bytes --]

Hi Vijay,



I try to upgrade BIOS from BMC, but I do not know how to set the mode of  ME/NM.

Can you share the process how you set ME/NM to recovery mode and operation mode?



Thanks.



B.R.

Medad

From: Vijay Khemka <vijaykhemka@fb.com<mailto:vijaykhemka@fb.com>>

To: OpenBMC Maillist <openbmc@lists.ozlabs.org<mailto:openbmc@lists.ozlabs.org>>

Subject: Bios upgrade from BMC

Hi,
I am writing an application for Bios upgrade. Currently I have created a hook to our bmc updater which expects a systemd unit file to be run at the time of activating update. But I am thinking of adding some of this unit file functionality in updater itself and wanted to run by you all if it is common procedure used by every platform. Below are the process we use in our platform as a part of bios upgrade.


  1.  Power off host server.
  2.  Set ME/NM (Management engine or Node manager in x86) to recovery mode
  3.  Flip GPIO to access SPI flash used by host.
  4.  Bind spi driver to access flash
  5.  Flashcp image to device.
  6.  Unbind spi driver
  7.  Flip GPIO back for host to access SPI flash
  8.  Set ME/NM to operational mode
  9.  Power on server.

I can have some flexibility in this sequence based on each platform configuration. Looking forward to your suggestions.

Regards
-Vijay
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Nuvoton is strictly prohibited; and any information in this email irrelevant to the official business of Nuvoton shall be deemed as neither given nor endorsed by Nuvoton.

[-- Attachment #2: Type: text/html, Size: 9248 bytes --]

RE: Bios upgrade from BMC

[Lawniczak, Maciej]

[-- Attachment #1: Type: text/plain, Size: 3297 bytes --]

Hi Medad,

To set up ME in Recovery mode use IPMI command “Force ME Recovery” – DFh (byte4=01h)
To set up ME in Operational again use IPMI “Cold Reset” command” – 02h

For more details check: https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/intel-power-node-manager-v3-spec.pdf

Regards,
Maciej
________________________________

Hi Vijay,



I try to upgrade BIOS from BMC, but I do not know how to set the mode of  ME/NM.

Can you share the process how you set ME/NM to recovery mode and operation mode?



Thanks.



B.R.

Medad



From: Vijay Khemka <vijaykhemka@fb.com<mailto:vijaykhemka@fb.com>>

To: OpenBMC Maillist <openbmc@lists.ozlabs.org<mailto:openbmc@lists.ozlabs.org>>

Subject: Bios upgrade from BMC

Hi,
I am writing an application for Bios upgrade. Currently I have created a hook to our bmc updater which expects a systemd unit file to be run at the time of activating update. But I am thinking of adding some of this unit file functionality in updater itself and wanted to run by you all if it is common procedure used by every platform. Below are the process we use in our platform as a part of bios upgrade.


  1.  Power off host server.
  2.  Set ME/NM (Management engine or Node manager in x86) to recovery mode
  3.  Flip GPIO to access SPI flash used by host.
  4.  Bind spi driver to access flash
  5.  Flashcp image to device.
  6.  Unbind spi driver
  7.  Flip GPIO back for host to access SPI flash
  8.  Set ME/NM to operational mode
  9.  Power on server.

I can have some flexibility in this sequence based on each platform configuration. Looking forward to your suggestions.

Regards
-Vijay
________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Nuvoton is strictly prohibited; and any information in this email irrelevant to the official business of Nuvoton shall be deemed as neither given nor endorsed by Nuvoton.
--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

[-- Attachment #2: Type: text/html, Size: 13709 bytes --]

RE: Bios upgrade from BMC

[Neeraj Ladkani]

Vijay, 

- How are you able to determine flash ranges used by UEFI in case we may not want to update ME but only UEFI regions?  
- How is this interfaced from redfish/IPMI ? 

Is there any design document for this feature that we can review? 

Neeraj


From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> On Behalf Of Lawniczak, Maciej
Sent: Sunday, January 19, 2020 11:35 PM
To: CS20 CTCchien <CTCCHIEN@nuvoton.com>; openbmc@lists.ozlabs.org; Vijay Khemka <vijaykhemka@fb.com>
Subject: [EXTERNAL] RE: Bios upgrade from BMC

Hi Medad,

To set up ME in Recovery mode use IPMI command “Force ME Recovery” – DFh (byte4=01h)
To set up ME in Operational again use IPMI “Cold Reset” command” – 02h

For more details check: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.intel.com%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%2Ftechnical-specifications%2Fintel-power-node-manager-v3-spec.pdf&data=02%7C01%7Cneladk%40microsoft.com%7Cadc15cf0148e432110b508d79d7b6a38%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637151026289126945&sdata=2TLHwCsXMK2vg%2FfBMrKIPyNoPUG52RjYq33prp9xpsc%3D&reserved=0

Regards,
Maciej
________________________________________
Hi Vijay,

I try to upgrade BIOS from BMC, but I do not know how to set the mode of  ME/NM.
Can you share the process how you set ME/NM to recovery mode and operation mode?

Thanks.

B.R.
Medad

From: Vijay Khemka <mailto:vijaykhemka@fb.com>
To: OpenBMC Maillist <mailto:openbmc@lists.ozlabs.org>
Subject: Bios upgrade from BMC

Hi,
I am writing an application for Bios upgrade. Currently I have created a hook to our bmc updater which expects a systemd unit file to be run at the time of activating update. But I am thinking of adding some of this unit file functionality in updater itself and wanted to run by you all if it is common procedure used by every platform. Below are the process we use in our platform as a part of bios upgrade.

1. Power off host server.
2. Set ME/NM (Management engine or Node manager in x86) to recovery mode
3. Flip GPIO to access SPI flash used by host.
4. Bind spi driver to access flash
5. Flashcp image to device.
6. Unbind spi driver
7. Flip GPIO back for host to access SPI flash
8. Set ME/NM to operational mode
9. Power on server.

I can have some flexibility in this sequence based on each platform configuration. Looking forward to your suggestions.

Regards
-Vijay
________________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Nuvoton is strictly prohibited; and any information in this email irrelevant to the official business of Nuvoton shall be deemed as neither given nor endorsed by Nuvoton. 
---------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP 957-07-52-316 | Kapitał zakładowy 200.000 PLN.
Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.

Re: Bios upgrade from BMC

[Vijay Khemka]

On 1/20/20, 10:54 PM, "Neeraj Ladkani" <neladk@microsoft.com> wrote:

    Vijay, 
    
    - How are you able to determine flash ranges used by UEFI in case we may not want to update ME but only UEFI regions?  
I am not defining any range in flash. Currently it is just going to use flashcp command to flash image. I can accept this final stage of flashing as system unit file or command through config option.

    - How is this interfaced from redfish/IPMI ? 
It will interface same as BMC upgrade. Considering redfish/IPMI only, I am considering this this as a part of same image updater.
    
    Is there any design document for this feature that we can review? 
No, There is no design document just below steps and wanted to check if it is a common sequence everyone use or we need some flexibility.
    
    Neeraj
    
    
    From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> On Behalf Of Lawniczak, Maciej
    Sent: Sunday, January 19, 2020 11:35 PM
    To: CS20 CTCchien <CTCCHIEN@nuvoton.com>; openbmc@lists.ozlabs.org; Vijay Khemka <vijaykhemka@fb.com>
    Subject: [EXTERNAL] RE: Bios upgrade from BMC
    
    Hi Medad,
    
    To set up ME in Recovery mode use IPMI command “Force ME Recovery” – DFh (byte4=01h)
    To set up ME in Operational again use IPMI “Cold Reset” command” – 02h
    
    For more details check: https://urldefense.proofpoint.com/v2/url?u=https-3A__nam06.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fwww.intel.com-252Fcontent-252Fdam-252Fwww-252Fpublic-252Fus-252Fen-252Fdocuments-252Ftechnical-2Dspecifications-252Fintel-2Dpower-2Dnode-2Dmanager-2Dv3-2Dspec.pdf-26data-3D02-257C01-257Cneladk-2540microsoft.com-257Cadc15cf0148e432110b508d79d7b6a38-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-257C637151026289126945-26sdata-3D2TLHwCsXMK2vg-252FfBMrKIPyNoPUG52RjYq33prp9xpsc-253D-26reserved-3D0&d=DwIGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=v9MU0Ki9pWnTXCWwjHPVgpnCR80vXkkcrIaqU7USl5g&m=5Pb3J8Ryu1wP_3jCzVlxAbutqUd0mniJjRC5jlcl0mg&s=e4TQhEF7P9P3pjD1qc8cZ4G4_gqi-yAiGee9pTHUuNs&e= 
    
    Regards,
    Maciej
    ________________________________________
    Hi Vijay,
    
    I try to upgrade BIOS from BMC, but I do not know how to set the mode of  ME/NM.
    Can you share the process how you set ME/NM to recovery mode and operation mode?
    
    Thanks.
    
    B.R.
    Medad
    
    From: Vijay Khemka <mailto:vijaykhemka@fb.com>
    To: OpenBMC Maillist <mailto:openbmc@lists.ozlabs.org>
    Subject: Bios upgrade from BMC
    
    Hi,
    I am writing an application for Bios upgrade. Currently I have created a hook to our bmc updater which expects a systemd unit file to be run at the time of activating update. But I am thinking of adding some of this unit file functionality in updater itself and wanted to run by you all if it is common procedure used by every platform. Below are the process we use in our platform as a part of bios upgrade.
    
    1. Power off host server.
    2. Set ME/NM (Management engine or Node manager in x86) to recovery mode
    3. Flip GPIO to access SPI flash used by host.
    4. Bind spi driver to access flash
    5. Flashcp image to device.
    6. Unbind spi driver
    7. Flip GPIO back for host to access SPI flash
    8. Set ME/NM to operational mode
    9. Power on server.
    
    I can have some flexibility in this sequence based on each platform configuration. Looking forward to your suggestions.
    
    Regards
    -Vijay
    ________________________________________
    The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Nuvoton is strictly prohibited; and any information in this email irrelevant to the official business of Nuvoton shall be deemed as neither given nor endorsed by Nuvoton. 
    ---------------------------------------------------------------------
    Intel Technology Poland sp. z o.o.
    ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP 957-07-52-316 | Kapitał zakładowy 200.000 PLN.
    Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione.
    This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
    
    

Re: Bios upgrade from BMC

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 3095 bytes --]

Hi Vijay,

On Fri, Jan 17, 2020 at 07:04:51PM +0000, Vijay Khemka wrote:
> Hi,
> I am writing an application for Bios upgrade. Currently I have created a hook to our bmc updater which expects a systemd unit file to be run at the time of activating update. But I am thinking of adding some of this unit file functionality in updater itself and wanted to run by you all if it is common procedure used by every platform. Below are the process we use in our platform as a part of bios upgrade.
> 
> 
>   1.  Power off host server.

In the current code-update implementations, I don't believe that
powering off the Host is ever done automatically, but the expectation is
that the user has put the system into the right state first.  Then the
code-update implementation blocks any power-on until the activation is
complete [1].

I know the facebook/openbmc implementation automatically powers off, but
I think this is dangerous for general purpose commercial server.
Customers tend to get angry if a server powered off and they weren't
expecting it.

I will admit I don't know the Intel architecture well enough yet, but is
powering off the server prior to BIOS update actually required?  Is the
BIOS NOR chip always mapped into a physical address and used, or is the
BIOS at some point loaded and resident?  Are there stable points where
it is safe to perform an update?  Can we monitor POST code to know when
the BIOS is completed?

>   2.  Set ME/NM (Management engine or Node manager in x86) to recovery mode

Is this specific to the BIOS update path or is this something we should
do whenever the Host is powered off?  In either case I guess you can
make it a dependency on the systemd unit file, but it seems like it
would be nice if it were able to be generically applied to all power
on/off paths.

>   3.  Flip GPIO to access SPI flash used by host.
>   4.  Bind spi driver to access flash

This is another thing that seems like we could do generically on all
power on / power off paths?  Any time the host isn't running we can hit
the GPIO to put ownership at the BMC.  Is there any disadvantage to
that?

Is the GPIO something unique to Facebook's machines or do most other
Intel machines have the same requirements?

>   5.  Flashcp image to device.

I don't think `flashcp` is used today, or at least not in my
recollection of the previous Witherspoon implementation.  Is there any
advantage to it over `dd` to the raw mtdblock device?

>   6.  Unbind spi driver
>   7.  Flip GPIO back for host to access SPI flash
>   8.  Set ME/NM to operational mode
>   9.  Power on server.

Doesn't seem like "power on" should be a side-effect of a BIOS update.
Is this intended to be "go back to the previous power state"?

> I can have some flexibility in this sequence based on each platform configuration. Looking forward to your suggestions.
> 
> Regards
> -Vijay

[1] https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/ActivationBlocksTransition.interface.yaml

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Bios upgrade from BMC

[Oskar Senft]

[-- Attachment #1: Type: text/plain, Size: 4266 bytes --]

Hi Patrick

Here some thoughts:

>   1.  Power off host server.
>
> I will admit I don't know the Intel architecture well enough yet, but is
> powering off the server prior to BIOS update actually required?  Is the
> BIOS NOR chip always mapped into a physical address and used, or is the
> BIOS at some point loaded and resident?  Are there stable points where
> it is safe to perform an update?  Can we monitor POST code to know when
> the BIOS is completed?
>
There are two issues:

   - The host may access the BIOS SPI flash at any time by making BIOS
   calls. UEFI variables are such an example. The problem is that the BIOS
   code that executes these requests does not handle cases at all where the
   BIOS SPI flash becomes inaccessible. This results in an immediate crash of
   the host.
   - With ME in operational mode, we cannot guarantee that ME would not
   attempt to read/write from the SPI flash while the host is running. I'm not
   sure if it's possible to put ME into recovery mode WHILE the host is
   running or if the host needs to be shut down for that.

My understanding is that the only way to safely write the BIOS SPI flash
from the BMC is to shut the host down and put ME into recovery mode.
Alternatively, hold the host in full reset via RSMRST.


> >   2.  Set ME/NM (Management engine or Node manager in x86) to recovery
> mode
>
> Is this specific to the BIOS update path or is this something we should
> do whenever the Host is powered off?  In either case I guess you can
> make it a dependency on the systemd unit file, but it seems like it
> would be nice if it were able to be generically applied to all power
> on/off paths.
>
This question opens a can of worms. There are people who say that ME should
always be run in recovery mode ...


> >   3.  Flip GPIO to access SPI flash used by host.
> >   4.  Bind spi driver to access flash
>
> This is another thing that seems like we could do generically on all
> power on / power off paths?  Any time the host isn't running we can hit
> the GPIO to put ownership at the BMC.  Is there any disadvantage to
> that?
>
Yes. You cannot turn the host on via a power button if the PCH cannot
access the SPI flash. You'd have to catch that signal in the BMC and do the
right thing.

What's the advantage of having the BIOS SPI flash always connected to the
BMC when the host is off? That seems to be making things more complicated
to me.



> Is the GPIO something unique to Facebook's machines or do most other
> Intel machines have the same requirements?
>

I'm not sure if it was explained what the GPIO does:
Since the SPI flash can only have one master, a "mux" (it's really a
digital switch, or a pair of digital switches) connect the SPI flash either
to the PCH for access by the ME / host or to the BMC. The GPIO or pair of
GPIOs is used to control the mux / bus switches.

If the SPI flash is connected to the BMC, the ME / host cannot access it at
all. As it turns out, the PCH needs to be able to read the SPI flash to be
able to "turn on" the host.


>
> >   5.  Flashcp image to device.
>
> I don't think `flashcp` is used today, or at least not in my
> recollection of the previous Witherspoon implementation.  Is there any
> advantage to it over `dd` to the raw mtdblock device?
>
I'm new to this, too, and found this explanation:
https://unix.stackexchange.com/questions/274217/how-is-erasing-mtd-with-dd-if-dev-zero-different-from-flash-eraseall

This question was asked in the context of erase, but it applies to writes,
too.


> >   9.  Power on server.
>
> Doesn't seem like "power on" should be a side-effect of a BIOS update.
> Is this intended to be "go back to the previous power state"?
>
+1

Having said all that, I was experimenting with pretty much the same flow
but ended up with unreliable writes with individual bit flips. I'm pretty
sure the HW is fine, since the original (AMI) stock firmware that comes
with the board can do it just fine. This is with an Aspeed AST2500, a C620
PCH and a Dediprog EM100 SPI flash emulator.

I had even tried to change the SPI flash clock from the Aspeed down to the
minimum, with no change :-/ I already hooked up a logic analyzer to see
what's going on but haven't had a chance to investigate yet. Any ideas?

Oskar.

[-- Attachment #2: Type: text/html, Size: 5878 bytes --]

Re: Bios upgrade from BMC

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 5855 bytes --]

On Thu, Jan 30, 2020 at 11:30:10AM -0500, Oskar Senft wrote:
> Hi Patrick
> 
> Here some thoughts:
> 
> >   1.  Power off host server.
> >
> > I will admit I don't know the Intel architecture well enough yet, but is
> > powering off the server prior to BIOS update actually required?  Is the
> > BIOS NOR chip always mapped into a physical address and used, or is the
> > BIOS at some point loaded and resident?  Are there stable points where
> > it is safe to perform an update?  Can we monitor POST code to know when
> > the BIOS is completed?
> >
> There are two issues:
> 
>    - The host may access the BIOS SPI flash at any time by making BIOS
>    calls. UEFI variables are such an example. The problem is that the BIOS
>    code that executes these requests does not handle cases at all where the
>    BIOS SPI flash becomes inaccessible. This results in an immediate crash of
>    the host.
>    - With ME in operational mode, we cannot guarantee that ME would not
>    attempt to read/write from the SPI flash while the host is running. I'm not
>    sure if it's possible to put ME into recovery mode WHILE the host is
>    running or if the host needs to be shut down for that.
> 
> My understanding is that the only way to safely write the BIOS SPI flash
> from the BMC is to shut the host down and put ME into recovery mode.
> Alternatively, hold the host in full reset via RSMRST.

Good to know, thanks.

> > >   2.  Set ME/NM (Management engine or Node manager in x86) to recovery
> > mode
> >
> > Is this specific to the BIOS update path or is this something we should
> > do whenever the Host is powered off?  In either case I guess you can
> > make it a dependency on the systemd unit file, but it seems like it
> > would be nice if it were able to be generically applied to all power
> > on/off paths.
> >
> This question opens a can of worms. There are people who say that ME should
> always be run in recovery mode ...

Hah.  I think it is worth answering if the ME provides any useful
function when the server is powered off though.  I don't know, but it
would potentially simplify the BIOS update flow if Host Off => ME in
Recovery.

> > >   3.  Flip GPIO to access SPI flash used by host.
> > >   4.  Bind spi driver to access flash
> >
> > This is another thing that seems like we could do generically on all
> > power on / power off paths?  Any time the host isn't running we can hit
> > the GPIO to put ownership at the BMC.  Is there any disadvantage to
> > that?
> >
> Yes. You cannot turn the host on via a power button if the PCH cannot
> access the SPI flash. You'd have to catch that signal in the BMC and do the
> right thing.
> 
> What's the advantage of having the BIOS SPI flash always connected to the
> BMC when the host is off? That seems to be making things more complicated
> to me.

It was just another simplification.  Usually we have special user
utilities to steal the flash to the BMC and we have this logic in BIOS
update path.  Again, if Host Off => BIOS SPI owned by BMC, it simplifies
/ eliminates logic.

> > Is the GPIO something unique to Facebook's machines or do most other
> > Intel machines have the same requirements?
> >
> 
> I'm not sure if it was explained what the GPIO does:
> Since the SPI flash can only have one master, a "mux" (it's really a
> digital switch, or a pair of digital switches) connect the SPI flash either
> to the PCH for access by the ME / host or to the BMC. The GPIO or pair of
> GPIOs is used to control the mux / bus switches.
> 
> If the SPI flash is connected to the BMC, the ME / host cannot access it at
> all. As it turns out, the PCH needs to be able to read the SPI flash to be
> able to "turn on" the host.

Yep, I'm aware of the mux (on Facebook systems).  I wasn't sure if this
was common or typical Intel architecture feature or something we
specifically had on our Facebook systems.

> >
> > >   5.  Flashcp image to device.
> >
> > I don't think `flashcp` is used today, or at least not in my
> > recollection of the previous Witherspoon implementation.  Is there any
> > advantage to it over `dd` to the raw mtdblock device?
> >
> I'm new to this, too, and found this explanation:
> https://unix.stackexchange.com/questions/274217/how-is-erasing-mtd-with-dd-if-dev-zero-different-from-flash-eraseall
> 
> This question was asked in the context of erase, but it applies to writes,
> too.

The stackexchange here is referring to /dev/mtdN devices and not
/dev/mtdblockN devices (and I agree for plain-mtd).  mtdblock
specifically has the extra logic to deal with erasing and writing in
pages as appropriate.

> > >   9.  Power on server.
> >
> > Doesn't seem like "power on" should be a side-effect of a BIOS update.
> > Is this intended to be "go back to the previous power state"?
> >
> +1
> 
> Having said all that, I was experimenting with pretty much the same flow
> but ended up with unreliable writes with individual bit flips. I'm pretty
> sure the HW is fine, since the original (AMI) stock firmware that comes
> with the board can do it just fine. This is with an Aspeed AST2500, a C620
> PCH and a Dediprog EM100 SPI flash emulator.
> 
> I had even tried to change the SPI flash clock from the Aspeed down to the
> minimum, with no change :-/ I already hooked up a logic analyzer to see
> what's going on but haven't had a chance to investigate yet. Any ideas?

Sorry, I've got nothing except maybe the original code retries a bunch
to get past random flips?  If you are seeing bit-flips even with the
Dediprog, are you sure the bus is any good?  Did you solder on headers
to be able to affix the Dediprog?  That might have changed the
capacitance enough to affect SPI activity.

> Oskar.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Bios upgrade from BMC

[Oskar Senft]

[-- Attachment #1: Type: text/plain, Size: 6595 bytes --]

Hi everyone

I finally figured it out. And with "it" I mean the problem
that reading/writing to the BIOS flash ROM from an AST2500 was unreliable.
The problem was that Linux on the BMC didn't like when I changed the flash
image while the BMC was running. I was using a Dediprog EM100 flash
emulator, which allows to do that safely. The fix is to either not do that
(which wouldn't happen in production anyway), or to explicitly unbind +
bind the mtd before using it again after changing the flash contents. Yeah!

Oskar.

On Thu, Jan 30, 2020 at 7:22 PM Patrick Williams <patrick@stwcx.xyz> wrote:

> On Thu, Jan 30, 2020 at 11:30:10AM -0500, Oskar Senft wrote:
> > Hi Patrick
> >
> > Here some thoughts:
> >
> > >   1.  Power off host server.
> > >
> > > I will admit I don't know the Intel architecture well enough yet, but
> is
> > > powering off the server prior to BIOS update actually required?  Is the
> > > BIOS NOR chip always mapped into a physical address and used, or is the
> > > BIOS at some point loaded and resident?  Are there stable points where
> > > it is safe to perform an update?  Can we monitor POST code to know when
> > > the BIOS is completed?
> > >
> > There are two issues:
> >
> >    - The host may access the BIOS SPI flash at any time by making BIOS
> >    calls. UEFI variables are such an example. The problem is that the
> BIOS
> >    code that executes these requests does not handle cases at all where
> the
> >    BIOS SPI flash becomes inaccessible. This results in an immediate
> crash of
> >    the host.
> >    - With ME in operational mode, we cannot guarantee that ME would not
> >    attempt to read/write from the SPI flash while the host is running.
> I'm not
> >    sure if it's possible to put ME into recovery mode WHILE the host is
> >    running or if the host needs to be shut down for that.
> >
> > My understanding is that the only way to safely write the BIOS SPI flash
> > from the BMC is to shut the host down and put ME into recovery mode.
> > Alternatively, hold the host in full reset via RSMRST.
>
> Good to know, thanks.
>
> > > >   2.  Set ME/NM (Management engine or Node manager in x86) to
> recovery
> > > mode
> > >
> > > Is this specific to the BIOS update path or is this something we should
> > > do whenever the Host is powered off?  In either case I guess you can
> > > make it a dependency on the systemd unit file, but it seems like it
> > > would be nice if it were able to be generically applied to all power
> > > on/off paths.
> > >
> > This question opens a can of worms. There are people who say that ME
> should
> > always be run in recovery mode ...
>
> Hah.  I think it is worth answering if the ME provides any useful
> function when the server is powered off though.  I don't know, but it
> would potentially simplify the BIOS update flow if Host Off => ME in
> Recovery.
>
> > > >   3.  Flip GPIO to access SPI flash used by host.
> > > >   4.  Bind spi driver to access flash
> > >
> > > This is another thing that seems like we could do generically on all
> > > power on / power off paths?  Any time the host isn't running we can hit
> > > the GPIO to put ownership at the BMC.  Is there any disadvantage to
> > > that?
> > >
> > Yes. You cannot turn the host on via a power button if the PCH cannot
> > access the SPI flash. You'd have to catch that signal in the BMC and do
> the
> > right thing.
> >
> > What's the advantage of having the BIOS SPI flash always connected to the
> > BMC when the host is off? That seems to be making things more complicated
> > to me.
>
> It was just another simplification.  Usually we have special user
> utilities to steal the flash to the BMC and we have this logic in BIOS
> update path.  Again, if Host Off => BIOS SPI owned by BMC, it simplifies
> / eliminates logic.
>
> > > Is the GPIO something unique to Facebook's machines or do most other
> > > Intel machines have the same requirements?
> > >
> >
> > I'm not sure if it was explained what the GPIO does:
> > Since the SPI flash can only have one master, a "mux" (it's really a
> > digital switch, or a pair of digital switches) connect the SPI flash
> either
> > to the PCH for access by the ME / host or to the BMC. The GPIO or pair of
> > GPIOs is used to control the mux / bus switches.
> >
> > If the SPI flash is connected to the BMC, the ME / host cannot access it
> at
> > all. As it turns out, the PCH needs to be able to read the SPI flash to
> be
> > able to "turn on" the host.
>
> Yep, I'm aware of the mux (on Facebook systems).  I wasn't sure if this
> was common or typical Intel architecture feature or something we
> specifically had on our Facebook systems.
>
> > >
> > > >   5.  Flashcp image to device.
> > >
> > > I don't think `flashcp` is used today, or at least not in my
> > > recollection of the previous Witherspoon implementation.  Is there any
> > > advantage to it over `dd` to the raw mtdblock device?
> > >
> > I'm new to this, too, and found this explanation:
> >
> https://unix.stackexchange.com/questions/274217/how-is-erasing-mtd-with-dd-if-dev-zero-different-from-flash-eraseall
> >
> > This question was asked in the context of erase, but it applies to
> writes,
> > too.
>
> The stackexchange here is referring to /dev/mtdN devices and not
> /dev/mtdblockN devices (and I agree for plain-mtd).  mtdblock
> specifically has the extra logic to deal with erasing and writing in
> pages as appropriate.
>
> > > >   9.  Power on server.
> > >
> > > Doesn't seem like "power on" should be a side-effect of a BIOS update.
> > > Is this intended to be "go back to the previous power state"?
> > >
> > +1
> >
> > Having said all that, I was experimenting with pretty much the same flow
> > but ended up with unreliable writes with individual bit flips. I'm pretty
> > sure the HW is fine, since the original (AMI) stock firmware that comes
> > with the board can do it just fine. This is with an Aspeed AST2500, a
> C620
> > PCH and a Dediprog EM100 SPI flash emulator.
> >
> > I had even tried to change the SPI flash clock from the Aspeed down to
> the
> > minimum, with no change :-/ I already hooked up a logic analyzer to see
> > what's going on but haven't had a chance to investigate yet. Any ideas?
>
> Sorry, I've got nothing except maybe the original code retries a bunch
> to get past random flips?  If you are seeing bit-flips even with the
> Dediprog, are you sure the bus is any good?  Did you solder on headers
> to be able to affix the Dediprog?  That might have changed the
> capacitance enough to affect SPI activity.
>
> > Oskar.
>
> --
> Patrick Williams
>

[-- Attachment #2: Type: text/html, Size: 8095 bytes --]