Design proposal for dual BMC flash with "golden image"

Design proposal for dual BMC flash with "golden image"

[Lei Yu]

We are working on a system that has two BMC chips, and one of them is
designed to be used as a golden image.

The current OpenBMC does not support this usage, so a design proposal
is sent to gerrit, please help to review if you are interested.

https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/36407

Thanks!

-- 
BRs,
Lei YU

Re: Design proposal for dual BMC flash with "golden image"

[Alexander Amelkin]

Hi Lei!

11.09.2020 11:49, Lei Yu пишет:
> We are working on a system that has two BMC chips, and one of them is
> designed to be used as a golden image.
>
> The current OpenBMC does not support this usage, so a design proposal
> is sent to gerrit, please help to review if you are interested.
>
> https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/36407
>
> Thanks!

We have this support implemented for YADRO VESNIN in our private repository.

Ivan will send a description of how it is implemented for us.

Alexander.

Re: Design proposal for dual BMC flash with "golden image"

[Lei Yu]

On Fri, Sep 11, 2020 at 7:48 PM Alexander Amelkin <a.amelkin@yadro.com> wrote:
>
> Hi Lei!
>
> 11.09.2020 11:49, Lei Yu пишет:
> > We are working on a system that has two BMC chips, and one of them is
> > designed to be used as a golden image.
> >
> > The current OpenBMC does not support this usage, so a design proposal
> > is sent to gerrit, please help to review if you are interested.
> >
> > https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/36407
> >
> > Thanks!
>
> We have this support implemented for YADRO VESNIN in our private repository.
>

That's good to know! Could you comment on the design doc and see how
much difference it has?
Also, may I ask if Yadro's implementation could be open sourced or not?

> Ivan will send a description of how it is implemented for us.

Thanks.


-- 
BRs,
Lei YU

Re: Design proposal for dual BMC flash with "golden image"

[Ivan Mikhaylov]

Hey Lei, the main idea is to provide user transparent and safe access to
the golden side with, and to give our manufacturing department (and possibly the
support engineers too) the write lock control and possibility to flash the
golden chip during server assembly and even later if the need arises (like major
bugs discovered in the golden firmware version).

To achieve these goals we've made a utility and some kernel patches for watchdog
and spi nor driver:

- A series of patches for aspeed watchdog and watchdog api.
  A new ioctl was added into the watchdog api that sets a
  `start on reboot` status. It grants us transparent and safe switch into
  golden side without worrying about breaking something on the main spi flash
  side if we want to force-switch to golden side for testing.

- A series of patches for implementing flash locks for Macronix chips that we
use
- A utility which gives control of golden side, called `gftool` and providing   
  abilities to:
  - lock/unlock golden side
  - reboot into golden side
  - reflash golden side from main

> That's good to know! Could you comment on the design doc and see how
> much difference it has?

Yes, I will do on monday.

> Also, may I ask if Yadro's implementation could be open sourced or not?

Yes, but the watchdog series of patches is more like a hack for now.
We're expanding the watchdog api so that it grants the user control of WDT2 via
ioctl(watchdog start on reboot) and allows for rebooting into golden chip using
WDT2. However the upstream kernel maintainers won't accept it as they say it's a
driver level decision, and we're exporting it to the user. That's something they
don't want to accept. That may be fine though as theoretically the patch set may
be kept only in the openbmc linux kernel, but that will put an additional
support burden on Joel and the community. I'm discussing that with Joel.

Thanks.

Re: Design proposal for dual BMC flash with "golden image"

[Lei Yu]

> - A series of patches for implementing flash locks for Macronix chips that we
> use
> - A utility which gives control of golden side, called `gftool` and providing
>   abilities to:
>   - lock/unlock golden side
>   - reboot into golden side
>   - reflash golden side from main
>

Does the lock/unlock function work on the Macronix chip only? Does it
apply to other chips?

> > That's good to know! Could you comment on the design doc and see how
> > much difference it has?
>
> Yes, I will do on monday.

Thanks, let's discuss the design doc in gerrit.

> Yes, but the watchdog series of patches is more like a hack for now.
> We're expanding the watchdog api so that it grants the user control of WDT2 via
> ioctl(watchdog start on reboot) and allows for rebooting into golden chip using
> WDT2. However the upstream kernel maintainers won't accept it as they say it's a
> driver level decision, and we're exporting it to the user. That's something they
> don't want to accept. That may be fine though as theoretically the patch set may
> be kept only in the openbmc linux kernel, but that will put an additional
> support burden on Joel and the community. I'm discussing that with Joel.

OK, thanks.
For now, we use "devmem" to manipulate the registers for testing purpose.
It's nice to have that driver, but in productions there will be no
need to use devmem nor the ioctl on watchdog, so it's not a must for
us to use the driver.


-- 
BRs,
Lei YU

Re: Design proposal for dual BMC flash with "golden image"

[Ivan Mikhaylov]

On Tue, 2020-09-15 at 11:38 +0800, Lei Yu wrote:
> > - A series of patches for implementing flash locks for Macronix chips that
> > we
> > use
> > - A utility which gives control of golden side, called `gftool` and
> > providing
> >   abilities to:
> >   - lock/unlock golden side
> >   - reboot into golden side
> >   - reflash golden side from main
> > 
> 
> Does the lock/unlock function work on the Macronix chip only? Does it
> apply to other chips?

It can be applied to different chips, macronix locks are not supported for now
in upstream. You can check here - 
https://elixir.bootlin.com/linux/v5.9-rc5/source/drivers/mtd/spi-nor , chipswith SPI_NOR_HAS_LOCK.  

> > Yes, but the watchdog series of patches is more like a hack for now.
> > We're expanding the watchdog api so that it grants the user control of WDT2
> > via
> > ioctl(watchdog start on reboot) and allows for rebooting into golden chip
> > using
> > WDT2. However the upstream kernel maintainers won't accept it as they say
> > it's a
> > driver level decision, and we're exporting it to the user. That's something
> > they
> > don't want to accept. That may be fine though as theoretically the patch set
> > may
> > be kept only in the openbmc linux kernel, but that will put an additional
> > support burden on Joel and the community. I'm discussing that with Joel.
> 
> OK, thanks.
> For now, we use "devmem" to manipulate the registers for testing purpose.
> It's nice to have that driver, but in productions there will be no
> need to use devmem nor the ioctl on watchdog, so it's not a must for
> us to use the driver.
> 

And how you switch safely into golden side in this case?

Thanks.

Re: Design proposal for dual BMC flash with "golden image"

[Lei Yu]

On Wed, Sep 16, 2020 at 5:55 PM Ivan Mikhaylov <i.mikhaylov@yadro.com> wrote:
> > For now, we use "devmem" to manipulate the registers for testing purpose.
> > It's nice to have that driver, but in productions there will be no
> > need to use devmem nor the ioctl on watchdog, so it's not a must for
> > us to use the driver.
> >
>
> And how you switch safely into golden side in this case?
>

The plan is:
1. When the primary flash is broken and u-boot could not be started,
aspeed will switch to the golden side automatically.
2. When the primary flash's u-boot is OK, but the kernel/rofs fails a
couple of times, the u-boot could detect this and switch to the golden
side by setting the related registers. See example in
https://github.com/openbmc/openbmc/blob/master/meta-phosphor/aspeed-layer/recipes-bsp/u-boot/files/0005-config-ast-common-Fall-back-to-secondary-flash-on-fa.patch

Re: Design proposal for dual BMC flash with "golden image"

[Alexander Amelkin]

16.09.2020 13:18, Lei Yu пишет:
> Does the lock/unlock function work on the Macronix chip only? Does it
> apply to other chips?

As Ivan said, it is supported by some other chips as well. At least some 
ST chips have that feature.

Nonetheless, that feature is not a must. We just had to implement 
support for it for the Macronix chips
that we use, or otherwise they would just ignore the #WP pin. Most chips 
that don't have the software
lock/unlock feature can be properly protected/unprotected using the 
traditional #WP pin mechanism.

> On Wed, Sep 16, 2020 at 5:55 PM Ivan Mikhaylov <i.mikhaylov@yadro.com> 
> wrote:
>>> For now, we use "devmem" to manipulate the registers for testing purpose.
>>> It's nice to have that driver, but in productions there will be no
>>> need to use devmem nor the ioctl on watchdog, so it's not a must for
>>> us to use the driver.
>>>
>> And how you switch safely into golden side in this case?
>>
> The plan is:
> 1. When the primary flash is broken and u-boot could not be started,
> aspeed will switch to the golden side automatically.
> 2. When the primary flash's u-boot is OK, but the kernel/rofs fails a
> couple of times, the u-boot could detect this and switch to the golden
> side by setting the related registers. See example in
> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/aspeed-layer/recipes-bsp/u-boot/files/0005-config-ast-common-Fall-back-to-secondary-flash-on-fa.patch

The main idea behind the new ioctl() is to allow for easy switching into 
the golden chip
for testing purposes when both the u-boot and the kernel/rofs on the 
primary chip are fine.

Alexander

Re: Design proposal for dual BMC flash with "golden image"

[Ivan Mikhaylov]

On Wed, 2020-09-16 at 18:18 +0800, Lei Yu wrote:
> On Wed, Sep 16, 2020 at 5:55 PM Ivan Mikhaylov <i.mikhaylov@yadro.com> wrote:
> > > For now, we use "devmem" to manipulate the registers for testing purpose.
> > > It's nice to have that driver, but in productions there will be no
> > > need to use devmem nor the ioctl on watchdog, so it's not a must for
> > > us to use the driver.
> > > 
> > 
> > And how you switch safely into golden side in this case?
> > 
> 
> The plan is:
> 1. When the primary flash is broken and u-boot could not be started,
> aspeed will switch to the golden side automatically.
> 2. When the primary flash's u-boot is OK, but the kernel/rofs fails a
> couple of times, the u-boot could detect this and switch to the golden
> side by setting the related registers. See example in
> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/aspeed-layer/recipes-bsp/u-boot/files/0005-config-ast-common-Fall-back-to-secondary-flash-on-fa.patch

That's the standard procedure, I'm saying that you may need the access to the
golden side at any time for different purposes(support engineers, manufacturing
department, some problems in field) and how you can achieve that? With u-boot
env set or manual watchdog trigger?

Thanks.

Re: Design proposal for dual BMC flash with "golden image"

[Lei Yu]

On Wed, Sep 16, 2020 at 8:02 PM Ivan Mikhaylov <i.mikhaylov@yadro.com> wrote:
> > The plan is:
> > 1. When the primary flash is broken and u-boot could not be started,
> > aspeed will switch to the golden side automatically.
> > 2. When the primary flash's u-boot is OK, but the kernel/rofs fails a
> > couple of times, the u-boot could detect this and switch to the golden
> > side by setting the related registers. See example in
> > https://github.com/openbmc/openbmc/blob/master/meta-phosphor/aspeed-layer/recipes-bsp/u-boot/files/0005-config-ast-common-Fall-back-to-secondary-flash-on-fa.patch
>
> That's the standard procedure, I'm saying that you may need the access to the
> golden side at any time for different purposes(support engineers, manufacturing
> department, some problems in field) and how you can achieve that? With u-boot
> env set or manual watchdog trigger?

In that case, we could manually trigger watchdog by:
1. Enable devmem by appending mem.devmem in bootargs in u-boot, and reboot
2. Now devmem could be used, manually trigger wdt2

Be noted that such case should be rare in the field.
In the lab, we could enable devmem by default and just use it.

Re: Design proposal for dual BMC flash with "golden image"

[Ivan Mikhaylov]

On Wed, 2020-09-16 at 20:13 +0800, Lei Yu wrote:
> On Wed, Sep 16, 2020 at 8:02 PM Ivan Mikhaylov <i.mikhaylov@yadro.com> wrote:
> > > The plan is:
> > > 1. When the primary flash is broken and u-boot could not be started,
> > > aspeed will switch to the golden side automatically.
> > > 2. When the primary flash's u-boot is OK, but the kernel/rofs fails a
> > > couple of times, the u-boot could detect this and switch to the golden
> > > side by setting the related registers. See example in
> > > https://github.com/openbmc/openbmc/blob/master/meta-phosphor/aspeed-layer/recipes-bsp/u-boot/files/0005-config-ast-common-Fall-back-to-secondary-flash-on-fa.patch
> > 
> > That's the standard procedure, I'm saying that you may need the access to
> > the
> > golden side at any time for different purposes(support engineers,
> > manufacturing
> > department, some problems in field) and how you can achieve that? With u-
> > boot
> > env set or manual watchdog trigger?
> 
> In that case, we could manually trigger watchdog by:
> 1. Enable devmem by appending mem.devmem in bootargs in u-boot, and reboot
> 2. Now devmem could be used, manually trigger wdt2
> 
> Be noted that such case should be rare in the field.
> In the lab, we could enable devmem by default and just use it.

I'm trying to say that if it would be done by low level application, would it be
much better without additional reboots? Just app execution like:
 <app> --to-golden-side

It will remove all these steps with devmem hacks. 

Thanks.