OpenBMC Image Management

OpenBMC Image Management

[Adriana Kobylak]

Hi,

Here is a first draft proposal for image management (code update) in OpenBMC for feedback.

General:
* Move to UBI volume management (on the BMC and PNOR chips), which provides dynamic partitioning and wear-leveling. In a limited flash space environment there might be the need to re-allocate partition space and resizing in a static partitioning implementation can be very painful.
* Use CRAMFS for read-only partitions, and UBIFS for read-write partitions.

PNOR:
* Implement a new mboxd based on Cyril’s (see Cyril’s email “Mailbox daemon”) that handles dbus objects, generates a virtual PNOR for UBI content, etc.
* Move to memory-based access instead of the current LPC-to-AHD path.
* Allocate just enough space in the initial window for Hostboot’s TOC, HBB, and HBI partitions, grow as required.
* Ability to ‘patch’ by copying a Hostboot image *.bin into a designated directory (/usr/local/ for example).
* Tool to write from PNOR image to BMC format. Implement UBI initially but it could be extended to support different volume managements on other BMCs.

BMC:
*Save multiple firmware versions, starting with 2, to provide the ability to roll-back if needed. If single BMC chip system, save both versions there. If two BMC chip system, save other version in 2nd chip.
* Implement various levels of ‘persistency’, such as dev, factory, field. Dev persistency would allow for local patches (/usr/local/ for example) that can be cleared before shipping to customers. Factory mode could delete the location where user data such as network settings resides but preserves the mac address and uuid for example. Etc.

Thanks,

Adriana

Re: OpenBMC Image Management

[Chris Austen]

[-- Attachment #1: Type: text/plain, Size: 2591 bytes --]

"openbmc" <openbmc-bounces+austenc=us.ibm.com@lists.ozlabs.org> wrote on
01/25/2017 04:15:27 PM:

> From: Adriana Kobylak <anoo@linux.vnet.ibm.com>
> To: openbmc@lists.ozlabs.org
> Date: 01/25/2017 04:32 PM
> Subject: OpenBMC Image Management
> Sent by: "openbmc" <openbmc-bounces+austenc=us.ibm.com@lists.ozlabs.org>
>
> Hi,
>
> Here is a first draft proposal for image management (code update) in
> OpenBMC for feedback.
>
> General:
> * Move to UBI volume management (on the BMC and PNOR chips), which
> provides dynamic partitioning and wear-leveling. In a limited flash
> space environment there might be the need to re-allocate partition
> space and resizing in a static partitioning implementation can be
> very painful.
> * Use CRAMFS for read-only partitions, and UBIFS for read-write
partitions.
>
> PNOR:
> * Implement a new mboxd based on Cyril’s (see Cyril’s email “Mailbox
> daemon”) that handles dbus objects, generates a virtual PNOR for UBI
> content, etc.
> * Move to memory-based access instead of the current LPC-to-AHD path.
> * Allocate just enough space in the initial window for Hostboot’s
> TOC, HBB, and HBI partitions, grow as required.
> * Ability to ‘patch’ by copying a Hostboot image *.bin into a
> designated directory (/usr/local/ for example).
> * Tool to write from PNOR image to BMC format. Implement UBI
> initially but it could be extended to support different volume
> managements on other BMCs.
>
> BMC:
> *Save multiple firmware versions, starting with 2, to provide the
> ability to roll-back if needed. If single BMC chip system, save both
> versions there. If two BMC chip system, save other version in 2nd chip.
> * Implement various levels of ‘persistency’, such as dev, factory,
> field. Dev persistency would allow for local patches (/usr/local/
> for example) that can be cleared before shipping to customers.
> Factory mode could delete the location where user data such as
> network settings resides but preserves the mac address and uuid for
> example. Etc.
>
> Thanks,
>
> Adriana
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc


So are the goals of Image Management..

1) Actively manage wear-leveling
2) Provide an api for BIOS/Host inband update
3) Provide a way to boot an image that is not from flash
4) Duel image support
5) Duel flash chip support
6) Support multiple tiers of sanitization

Are there any security goals that need to be considered?


[-- Attachment #2: Type: text/html, Size: 3155 bytes --]

Re: OpenBMC Image Management

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2803 bytes --]

On Wed, Jan 25, 2017 at 05:50:46PM -0600, Chris Austen wrote:
> "openbmc" <openbmc-bounces+austenc=us.ibm.com@lists.ozlabs.org> wrote on
> 01/25/2017 04:15:27 PM:
> 
> Are there any security goals that need to be considered?
> 

There are a few different aspects to security that I can think of:

1. Is there a way to identify and reject an invalid image (Define
"invalid") before it is applied onto the system?

2. Is there a way to identify an applied image has been tampered with?

3. Is there a way for an image to expose a security flaw in the code
itself (such as by "fuzzing") to cause unintended effects?

A few statements to answer your question:

    * If there is a fundamental flaw in any of these regards with our design,
      we would like to know about it and will fix it.

    * #1 is typically solved through image signing and a one-time 
      verification at the time an image is applied.  Issue
      openbmc/openbmc#356 is meant to implement this and would be a
      later feature on top of Adriana's proposed work.

    * #2 is typically solved through "Secureboot" or similar
      functionality.  
        * The Power9 processor can implement Secureboot itself, so the IBM
          team currently has no plans to implement additional per-use
          verification of the Host firmware contents [in PNOR] by the BMC.
        * IBM also does not currently plan to include BMC Secureboot for
          the Witherspoon machine's initial delivery.
        * Rick Altherr from Google has been contributing support for
          U-Boot "FIT" images, which provide something like Secureboot
          verification for the kernel and initramfs images.
        * The project would certainly be interested in further
          development here, but hopefully like most of our features, it
          can be selectively enabled or disabled for different machines.
          In particular, given that Power9 is itself performing
          verification of images, verification of the PNOR by the BMC
          should be a choice by the vendor weighing boot-speed and
          security.
        * Adriana is not intending on doing anything explicit to enable
          or impede enablement of this.  If there is something in the
          design or implementation that would significantly impede
          enablement, or if there is a relatively minor improvement that
          can be done to aid future enablement, we are very interested
          in this feedback.

    * We don't intend to have purposeful flaws in the area of #3 (does
      anyone?).  No one has committed to doing explicit penetration
      testing here, but we certainly will accept vulnerability reports
      and, even better, fixes.

-- 
Patrick Williams

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: OpenBMC Image Management

[Stewart Smith]

Patrick Williams <patrick@stwcx.xyz> writes:
> On Wed, Jan 25, 2017 at 05:50:46PM -0600, Chris Austen wrote:
>> "openbmc" <openbmc-bounces+austenc=us.ibm.com@lists.ozlabs.org> wrote on
>> 01/25/2017 04:15:27 PM:
>> 
>> Are there any security goals that need to be considered?
>> 
>
> There are a few different aspects to security that I can think of:
>
> 1. Is there a way to identify and reject an invalid image (Define
> "invalid") before it is applied onto the system?
>
> 2. Is there a way to identify an applied image has been tampered with?
>
> 3. Is there a way for an image to expose a security flaw in the code
> itself (such as by "fuzzing") to cause unintended effects?

I think the biggest opportunity for fuzzing and security analysis will
be in BMC<->HOST interfaces.

It'd be great if every BMC<->HOST interface could be fuzzed in sim or in
userspace.


> A few statements to answer your question:
>
>     * If there is a fundamental flaw in any of these regards with our design,
>       we would like to know about it and will fix it.
>
>     * #1 is typically solved through image signing and a one-time 
>       verification at the time an image is applied.  Issue
>       openbmc/openbmc#356 is meant to implement this and would be a
>       later feature on top of Adriana's proposed work.
>
>     * #2 is typically solved through "Secureboot" or similar
>       functionality.  
>         * The Power9 processor can implement Secureboot itself, so the IBM
>           team currently has no plans to implement additional per-use
>           verification of the Host firmware contents [in PNOR] by the BMC.
>         * IBM also does not currently plan to include BMC Secureboot for
>           the Witherspoon machine's initial delivery.

dm-verity (a device-mapper target taht cryptographically verifies each
filesystem block) could be a way to very easily get most of what's
needed here.

https://lwn.net/Articles/459420/

https://source.android.com/security/verifiedboot/


>         * Rick Altherr from Google has been contributing support for
>           U-Boot "FIT" images, which provide something like Secureboot
>           verification for the kernel and initramfs images.

Combined with dm-verity, we'd be a long way towards a remotely
trustworthy BMC (well, trust-worthy in the way that it's running a
*known* set of vulnerabilities :)

-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

On Mon, Jan 30, 2017 at 04:47:13PM +1100, Stewart Smith wrote:
> dm-verity (a device-mapper target taht cryptographically verifies each
> filesystem block) could be a way to very easily get most of what's
> needed here.
> 
> https://lwn.net/Articles/459420/
> 
> https://source.android.com/security/verifiedboot/
> 

Any ideas on how nicely that plays with mtd/ubi?  I don't see anything
about it.  I do see some dm-verity presentations claiming that IMA is
slow and dm-verity is much faster.

We should have all code in a SquashFS image anyhow.  Signing / verifying
that whole image might be reasonable as well.

-- 
Patrick Williams

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: OpenBMC Image Management

[Rick Altherr]

[-- Attachment #1: Type: text/plain, Size: 1285 bytes --]

dm-verity works by hashing each 4k block of the raw block device.  The
resulting tree of hashes is stored on the block device outside the
filesystem.  That should work fine for MTD.  I suspect it will work with
UBI as well since UBI is just another block device type.  So, we can
probably store FIT+squashfs+dm-verity(of squashfs)+UBIFS(r/w) in UBI.
It'll take some experimentation.

Rick

On Tue, Jan 31, 2017 at 10:16 AM, Patrick Williams <patrick@stwcx.xyz>
wrote:

> On Mon, Jan 30, 2017 at 04:47:13PM +1100, Stewart Smith wrote:
> > dm-verity (a device-mapper target taht cryptographically verifies each
> > filesystem block) could be a way to very easily get most of what's
> > needed here.
> >
> > https://lwn.net/Articles/459420/
> >
> > https://source.android.com/security/verifiedboot/
> >
>
> Any ideas on how nicely that plays with mtd/ubi?  I don't see anything
> about it.  I do see some dm-verity presentations claiming that IMA is
> slow and dm-verity is much faster.
>
> We should have all code in a SquashFS image anyhow.  Signing / verifying
> that whole image might be reasonable as well.
>
> --
> Patrick Williams
>
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc
>
>

[-- Attachment #2: Type: text/html, Size: 2131 bytes --]

Re: OpenBMC Image Management

[Adriana Kobylak]

[-- Attachment #1: Type: text/html, Size: 1695 bytes --]

Re: OpenBMC Image Management

[Stewart Smith]

Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
> Regarding the mailbox daemon, the mboxbridge repository is staying as a reference
> implementation and has been cloned into phosphor-mboxd. Any system using mailbox and
> running with a flashed ffs pnor image (instead of updating to the ubi layout) would still need
> the reference mboxbridge implementation. The openbmc build is being updated to build
> phosphor-mboxd as the default.

Why is there mboxbridge separate from phosphor-mboxd?


-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management

[Adriana Kobylak]

> On Jun 13, 2017, at 4:45 PM, Stewart Smith <stewart@linux.vnet.ibm.com> wrote:
> 
> Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
>> Regarding the mailbox daemon, the mboxbridge repository is staying as a reference
>> implementation and has been cloned into phosphor-mboxd. Any system using mailbox and
>> running with a flashed ffs pnor image (instead of updating to the ubi layout) would still need
>> the reference mboxbridge implementation. The openbmc build is being updated to build
>> phosphor-mboxd as the default.
> 
> Why is there mboxbridge separate from phosphor-mboxd?

Because the mboxbridge is being used by other companies as a reference to be able to implement the mailbox daemon in their own BMC firmware stack for other openpower systems, we didn’t want to “pollute” the repository with openbmc-specific and c++ implementation that could confuse them.

> 
> 
> -- 
> Stewart Smith
> OPAL Architect, IBM.

Re: OpenBMC Image Management

[Stewart Smith]

Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
>> Why is there mboxbridge separate from phosphor-mboxd?
>
> Because the mboxbridge is being used by other companies as a reference
> to be able to implement the mailbox daemon in their own BMC firmware
> stack for other openpower systems, we didn’t want to “pollute” the
> repository with openbmc-specific and c++ implementation that could
> confuse them.

Why are there OpenBMC specific things in mboxd?

Why can't the reference implementation be used? What's deficient in it?

-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management

[Adriana Kobylak]

> On Jun 19, 2017, at 12:44 AM, Stewart Smith <stewart@linux.vnet.ibm.com> wrote:
> 
> Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
>>> Why is there mboxbridge separate from phosphor-mboxd?
>> 
>> Because the mboxbridge is being used by other companies as a reference
>> to be able to implement the mailbox daemon in their own BMC firmware
>> stack for other openpower systems, we didn’t want to “pollute” the
>> repository with openbmc-specific and c++ implementation that could
>> confuse them.
> 
> Why are there OpenBMC specific things in mboxd?
> 
> Why can't the reference implementation be used? What's deficient in it?

For OpenBMC, we’re having the PNOR chip with a filesystem, so mboxd would read/write files instead of looking for the data at an offset in the chip. Nothing deficient with it, just a different implementation that needs to be handled.

> 
> -- 
> Stewart Smith
> OPAL Architect, IBM.

Re: OpenBMC Image Management

[Chris Austen]

[-- Attachment #1: Type: text/plain, Size: 1688 bytes --]



"openbmc" <openbmc-bounces+austenc=us.ibm.com@lists.ozlabs.org> wrote on
06/20/2017 09:38:40 AM:

> From: Adriana Kobylak <anoo@linux.vnet.ibm.com>
> To: Stewart Smith <stewart@linux.vnet.ibm.com>
> Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org>
> Date: 06/20/2017 09:38 AM
> Subject: Re: OpenBMC Image Management
> Sent by: "openbmc" <openbmc-bounces+austenc=us.ibm.com@lists.ozlabs.org>
>
>
> > On Jun 19, 2017, at 12:44 AM, Stewart Smith
> <stewart@linux.vnet.ibm.com> wrote:
> >
> > Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
> >>> Why is there mboxbridge separate from phosphor-mboxd?
> >>
> >> Because the mboxbridge is being used by other companies as a reference
> >> to be able to implement the mailbox daemon in their own BMC firmware
> >> stack for other openpower systems, we didn’t want to “pollute” the
> >> repository with openbmc-specific and c++ implementation that could
> >> confuse them.
> >
> > Why are there OpenBMC specific things in mboxd?
> >
> > Why can't the reference implementation be used? What's deficient in it?
>
> For OpenBMC, we’re having the PNOR chip with a filesystem, so mboxd
> would read/write files instead of looking for the data at an offset
> in the chip. Nothing deficient with it, just a different
> implementation that needs to be handled.

I have been recommending that if you want to try OpenBMC on your system
(that is not running  OpenBMC now) that the BMC chip be swapped.  Does the
offset vs filesystem change force the PNOR flash image to be forever
changed too?

>
> >
> > --
> > Stewart Smith
> > OPAL Architect, IBM.
>
Chris Austen
POWER Systems Enablement Manager

[-- Attachment #2: Type: text/html, Size: 2395 bytes --]

Re: OpenBMC Image Management

[Adriana Kobylak]

[-- Attachment #1: Type: text/plain, Size: 1324 bytes --]


> On Jun 20, 2017, at 10:03 AM, Chris Austen <austenc@us.ibm.com> wrote:
> 
> "openbmc" <openbmc-bounces+austenc=us.ibm.com@lists.ozlabs.org <mailto:openbmc-bounces+austenc=us.ibm.com@lists.ozlabs.org>> wrote on 06/20/2017 09:38:40 AM:
> 
> 
> > 
> > For OpenBMC, we’re having the PNOR chip with a filesystem, so mboxd 
> > would read/write files instead of looking for the data at an offset 
> > in the chip. Nothing deficient with it, just a different 
> > implementation that needs to be handled.
> 
> I have been recommending that if you want to try OpenBMC on your system (that is not running  OpenBMC now) that the BMC chip be swapped.  Does the offset vs filesystem change force the PNOR flash image to be forever changed too?
> 
> 
Not forever, it can be reflashed back to ffs using pflash. If the virtual pnor is not desired to be used on that system (since in this example the user just wants to try out openbmc) it can be disabled via a config flag when building the bmc image. Ex:

https://github.com/openbmc/openbmc/commit/dd05b963e1847a1f56b11825a7d39602b6151165 <https://github.com/openbmc/openbmc/commit/dd05b963e1847a1f56b11825a7d39602b6151165>

> > 
> > > 
> > > -- 
> > > Stewart Smith
> > > OPAL Architect, IBM.
> > 
> Chris Austen
> POWER Systems Enablement Manager 
> 


[-- Attachment #2: Type: text/html, Size: 3402 bytes --]

Re: OpenBMC Image Management

[Stewart Smith]

Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
>> On Jun 19, 2017, at 12:44 AM, Stewart Smith <stewart@linux.vnet.ibm.com> wrote:
>> 
>> Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
>>>> Why is there mboxbridge separate from phosphor-mboxd?
>>> 
>>> Because the mboxbridge is being used by other companies as a reference
>>> to be able to implement the mailbox daemon in their own BMC firmware
>>> stack for other openpower systems, we didn’t want to “pollute” the
>>> repository with openbmc-specific and c++ implementation that could
>>> confuse them.
>> 
>> Why are there OpenBMC specific things in mboxd?
>> 
>> Why can't the reference implementation be used? What's deficient in it?
>
> For OpenBMC, we’re having the PNOR chip with a filesystem, so mboxd
> would read/write files instead of looking for the data at an offset in
> the chip. Nothing deficient with it, just a different implementation
> that needs to be handled.

Why not have an MBOX protocol that supports partitions/files natively
then? Why not develop this upstream? Surely this is something others
could be interested in.

It seems really backwards having a bunch of things convert to/from FFS
for no reason.

The host already has such abstractions as we have it for FSP systems.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management

[Adriana Kobylak]

[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]


> On Jun 21, 2017, at 12:42 AM, Stewart Smith <stewart@linux.vnet.ibm.com> wrote:
> 
> Adriana Kobylak <anoo@linux.vnet.ibm.com <mailto:anoo@linux.vnet.ibm.com>> writes:
>>> On Jun 19, 2017, at 12:44 AM, Stewart Smith <stewart@linux.vnet.ibm.com> wrote:
>>> 
>>> Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
>>>>> Why is there mboxbridge separate from phosphor-mboxd?
>>>> 
>>>> Because the mboxbridge is being used by other companies as a reference
>>>> to be able to implement the mailbox daemon in their own BMC firmware
>>>> stack for other openpower systems, we didn’t want to “pollute” the
>>>> repository with openbmc-specific and c++ implementation that could
>>>> confuse them.
>>> 
>>> Why are there OpenBMC specific things in mboxd?
>>> 
>>> Why can't the reference implementation be used? What's deficient in it?
>> 
>> For OpenBMC, we’re having the PNOR chip with a filesystem, so mboxd
>> would read/write files instead of looking for the data at an offset in
>> the chip. Nothing deficient with it, just a different implementation
>> that needs to be handled.
> 
> Why not have an MBOX protocol that supports partitions/files natively
> then? Why not develop this upstream? Surely this is something others
> could be interested in.
> 
It’d be available in the openbmc/phosphor-mboxd repo for anybody that wants to use it. Or what do you mean by develop it upstream?
 
> It seems really backwards having a bunch of things convert to/from FFS
> for no reason.

We’re not converting back and forth, if the pnor is blank, we’ll attach ubi to it, and if the pnor is formatted as ffs it’ll be reformatted once to ubi.

> 
> The host already has such abstractions as we have it for FSP systems.
> 
> -- 
> Stewart Smith
> OPAL Architect, IBM.


[-- Attachment #2: Type: text/html, Size: 10789 bytes --]

Re: OpenBMC Image Management

[Stewart Smith]

Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
>> On Jun 21, 2017, at 12:42 AM, Stewart Smith <stewart@linux.vnet.ibm.com> wrote:
>> 
>> Adriana Kobylak <anoo@linux.vnet.ibm.com <mailto:anoo@linux.vnet.ibm.com>> writes:
>>>> On Jun 19, 2017, at 12:44 AM, Stewart Smith <stewart@linux.vnet.ibm.com> wrote:
>>>> 
>>>> Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
>>>>>> Why is there mboxbridge separate from phosphor-mboxd?
>>>>> 
>>>>> Because the mboxbridge is being used by other companies as a reference
>>>>> to be able to implement the mailbox daemon in their own BMC firmware
>>>>> stack for other openpower systems, we didn’t want to “pollute” the
>>>>> repository with openbmc-specific and c++ implementation that could
>>>>> confuse them.
>>>> 
>>>> Why are there OpenBMC specific things in mboxd?
>>>> 
>>>> Why can't the reference implementation be used? What's deficient in it?
>>> 
>>> For OpenBMC, we’re having the PNOR chip with a filesystem, so mboxd
>>> would read/write files instead of looking for the data at an offset in
>>> the chip. Nothing deficient with it, just a different implementation
>>> that needs to be handled.
>> 
>> Why not have an MBOX protocol that supports partitions/files natively
>> then? Why not develop this upstream? Surely this is something others
>> could be interested in.
>> 
> It’d be available in the openbmc/phosphor-mboxd repo for anybody that
> wants to use it. Or what do you mean by develop it upstream?

I mean in https://github.com/openbmc/mboxbridge - the reference
implementation.

>> It seems really backwards having a bunch of things convert to/from FFS
>> for no reason.
>
> We’re not converting back and forth, if the pnor is blank, we’ll
> attach ubi to it, and if the pnor is formatted as ffs it’ll be
> reformatted once to ubi.

I mean that I don't see any patches changing how we build things in
op-build, so there has to be some unpacking of the content going on
there.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management

[Adriana Kobylak]

[-- Attachment #1: Type: text/html, Size: 3106 bytes --]

Re: OpenBMC Image Management - Witherspoon

[Adriana Kobylak]

[-- Attachment #1: Type: text/plain, Size: 1797 bytes --]

This week, the support for virtual pnor on witherspoon (which reformats the pnor chip to ubi) will be turned on.
REST interfaces and copying partition files for patching replace pflash for updating and patching the pnor.

Additional details of the implementation and instructions are documented here:
https://gerrit.openbmc-project.xyz/#/c/4756/ <https://gerrit.openbmc-project.xyz/#/c/4756/>
https://gerrit.openbmc-project.xyz/#/c/5433/ <https://gerrit.openbmc-project.xyz/#/c/5433/>


> On Jun 13, 2017, at 1:10 PM, Adriana Kobylak <anoo@linux.vnet.ibm.com> wrote:
> 
> In the next coming days, the changes to the host bios update procedure will be available for use.
> 
> Notable changes are using a squashfs image that includes individual pnor partition files instead of the ffs (.pnor) image, formatting the pnor chip as ubi to store these files (allowing for the storage of more than one image), and mailbox daemon updates to support these changes (new phosphor-mboxd repo).
> 
> Regarding the mailbox daemon, the mboxbridge repository is staying as a reference implementation and has been cloned into phosphor-mboxd. Any system using mailbox and running with a flashed ffs pnor image (instead of updating to the ubi layout) would still need the reference mboxbridge implementation. The openbmc build is being updated to build phosphor-mboxd as the default.
> 
> A system can be updated transparently from running with an ffs pnor image to the squashfs image via the Software dbus interfaces. In a similar way, a system with a ubi-formatted pnor can be flashed with an ffs pnor image using pflash.
> 
> The first round of documentation updates are here: https://gerrit.openbmc-project.xyz/#/c/4756/ <https://gerrit.openbmc-project.xyz/#/c/4756/>
> Reviewers welcome :)


[-- Attachment #2: Type: text/html, Size: 2896 bytes --]

Re: OpenBMC Image Management - Witherspoon

[Stewart Smith]

Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
> This week, the support for virtual pnor on witherspoon (which reformats the pnor chip to ubi) will be turned on.
> REST interfaces and copying partition files for patching replace pflash for updating and patching the pnor.
>
> Additional details of the implementation and instructions are documented here:
> https://gerrit.openbmc-project.xyz/#/c/4756/ <https://gerrit.openbmc-project.xyz/#/c/4756/>
> https://gerrit.openbmc-project.xyz/#/c/5433/ <https://gerrit.openbmc-project.xyz/#/c/5433/>
>
>
>> On Jun 13, 2017, at 1:10 PM, Adriana Kobylak <anoo@linux.vnet.ibm.com> wrote:
>> 
>> In the next coming days, the changes to the host bios update procedure will be available for use.
>> 
>> Notable changes are using a squashfs image that includes individual pnor partition files instead of the ffs (.pnor) image, formatting the pnor chip as ubi to store these files (allowing for the storage of more than one image), and mailbox daemon updates to support these changes (new phosphor-mboxd repo).
>> 
>> Regarding the mailbox daemon, the mboxbridge repository is staying as a reference implementation and has been cloned into phosphor-mboxd. Any system using mailbox and running with a flashed ffs pnor image (instead of updating to the ubi layout) would still need the reference mboxbridge implementation. The openbmc build is being updated to build phosphor-mboxd as the default.
>> 
>> A system can be updated transparently from running with an ffs pnor image to the squashfs image via the Software dbus interfaces. In a similar way, a system with a ubi-formatted pnor can be flashed with an ffs pnor image using pflash.
>> 
>> The first round of documentation updates are here: https://gerrit.openbmc-project.xyz/#/c/4756/ <https://gerrit.openbmc-project.xyz/#/c/4756/>
>> Reviewers welcome :)

This breaks all our scripts and workflow.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management - Witherspoon

[Andrew Geissler]

This went live tonight in master with commit 0bf11809a80f88
(https://github.com/openbmc/openbmc/commit/0bf11809a80f8829ffc47fd0ea06ffab93199638).
This commit and any after it will now require the new PNOR update
images and process documented by Adriana below.

To make it all work well, first upgrade the BMC to the new level, then
upgrade the PNOR using the new squashfs.tar file (see directions in
doc updates on getting this).  The first time you update to using this
new PNOR image type you need to reboot the BMC after you've done the
bmc and pnor update.  After that, just boot the system up like normal.
PNOR updates now take on the order of 30s vs. 4.5 minutes!  You can
have multiple versions of PNOR loaded, with the ability to choose
which one you want at boot time, and the ability to easily patch any
of the partitions in PNOR via simple filesystem operations.

This robot test can be used directly to do the updates, or as a
reference - https://github.com/openbmc/openbmc-test-automation/blob/master/extended/code_update/code_update.robot

Switching back to older BMC code is pretty painless.  Just flash the
old BMC level then pflash your PNOR like the old days.

On Wed, Aug 2, 2017 at 12:23 PM, Adriana Kobylak
<anoo@linux.vnet.ibm.com> wrote:
> This week, the support for virtual pnor on witherspoon (which reformats the
> pnor chip to ubi) will be turned on.
> REST interfaces and copying partition files for patching replace pflash for
> updating and patching the pnor.
>
> Additional details of the implementation and instructions are documented
> here:
> https://gerrit.openbmc-project.xyz/#/c/4756/
> https://gerrit.openbmc-project.xyz/#/c/5433/
>
>
> On Jun 13, 2017, at 1:10 PM, Adriana Kobylak <anoo@linux.vnet.ibm.com>
> wrote:
>
> In the next coming days, the changes to the host bios update procedure will
> be available for use.
>
> Notable changes are using a squashfs image that includes individual pnor
> partition files instead of the ffs (.pnor) image, formatting the pnor chip
> as ubi to store these files (allowing for the storage of more than one
> image), and mailbox daemon updates to support these changes (new
> phosphor-mboxd repo).
>
> Regarding the mailbox daemon, the mboxbridge repository is staying as a
> reference implementation and has been cloned into phosphor-mboxd. Any system
> using mailbox and running with a flashed ffs pnor image (instead of updating
> to the ubi layout) would still need the reference mboxbridge implementation.
> The openbmc build is being updated to build phosphor-mboxd as the default.
>
> A system can be updated transparently from running with an ffs pnor image to
> the squashfs image via the Software dbus interfaces. In a similar way, a
> system with a ubi-formatted pnor can be flashed with an ffs pnor image using
> pflash.
>
> The first round of documentation updates are here:
> https://gerrit.openbmc-project.xyz/#/c/4756/
> Reviewers welcome :)
>
>

Re: OpenBMC Image Management - Witherspoon

[Adriana Kobylak]

[-- Attachment #1: Type: text/plain, Size: 3055 bytes --]

Today, the support for UBI on BMC will be turned on for the Witherspoon platform. This support (that can be enabled on any system via the machine feature "obmc-ubi-fs”) allows for the storage of the BMC image files in UBI volumes and takes advantage of the 2nd BMC chip on the system to support dual images.

The method for updating the BMC are the same as updating the virtual PNOR where the BMC tarfile is uploaded to the BMC, followed by setting the RequestedActivation property to Active. After the activation is successful, the BMC can be rebooted.

The openbmc-sfw tool located here helps with the querying and updating process: https://github.com/geissonator/openbmc-events <https://github.com/geissonator/openbmc-events>

To move from the existing static layout for the first time, this script can be used (this will reformat the BMC chip clearing all user settings): https://gist.github.com/charleshofer/e753f88deff0c75eda196eb24877fa0c <https://gist.github.com/charleshofer/e753f88deff0c75eda196eb24877fa0c>

I’ll be updating the docs/code-update.md file to reflect this change.



> On Aug 2, 2017, at 12:23 PM, Adriana Kobylak <anoo@linux.vnet.ibm.com> wrote:
> 
> This week, the support for virtual pnor on witherspoon (which reformats the pnor chip to ubi) will be turned on.
> REST interfaces and copying partition files for patching replace pflash for updating and patching the pnor.
> 
> Additional details of the implementation and instructions are documented here:
> https://gerrit.openbmc-project.xyz/#/c/4756/ <https://gerrit.openbmc-project.xyz/#/c/4756/>
> https://gerrit.openbmc-project.xyz/#/c/5433/ <https://gerrit.openbmc-project.xyz/#/c/5433/>
> 
> 
>> On Jun 13, 2017, at 1:10 PM, Adriana Kobylak <anoo@linux.vnet.ibm.com <mailto:anoo@linux.vnet.ibm.com>> wrote:
>> 
>> In the next coming days, the changes to the host bios update procedure will be available for use.
>> 
>> Notable changes are using a squashfs image that includes individual pnor partition files instead of the ffs (.pnor) image, formatting the pnor chip as ubi to store these files (allowing for the storage of more than one image), and mailbox daemon updates to support these changes (new phosphor-mboxd repo).
>> 
>> Regarding the mailbox daemon, the mboxbridge repository is staying as a reference implementation and has been cloned into phosphor-mboxd. Any system using mailbox and running with a flashed ffs pnor image (instead of updating to the ubi layout) would still need the reference mboxbridge implementation. The openbmc build is being updated to build phosphor-mboxd as the default.
>> 
>> A system can be updated transparently from running with an ffs pnor image to the squashfs image via the Software dbus interfaces. In a similar way, a system with a ubi-formatted pnor can be flashed with an ffs pnor image using pflash.
>> 
>> The first round of documentation updates are here: https://gerrit.openbmc-project.xyz/#/c/4756/ <https://gerrit.openbmc-project.xyz/#/c/4756/>
>> Reviewers welcome :)
> 


[-- Attachment #2: Type: text/html, Size: 4853 bytes --]

Re: OpenBMC Image Management

[Anton D. Kachalov]

Hello.

26.01.2017, 01:32, "Adriana Kobylak" <anoo@linux.vnet.ibm.com>:
> General:
> * Use CRAMFS for read-only partitions, and UBIFS for read-write partitions.

Why CRAMFS? It's small, but supports only zlib compression, has a 16MB file size limitation and marked as obsolete in kernel tree. Why not SquashFS? It's better for larger projects, has a number of modern compression algos.

> PNOR:
> * Ability to ‘patch’ by copying a Hostboot image *.bin into a designated directory (/usr/local/ for example).

Would it be good to add support for opkg-based packaging to incrementally (hotfix) system's update? I use it in OpenWRT.

> * Tool to write from PNOR image to BMC format. Implement UBI initially but it could be extended to support different volume managements on other BMCs.
>
> BMC:
> *Save multiple firmware versions, starting with 2, to provide the ability to roll-back if needed. If single BMC chip system, save both versions there. If two BMC chip system, save other version in 2nd chip.
> * Implement various levels of ‘persistency’, such as dev, factory, field. Dev persistency would allow for local patches (/usr/local/ for example) that can be cleared before shipping to customers. Factory mode could delete the location where user data such as network settings resides but preserves the mac address and uuid for example. Etc.
>
> Thanks,
>
> Adriana
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc

-- 
Anton D. Kachalov

Re: OpenBMC Image Management

[Adriana Kobylak]

[-- Attachment #1: Type: text/plain, Size: 2545 bytes --]


> On Jan 26, 2017, at 4:43 AM, Anton D. Kachalov <mouse@yandex-team.ru> wrote:
> 
> Hello.
> 
> 26.01.2017, 01:32, "Adriana Kobylak" <anoo@linux.vnet.ibm.com <mailto:anoo@linux.vnet.ibm.com>>:
>> General:
>> * Use CRAMFS for read-only partitions, and UBIFS for read-write partitions.
> 
> Why CRAMFS? It's small, but supports only zlib compression, has a 16MB file size limitation and marked as obsolete in kernel tree. Why not SquashFS? It's better for larger projects, has a number of modern compression algos.

You’re right Anton, typo mistake, it should be SquashFS, not CRAMFS.

> 
>> PNOR:
>> * Ability to ‘patch’ by copying a Hostboot image *.bin into a designated directory (/usr/local/ for example).
> 
> Would it be good to add support for opkg-based packaging to incrementally (hotfix) system's update? I use it in OpenWRT.

It is a good point, for this we’d need a read-write filesystem to contain the code, and be more formal in updating the revision number in bitbake for the images (currently is set to 0). But since our current use case is about applying temporary (development) patches, not permanent fixes, these changes wouldn’t be a priority at the moment. Also for PNOR images, we don’t own the packaging process (it’s done by the hostboot build).

As additional information, I’ll be sending a more detailed flow of the PNOR update, including how the dbus objects in Patrick’s Software.Version yaml specification would fit, and how to create the virtual PNOR for the host (memory piece) that the mboxd will be interacting with.

> 
>> * Tool to write from PNOR image to BMC format. Implement UBI initially but it could be extended to support different volume managements on other BMCs.
>> 
>> BMC:
>> *Save multiple firmware versions, starting with 2, to provide the ability to roll-back if needed. If single BMC chip system, save both versions there. If two BMC chip system, save other version in 2nd chip.
>> * Implement various levels of ‘persistency’, such as dev, factory, field. Dev persistency would allow for local patches (/usr/local/ for example) that can be cleared before shipping to customers. Factory mode could delete the location where user data such as network settings resides but preserves the mac address and uuid for example. Etc.
>> 
>> Thanks,
>> 
>> Adriana
>> _______________________________________________
>> openbmc mailing list
>> openbmc@lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/openbmc
> 
> -- 
> Anton D. Kachalov


[-- Attachment #2: Type: text/html, Size: 11008 bytes --]

Re: OpenBMC Image Management

[Rick Altherr]

[-- Attachment #1: Type: text/plain, Size: 3066 bytes --]

Applying patches makes for a security problem as well. Verifying integrity
and authenticity of the rootfs requires having trusted hashes. dm-verity
does this at a filesystem level. Patching individual binaries would
invalidate the signatures.

On Jan 26, 2017 7:36 AM, "Adriana Kobylak" <anoo@linux.vnet.ibm.com> wrote:

>
> On Jan 26, 2017, at 4:43 AM, Anton D. Kachalov <mouse@yandex-team.ru>
> wrote:
>
> Hello.
>
> 26.01.2017, 01:32, "Adriana Kobylak" <anoo@linux.vnet.ibm.com>:
>
> General:
> * Use CRAMFS for read-only partitions, and UBIFS for read-write partitions.
>
>
> Why CRAMFS? It's small, but supports only zlib compression, has a 16MB
> file size limitation and marked as obsolete in kernel tree. Why not
> SquashFS? It's better for larger projects, has a number of modern
> compression algos.
>
>
> You’re right Anton, typo mistake, it should be SquashFS, not CRAMFS.
>
>
> PNOR:
> * Ability to ‘patch’ by copying a Hostboot image *.bin into a designated
> directory (/usr/local/ for example).
>
>
> Would it be good to add support for opkg-based packaging to incrementally
> (hotfix) system's update? I use it in OpenWRT.
>
>
> It is a good point, for this we’d need a read-write filesystem to contain
> the code, and be more formal in updating the revision number in bitbake for
> the images (currently is set to 0). But since our current use case is about
> applying temporary (development) patches, not permanent fixes, these
> changes wouldn’t be a priority at the moment. Also for PNOR images, we
> don’t own the packaging process (it’s done by the hostboot build).
>
> As additional information, I’ll be sending a more detailed flow of the
> PNOR update, including how the dbus objects in Patrick’s Software.Version
> yaml specification would fit, and how to create the virtual PNOR for the
> host (memory piece) that the mboxd will be interacting with.
>
>
> * Tool to write from PNOR image to BMC format. Implement UBI initially but
> it could be extended to support different volume managements on other BMCs.
>
> BMC:
> *Save multiple firmware versions, starting with 2, to provide the ability
> to roll-back if needed. If single BMC chip system, save both versions
> there. If two BMC chip system, save other version in 2nd chip.
> * Implement various levels of ‘persistency’, such as dev, factory, field.
> Dev persistency would allow for local patches (/usr/local/ for example)
> that can be cleared before shipping to customers. Factory mode could delete
> the location where user data such as network settings resides but preserves
> the mac address and uuid for example. Etc.
>
> Thanks,
>
> Adriana
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc
>
>
> --
> Anton D. Kachalov
>
>
>
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc
>
>

[-- Attachment #2: Type: text/html, Size: 9461 bytes --]

Re: OpenBMC Image Management

[Stewart Smith]

Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
>>> PNOR:
>>> * Ability to ‘patch’ by copying a Hostboot image *.bin into a designated directory (/usr/local/ for example).
>> 
>> Would it be good to add support for opkg-based packaging to incrementally (hotfix) system's update? I use it in OpenWRT.
>
> It is a good point, for this we’d need a read-write filesystem to
> contain the code, and be more formal in updating the revision number
> in bitbake for the images (currently is set to 0). But since our
> current use case is about applying temporary (development) patches,
> not permanent fixes, these changes wouldn’t be a priority at the
> moment. Also for PNOR images, we don’t own the packaging process (it’s
> done by the hostboot build).

Err... actually by op-build ( https://github.com/open-power/op-build/  )
- and that's *only* for POWER.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management

[Stewart Smith]

"Anton D. Kachalov" <mouse@yandex-team.ru> writes:
>> PNOR:
>> * Ability to ‘patch’ by copying a Hostboot image *.bin into a designated directory (/usr/local/ for example).
>
> Would it be good to add support for opkg-based packaging to
> incrementally (hotfix) system's update? I use it in OpenWRT.

Since discovering dm-verity, I've hoped that one day the BMC would use
it to verify what it's running, and have no executable code on other
partitions. i.e. everything running on BMC and host has been
cryptographically verified.

I imagine that using opkg based packaging or something else similar
would put a spanner in the works of using dm-verity?

-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management

[Stewart Smith]

Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
> BMC:
> *Save multiple firmware versions, starting with 2, to provide the
> ability to roll-back if needed. If single BMC chip system, save both
> versions there. If two BMC chip system, save other version in 2nd
> chip.

two pnor chip or two bmc chips?

with this, is there a mitiigation for downgrade attacks?

> * Implement various levels of ‘persistency’, such as dev, factory,
> field. Dev persistency would allow for local patches (/usr/local/ for
> example) that can be cleared before shipping to customers. Factory
> mode could delete the location where user data such as network
> settings resides but preserves the mac address and uuid for
> example. Etc.

patches to what?

-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management

[Stewart Smith]

Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
> * Tool to write from PNOR image to BMC format. Implement UBI initially
> but it could be extended to support different volume managements on
> other BMCs.

What are these formats? Is this somehow extracting the FFS image back
into files?

-- 
Stewart Smith
OPAL Architect, IBM.

Re: OpenBMC Image Management

[Maxim Sloyko]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

Hi Adriana,

On Wed, Jan 25, 2017 at 2:15 PM, Adriana Kobylak <anoo@linux.vnet.ibm.com>
wrote:

> Hi,
>
> Here is a first draft proposal for image management (code update) in
> OpenBMC for feedback.
>
> General:
> * Move to UBI volume management (on the BMC and PNOR chips), which
> provides dynamic partitioning and wear-leveling. In a limited flash space
> environment there might be the need to re-allocate partition space and
> resizing in a static partitioning implementation can be very painful.
> * Use CRAMFS for read-only partitions, and UBIFS for read-write partitions.
>
> PNOR:
> * Implement a new mboxd based on Cyril’s (see Cyril’s email “Mailbox
> daemon”) that handles dbus objects, generates a virtual PNOR for UBI
> content, etc.
> * Move to memory-based access instead of the current LPC-to-AHD path.
> * Allocate just enough space in the initial window for Hostboot’s TOC,
> HBB, and HBI partitions, grow as required.
> * Ability to ‘patch’ by copying a Hostboot image *.bin into a designated
> directory (/usr/local/ for example).
> * Tool to write from PNOR image to BMC format. Implement UBI initially but
> it could be extended to support different volume managements on other BMCs.
>
> BMC:
> *Save multiple firmware versions, starting with 2, to provide the ability
> to roll-back if needed. If single BMC chip system, save both versions
> there. If two BMC chip system, save other version in 2nd chip.
> * Implement various levels of ‘persistency’, such as dev, factory, field.
> Dev persistency would allow for local patches (/usr/local/ for example)
> that can be cleared before shipping to customers. Factory mode could delete
> the location where user data such as network settings resides but preserves
> the mac address and uuid for example. Etc.
>

I'm interested in roll-back feature, do you have something working already
or is this still under construction?

Sorry, I could not find anything in the docs or code about it.

Thanks


>
> Thanks,
>
> Adriana
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc
>



-- 
*M*axim *S*loyko

[-- Attachment #2: Type: text/html, Size: 3003 bytes --]

Re: OpenBMC Image Management

[Adriana Kobylak]

[-- Attachment #1: Type: text/plain, Size: 2806 bytes --]


> On Jun 16, 2017, at 5:18 PM, Maxim Sloyko <maxims@google.com> wrote:
> 
> Hi Adriana,
> 
> On Wed, Jan 25, 2017 at 2:15 PM, Adriana Kobylak <anoo@linux.vnet.ibm.com <mailto:anoo@linux.vnet.ibm.com>> wrote:
> Hi,
> 
> Here is a first draft proposal for image management (code update) in OpenBMC for feedback.
> 
> General:
> * Move to UBI volume management (on the BMC and PNOR chips), which provides dynamic partitioning and wear-leveling. In a limited flash space environment there might be the need to re-allocate partition space and resizing in a static partitioning implementation can be very painful.
> * Use CRAMFS for read-only partitions, and UBIFS for read-write partitions.
> 
> PNOR:
> * Implement a new mboxd based on Cyril’s (see Cyril’s email “Mailbox daemon”) that handles dbus objects, generates a virtual PNOR for UBI content, etc.
> * Move to memory-based access instead of the current LPC-to-AHD path.
> * Allocate just enough space in the initial window for Hostboot’s TOC, HBB, and HBI partitions, grow as required.
> * Ability to ‘patch’ by copying a Hostboot image *.bin into a designated directory (/usr/local/ for example).
> * Tool to write from PNOR image to BMC format. Implement UBI initially but it could be extended to support different volume managements on other BMCs.
> 
> BMC:
> *Save multiple firmware versions, starting with 2, to provide the ability to roll-back if needed. If single BMC chip system, save both versions there. If two BMC chip system, save other version in 2nd chip.
> * Implement various levels of ‘persistency’, such as dev, factory, field. Dev persistency would allow for local patches (/usr/local/ for example) that can be cleared before shipping to customers. Factory mode could delete the location where user data such as network settings resides but preserves the mac address and uuid for example. Etc.
> 
> I'm interested in roll-back feature, do you have something working already or is this still under construction?

Hi Maxim, the roll-back feature for the host bios is under review: https://gerrit.openbmc-project.xyz/#/c/4675/ <https://gerrit.openbmc-project.xyz/#/c/4675/>
In summary, the user would set the RedundancyPriority value to 0 for a previously activated image to make it active.
> 
> Sorry, I could not find anything in the docs or code about it.

I’ll add the RedundancyPriority details to my doc update commit: https://gerrit.openbmc-project.xyz/#/c/4756/
> 
> Thanks 
>  
> 
> Thanks,
> 
> Adriana
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org <mailto:openbmc@lists.ozlabs.org>
> https://lists.ozlabs.org/listinfo/openbmc <https://lists.ozlabs.org/listinfo/openbmc>
> 
> 
> 
> -- 
> Maxim Sloyko


[-- Attachment #2: Type: text/html, Size: 5984 bytes --]

Re: OpenBMC Image Management

[Adriana Kobylak]

[-- Attachment #1: Type: text/plain, Size: 3065 bytes --]


> On Jun 16, 2017, at 5:18 PM, Maxim Sloyko <maxims@google.com> wrote:
> 
> Hi Adriana,
> 
> On Wed, Jan 25, 2017 at 2:15 PM, Adriana Kobylak <anoo@linux.vnet.ibm.com <mailto:anoo@linux.vnet.ibm.com>> wrote:
> Hi,
> 
> Here is a first draft proposal for image management (code update) in OpenBMC for feedback.
> 
> General:
> * Move to UBI volume management (on the BMC and PNOR chips), which provides dynamic partitioning and wear-leveling. In a limited flash space environment there might be the need to re-allocate partition space and resizing in a static partitioning implementation can be very painful.
> * Use CRAMFS for read-only partitions, and UBIFS for read-write partitions.
> 
> PNOR:
> * Implement a new mboxd based on Cyril’s (see Cyril’s email “Mailbox daemon”) that handles dbus objects, generates a virtual PNOR for UBI content, etc.
> * Move to memory-based access instead of the current LPC-to-AHD path.
> * Allocate just enough space in the initial window for Hostboot’s TOC, HBB, and HBI partitions, grow as required.
> * Ability to ‘patch’ by copying a Hostboot image *.bin into a designated directory (/usr/local/ for example).
> * Tool to write from PNOR image to BMC format. Implement UBI initially but it could be extended to support different volume managements on other BMCs.
> 
> BMC:
> *Save multiple firmware versions, starting with 2, to provide the ability to roll-back if needed. If single BMC chip system, save both versions there. If two BMC chip system, save other version in 2nd chip.
> * Implement various levels of ‘persistency’, such as dev, factory, field. Dev persistency would allow for local patches (/usr/local/ for example) that can be cleared before shipping to customers. Factory mode could delete the location where user data such as network settings resides but preserves the mac address and uuid for example. Etc.
> 
> I'm interested in roll-back feature, do you have something working already or is this still under construction?
> 
> Sorry, I could not find anything in the docs or code about it.

For the roll-back feature, the user would use the RedundancyPriority interface (https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/RedundancyPriority.interface.yaml
), to indicate which version they want to re-activate. For example, the version1 is activated, the priority for it is 0, then version 2 is activated which sets its priority to 0 and version 1 is set to priority 1. The user can set the priority of version 1 to 0 to indicate that version 1 should be activated thus triggering a roll back. This takes effect on power on for pnor version, and on bmc reboots (still been worked) for bmc versions.
> 
> Thanks 
>  
> 
> Thanks,
> 
> Adriana
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org <mailto:openbmc@lists.ozlabs.org>
> https://lists.ozlabs.org/listinfo/openbmc <https://lists.ozlabs.org/listinfo/openbmc>
> 
> 
> 
> -- 
> Maxim Sloyko


[-- Attachment #2: Type: text/html, Size: 5999 bytes --]

Re: OpenBMC Image Management

[Benjamin Herrenschmidt]

On Wed, 2017-08-02 at 12:15 -0500, Adriana Kobylak wrote:
> General:
> * Move to UBI volume management (on the BMC and PNOR chips), which
> provides dynamic partitioning and wear-leveling. In a limited flash
> space environment there might be the need to re-allocate partition
> space and resizing in a static partitioning implementation can be
> very painful.
> * Use CRAMFS for read-only partitions, and UBIFS for read-write
> partitions.

I still don't understand what is the point of using UBI on a 32M or 64M
NOR flash. Other than wasting space and time.

Ben.

Re: OpenBMC Image Management

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2058 bytes --]

On Thu, Aug 03, 2017 at 10:14:55AM +1000, Benjamin Herrenschmidt wrote:
> On Wed, 2017-08-02 at 12:15 -0500, Adriana Kobylak wrote:
> > General:
> > * Move to UBI volume management (on the BMC and PNOR chips), which
> > provides dynamic partitioning and wear-leveling. In a limited flash
> > space environment there might be the need to re-allocate partition
> > space and resizing in a static partitioning implementation can be
> > very painful.
> > * Use CRAMFS for read-only partitions, and UBIFS for read-write
> > partitions.
> 
> I still don't understand what is the point of using UBI on a 32M or 64M
> NOR flash. Other than wasting space and time.

Ben,

"Still don't"?  You've never actually asked any questions even though
this was first proposed publicly on the mailing list on Jan 26th:

    https://lists.ozlabs.org/pipermail/openbmc/2017-January/006298.html

No one questioned UBI in that original thread, or really since then,
until now.  Kernel changes were proposed and merged back in early Feb
and again there was no discussion or questions:

    https://lists.ozlabs.org/pipermail/openbmc/2017-February/006457.html

But ok, I'll bite.  There are two main reasons we went this path:

    *  UBIFS is, to the best of my knowledge, a better file system than
       JFFS2 in pretty much every metric.

    *  Dynamic volumes are easier to deal with than static volumes,
       which Adriana spelled out above.

Is your real question why do we have a file system instead of a raw FFS
image?  That could have been implemented with static volumes instead of
UBI, and someone still could go implement code to do that instead if
they are interested.  A few advantages of a file system over raw FFS are:

    * Measurably faster boot and update performance.
    * Ability to support a true "factory reset" operation, unlike the
      competitive BMC implementations of raw FFS.
    * Easier and more obvious removal of development patches from a
      machine.

> 
> Ben.
> 

-- 
Patrick Williams

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OpenBMC Image Management

[Benjamin Herrenschmidt]

On Thu, 2017-08-03 at 00:55 -0500, Patrick Williams wrote:
> But ok, I'll bite.  There are two main reasons we went this path:
> 
>     *  UBIFS is, to the best of my knowledge, a better file system than
>        JFFS2 in pretty much every metric.
> 
>     *  Dynamic volumes are easier to deal with than static volumes,
>        which Adriana spelled out above.
> 
> Is your real question why do we have a file system instead of a raw FFS
> image? 

No that wasn't my real question. I was just wondering why ubifs
compared to jffs2, I was under the impression that ubifs was designed
for large NAND flashes.

I might be wrong, mind you, but I remember reading that it had
significant overhead for a small flash.

Cheers,
Ben.

Re: OpenBMC Image Management

[Milton Miller II]

On 08/03/2017 around 1:00am in some time zone, Benjamin Herrenschmidt wrote:
>On Thu, 2017-08-03 at 00:55 -0500, Patrick Williams wrote:
>> But ok, I'll bite.  There are two main reasons we went this path:
>> 
>>     *  UBIFS is, to the best of my knowledge, a better file system
>than
>>        JFFS2 in pretty much every metric.
>> 
>>     *  Dynamic volumes are easier to deal with than static volumes,
>>        which Adriana spelled out above.
>> 
>> Is your real question why do we have a file system instead of a raw
>FFS
>> image? 
>
>No that wasn't my real question. I was just wondering why ubifs
>compared to jffs2, I was under the impression that ubifs was designed
>for large NAND flashes.
>
>I might be wrong, mind you, but I remember reading that it had
>significant overhead for a small flash.


There was some discussion about the OLPC decision to not use UBI on 
their flash because of space.

UBI requires two write programming units to hold two units of metadata 
for each erase block.  One block holds the wear leveling counter and is 
written immediately after erase, and the second holds the assigned volume 
and logical erase block theirin.  On NAND, each of these are a write 
program unit and can be upuwards of 256 bytes, and erase units might be 
a few kbytes.  On NOR, which we are using, these are each assigned a 
minimum size of 64 bytes.  Also ubi enforces each erase block must 
be at least 16kB, this means we have to use 64kB sectors not 4kB.

Thus the overhead for ubi is 128 bytes per 65536 bytes, and in return 
we get volume management and wear leveling across the whole device.

We have to use 64k blocks vs 4k but can share 1 floating block across
all volumes in the device.

I have not analyzed the efficency of jffs2 vs ubifs for storage and 
journaling.   jffs2 mantains all inodes in ram, it appears that ubifs
presently does but would not be required to.  Both have shutdown fast
mount support and both have some degree of support for compressed file
fragments.

jffs2 only supports synchronous updates to the flash, ubifs has a mount 
option to do the same but can do async writeback to the flash.  This may 
expose our applications to needing to add fsync code.

milton