Packaging and deploying multiple firmware image types in one

Packaging and deploying multiple firmware image types in one

[Adriana Kobylak]

There was some discussion in the "PSU firmware update" proposal[1] about 
bundling firmware images together such as the BMC and PSU firmware. The 
Software interface defines these bundles as a "System" version[2].

At IBM, we're starting to create a "System" bundle of BMC + host 
firmware by adding a recipe that would add the host's firmware files to 
the BMC's squashfs[3].

One of the pending items is determining how to showcase the version of 
these System images in Redfish. We can assume that the version of a 
System image corresponds to the BMC's "FirmwareVersion", but it's 
unknown what else the bundle contains (host fw, PSU fw, PCIe fw,..), in 
order to populate the other version fields such as "BiosVersion". 
There's a D-Bus "Extended Version" property[4] that could be leveraged 
to add the additional image information.

Feel free to share if you have any thoughts on this item or any aspect 
of these "System" images.


---
[1] https://lists.ozlabs.org/pipermail/openbmc/2019-June/016573.html
[2] 
https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/Version.interface.yaml
[3] 
https://github.com/openbmc/meta-openpower/blob/master/recipes-phosphor/flash/host-fw_git.bb
[4] 
https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/ExtendedVersion.interface.yaml

Re: Packaging and deploying multiple firmware image types in one

[Alexander A. Filippov]

On Mon, Nov 11, 2019 at 01:28:11PM -0600, Adriana Kobylak wrote:
> There was some discussion in the "PSU firmware update" proposal[1] about
> bundling firmware images together such as the BMC and PSU firmware. The
> Software interface defines these bundles as a "System" version[2].
> 
> At IBM, we're starting to create a "System" bundle of BMC + host firmware by
> adding a recipe that would add the host's firmware files to the BMC's
> squashfs[3].
> 
> One of the pending items is determining how to showcase the version of these
> System images in Redfish. We can assume that the version of a System image
> corresponds to the BMC's "FirmwareVersion", but it's unknown what else the
> bundle contains (host fw, PSU fw, PCIe fw,..), in order to populate the
> other version fields such as "BiosVersion". There's a D-Bus "Extended
> Version" property[4] that could be leveraged to add the additional image
> information.
> 
> Feel free to share if you have any thoughts on this item or any aspect of
> these "System" images.
> 

We use the system bundle of BMC + Host firmware on our VESNIN hardware.
There are some things which cause discomfort a little bit:
- The uploaded system bundle isn't shown in the WebUI.
- The system bundle has only one version field which is common for BMC and Host
  firmwares.
- After rebooting BMC, which is required to complete update the BMC firmware
  the system bundle turns to two separated instances in D-Bus which has its own
  real versions.

Thus, I thought about putting the separate manifests for each part of the
bundle.

> 
> ---
> [1] https://lists.ozlabs.org/pipermail/openbmc/2019-June/016573.html
> [2] https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/Version.interface.yaml
> [3] https://github.com/openbmc/meta-openpower/blob/master/recipes-phosphor/flash/host-fw_git.bb
> [4] https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/ExtendedVersion.interface.yaml

Re: Packaging and deploying multiple firmware image types in one

[Gunnar Mills]

On 11/12/2019 1:49 AM, Alexander A. Filippov wrote:
> On Mon, Nov 11, 2019 at 01:28:11PM -0600, Adriana Kobylak wrote:
>
>
> At IBM, we're starting to create a "System" bundle of BMC + host firmware by
> adding a recipe that would add the host's firmware files to the BMC's
> squashfs[3].
>
>
> Feel free to share if you have any thoughts on this item or any aspect of
> these "System" images.
>
> We use the system bundle of BMC + Host firmware on our VESNIN hardware.
> There are some things which cause discomfort a little bit:
> - The uploaded system bundle isn't shown in the WebUI.

IBM is planning on supporting a bundled "System" firmware form the WebUI.
The design is here:
https://ibm.invisionapp.com/share/4XNZ0JAMJ7B#/screens/319215228

>> [3] https://github.com/openbmc/meta-openpower/blob/master/recipes-phosphor/flash/host-fw_git.bb
>>

Re: Packaging and deploying multiple firmware image types in one

[Adriana Kobylak]

On 2019-11-12 01:49, Alexander A. Filippov wrote:
> On Mon, Nov 11, 2019 at 01:28:11PM -0600, Adriana Kobylak wrote:
> 
> We use the system bundle of BMC + Host firmware on our VESNIN hardware.

How are you currently building the bundled image, do you include the 
Host firmware in the BMC rootfs, or do you have separate image files 
(bmc fw file, host fw file) in a single tarball?

> There are some things which cause discomfort a little bit:
> - The uploaded system bundle isn't shown in the WebUI.
> - The system bundle has only one version field which is common for BMC 
> and Host
>   firmwares.

Do you think the ExtendedVersion d-bus property could help in this case? 
For example adding to the manifest "extended_version=host-v1.2."

> - After rebooting BMC, which is required to complete update the BMC 
> firmware
>   the system bundle turns to two separated instances in D-Bus which has 
> its own
>   real versions.

Yeah, the purpose is not currently preserved across reboots. I have a 
change here for that: 
https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-bmc-code-mgmt/+/27045

> 
> Thus, I thought about putting the separate manifests for each part of 
> the
> bundle.

If you go the route of adding a second manifest, would you have them in 
a separate tarball (bmc image + manifest) and (host image + manifest) 
then put those tarballs inside a tarball? since the manifest file name 
would be the same.

> 
>> 
>> ---
>> [1] https://lists.ozlabs.org/pipermail/openbmc/2019-June/016573.html
>> [2] 
>> https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/Version.interface.yaml
>> [3] 
>> https://github.com/openbmc/meta-openpower/blob/master/recipes-phosphor/flash/host-fw_git.bb
>> [4] 
>> https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/ExtendedVersion.interface.yaml

Re: Packaging and deploying multiple firmware image types in one

[Adriana Kobylak]

On 2019-11-12 15:38, Gunnar Mills wrote:
> On 11/12/2019 1:49 AM, Alexander A. Filippov wrote:
>> On Mon, Nov 11, 2019 at 01:28:11PM -0600, Adriana Kobylak wrote:
>> 
>> We use the system bundle of BMC + Host firmware on our VESNIN 
>> hardware.
>> There are some things which cause discomfort a little bit:
>> - The uploaded system bundle isn't shown in the WebUI.
> 
> IBM is planning on supporting a bundled "System" firmware form the 
> WebUI.
> The design is here:
> https://ibm.invisionapp.com/share/4XNZ0JAMJ7B#/screens/319215228
> 

Maybe in the mean time we can have the WebUI display the "System" 
versions alongside the "BMC" ones. I'll check if it's as simple as "if 
purpose is BMC or System" do what it currently does.

>>> [3] 
>>> https://github.com/openbmc/meta-openpower/blob/master/recipes-phosphor/flash/host-fw_git.bb
>>> 

Re: Packaging and deploying multiple firmware image types in one

[Alexander A. Filippov]

On Tue, Nov 12, 2019 at 04:54:22PM -0600, Adriana Kobylak wrote:
> On 2019-11-12 01:49, Alexander A. Filippov wrote:
> > On Mon, Nov 11, 2019 at 01:28:11PM -0600, Adriana Kobylak wrote:
> > 
> > We use the system bundle of BMC + Host firmware on our VESNIN hardware.
> 
> How are you currently building the bundled image, do you include the Host
> firmware in the BMC rootfs, or do you have separate image files (bmc fw
> file, host fw file) in a single tarball?

We put separate image files in a single tarball.

> 
> > There are some things which cause discomfort a little bit:
> > - The uploaded system bundle isn't shown in the WebUI.
> > - The system bundle has only one version field which is common for BMC
> > and Host
> >   firmwares.
> 
> Do you think the ExtendedVersion d-bus property could help in this case? For
> example adding to the manifest "extended_version=host-v1.2."

I think, it is right only if the system bundle will be processed by its own
service, which creates corresponding objects for each part of the bundle with
their own hash, version's data and purpose.

E.g.: as you've proposed below with separate tarballs in one common tarball.

> 
> > - After rebooting BMC, which is required to complete update the BMC
> > firmware
> >   the system bundle turns to two separated instances in D-Bus which has
> > its own
> >   real versions.
> 
> Yeah, the purpose is not currently preserved across reboots. I have a change
> here for that:
> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-bmc-code-mgmt/+/27045
> 

We use static filesystem layout on our VESNIN hardware. 
So, the version of active BMC is read from /etc/os-release and always has the
BMC purpose. 
The same way with the host firmware: it is read directly from the flash, which
has a special partition with version's data and always has the Host purpose.

> > 
> > Thus, I thought about putting the separate manifests for each part of
> > the
> > bundle.
> 
> If you go the route of adding a second manifest, would you have them in a
> separate tarball (bmc image + manifest) and (host image + manifest) then put
> those tarballs inside a tarball? since the manifest file name would be the
> same.
>

Yeah, it's possible.
But when I researched that, I had found that the phosphor-image-updater and the
openpower-update-manager both already have support for the system bundle. I
decided, we shouldn't create any other implementation, but follow the upstream's
way. So, it has led to the state I described earlier.

> > 
> > > 
> > > ---
> > > [1] https://lists.ozlabs.org/pipermail/openbmc/2019-June/016573.html
> > > [2] https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/Version.interface.yaml
> > > [3] https://github.com/openbmc/meta-openpower/blob/master/recipes-phosphor/flash/host-fw_git.bb
> > > [4] https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/ExtendedVersion.interface.yaml

Re: Packaging and deploying multiple firmware image types in one

[Alexander A. Filippov]

On Tue, Nov 12, 2019 at 04:54:22PM -0600, Adriana Kobylak wrote:
> 
> Yeah, the purpose is not currently preserved across reboots. I have a change
> here for that:
> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-bmc-code-mgmt/+/27045
>

On Thu, Nov 14, 2019 at 03:14:41AM +0000, Adriana Kobylak (Code Review) wrote:
> ...
> but let's continue on the mailing list about your thoughts on how you think
> the tarball of tarballs should be handled.
>

Ok, here are my thoughts:
The phosphor-version-software-manager might put all internal tarballs in the
/tmp/images folder during processing the top level tarball. That will lead to
creation of corresponding D-Bus objects. Each of them will have their own
purpose, version, object path and so on.

The root D-Bus object and their folder might be removed after that  to reduce a
used file system space.

Re: Packaging and deploying multiple firmware image types in one

[Adriana Kobylak]

On 2019-11-14 01:51, Alexander A. Filippov wrote:
> On Tue, Nov 12, 2019 at 04:54:22PM -0600, Adriana Kobylak wrote:
>> 
>> Yeah, the purpose is not currently preserved across reboots. I have a 
>> change
>> here for that:
>> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-bmc-code-mgmt/+/27045
>> 
> 
> On Thu, Nov 14, 2019 at 03:14:41AM +0000, Adriana Kobylak (Code Review) 
> wrote:
>> ...
>> but let's continue on the mailing list about your thoughts on how you 
>> think
>> the tarball of tarballs should be handled.
>> 
> 
> Ok, here are my thoughts:
> The phosphor-version-software-manager might put all internal tarballs 
> in the
> /tmp/images folder during processing the top level tarball. That will 
> lead to
> creation of corresponding D-Bus objects. Each of them will have their 
> own
> purpose, version, object path and so on.

Yeah agree. We may still need some way to let the 
phosphor-version-software-manager
know that it needs to untar the internal tarballs, maybe a very simple 
MANIFEST
with a new field, then each individual tarball would have their own 
MANIFEST that
creates the D-Bus versions like you mentioned.

> 
> The root D-Bus object and their folder might be removed after that  to 
> reduce a
> used file system space.

Re: Packaging and deploying multiple firmware image types in one

[Adriana Kobylak]

>> Ok, here are my thoughts:
>> The phosphor-version-software-manager might put all internal tarballs 
>> in the
>> /tmp/images folder during processing the top level tarball. That will 
>> lead to
>> creation of corresponding D-Bus objects. Each of them will have their 
>> own
>> purpose, version, object path and so on.
> 
> Yeah agree. We may still need some way to let the
> phosphor-version-software-manager
> know that it needs to untar the internal tarballs, maybe a very simple 
> MANIFEST
> with a new field, then each individual tarball would have their own
> MANIFEST that
> creates the D-Bus versions like you mentioned.
> 

For the bmcweb change to support the System purpose 
(https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/27144), Andrew 
had the comment about if we could have the software manager determine 
the .BMC, .Host, .PSU details out of a System image and populate the 
individual version interfaces, which further confirms that the current 
Version D-Bus interface is insufficient to detail the individual 
components of a "System" firmware image, and that workarounds like 
adding individual Manifest files that Alexander proposed are needed.

Therefore, wanted to see what people thought about expanding the 
software D-Bus interface to add a 2nd version id to the path of a System 
version, which would contain the version information for each 
sub-element, but would not contain an Activation object:

--
Current, leave as is for individual firmware images:

/xyz/openbmc_project/software/<bmc-version-id>/
   - Activation: Ready
   - Purpose: .BMC
   - Version: 2.8.0

/xyz/openbmc_project/software/<host-version-id>/
   - Activation: Ready
   - Purpose: .Host
   - Version: 1.2.3

--
New expanded option for bundled images:

/xyz/openbmc_project/software/<system-version-id>/
   - Activation: Ready
   - Purpose: .System
   - Version: CompanyZ-v1.0
   /xyz/openbmc_project/software/<sysrem-version-id>/<bmc-version-id>/
     - Purpose: .BMC
     - Version: 2.8.0
   /xyz/openbmc_project/software/<sysrem-version-id>/<host-version-id>/
     - Purpose: .Host
     - Version: 1.2.3

Re: Packaging and deploying multiple firmware image types in one

[Andrew Geissler]

On Fri, Nov 15, 2019 at 8:50 AM Adriana Kobylak <anoo@linux.ibm.com> wrote:

>
> For the bmcweb change to support the System purpose
> (https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/27144), Andrew
> had the comment about if we could have the software manager determine
> the .BMC, .Host, .PSU details out of a System image and populate the
> individual version interfaces, which further confirms that the current
> Version D-Bus interface is insufficient to detail the individual
> components of a "System" firmware image, and that workarounds like
> adding individual Manifest files that Alexander proposed are needed.
>
> Therefore, wanted to see what people thought about expanding the
> software D-Bus interface to add a 2nd version id to the path of a System
> version, which would contain the version information for each
> sub-element, but would not contain an Activation object:
>
> --
> Current, leave as is for individual firmware images:
>
> /xyz/openbmc_project/software/<bmc-version-id>/
>    - Activation: Ready
>    - Purpose: .BMC
>    - Version: 2.8.0
>
> /xyz/openbmc_project/software/<host-version-id>/
>    - Activation: Ready
>    - Purpose: .Host
>    - Version: 1.2.3
>
> --
> New expanded option for bundled images:
>
> /xyz/openbmc_project/software/<system-version-id>/
>    - Activation: Ready
>    - Purpose: .System
>    - Version: CompanyZ-v1.0
>    /xyz/openbmc_project/software/<sysrem-version-id>/<bmc-version-id>/
>      - Purpose: .BMC
>      - Version: 2.8.0
>    /xyz/openbmc_project/software/<sysrem-version-id>/<host-version-id>/
>      - Purpose: .Host
>      - Version: 1.2.3

Makes sense to me. The phosphor-webui code could look for a system object
and if found, only allow the user to update/switch images based on system
level. They could still easily show the BMC, host, PSU levels though.
Or maybe this will just be a config option for the GUI? Either way I
think the GUI would want to show both the system and individual levels
of firmware.

Redfish code in bmcweb can continue to work as-is for bmc, host, psu
firmware inventory. I think we should see if we can get the DMTF to
add a FirmwareVersion property to the ComputerSystem object where
we could store this new system version.

Re: Packaging and deploying multiple firmware image types in one

[Adriana Kobylak]

On 2019-11-15 09:46, Andrew Geissler wrote:
> On Fri, Nov 15, 2019 at 8:50 AM Adriana Kobylak <anoo@linux.ibm.com> 
> wrote:
> 
>> New expanded option for bundled images:
>> 
>> /xyz/openbmc_project/software/<system-version-id>/
>>    - Activation: Ready
>>    - Purpose: .System
>>    - Version: CompanyZ-v1.0
>>    /xyz/openbmc_project/software/<sysrem-version-id>/<bmc-version-id>/
>>      - Purpose: .BMC
>>      - Version: 2.8.0
>>    
>> /xyz/openbmc_project/software/<sysrem-version-id>/<host-version-id>/
>>      - Purpose: .Host
>>      - Version: 1.2.3

Ended up adding an array instead for the proposal, to avoid confusion 
with multiple version ids:

https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-dbus-interfaces/+/27395

       "Purpose": 
"xyz.openbmc_project.Software.Version.VersionPurpose.System",
       "Version": "2.8.0-dev-882-gf0cbf42",
       "VersionAggregate": [
         [
           "xyz.openbmc_project.Software.Version.VersionPurpose.BMC",
           "2.8.0-dev",
           ""
         ],
         [
           "xyz.openbmc_project.Software.Version.VersionPurpose.Host",
           "open-power-witherspoon-v2.2",
           
"buildroot-2018.11.1-7-g5d7cc8c,skiboot-v6.2,hostboot-17ba81e-pb264a73,occ-12c8088"
         ]
       ]

> 
> Makes sense to me. The phosphor-webui code could look for a system 
> object
> and if found, only allow the user to update/switch images based on 
> system
> level. They could still easily show the BMC, host, PSU levels though.
> Or maybe this will just be a config option for the GUI? Either way I
> think the GUI would want to show both the system and individual levels
> of firmware.

Yeah, it'd make sense to also show the individual levels to provide the 
user with more information about the image.

> 
> Redfish code in bmcweb can continue to work as-is for bmc, host, psu
> firmware inventory.

Yeah, I'd just need to make a change in bmcweb to get the version string 
from that new Aggregate interface to populate the BMC FirmwareVersion 
field.

> I think we should see if we can get the DMTF to
> add a FirmwareVersion property to the ComputerSystem object where
> we could store this new system version.

Agree.