BMC Image Signing Proposal

BMC Image Signing Proposal

[anoo]

Hi all,

During the hackaton meetup, we touched on BMC image verification and 
signing and concluded that the community would like to see two 
verification steps, one prior to writing the image to flash (via digital 
signature verification), and another one by checking FIT in U-Boot prior 
to booting from the new image.

The proposal would be to implement the digital signature verification 
first.

At a high level, during the build:
* A SHA-256 hash would be calculated over tbd files that make up the 
firmware image.
* The hash would be signed by a private key that's part of the 
repository (community key). Companies could overwrite it with their own 
private key when building production images.
* The encrypted hash (digital signature) and corresponding public key 
would be added to the firmware image.
* Yocto may already provide a way to sign images and generate keys.

On the BMC:
* The hash would be calculated on the image files that were uploaded to 
the BMC.
* The signature would be decrypted using an existing public key on the 
BMC (this validates the new public key delivered with the image is also 
valid).
* Accept image if both values are the same.

Any thoughts or comments?

Re: BMC Image Signing Proposal

[Alexander Amelkin]

Hi, Anoo!

The thoughts are as follows:

1. BMC usually runs in a secured environment where probability of 
tampering with flash IC contents by means other than BMC's firmware 
itself is negligible.

2. U-Boot already performs image checksum validation before booting a 
FIT image

3. User input really needs validation, at least to make the system 
fool-proof

Having said that, I suggest that the only thing that really needs doing 
is signing (and checking) of the overall firmware image file that is 
supplied by the user (admin) during the firmware upgrade procedure. 
Applying asymmetric cryptography to a digest hash looks to me like a 
good idea as it indeed allows for verifying the supplier of the firmware 
image.

Regards,
Alexander Amelkin
YADRO

26.01.2018 00:15, anoo wrote:
> Hi all,
>
> During the hackaton meetup, we touched on BMC image verification and 
> signing and concluded that the community would like to see two 
> verification steps, one prior to writing the image to flash (via 
> digital signature verification), and another one by checking FIT in 
> U-Boot prior to booting from the new image.
>
> The proposal would be to implement the digital signature verification 
> first.
>
> At a high level, during the build:
> * A SHA-256 hash would be calculated over tbd files that make up the 
> firmware image.
> * The hash would be signed by a private key that's part of the 
> repository (community key). Companies could overwrite it with their 
> own private key when building production images.
> * The encrypted hash (digital signature) and corresponding public key 
> would be added to the firmware image.
> * Yocto may already provide a way to sign images and generate keys.
>
> On the BMC:
> * The hash would be calculated on the image files that were uploaded 
> to the BMC.
> * The signature would be decrypted using an existing public key on the 
> BMC (this validates the new public key delivered with the image is 
> also valid).
> * Accept image if both values are the same.
>
> Any thoughts or comments?
>

Re: BMC Image Signing Proposal

[Andrew Jeffery]

[-- Attachment #1: Type: text/plain, Size: 1966 bytes --]

On Thu, 2018-01-25 at 15:15 -0600, anoo wrote:
> Hi all,
> 
> During the hackaton meetup, we touched on BMC image verification and 
> signing and concluded that the community would like to see two 
> verification steps, one prior to writing the image to flash (via digital 
> signature verification), and another one by checking FIT in U-Boot prior 
> to booting from the new image.
> 
> The proposal would be to implement the digital signature verification 
> first.
> 
> At a high level, during the build:
> * A SHA-256 hash would be calculated over tbd files that make up the 
> firmware image.
> * The hash would be signed by a private key that's part of the 
> repository (community key). Companies could overwrite it with their own 
> private key when building production images.
> * The encrypted hash (digital signature) and corresponding public key 
> would be added to the firmware image.
> * Yocto may already provide a way to sign images and generate keys.

I'm starting to look into signing of artefacts as part of the build
process. Some quick searching suggests Yocto doesn't have support for
it out of the box, but there is the swupdate project[1] and the
corresponding meta-swupdate[2] Yocto layer that sound interesting.
However, it may require a rework of how we manage image updates if we
were to make use of it. I'll keep poking around, but if anyone has any
other pointers please let me know.

[1] https://github.com/sbabic/swupdate
[2] https://github.com/sbabic/meta-swupdate

> 
> On the BMC:
> * The hash would be calculated on the image files that were uploaded to 
> the BMC.
> * The signature would be decrypted using an existing public key on the 
> BMC (this validates the new public key delivered with the image is also 
> valid).
> * Accept image if both values are the same.
> 
> Any thoughts or comments?

Can you take a look at swupdate and reply with your thoughts?

Cheers,

Andrew

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: BMC Image Signing Proposal

[Andrew Jeffery]

[-- Attachment #1: Type: text/plain, Size: 2258 bytes --]

Hi Alexander,

On Fri, 2018-01-26 at 14:07 +0300, Alexander Amelkin wrote:
> Hi, Anoo!
> 
> The thoughts are as follows:
> 
> 1. BMC usually runs in a secured environment where probability of 
> tampering with flash IC contents by means other than BMC's firmware 
> itself is negligible.

This does bring up the issue of developing a threat model to develop
against; we should probably do that. However, without one, I feel like
we should design *for* defense in depth and allow people to remove
protections as they see fit for their environment, rather than make
potentially compromising assumptions about what that environment is at
the outset.

For instance, whilst BMCs might typically be isolated from customer
traffic on a separate LAN, there's still the in-band interface which
can be used to poke at the BMC. Vulnerabilities in the IPMI stack could
lead to BMC compromise and undesirable flash writes*. Therefore BMC
flash integrity remains a valid concern despite network isolation.

* This ties into Michael Brown's talks of isolating daemons to their
own user/group and enforcing SELinux policy against them.

> 
> 2. U-Boot already performs image checksum validation before booting a 
> FIT image

Typically the rootfs is not part of the FIT, so it will not be checked.
    Some systems supported by OpenBMC directly mount the rootfs rather
than booting through an initrd, which makes rootfs authentication
somewhat tricky. Regardless, with signed images we should expand the
FIT hash check to be a full signature check.

> 
> 3. User input really needs validation, at least to make the system 
> fool-proof
> 
> Having said that, I suggest that the only thing that really needs doing 
> is signing (and checking) of the overall firmware image file that is 
> supplied by the user (admin) during the firmware upgrade procedure. 
> Applying asymmetric cryptography to a digest hash looks to me like a 
> good idea as it indeed allows for verifying the supplier of the firmware 
> image.

Agreed - do Yadro do this already? If so, what did you do to integrate
image signing into the build process? If not, then I'd be keen to keep
the discussion alive on how we can achieve it.

Cheers,

Andrew

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: BMC Image Signing Proposal

[Avi Fishman]

Hi Anoo and all,

Regarding "The encrypted hash (digital signature) and corresponding
public key would be added to the firmware image":
If both the decrypted signature and the public key are in flash, if
flash was replaced (by hacking or physicaly) a malicious public key
can be used with a  signature that was decrypted by the paired
malicious private key of the stored public key.

To overcome that the public key should be stored in OTP.
We use this method on our Nuvoton Poleg BMC.

Does this sound reasonable?

Regards,
Avi


On Thu, Jan 25, 2018 at 11:15 PM, anoo <anoo@linux.vnet.ibm.com> wrote:
> Hi all,
>
> During the hackaton meetup, we touched on BMC image verification and signing
> and concluded that the community would like to see two verification steps,
> one prior to writing the image to flash (via digital signature
> verification), and another one by checking FIT in U-Boot prior to booting
> from the new image.
>
> The proposal would be to implement the digital signature verification first.
>
> At a high level, during the build:
> * A SHA-256 hash would be calculated over tbd files that make up the
> firmware image.
> * The hash would be signed by a private key that's part of the repository
> (community key). Companies could overwrite it with their own private key
> when building production images.
> * The encrypted hash (digital signature) and corresponding public key would
> be added to the firmware image.
> * Yocto may already provide a way to sign images and generate keys.
>
> On the BMC:
> * The hash would be calculated on the image files that were uploaded to the
> BMC.
> * The signature would be decrypted using an existing public key on the BMC
> (this validates the new public key delivered with the image is also valid).
> * Accept image if both values are the same.
>
> Any thoughts or comments?
>

RE: BMC Image Signing Proposal

[Eugene.Cho]

Dell - Internal Use - Confidential  

The "firmware update" public key in flash (and firmware update code itself) would be authenticated by BMC secure boot.

Also - has anybody mentioned using the FIT as a firmware update install image, in addition to using to boot the kernel? (i.e. FIT within a FIT). Think the native DFU command in u-boot supports this method. Just a thought
 

-----Original Message-----
From: openbmc [mailto:openbmc-bounces+eugene.cho=dell.com@lists.ozlabs.org] On Behalf Of Avi Fishman
Sent: Monday, January 29, 2018 4:45 AM
To: anoo <anoo@linux.vnet.ibm.com>
Cc: eyal.cohen@nuvoton.com; OpenBMC Maillist <openbmc@lists.ozlabs.org>; yoel.hayon@nuvoton.com; uri.trichter@nuvoton.com
Subject: Re: BMC Image Signing Proposal

Hi Anoo and all,

Regarding "The encrypted hash (digital signature) and corresponding public key would be added to the firmware image":
If both the decrypted signature and the public key are in flash, if flash was replaced (by hacking or physicaly) a malicious public key can be used with a  signature that was decrypted by the paired malicious private key of the stored public key.

To overcome that the public key should be stored in OTP.
We use this method on our Nuvoton Poleg BMC.

Does this sound reasonable?

Regards,
Avi


On Thu, Jan 25, 2018 at 11:15 PM, anoo <anoo@linux.vnet.ibm.com> wrote:
> Hi all,
>
> During the hackaton meetup, we touched on BMC image verification and 
> signing and concluded that the community would like to see two 
> verification steps, one prior to writing the image to flash (via 
> digital signature verification), and another one by checking FIT in 
> U-Boot prior to booting from the new image.
>
> The proposal would be to implement the digital signature verification first.
>
> At a high level, during the build:
> * A SHA-256 hash would be calculated over tbd files that make up the 
> firmware image.
> * The hash would be signed by a private key that's part of the 
> repository (community key). Companies could overwrite it with their 
> own private key when building production images.
> * The encrypted hash (digital signature) and corresponding public key 
> would be added to the firmware image.
> * Yocto may already provide a way to sign images and generate keys.
>
> On the BMC:
> * The hash would be calculated on the image files that were uploaded 
> to the BMC.
> * The signature would be decrypted using an existing public key on the 
> BMC (this validates the new public key delivered with the image is also valid).
> * Accept image if both values are the same.
>
> Any thoughts or comments?
>

Re: BMC Image Signing Proposal

[Simon Glass]

Hi,

On 28 January 2018 at 23:30, Andrew Jeffery <andrew@aj.id.au> wrote:
> Hi Alexander,
>
> On Fri, 2018-01-26 at 14:07 +0300, Alexander Amelkin wrote:
>> Hi, Anoo!
>>
>> The thoughts are as follows:
>>
>> 1. BMC usually runs in a secured environment where probability of
>> tampering with flash IC contents by means other than BMC's firmware
>> itself is negligible.
>
> This does bring up the issue of developing a threat model to develop
> against; we should probably do that. However, without one, I feel like
> we should design *for* defense in depth and allow people to remove
> protections as they see fit for their environment, rather than make
> potentially compromising assumptions about what that environment is at
> the outset.
>
> For instance, whilst BMCs might typically be isolated from customer
> traffic on a separate LAN, there's still the in-band interface which
> can be used to poke at the BMC. Vulnerabilities in the IPMI stack could
> lead to BMC compromise and undesirable flash writes*. Therefore BMC
> flash integrity remains a valid concern despite network isolation.
>
> * This ties into Michael Brown's talks of isolating daemons to their
> own user/group and enforcing SELinux policy against them.
>
>>
>> 2. U-Boot already performs image checksum validation before booting a
>> FIT image
>
> Typically the rootfs is not part of the FIT, so it will not be checked.
>     Some systems supported by OpenBMC directly mount the rootfs rather
> than booting through an initrd, which makes rootfs authentication
> somewhat tricky. Regardless, with signed images we should expand the
> FIT hash check to be a full signature check.

If you use an initrd in the FIT, it can be checked in the same way as
the kernel. FIT supports a full signature check.

If you use a separate rootfs, verity can be used if it is read-only.
Otherwise I am not sure.

>
>>
>> 3. User input really needs validation, at least to make the system
>> fool-proof
>>
>> Having said that, I suggest that the only thing that really needs doing
>> is signing (and checking) of the overall firmware image file that is
>> supplied by the user (admin) during the firmware upgrade procedure.
>> Applying asymmetric cryptography to a digest hash looks to me like a
>> good idea as it indeed allows for verifying the supplier of the firmware
>> image.
>
> Agreed - do Yadro do this already? If so, what did you do to integrate
> image signing into the build process? If not, then I'd be keen to keep
> the discussion alive on how we can achieve it.
>
> Cheers,
>
> Andrew

Regards,
Simon

Re: BMC Image Signing Proposal

[Vernon Mauery]

On 29-Jan-2018 08:50 AM, Simon Glass wrote:
>Hi,
>
>On 28 January 2018 at 23:30, Andrew Jeffery <andrew@aj.id.au> wrote:
>> Hi Alexander,
>>
>> On Fri, 2018-01-26 at 14:07 +0300, Alexander Amelkin wrote:
>>> Hi, Anoo!
>>>
>>> The thoughts are as follows:
>>>
>>> 1. BMC usually runs in a secured environment where probability of
>>> tampering with flash IC contents by means other than BMC's firmware
>>> itself is negligible.
>>
>> This does bring up the issue of developing a threat model to develop
>> against; we should probably do that. However, without one, I feel like
>> we should design *for* defense in depth and allow people to remove
>> protections as they see fit for their environment, rather than make
>> potentially compromising assumptions about what that environment is at
>> the outset.
>>
>> For instance, whilst BMCs might typically be isolated from customer
>> traffic on a separate LAN, there's still the in-band interface which
>> can be used to poke at the BMC. Vulnerabilities in the IPMI stack could
>> lead to BMC compromise and undesirable flash writes*. Therefore BMC
>> flash integrity remains a valid concern despite network isolation.
>>
>> * This ties into Michael Brown's talks of isolating daemons to their
>> own user/group and enforcing SELinux policy against them.

We would prefer that no environment is used for any action of 
consequence. It is completely untrusted, and only used for things like 
hints of preferences. For example, we would store the last preferred 
image (A or B), but ultimately, the one chosen would have to be 
authenticated before booting anyway; it is just the preferred image 
would be checked first and only fall back to the other one if 
verification failed.

Along the same lines, if the user stopped the BMC from automatically 
booting, we are now in a situation where the Linux runtime's root of 
trust could be compromised (specifically by forcing a boot of something 
that does not verify or by passing custom kernel command line 
arguments). I would prefer if there was a way that if stopped in u-boot, 
you would have to reset the BMC to actually boot to the kernel.

>>>
>>> 2. U-Boot already performs image checksum validation before booting a
>>> FIT image
>>
>> Typically the rootfs is not part of the FIT, so it will not be checked.
>>     Some systems supported by OpenBMC directly mount the rootfs rather
>> than booting through an initrd, which makes rootfs authentication
>> somewhat tricky. Regardless, with signed images we should expand the
>> FIT hash check to be a full signature check.
>
>If you use an initrd in the FIT, it can be checked in the same way as
>the kernel. FIT supports a full signature check.
>
>If you use a separate rootfs, verity can be used if it is read-only.
>Otherwise I am not sure.

Historically, we have just booted with an initramfs (squashfs) that is 
the root filesystem. This means that it is possible to update in place 
without worrying about corrupting the current runtime. Normally our 
updates are A->B->A, but we do have one mode that is always in place.

Our current plan was to have our FIT include U-Boot, Linux, rootfs (as 
squashfs), and DTB, with a signature on U-Boot individually, and then 
another signature on all the parts. We would use an SPL to verify the 
signature of U-Boot and then boot it, and then U-Boot would verify the 
signature of the full FIT before booting Linux.

--Vernon

>>>
>>> 3. User input really needs validation, at least to make the system
>>> fool-proof
>>>
>>> Having said that, I suggest that the only thing that really needs doing
>>> is signing (and checking) of the overall firmware image file that is
>>> supplied by the user (admin) during the firmware upgrade procedure.
>>> Applying asymmetric cryptography to a digest hash looks to me like a
>>> good idea as it indeed allows for verifying the supplier of the firmware
>>> image.
>>
>> Agreed - do Yadro do this already? If so, what did you do to integrate
>> image signing into the build process? If not, then I'd be keen to keep
>> the discussion alive on how we can achieve it.
>>
>> Cheers,
>>
>> Andrew
>
>Regards,
>Simon

Re: BMC Image Signing Proposal

[Vernon Mauery]

On 29-Jan-2018 04:26 PM, Andrew Jeffery wrote:
>On Thu, 2018-01-25 at 15:15 -0600, anoo wrote:
>> Hi all,
>>
>> During the hackaton meetup, we touched on BMC image verification and
>> signing and concluded that the community would like to see two
>> verification steps, one prior to writing the image to flash (via digital
>> signature verification), and another one by checking FIT in U-Boot prior
>> to booting from the new image.
>>
>> The proposal would be to implement the digital signature verification
>> first.
>>
>> At a high level, during the build:
>> * A SHA-256 hash would be calculated over tbd files that make up the
>> firmware image.
>> * The hash would be signed by a private key that's part of the
>> repository (community key). Companies could overwrite it with their own
>> private key when building production images.

Can we not store any key in the tree at all, please. If we do, somebody 
will inevitably use that key for production images and be totally 
insecure. Generate a key by default and print a big warning that it is 
being signed by a local (insecure) key and that they should set up keys 
according to some documentation.

Provide a script that can be used to (somewhat) securely store openssl 
keys (with a passphrase) and then request the passphrase for each build. 
This could also be used to securely store keys on an HSM that openssl 
has ENGINE support for.

>> * The encrypted hash (digital signature) and corresponding public key
>> would be added to the firmware image.
>> * Yocto may already provide a way to sign images and generate keys.
>
>I'm starting to look into signing of artefacts as part of the build
>process. Some quick searching suggests Yocto doesn't have support for
>it out of the box, but there is the swupdate project[1] and the
>corresponding meta-swupdate[2] Yocto layer that sound interesting.
>However, it may require a rework of how we manage image updates if we
>were to make use of it. I'll keep poking around, but if anyone has any
>other pointers please let me know.
>
>[1] https://github.com/sbabic/swupdate
>[2] https://github.com/sbabic/meta-swupdate

Whatever mechanism is used to generate the signature should be able to 
be overridden by an environment variable or something. We already do 
something like this for our pre-openbmc generations where a common 
developer key is stored outside the tree, then the image gets re-signed 
with the new key to turn it into a production-signed image. This way, we 
can use different mechanisms for generating the development and 
production signatures (openssl and a remote signing service).

--Vernon


>>
>> On the BMC:
>> * The hash would be calculated on the image files that were uploaded to
>> the BMC.
>> * The signature would be decrypted using an existing public key on the
>> BMC (this validates the new public key delivered with the image is also
>> valid).
>> * Accept image if both values are the same.
>>
>> Any thoughts or comments?
>
>Can you take a look at swupdate and reply with your thoughts?
>
>Cheers,
>
>Andrew

Re: BMC Image Signing Proposal

[Stewart Smith]

Alexander Amelkin <a.amelkin@yadro.com> writes:
> 1. BMC usually runs in a secured environment where probability of 
> tampering with flash IC contents by means other than BMC's firmware 
> itself is negligible.

Consider the host to be actively hostile.

In any metal-as-a-service environment, a privilege escalation is to get
code onto the BMC, and thus survive to the next tenant. While the bmc to
host interfaces *should* be nice and secure, they are software, and thus
are *going* to have security issues. A solid secure-boot story on the
BMC would add defense in depth.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: BMC Image Signing Proposal

[Stewart Smith]

Andrew Jeffery <andrew@aj.id.au> writes:
> On Fri, 2018-01-26 at 14:07 +0300, Alexander Amelkin wrote:
>> Hi, Anoo!
>> 
>> The thoughts are as follows:
>> 
>> 1. BMC usually runs in a secured environment where probability of 
>> tampering with flash IC contents by means other than BMC's firmware 
>> itself is negligible.
>
> This does bring up the issue of developing a threat model to develop
> against; we should probably do that. However, without one, I feel like
> we should design *for* defense in depth and allow people to remove
> protections as they see fit for their environment, rather than make
> potentially compromising assumptions about what that environment is at
> the outset.
>
> For instance, whilst BMCs might typically be isolated from customer
> traffic on a separate LAN, there's still the in-band interface which
> can be used to poke at the BMC. Vulnerabilities in the IPMI stack could
> lead to BMC compromise and undesirable flash writes*. Therefore BMC
> flash integrity remains a valid concern despite network isolation.
>
> * This ties into Michael Brown's talks of isolating daemons to their
> own user/group and enforcing SELinux policy against them.

using isolation technologies that are used by containers may also add
extra depth, and be a good interim solution as SELinux policy is
developed and tuned.

>> 2. U-Boot already performs image checksum validation before booting a 
>> FIT image
>
> Typically the rootfs is not part of the FIT, so it will not be checked.
>     Some systems supported by OpenBMC directly mount the rootfs rather
> than booting through an initrd, which makes rootfs authentication
> somewhat tricky. Regardless, with signed images we should expand the
> FIT hash check to be a full signature check.

dm-verity would solve that (for a ro rootfs). The read/write parts will
have to be carefully controlled of course, noexec is an obvious
thing. Oh, and not having configuration as executable code and being
*paranoid* about parsing config.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: BMC Image Signing Proposal

[Joel Stanley]

On Tue, Jan 30, 2018 at 3:17 PM, Stewart Smith
<stewart@linux.vnet.ibm.com> wrote:
> Andrew Jeffery <andrew@aj.id.au> writes:
>> On Fri, 2018-01-26 at 14:07 +0300, Alexander Amelkin wrote:

>>> 2. U-Boot already performs image checksum validation before booting a
>>> FIT image
>>
>> Typically the rootfs is not part of the FIT, so it will not be checked.
>>     Some systems supported by OpenBMC directly mount the rootfs rather
>> than booting through an initrd, which makes rootfs authentication
>> somewhat tricky. Regardless, with signed images we should expand the
>> FIT hash check to be a full signature check.
>
> dm-verity would solve that (for a ro rootfs).

dm-verity is a worthwhile technology, but being based on device mapper
and therefore block devices, we can't use it for MTD devices, which is
all of the upstream OpenBMC machines at this moment.

I would suggest using some kind of pre-mount verification of the raw
MTD device against a stored checksum would be the way to go. This
would imply the use of an initrd (as we would need somewhere to store
the tools that do the verification). The initrd itself would be
verified by u-boot checking the FIT.

Future contributors to OpenBMC that have eMMC hardware do have the
option of using dm-verity.

Cheers,

Joel

Re: BMC Image Signing Proposal

[Simon Glass]

Hi Joel,

On 29 January 2018 at 23:18, Joel Stanley <joel@jms.id.au> wrote:
>
> On Tue, Jan 30, 2018 at 3:17 PM, Stewart Smith
> <stewart@linux.vnet.ibm.com> wrote:
> > Andrew Jeffery <andrew@aj.id.au> writes:
> >> On Fri, 2018-01-26 at 14:07 +0300, Alexander Amelkin wrote:
>
> >>> 2. U-Boot already performs image checksum validation before booting a
> >>> FIT image
> >>
> >> Typically the rootfs is not part of the FIT, so it will not be checked.
> >>     Some systems supported by OpenBMC directly mount the rootfs rather
> >> than booting through an initrd, which makes rootfs authentication
> >> somewhat tricky. Regardless, with signed images we should expand the
> >> FIT hash check to be a full signature check.
> >
> > dm-verity would solve that (for a ro rootfs).
>
> dm-verity is a worthwhile technology, but being based on device mapper
> and therefore block devices, we can't use it for MTD devices, which is
> all of the upstream OpenBMC machines at this moment.

You could use ubi to provide a block device, I haven't tried it though.

>
> I would suggest using some kind of pre-mount verification of the raw
> MTD device against a stored checksum would be the way to go. This
> would imply the use of an initrd (as we would need somewhere to store
> the tools that do the verification). The initrd itself would be
> verified by u-boot checking the FIT.
>
> Future contributors to OpenBMC that have eMMC hardware do have the
> option of using dm-verity.

Are you saying that dm-verity does not work with eMMC, or something else?

Regards,
SImon

Re: BMC Image Signing Proposal

[Stewart Smith]

Joel Stanley <joel@jms.id.au> writes:
> On Tue, Jan 30, 2018 at 3:17 PM, Stewart Smith
> <stewart@linux.vnet.ibm.com> wrote:
>> Andrew Jeffery <andrew@aj.id.au> writes:
>>> On Fri, 2018-01-26 at 14:07 +0300, Alexander Amelkin wrote:
>
>>>> 2. U-Boot already performs image checksum validation before booting a
>>>> FIT image
>>>
>>> Typically the rootfs is not part of the FIT, so it will not be checked.
>>>     Some systems supported by OpenBMC directly mount the rootfs rather
>>> than booting through an initrd, which makes rootfs authentication
>>> somewhat tricky. Regardless, with signed images we should expand the
>>> FIT hash check to be a full signature check.
>>
>> dm-verity would solve that (for a ro rootfs).
>
> dm-verity is a worthwhile technology, but being based on device mapper
> and therefore block devices, we can't use it for MTD devices, which is
> all of the upstream OpenBMC machines at this moment.
>
> I would suggest using some kind of pre-mount verification of the raw
> MTD device against a stored checksum would be the way to go. This
> would imply the use of an initrd (as we would need somewhere to store
> the tools that do the verification). The initrd itself would be
> verified by u-boot checking the FIT.

mtdblock could end up being okay for the dm-verity case? There's no
writes there at least... although I haven't spent much/any time thinking
of any implications to that - you're the bigger expert there than I.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: BMC Image Signing Proposal

[Adriana Kobylak]

Thanks everybody for your thoughts so far. I'll try to comment on the 
remaining open questions.

Summarizing, there are 2 different verifications that are desired, one 
prior to writing the image to flash (using signature check), and one 
prior to booting the image (FIT verification).

1. Signature check

> Can you take a look at swupdate and reply with your thoughts?
Yes, thanks Andrew for looking and suggesting this.

> To overcome that the public key should be stored in OTP.
> We use this method on our Nuvoton Poleg BMC.
One concern would be if the key is deemed compromised. Avi, does the 
Nuvoton system have a mechanism for invalidating a key stored in OTP?
>> The "firmware update" public key in flash (and firmware update code 
>> itself) would be authenticated by BMC secure boot.
Per Eugene's comment, maybe we're ok with the key in flash?

> Whatever mechanism is used to generate the signature should be able to 
> be overridden by an environment variable or something.
Yes, good point Vernon.


2. FIT verification

> Regardless, with signed images we should expand the
> FIT hash check to be a full signature check.
Yes, I think there was some agreement on this statement during the 
hackaton.

> Also - has anybody mentioned using the FIT as a firmware update install 
> image, in addition to using to boot the kernel?(i.e. FIT within a FIT). 
> Think the native DFU command in u-boot supports this method. Just a 
> thought
Yes, ideally we'd implement FIT verification of the rootfs in addition 
to the kernel to verify an image before booting.

Re: BMC Image Signing Proposal

[Adriana Kobylak]

Here are some charts with the image signing flow for comment:

https://drive.google.com/file/d/1IxfMYRttN8RbhRY7PwBmXsqCBvtv_yLJ/view?usp=sharing

RE: BMC Image Signing Proposal

[Yugi Mani]

On Thursday, February 8, 2018 12:27 PM, Adriana Kobylak wrote:
> Here are some charts with the image signing flow for comment:
> 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdri
> ve.google.com%2Ffile%2Fd%2F1IxfMYRttN8RbhRY7PwBmXsqCBvtv_yLJ
> %2Fview%3Fusp%3Dsharing&data=04%7C01%7Cyupalani%40microsoft.c
> om%7C9a2d9f45a2144c59bc5908d56f3235c7%7Cee3303d7fb734b0c8589b
> cd847f1c277%7C1%7C0%7C636537183804895636%7CUnknown%7CTWFp
> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ
> %3D%3D%7C-
> 1&sdata=C0RBtfmjE3VHH1A8A8MwCBVFCUQyvF3WEdWH7bgjRC0%3D&
> reserved=0

Thanks for putting together a chart. 

We should consider both of these requirements for image signing:
1. Update verification
2. Boot Verification

Appending signature to image meets verification during firmware update. To do verification on every boot, we need something like FIT.  
https://chromium.googlesource.com/chromiumos/third_party/u-boot-next/+/chromeos-v2013.06/doc/uImage.FIT

As far as actual signing is concerned, we don't have access to private key for security reasons. We should support two models:
Model 1:
Source code has private key and signing is part of build process ("bitbake obmc-phosphor-image")

Model 2:
Source code does not have private key, Signing is done externally and some post-processing is done to add hash to image.  (maybe a new task, "bitbake obmc-phosphor-image -c add_hash")

RE: BMC Image Signing Proposal

[Adriana Kobylak]

On 2018-02-09 19:36, Yugi Mani wrote:

> We should consider both of these requirements for image signing:
> 1. Update verification
> 2. Boot Verification
> 
Yes, the boot verification via FIT is being tracked via issue 
https://github.com/openbmc/openbmc/issues/2829, it's planned to be 
implemented after the update verification.

> Appending signature to image meets verification during firmware
> update. To do verification on every boot, we need something like FIT.
> https://chromium.googlesource.com/chromiumos/third_party/u-boot-next/+/chromeos-v2013.06/doc/uImage.FIT
> 
Thanks, I've added the link to the issue 2829 for reference.

> As far as actual signing is concerned, we don't have access to private
> key for security reasons. We should support two models:
> Model 1:
> Source code has private key and signing is part of build process
> ("bitbake obmc-phosphor-image")
> 
> Model 2:
> Source code does not have private key, Signing is done externally and
> some post-processing is done to add hash to image.  (maybe a new task,
> "bitbake obmc-phosphor-image -c add_hash")
Yeah, these are good points, we were thinking also through a build 
environment variable this could be specified. Eddie is implementing this 
piece so he could expand on this.

RE: BMC Image Signing Proposal

[Adriana Kobylak]

For the location of the public keys on the BMC, seems /etc/ would be an 
appropriate place as the keys are static and unshareable (reference: 
http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s07.html).

Any thoughts on storing the public keys on /etc/ssh/ for example?

Re: BMC Image Signing Proposal

[Joel Stanley]

On Wed, Feb 14, 2018 at 9:04 AM, Adriana Kobylak
<anoo@linux.vnet.ibm.com> wrote:
> For the location of the public keys on the BMC, seems /etc/ would be an
> appropriate place as the keys are static and unshareable (reference:
> http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s07.html).

What do you mean unsharable?

> Any thoughts on storing the public keys on /etc/ssh/ for example?

This is where ssh keys go :)

Storing them in /etc/ makes sense though.

Cheers,

Joel

Re: BMC Image Signing Proposal

[Joel Stanley]

On Fri, Feb 9, 2018 at 6:57 AM, Adriana Kobylak <anoo@linux.vnet.ibm.com> wrote:
> Here are some charts with the image signing flow for comment:
>
> https://drive.google.com/file/d/1IxfMYRttN8RbhRY7PwBmXsqCBvtv_yLJ/view?usp=sharing
>

Nancy, does anyone from your team want to have input on this?

Re: BMC Image Signing Proposal

[Adriana Kobylak]

> 
> What do you mean unsharable?
> 

The public keys only apply to the BMC where the firmware image is being 
installed on and not are not intended to be shared with other BMCs / 
hosts. "Shareable files can be stored on one host and used on several 
others" (http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch02.html).


On another note, I'll be adding the gerrit links to the main epic 
(https://github.com/openbmc/openbmc/issues/356) as changes start getting 
submitted for anyone that wants to leave comments or follow along. The 
first ones are:

https://gerrit.openbmc-project.xyz/#/c/8949/ - Add image signing 
framework and open keys
https://gerrit.openbmc-project.xyz/#/c/9025/ - BMC Digital Signature 
documentation

RE: BMC Image Signing Proposal

[Stewart Smith]

Yugi Mani <yupalani@microsoft.com> writes:
> We should consider both of these requirements for image signing:
> 1. Update verification
> 2. Boot Verification
>
> Appending signature to image meets verification during firmware update. To do verification on every boot, we need something like FIT.  
> https://chromium.googlesource.com/chromiumos/third_party/u-boot-next/+/chromeos-v2013.06/doc/uImage.FIT
>
> As far as actual signing is concerned, we don't have access to private key for security reasons. We should support two models:
> Model 1:
> Source code has private key and signing is part of build process ("bitbake obmc-phosphor-image")
>
> Model 2:
> Source code does not have private key, Signing is done externally and
> some post-processing is done to add hash to image.  (maybe a new task,
> "bitbake obmc-phosphor-image -c add_hash")

For reference, for OpenPOWER host firmware, we support three models:

Local mode (a.k.a. development mode) — Build the container and sign
using locally available private keys. Signatures are generated using
simple openssl operations. Because the private keys are exposed on the
local system (the build machine), this mode should be used only for
development signing, or when the user is confident that the build
machine is secure against unauthorized access.

Independent mode — Generate the signing requests locally and export the
requests for signing externally. External signing is by user's method of
choice: any method capable of generating a ECDSA p521 signature (the
built-in support uses openssl). Resulting signatures are re-imported to
the container build process, to create the completed container. No
private or privileged information is exposed at the build machine.

Production mode — Build the container locally and interface with the
remote signframework to retrieve signatures and (public) keys as
needed. Signing is done remotely on a secure signing server using a
hardware security module (HSM). Private keys are stored securely in the
HSM and never exposed. Completed signatures are returned by the
signframework and integrated into the container.

(gratuitously copy&pasted from Nick's great docs up at
https://github.com/open-power/sb-signing-utils/wiki )

-- 
Stewart Smith
OPAL Architect, IBM.

Re: BMC Image Signing Proposal

[Stewart Smith]

Adriana Kobylak <anoo@linux.vnet.ibm.com> writes:
> Here are some charts with the image signing flow for comment:
>
> https://drive.google.com/file/d/1IxfMYRttN8RbhRY7PwBmXsqCBvtv_yLJ/view?usp=sharing

Why are there changes to host pnor signing?

We already have secure boot and signatures on the host side, I'm not
keen on adding in another set of signatures into yet-another already non-standard
and undocumented file format.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: BMC Image Signing Proposal

[Vernon Mauery]

On 23-Feb-2018 12:44 PM, Stewart Smith wrote:
>Yugi Mani <yupalani@microsoft.com> writes:
>> We should consider both of these requirements for image signing:
>> 1. Update verification
>> 2. Boot Verification
>>
>> Appending signature to image meets verification during firmware update. To do verification on every boot, we need something like FIT.
>> https://chromium.googlesource.com/chromiumos/third_party/u-boot-next/+/chromeos-v2013.06/doc/uImage.FIT
>>
>> As far as actual signing is concerned, we don't have access to private key for security reasons. We should support two models:
>> Model 1:
>> Source code has private key and signing is part of build process ("bitbake obmc-phosphor-image")
>>
>> Model 2:
>> Source code does not have private key, Signing is done externally and
>> some post-processing is done to add hash to image.  (maybe a new task,
>> "bitbake obmc-phosphor-image -c add_hash")
>
>For reference, for OpenPOWER host firmware, we support three models:
>
>Local mode (a.k.a. development mode) — Build the container and sign
>using locally available private keys. Signatures are generated using
>simple openssl operations. Because the private keys are exposed on the
>local system (the build machine), this mode should be used only for
>development signing, or when the user is confident that the build
>machine is secure against unauthorized access.
>
>Independent mode — Generate the signing requests locally and export the
>requests for signing externally. External signing is by user's method of
>choice: any method capable of generating a ECDSA p521 signature (the
>built-in support uses openssl). Resulting signatures are re-imported to
>the container build process, to create the completed container. No
>private or privileged information is exposed at the build machine.
>
>Production mode — Build the container locally and interface with the
>remote signframework to retrieve signatures and (public) keys as
>needed. Signing is done remotely on a secure signing server using a
>hardware security module (HSM). Private keys are stored securely in the
>HSM and never exposed. Completed signatures are returned by the
>signframework and integrated into the container.
>
>(gratuitously copy&pasted from Nick's great docs up at
>https://github.com/open-power/sb-signing-utils/wiki )

Historically, we have supported two signing modes, similar to one and 
three:

Local mode (a.k.a. development mode) - similar to what you describe 
where the keys are insecure and should not be used to generate 
production images.

Re-Sign mode (for Production images) - takes a binary image, parses and 
deconstructs it, verifies the signature, and puts it back together with 
a new signature and public key certificate. This makes it so we only 
ever sign 'blessed' images with the production key, where this blessing 
comes from the validation team. The production build server initially 
signs the release image with a non-production key and that image gets 
tested. Then it gets re-signed with the production key only if it passes 
validation.

--Vernon

Re: BMC Image Signing Proposal

[Adriana Kobylak]

> 
> Why are there changes to host pnor signing?
> 
> We already have secure boot and signatures on the host side, I'm not
> keen on adding in another set of signatures into yet-another already
> non-standard
> and undocumented file format.

Secure boot validates individual pnor packages, so it could be possible 
to build an image with an older individual package (like an opal n-1) 
and still be able to boot the host. Signing the whole image would catch 
this scenario.
This image signing feature in general is an extension of the current 
implementation of the software dbus interfaces, which provides options 
for a user/company to use.

Re: BMC Image Signing Proposal

[Lei YU]

I'd like to bring this topic again.

As I know image signing feature is completed for both BMC and PNOR:
https://github.com/openbmc/phosphor-bmc-code-mgmt
https://github.com/openbmc/openpower-pnor-code-mgmt

However, the above repos are only for systems with UBI-FS feature.
Most of machines are still using the "legacy" obmc flash layout, and thus
they do not have image singing feature.

So I would like to ask for ideas about how to support image signing feature
for machines with "legacy" flash layout?
1. Should we use UBI-FS for machines that requires image signing feature?
2. Or should we implement image signing feature on "legacy" flash layout as
well?

Thanks!

RE: BMC Image Signing Proposal

[Yugi Mani]

Good point. We at MSFT are using legacy (non-UBI) layout. We have a manifest for boot verification and we append the hash to image for update verification. 
I can share details about the design/implementation, if you have any specific questions. 

> -----Original Message-----
> From: openbmc <openbmc-
> bounces+yupalani=microsoft.com@lists.ozlabs.org> On Behalf Of Lei YU
> Sent: Monday, May 14, 2018 7:06 PM
> To: Adriana Kobylak <anoo@linux.vnet.ibm.com>
> Cc: Stewart Smith <stewart@linux.vnet.ibm.com>; OpenBMC Maillist
> <openbmc@lists.ozlabs.org>
> Subject: Re: BMC Image Signing Proposal
> 
> I'd like to bring this topic again.
> 
> As I know image signing feature is completed for both BMC and PNOR:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> hub.com%2Fopenbmc%2Fphosphor-bmc-code-
> mgmt&data=02%7C01%7Cyupalani%40microsoft.com%7C12f250ae199b4
> 32be89a08d5ba089773%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C
> 0%7C636619468424007343&sdata=GHBBgm9g9kBpfNAzYxgveFoGWXXl1
> QOr8fHwZNJNvNA%3D&reserved=0
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> hub.com%2Fopenbmc%2Fopenpower-pnor-code-
> mgmt&data=02%7C01%7Cyupalani%40microsoft.com%7C12f250ae199b4
> 32be89a08d5ba089773%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C
> 0%7C636619468424007343&sdata=CR2o%2FgVnHPfMr7w8fwOYeRsbOdF
> B3VMiB%2FFUCnE2aHE%3D&reserved=0
> 
> However, the above repos are only for systems with UBI-FS feature.
> Most of machines are still using the "legacy" obmc flash layout, and thus
> they do not have image singing feature.
> 
> So I would like to ask for ideas about how to support image signing
> feature for machines with "legacy" flash layout?
> 1. Should we use UBI-FS for machines that requires image signing
> feature?
> 2. Or should we implement image signing feature on "legacy" flash layout
> as well?
> 
> Thanks!

Re: BMC Image Signing Proposal

[Stewart Smith]

Lei YU <mine260309@gmail.com> writes:
> I'd like to bring this topic again.
>
> As I know image signing feature is completed for both BMC and PNOR:
> https://github.com/openbmc/phosphor-bmc-code-mgmt
> https://github.com/openbmc/openpower-pnor-code-mgmt
>
> However, the above repos are only for systems with UBI-FS feature.
> Most of machines are still using the "legacy" obmc flash layout, and thus
> they do not have image singing feature.
>
> So I would like to ask for ideas about how to support image signing feature
> for machines with "legacy" flash layout?
> 1. Should we use UBI-FS for machines that requires image signing feature?
> 2. Or should we implement image signing feature on "legacy" flash layout as
> well?

Either option is valid. I think we should get it gonig for non-UBI-FS
systems though, and we should do that through the host

-- 
Stewart Smith
OPAL Architect, IBM.

RE: BMC Image Signing Proposal

[Stewart Smith]

Yugi Mani <yupalani@microsoft.com> writes:
> Good point. We at MSFT are using legacy (non-UBI) layout. We have a manifest for boot verification and we append the hash to image for update verification. 
> I can share details about the design/implementation, if you have any
> specific questions.

I'd be interested in what you came up with as we have the same two
issues with OpenPOWER: update verification (not security so much as
stopping people doing something silly like flashing the wrong firmware
image), as well as partial downgrade protection (i.e. the host has to
verify the integrity of the whole image, not just individual components)


-- 
Stewart Smith
OPAL Architect, IBM.

Re: BMC Image Signing Proposal

[Vernon Mauery]

On 15-May-2018 06:18 PM, Yugi Mani wrote:
>Good point. We at MSFT are using legacy (non-UBI) layout. We have a manifest for boot verification and we append the hash to image for update verification.
>I can share details about the design/implementation, if you have any specific questions.

At Intel, we are using a legacy layout as well, either ping-ponging 
between partitions or using a active/temp/recovery partition scheme 
depending on the secure boot mechanism for that platform.

--Vernon

>> -----Original Message-----
>> From: openbmc <openbmc-
>> bounces+yupalani=microsoft.com@lists.ozlabs.org> On Behalf Of Lei YU
>> Sent: Monday, May 14, 2018 7:06 PM
>> To: Adriana Kobylak <anoo@linux.vnet.ibm.com>
>> Cc: Stewart Smith <stewart@linux.vnet.ibm.com>; OpenBMC Maillist
>> <openbmc@lists.ozlabs.org>
>> Subject: Re: BMC Image Signing Proposal
>>
>> I'd like to bring this topic again.
>>
>> As I know image signing feature is completed for both BMC and PNOR:
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
>> hub.com%2Fopenbmc%2Fphosphor-bmc-code-
>> mgmt&data=02%7C01%7Cyupalani%40microsoft.com%7C12f250ae199b4
>> 32be89a08d5ba089773%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C
>> 0%7C636619468424007343&sdata=GHBBgm9g9kBpfNAzYxgveFoGWXXl1
>> QOr8fHwZNJNvNA%3D&reserved=0
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
>> hub.com%2Fopenbmc%2Fopenpower-pnor-code-
>> mgmt&data=02%7C01%7Cyupalani%40microsoft.com%7C12f250ae199b4
>> 32be89a08d5ba089773%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C
>> 0%7C636619468424007343&sdata=CR2o%2FgVnHPfMr7w8fwOYeRsbOdF
>> B3VMiB%2FFUCnE2aHE%3D&reserved=0
>>
>> However, the above repos are only for systems with UBI-FS feature.
>> Most of machines are still using the "legacy" obmc flash layout, and thus
>> they do not have image singing feature.
>>
>> So I would like to ask for ideas about how to support image signing
>> feature for machines with "legacy" flash layout?
>> 1. Should we use UBI-FS for machines that requires image signing
>> feature?
>> 2. Or should we implement image signing feature on "legacy" flash layout
>> as well?
>>
>> Thanks!

Re: BMC Image Signing Proposal

[Lei YU]

On Thu, May 17, 2018 at 12:02 AM, Vernon Mauery
<vernon.mauery@linux.intel.com> wrote:
> On 15-May-2018 06:18 PM, Yugi Mani wrote:
>>
>> Good point. We at MSFT are using legacy (non-UBI) layout. We have a
>> manifest for boot verification and we append the hash to image for update
>> verification.
>> I can share details about the design/implementation, if you have any
>> specific questions.
>
>
> At Intel, we are using a legacy layout as well, either ping-ponging between
> partitions or using a active/temp/recovery partition scheme depending on the
> secure boot mechanism for that platform.
>

Thanks for info!
So I think it is better for OpenBMC project to have a common (or example)
image signing tools/code, not for a specific machine or product, but for the
general machines in this project using legacy flash layout.
Let's discuss and get a design proposal?

> --Vernon
>
>

Re: BMC Image Signing Proposal

[Adriana Kobylak]

On 2018-05-17 22:33, Lei YU wrote:

> So I think it is better for OpenBMC project to have a common (or 
> example)
> image signing tools/code, not for a specific machine or product, but 
> for the
> general machines in this project using legacy flash layout.
> Let's discuss and get a design proposal?
> 

Yeah, ideally we'd converge the legacy and ubi code update methods into 
one, to take advantage of the Software D-Bus interfaces and have the 
different features like signature verification, filesystem mirroring, 
etc be able to be picked up as separate packages. One starting point for 
the design proposal would be to determine how to separate in the build 
and the repo the different code update methods. Lei, want to take an 
initial stab at it? :)

Re: BMC Image Signing Proposal

[Vernon Mauery]

On 18-May-2018 11:01 AM, Adriana Kobylak wrote:
>On 2018-05-17 22:33, Lei YU wrote:
>
>>So I think it is better for OpenBMC project to have a common (or 
>>example)
>>image signing tools/code, not for a specific machine or product, but 
>>for the
>>general machines in this project using legacy flash layout.
>>Let's discuss and get a design proposal?
>>
>
>Yeah, ideally we'd converge the legacy and ubi code update methods 
>into one, to take advantage of the Software D-Bus interfaces and have 
>the different features like signature verification, filesystem 
>mirroring, etc be able to be picked up as separate packages. One 
>starting point for the design proposal would be to determine how to 
>separate in the build and the repo the different code update methods. 
>Lei, want to take an initial stab at it? :)

As far as the update methods go, just using a descriptive payload goes a 
long way. Historically, using a manifest type thing that told the update 
mechanism which bytes to write to where was pretty simple and very 
effective. The update mechanism would check the authenticity of the 
payload before trusting the manifest, but then the manifest and payload 
would all get written to the flash. This was for specific flash offsets 
and did not fully comprehend the notion of A/B or other redundant 
scenarios, so all the offsets were relative.

But doing something in the manifest so simple as this would work for a 
variety of scenarios:

MANIFEST:
purpose=xyz.openbmc_project.Software.Version.VersionPurpose.BMC
version=v2.2-32-gc6712d3-dirty
KeyType=OpenBMC
HashType=RSA-SHA256

# Expected is a list of files alongside the manifest
Expected=part1,part2,part3,partN

update_pre=<target to activate prior to fwupdate>

# obmc-flash-bmc-by-name knows where to place this part by name
# (e.g. u-boot always goes at 0x00000000)
part1_update=obmc-flash-bmc-by-name@.service

# obmc-flash-bmc-at-offset places a part at a fixed offset
# optionally relative to an optionally specified MTD partition
# (default relative to /dev/mtd0)
part2_update=obmc-flash-bmc-at-offset@.service 0x130000
# (default relative to /dev/mtd0)
part3_update=obmc-flash-bmc-at-offset@.service 0x130000

# not a ubi person, so I can't comment on all the options :)
part4_update=obmc-flash-bmc-ubi@.service

partN_update=<service for updating partN> 

update_post=<target to activate post fwupdate>


The idea is that we can have a series of service files that can handle 
the majority of operations generically and any special cases can supply 
their own. But since it is just a service, the dependencies are all 
taken care of.


Just some thoughts on how this might be done. This is actually something 
that I am working on right now, trying to get our redundant image 
booting working internally, so I am glad to hear that others are working 
toward similar goals.


--Vernon

Re: BMC Image Signing Proposal

[Lei YU]

On Sat, May 19, 2018 at 5:02 AM, Vernon Mauery
<vernon.mauery@linux.intel.com> wrote:
> On 18-May-2018 11:01 AM, Adriana Kobylak wrote:
>>
>> On 2018-05-17 22:33, Lei YU wrote:
>>
>>> So I think it is better for OpenBMC project to have a common (or example)
>>> image signing tools/code, not for a specific machine or product, but for
>>> the
>>> general machines in this project using legacy flash layout.
>>> Let's discuss and get a design proposal?
>>>
>>
>> Yeah, ideally we'd converge the legacy and ubi code update methods into
>> one, to take advantage of the Software D-Bus interfaces and have the
>> different features like signature verification, filesystem mirroring, etc be
>> able to be picked up as separate packages. One starting point for the design
>> proposal would be to determine how to separate in the build and the repo the
>> different code update methods. Lei, want to take an initial stab at it? :)
>
>
> As far as the update methods go, just using a descriptive payload goes a
> long way. Historically, using a manifest type thing that told the update
> mechanism which bytes to write to where was pretty simple and very
> effective. The update mechanism would check the authenticity of the payload
> before trusting the manifest, but then the manifest and payload would all
> get written to the flash. This was for specific flash offsets and did not
> fully comprehend the notion of A/B or other redundant scenarios, so all the
> offsets were relative.
>
> But doing something in the manifest so simple as this would work for a
> variety of scenarios:
>
> MANIFEST:
> purpose=xyz.openbmc_project.Software.Version.VersionPurpose.BMC
> version=v2.2-32-gc6712d3-dirty
> KeyType=OpenBMC
> HashType=RSA-SHA256
>
> # Expected is a list of files alongside the manifest
> Expected=part1,part2,part3,partN
>
> update_pre=<target to activate prior to fwupdate>
>
> # obmc-flash-bmc-by-name knows where to place this part by name
> # (e.g. u-boot always goes at 0x00000000)
> part1_update=obmc-flash-bmc-by-name@.service
>
> # obmc-flash-bmc-at-offset places a part at a fixed offset
> # optionally relative to an optionally specified MTD partition
> # (default relative to /dev/mtd0)
> part2_update=obmc-flash-bmc-at-offset@.service 0x130000
> # (default relative to /dev/mtd0)
> part3_update=obmc-flash-bmc-at-offset@.service 0x130000
>
> # not a ubi person, so I can't comment on all the options :)
> part4_update=obmc-flash-bmc-ubi@.service
>
> partN_update=<service for updating partN>
> update_post=<target to activate post fwupdate>

This is interesting, and it looks like a general method for code updating to
support both non-ubi and ubi layout. (Though it would require code changes in
phosphor-software-manager).

I have several questions though.

Background:
* For non-ubi, there are two ways to do code update:
   1. Copy image-bmc (or image-rofs, etc) to /run/initramfs, and reboot.
      A updater script in initramfs will run to program the image to flash.
      During the code update, BMC is NOT operational;
   2. Invoking "prepareForUpdate" method to ask BMC reboot into a ramfs, then
      invoking the updater script to program the image to flash.
      During the code update, BMC is working.
   3. There is not WebUI support (yet)
* For ubi:
   1. The update happens when BMC is working (due to the fact it assumes the
      flash contains enough space for two images).
   2. The WebUI will upload the image and "activate" it when BMC is running.

So here is my question: to support non-ubifs code update in a general way,
should it do "prepareForUpdate", making BMC to reboot into ramfs, or should it
do the update during reboot?

* If we prefer the "prepareForUpdate" way, then the WebUI should add this
   function to make BMC enter "update" mode;
* If we prefer to update during reboot, then the BMC will not be working for a
   few minutes during the code update, WebUI will not be working as well.

>
>
> The idea is that we can have a series of service files that can handle the
> majority of operations generically and any special cases can supply their
> own. But since it is just a service, the dependencies are all taken care of.
>
>
> Just some thoughts on how this might be done. This is actually something
> that I am working on right now, trying to get our redundant image booting
> working internally, so I am glad to hear that others are working toward
> similar goals.
>
>
> --Vernon

Re: BMC Image Signing Proposal

[Vernon Mauery]

On 22-May-2018 02:46 PM, Lei YU wrote:
>On Sat, May 19, 2018 at 5:02 AM, Vernon Mauery
><vernon.mauery@linux.intel.com> wrote:
>> On 18-May-2018 11:01 AM, Adriana Kobylak wrote:
>>>
>>> On 2018-05-17 22:33, Lei YU wrote:
>>>
>>>> So I think it is better for OpenBMC project to have a common (or example)
>>>> image signing tools/code, not for a specific machine or product, but for
>>>> the
>>>> general machines in this project using legacy flash layout.
>>>> Let's discuss and get a design proposal?
>>>>
>>>
>>> Yeah, ideally we'd converge the legacy and ubi code update methods into
>>> one, to take advantage of the Software D-Bus interfaces and have the
>>> different features like signature verification, filesystem mirroring, etc be
>>> able to be picked up as separate packages. One starting point for the design
>>> proposal would be to determine how to separate in the build and the repo the
>>> different code update methods. Lei, want to take an initial stab at it? :)
>>
>>
>> As far as the update methods go, just using a descriptive payload goes a
>> long way. Historically, using a manifest type thing that told the update
>> mechanism which bytes to write to where was pretty simple and very
>> effective. The update mechanism would check the authenticity of the payload
>> before trusting the manifest, but then the manifest and payload would all
>> get written to the flash. This was for specific flash offsets and did not
>> fully comprehend the notion of A/B or other redundant scenarios, so all the
>> offsets were relative.
>>
>> But doing something in the manifest so simple as this would work for a
>> variety of scenarios:
>>
>> MANIFEST:
>> purpose=xyz.openbmc_project.Software.Version.VersionPurpose.BMC
>> version=v2.2-32-gc6712d3-dirty
>> KeyType=OpenBMC
>> HashType=RSA-SHA256
>>
>> # Expected is a list of files alongside the manifest
>> Expected=part1,part2,part3,partN
>>
>> update_pre=<target to activate prior to fwupdate>
>>
>> # obmc-flash-bmc-by-name knows where to place this part by name
>> # (e.g. u-boot always goes at 0x00000000)
>> part1_update=obmc-flash-bmc-by-name@.service
>>
>> # obmc-flash-bmc-at-offset places a part at a fixed offset
>> # optionally relative to an optionally specified MTD partition
>> # (default relative to /dev/mtd0)
>> part2_update=obmc-flash-bmc-at-offset@.service 0x130000
>> # (default relative to /dev/mtd0)
>> part3_update=obmc-flash-bmc-at-offset@.service 0x130000
>>
>> # not a ubi person, so I can't comment on all the options :)
>> part4_update=obmc-flash-bmc-ubi@.service
>>
>> partN_update=<service for updating partN>
>> update_post=<target to activate post fwupdate>
>
>This is interesting, and it looks like a general method for code updating to
>support both non-ubi and ubi layout. (Though it would require code changes in
>phosphor-software-manager).

That was the idea :)

I think that if we really want to converge on some common code here, 
having the phosphor-software-manager handle several different flash 
formats is important. This is the sort of thing that could easily be 
selectable at build time, one method should have little impact on the 
other.

>I have several questions though.
>
>Background:
>* For non-ubi, there are two ways to do code update:
>   1. Copy image-bmc (or image-rofs, etc) to /run/initramfs, and reboot.
>      A updater script in initramfs will run to program the image to flash.
>      During the code update, BMC is NOT operational;
>   2. Invoking "prepareForUpdate" method to ask BMC reboot into a ramfs, then
>      invoking the updater script to program the image to flash.
>      During the code update, BMC is working.
>   3. There is not WebUI support (yet)
>* For ubi:
>   1. The update happens when BMC is working (due to the fact it assumes the
>      flash contains enough space for two images).
>   2. The WebUI will upload the image and "activate" it when BMC is running.
>
>So here is my question: to support non-ubifs code update in a general way,
>should it do "prepareForUpdate", making BMC to reboot into ramfs, or should it
>do the update during reboot?
>
>* If we prefer the "prepareForUpdate" way, then the WebUI should add this
>   function to make BMC enter "update" mode;

Intel has a requirement to do firmware updates while the BMC is active, 
so we will need to enable this feature in the WebUI. This is not a hard 
problem, given that a redfish mechanism is already in the works.

--Vernon

>* If we prefer to update during reboot, then the BMC will not be working for a
>   few minutes during the code update, WebUI will not be working as well.
>
>>
>>
>> The idea is that we can have a series of service files that can handle the
>> majority of operations generically and any special cases can supply their
>> own. But since it is just a service, the dependencies are all taken care of.
>>
>>
>> Just some thoughts on how this might be done. This is actually something
>> that I am working on right now, trying to get our redundant image booting
>> working internally, so I am glad to hear that others are working toward
>> similar goals.
>>
>>
>> --Vernon

Re: BMC Image Signing Proposal

[Vernon Mauery]

On 22-May-2018 08:30 AM, Vernon Mauery wrote:
>On 22-May-2018 02:46 PM, Lei YU wrote:
>>On Sat, May 19, 2018 at 5:02 AM, Vernon Mauery
>><vernon.mauery@linux.intel.com> wrote:
>>>On 18-May-2018 11:01 AM, Adriana Kobylak wrote:
>>>>
>>>>On 2018-05-17 22:33, Lei YU wrote:
>>>>
>>>>>So I think it is better for OpenBMC project to have a common (or example)
>>>>>image signing tools/code, not for a specific machine or product, but for
>>>>>the
>>>>>general machines in this project using legacy flash layout.
>>>>>Let's discuss and get a design proposal?
>>>>>
>>>>
>>>>Yeah, ideally we'd converge the legacy and ubi code update methods into
>>>>one, to take advantage of the Software D-Bus interfaces and have the
>>>>different features like signature verification, filesystem mirroring, etc be
>>>>able to be picked up as separate packages. One starting point for the design
>>>>proposal would be to determine how to separate in the build and the repo the
>>>>different code update methods. Lei, want to take an initial stab at it? :)
>>>
>>>
>>>As far as the update methods go, just using a descriptive payload goes a
>>>long way. Historically, using a manifest type thing that told the update
>>>mechanism which bytes to write to where was pretty simple and very
>>>effective. The update mechanism would check the authenticity of the payload
>>>before trusting the manifest, but then the manifest and payload would all
>>>get written to the flash. This was for specific flash offsets and did not
>>>fully comprehend the notion of A/B or other redundant scenarios, so all the
>>>offsets were relative.
>>>
>>>But doing something in the manifest so simple as this would work for a
>>>variety of scenarios:
>>>
>>>MANIFEST:
>>>purpose=xyz.openbmc_project.Software.Version.VersionPurpose.BMC
>>>version=v2.2-32-gc6712d3-dirty
>>>KeyType=OpenBMC
>>>HashType=RSA-SHA256
>>>
>>># Expected is a list of files alongside the manifest
>>>Expected=part1,part2,part3,partN
>>>
>>>update_pre=<target to activate prior to fwupdate>
>>>
>>># obmc-flash-bmc-by-name knows where to place this part by name
>>># (e.g. u-boot always goes at 0x00000000)
>>>part1_update=obmc-flash-bmc-by-name@.service
>>>
>>># obmc-flash-bmc-at-offset places a part at a fixed offset
>>># optionally relative to an optionally specified MTD partition
>>># (default relative to /dev/mtd0)
>>>part2_update=obmc-flash-bmc-at-offset@.service 0x130000
>>># (default relative to /dev/mtd0)
>>>part3_update=obmc-flash-bmc-at-offset@.service 0x130000
>>>
>>># not a ubi person, so I can't comment on all the options :)
>>>part4_update=obmc-flash-bmc-ubi@.service
>>>
>>>partN_update=<service for updating partN>
>>>update_post=<target to activate post fwupdate>
>>
>>This is interesting, and it looks like a general method for code updating to
>>support both non-ubi and ubi layout. (Though it would require code changes in
>>phosphor-software-manager).

One other thought I had was that we could make the manifest a JSON file 
which makes for very simple parsing (since the parser is already 
written).  Then we could go with something like this:

{
	'purpose': 'xyz.openbmc_project.Software.Version.VersionPurpose.BMC',
	'version': 'v2.2-32-gc6712d3-dirty',
	'KeyType': 'OpenBMC',
	'HashType': 'RSA-SHA256',
	'ImageParts': [
		'part1': {
			'update': {
				'service': 'obmc-flash-bmc-at-offset@.service','
				'args': [ 0x130000 ]
			}
		},
		'part2': {
			'update': {
				'service': 'obmc-flash-bmc-at-offset@.service','
				'args': [ '/dev/mtd/image-a', 0x80000 ]
			}
		},
		'part3': {
			'update': {
				'service': 'obmc-flash-bmc-ubi@.service',
			}
		}
	]
}

This makes it very clear what all the bits mean. We would need to 
document the format of the JSON very clearly and define behavior for 
malformed structures, but I like this better than just a key/value flat 
text file.

--Vernon

>That was the idea :)
>
>I think that if we really want to converge on some common code here, 
>having the phosphor-software-manager handle several different flash 
>formats is important. This is the sort of thing that could easily be 
>selectable at build time, one method should have little impact on the 
>other.

Re: BMC Image Signing Proposal

[Adriana Kobylak]

On 2018-05-22 13:28, Vernon Mauery wrote:
> 
> One other thought I had was that we could make the manifest a JSON
> file which makes for very simple parsing (since the parser is already
> written).  Then we could go with something like this:
> 

That's a good option, at least for the write to flash piece. We could
even extend the manifest to include the names of the service files to
delete/clean up the flash. Most of the rest of the code manages the
D-Bus objects so that'd be common with all flash layouts.

Another option, or combination with a json manifest, would be to have
different repos or different subdirectories for specific 
implementations.


Lei, thinking we could convert Romulus to ubi, and use the PNOR chip
to store the alternate BMC version. I think that'd be more straight fwd
and the advantage would be that the interfaces are tested and verified.
And on the side we can continue this discussion on how to make the
code more modular to support other layouts and we can start making
the changes but at least we can get Romulus using signature validation
in the mean time.

Re: BMC Image Signing Proposal

[Vernon Mauery]

On 24-May-2018 12:12 PM, Adriana Kobylak wrote:
>On 2018-05-22 13:28, Vernon Mauery wrote:
>>
>>One other thought I had was that we could make the manifest a JSON
>>file which makes for very simple parsing (since the parser is already
>>written).  Then we could go with something like this:
>>
>
>That's a good option, at least for the write to flash piece. We could
>even extend the manifest to include the names of the service files to
>delete/clean up the flash. Most of the rest of the code manages the
>D-Bus objects so that'd be common with all flash layouts.
>
>Another option, or combination with a json manifest, would be to have
>different repos or different subdirectories for specific 
>implementations.

I would prefer not to need to split the implementation, but that may not 
always be possible.

One thing along this line of thinking is that Intel's BMCs don't have 
the notion of field mode, which is baked into the software manager. So 
it might be helpful to have some way to deal with this.

One thought was that external service files executed from the manifest 
could deal with this (keep the OEM service files in external 
repositories).

--Vernon

>
>Lei, thinking we could convert Romulus to ubi, and use the PNOR chip
>to store the alternate BMC version. I think that'd be more straight fwd
>and the advantage would be that the interfaces are tested and verified.
>And on the side we can continue this discussion on how to make the
>code more modular to support other layouts and we can start making
>the changes but at least we can get Romulus using signature validation
>in the mean time.
>

Re: BMC Image Signing Proposal

[Lei YU]

On Fri, May 25, 2018 at 1:12 AM, Adriana Kobylak <anoo@linux.ibm.com> wrote:
> On 2018-05-22 13:28, Vernon Mauery wrote:
>>
>>
>> One other thought I had was that we could make the manifest a JSON
>> file which makes for very simple parsing (since the parser is already
>> written).  Then we could go with something like this:
>>
>
> That's a good option, at least for the write to flash piece. We could
> even extend the manifest to include the names of the service files to
> delete/clean up the flash. Most of the rest of the code manages the
> D-Bus objects so that'd be common with all flash layouts.
>
> Another option, or combination with a json manifest, would be to have
> different repos or different subdirectories for specific implementations.
>
>
> Lei, thinking we could convert Romulus to ubi, and use the PNOR chip
> to store the alternate BMC version. I think that'd be more straight fwd
> and the advantage would be that the interfaces are tested and verified.
> And on the side we can continue this discussion on how to make the
> code more modular to support other layouts and we can start making
> the changes but at least we can get Romulus using signature validation
> in the mean time.

By default Romulus has a single 32MiB for BMC and 64MiB for PNOR, there is not
enough room for Romulus to enable ubi with alternative BMC.
So I am afraid that it can not enable ubifs.

But I do have enabled image signature verification on fixed layout and tested
on Romulus.

The related changes are:
https://gerrit.openbmc-project.xyz/#/c/10765/
https://gerrit.openbmc-project.xyz/#/c/10801/
https://gerrit.openbmc-project.xyz/#/c/10768/
https://gerrit.openbmc-project.xyz/#/c/10800/

I have tested on Romulus that:
* Code update successful with valid image, both REST API and in WebUI;
* Code update failure with invalid image (signature verify failure), if
   FieldMode is enabled;

I you think that is OK, we can merge the changes, as it adds the new features
(code update and signature verify in phosphor-software-manager) without
breaking existing services.
It could be a start point for phosphor-software-manager to handle both ubifs
and non-ubifs.

Then we can continue to work on enhancing/refactoring to make it more generic
as Vernon suggested.