BMC update via TFTP

BMC update via TFTP

[rgrs]

[-- Attachment #1: Type: text/plain, Size: 289 bytes --]

Hi,

In BMC WebUI under "Download image file from TFTP server" section,
we have text fields "TFTP Server IP address" and "File name".
"File name" doesn't take folders in path. Is this a bug or expected behavior?

TFTP downloads work only if file is kept in root of tftp share.

Thanks,
Raj

[-- Attachment #2: Type: text/html, Size: 652 bytes --]

Re: BMC update via TFTP

[Gunnar Mills]

> In BMC WebUI under "Download image file from TFTP server" section,
> we have text fields "TFTP Server IP address" and "File name".
> "File name" doesn't take folders in path. Is this a bug or expected 
> behavior?
>
> TFTP downloads work only if file is kept in root of tftp share.
>
This is expected.
https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33

Re: BMC update via TFTP

[Gunnar Mills]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]


On 12/3/2019 10:12 AM, Gunnar Mills wrote:
>
>> In BMC WebUI under "Download image file from TFTP server" section,
>> we have text fields "TFTP Server IP address" and "File name".
>> "File name" doesn't take folders in path. Is this a bug or expected 
>> behavior?
>>
>> TFTP downloads work only if file is kept in root of tftp share.
>>
> This is expected.
> https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 
>

As long as we continue to sanitize the local filename, I don't see why 
we need to limit the source file name.
Patrick, Adriana, Any objection? Remember why we wrote it this way? I 
assume for simplicity..?

Thanks,
Gunnar


[-- Attachment #2: Type: text/html, Size: 1588 bytes --]

Re: BMC update via TFTP

[Adriana Kobylak]

On 2019-12-03 11:08, Gunnar Mills wrote:
> On 12/3/2019 10:12 AM, Gunnar Mills wrote:
> 
>>> In BMC WebUI under "Download image file from TFTP server" section,
>>> 
>>> we have text fields "TFTP Server IP address" and "File name".
>>> "File name" doesn't take folders in path. Is this a bug or
>>> expected behavior?
>>> 
>>> TFTP downloads work only if file is kept in root of tftp share.
>> This is expected.
>> 
> https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33
>> [1]
> 
> As long as we continue to sanitize the local filename, I don't see why
> we need to limit the source file name.
>  Patrick, Adriana, Any objection? Remember why we wrote it this way? I
> assume for simplicity..?

It was done for security to prevent users from specifying a file outside 
the tftp directory, such as /mydir/../root/system-file.
But seems the current file name handling breaks the ability to get files 
from a subdir like Raj pointed out, we should be able to fix that out, 
tftp supports having a file in subdirs such as 
/tftpboot/subdirectory/file, and passing /subdirectory/file to it as the 
path.

> 
>  Thanks,
>  Gunnar
> 
> 
> Links:
> ------
> [1]
> https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33

Re: BMC update via TFTP

[Joseph Reynolds]

On 12/3/19 1:55 PM, Adriana Kobylak wrote:
> On 2019-12-03 11:08, Gunnar Mills wrote:
>> On 12/3/2019 10:12 AM, Gunnar Mills wrote:
>>
>>>> In BMC WebUI under "Download image file from TFTP server" section,
>>>>
>>>> we have text fields "TFTP Server IP address" and "File name".
>>>> "File name" doesn't take folders in path. Is this a bug or
>>>> expected behavior?
>>>>
>>>> TFTP downloads work only if file is kept in root of tftp share.
>>> This is expected.
>>>
>> https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 
>>
>>> [1]
>>
>> As long as we continue to sanitize the local filename, I don't see why
>> we need to limit the source file name.
>>  Patrick, Adriana, Any objection? Remember why we wrote it this way? I
>> assume for simplicity..?
>
> It was done for security to prevent users from specifying a file 
> outside the tftp directory, such as /mydir/../root/system-file.
> But seems the current file name handling breaks the ability to get 
> files from a subdir like Raj pointed out, we should be able to fix 
> that out, tftp supports having a file in subdirs such as 
> /tftpboot/subdirectory/file, and passing /subdirectory/file to it as 
> the path.

It seems to me that the burden must be on the TFTP server to protect 
itself.  See the "Security Considerations" section of RFC 1350 - 
https://tools.ietf.org/html/rfc1350
Specifically, OpenBMC only needs read access (it does not need to write 
files to the TFTP server) and only needs to be able to access files the 
TFTP server intends to provide (typically: all the files under a 
specific directory).

To be more clear:
- OpenBMC does not provide a TFTP server, so this is guidance for 
someone setting a TFTP server for use with an OpenBMC system
- The TFTP server only needs to offer READ access.
- The TFTP server ought to prevent clients from using 
cleverly-constructed pathnames (for example, any file which begins with 
"/" or contains an ".." element) to access files outside of its sandbox.
- That said, I am not a TFTP expert, and whichever TFTP server is used, 
its security guidance should be followed.

What all of the above, I think it would be okay for OpenBMC to validate 
the pathname it sends to the TFTP server, but don't think it is necessary.

Finally, I hope we can move away from TFTP and more toward a secure 
image distribution approach.
Note that TFTP is disabled in OpenBMC by default.  See 
https://github.com/openbmc/bmcweb/blob/master/CMakeLists.txt

- Joseph

>>
>>  Thanks,
>>  Gunnar
>>
>>
>> Links:
>> ------
>> [1]
>> https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 
>>

Re: BMC update via TFTP

[Vernon Mauery]

On 04-Dec-2019 09:47 AM, Joseph Reynolds wrote:
>On 12/3/19 1:55 PM, Adriana Kobylak wrote:
>>On 2019-12-03 11:08, Gunnar Mills wrote:
>>>On 12/3/2019 10:12 AM, Gunnar Mills wrote:
>>>
>>>>>In BMC WebUI under "Download image file from TFTP server" section,
>>>>>
>>>>>we have text fields "TFTP Server IP address" and "File name".
>>>>>"File name" doesn't take folders in path. Is this a bug or
>>>>>expected behavior?
>>>>>
>>>>>TFTP downloads work only if file is kept in root of tftp share.
>>>>This is expected.
>>>>
>>>https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33
>>>
>>>>[1]
>>>
>>>As long as we continue to sanitize the local filename, I don't see why
>>>we need to limit the source file name.
>>> Patrick, Adriana, Any objection? Remember why we wrote it this way? I
>>>assume for simplicity..?
>>
>>It was done for security to prevent users from specifying a file 
>>outside the tftp directory, such as /mydir/../root/system-file.
>>But seems the current file name handling breaks the ability to get 
>>files from a subdir like Raj pointed out, we should be able to fix 
>>that out, tftp supports having a file in subdirs such as 
>>/tftpboot/subdirectory/file, and passing /subdirectory/file to it as 
>>the path.

While we are on the topic of security, should we do something about 
unsecured protocols for image transfers? Adding an HMAC and key along 
with the IP/path would then make sure the file that was fetched contains 
the contents that were expected.

Even if the BMC only accepts signed images, we want to make sure that 
the signed image the BMC fetches is the one that the administrator 
*wants* to be fetched. With tftp or http (or any insecure transport), 
one possible MiTM attack would be to substitute an alternate valid 
image. Let's say the admin wants to go from 1.18 to 1.20, bu the 
attacker wants to go to 1.16, which has a known vulnerability. The 
image would be authenticated by the signature, and will be accepted.

With an added HMAC, the MiTM attack would be thwarted. The HMAC and key 
would be transmitted to the BMC over a secure channel (https) so the 
attacker would not be able create a suitable replacement for the image 
in the insecure channel.

--Vernon

>It seems to me that the burden must be on the TFTP server to protect 
>itself.  See the "Security Considerations" section of RFC 1350 - 
>https://tools.ietf.org/html/rfc1350
>Specifically, OpenBMC only needs read access (it does not need to 
>write files to the TFTP server) and only needs to be able to access 
>files the TFTP server intends to provide (typically: all the files 
>under a specific directory).
>
>To be more clear:
>- OpenBMC does not provide a TFTP server, so this is guidance for 
>someone setting a TFTP server for use with an OpenBMC system
>- The TFTP server only needs to offer READ access.
>- The TFTP server ought to prevent clients from using 
>cleverly-constructed pathnames (for example, any file which begins 
>with "/" or contains an ".." element) to access files outside of its 
>sandbox.
>- That said, I am not a TFTP expert, and whichever TFTP server is 
>used, its security guidance should be followed.
>
>What all of the above, I think it would be okay for OpenBMC to 
>validate the pathname it sends to the TFTP server, but don't think it 
>is necessary.
>
>Finally, I hope we can move away from TFTP and more toward a secure 
>image distribution approach.
>Note that TFTP is disabled in OpenBMC by default.  See 
>https://github.com/openbmc/bmcweb/blob/master/CMakeLists.txt
>
>- Joseph
>
>>>
>>> Thanks,
>>> Gunnar
>>>
>>>
>>>Links:
>>>------
>>>[1]
>>>https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33
>>>
>

Re: BMC update via TFTP

[Alexander Tereschenko]

On 04-Dec-19 22:36, Vernon Mauery wrote:
> Even if the BMC only accepts signed images, we want to make sure that 
> the signed image the BMC fetches is the one that the administrator 
> *wants* to be fetched. With tftp or http (or any insecure transport), 
> one possible MiTM attack would be to substitute an alternate valid 
> image. Let's say the admin wants to go from 1.18 to 1.20, bu the 
> attacker wants to go to 1.16, which has a known vulnerability. The 
> image would be authenticated by the signature, and will be accepted. 

Thanks Vernon for raising this one - I think this is indeed a valid concern.

So there are essentially two things that we are talking about here, if 
we peel it a little bit:

1) an additional authorization by the BMC admin of specific images they 
want to run
2) plain integrity protection of the file being sent over insecure channel

The first one is a bit different from the second one (and frequently 
solved by co-signing these days), but in thise specific use case I'd say 
these two indeed overlap.

In this case, I'd rather suggest us to look into asymmetric crypto, so 
that no secret sharing/storage (with accompanying additional risk of 
compromise) is necessary. An admin would sign the image being sent + 
some context information (like time, to provide replay protection - also 
applicable to MAC case, otherwise the attacker can trivially replay the 
file/MAC) and the BMC would check the admin's signature using 
pre-provisioned public key (send over HTTPS, same as MAC key in your 
proposal - but only for integrity protection, con confidentiality is 
necessary) + verify that the context is "fresh" and then use the file.

I'd be happy to help out with elaborating details of the scheme, if 
that's deemed interesting enough for wider audience.

regards,
Alexander

Re: BMC update via TFTP

[Vernon Mauery]

On 05-Dec-2019 12:05 PM, Alexander Tereschenko wrote:
>On 04-Dec-19 22:36, Vernon Mauery wrote:
>>Even if the BMC only accepts signed images, we want to make sure 
>>that the signed image the BMC fetches is the one that the 
>>administrator *wants* to be fetched. With tftp or http (or any 
>>insecure transport), one possible MiTM attack would be to substitute 
>>an alternate valid image. Let's say the admin wants to go from 1.18 
>>to 1.20, bu the attacker wants to go to 1.16, which has a known 
>>vulnerability. The image would be authenticated by the signature, 
>>and will be accepted.
>
>Thanks Vernon for raising this one - I think this is indeed a valid concern.
>
>So there are essentially two things that we are talking about here, if 
>we peel it a little bit:
>
>1) an additional authorization by the BMC admin of specific images 
>they want to run
>2) plain integrity protection of the file being sent over insecure channel
>
>The first one is a bit different from the second one (and frequently 
>solved by co-signing these days), but in thise specific use case I'd 
>say these two indeed overlap.
>
>In this case, I'd rather suggest us to look into asymmetric crypto, so 
>that no secret sharing/storage (with accompanying additional risk of 
>compromise) is necessary. An admin would sign the image being sent + 
>some context information (like time, to provide replay protection - 
>also applicable to MAC case, otherwise the attacker can trivially 
>replay the file/MAC) and the BMC would check the admin's signature 
>using pre-provisioned public key (send over HTTPS, same as MAC key in 
>your proposal - but only for integrity protection, con confidentiality 
>is necessary) + verify that the context is "fresh" and then use the 
>file.

I am not convinced that it needs to be this elaborate. I don't see what 
it adds over the simple case of sending the key/hmac/uri encrypted with 
TLS to the BMC. There will be no replay attacks because TLS prevents it. 

Maybe I am missing something?

--Vernon

>I'd be happy to help out with elaborating details of the scheme, if 
>that's deemed interesting enough for wider audience.

Re: BMC update via TFTP

[Alexander Tereschenko]

On 05-Dec-19 23:37, Vernon Mauery wrote:
> On 05-Dec-2019 12:05 PM, Alexander Tereschenko wrote:
>> On 04-Dec-19 22:36, Vernon Mauery wrote:
>>> Even if the BMC only accepts signed images, we want to make sure 
>>> that the signed image the BMC fetches is the one that the 
>>> administrator *wants* to be fetched. With tftp or http (or any 
>>> insecure transport), one possible MiTM attack would be to substitute 
>>> an alternate valid image. Let's say the admin wants to go from 1.18 
>>> to 1.20, bu the attacker wants to go to 1.16, which has a known 
>>> vulnerability. The image would be authenticated by the signature, 
>>> and will be accepted.
>>
>> Thanks Vernon for raising this one - I think this is indeed a valid 
>> concern.
>>
>> So there are essentially two things that we are talking about here, 
>> if we peel it a little bit:
>>
>> 1) an additional authorization by the BMC admin of specific images 
>> they want to run
>> 2) plain integrity protection of the file being sent over insecure 
>> channel
>>
>> The first one is a bit different from the second one (and frequently 
>> solved by co-signing these days), but in thise specific use case I'd 
>> say these two indeed overlap.
>>
>> In this case, I'd rather suggest us to look into asymmetric crypto, 
>> so that no secret sharing/storage (with accompanying additional risk 
>> of compromise) is necessary. An admin would sign the image being sent 
>> + some context information (like time, to provide replay protection - 
>> also applicable to MAC case, otherwise the attacker can trivially 
>> replay the file/MAC) and the BMC would check the admin's signature 
>> using pre-provisioned public key (send over HTTPS, same as MAC key in 
>> your proposal - but only for integrity protection, con 
>> confidentiality is necessary) + verify that the context is "fresh" 
>> and then use the file.
>
> I am not convinced that it needs to be this elaborate. I don't see 
> what it adds over the simple case of sending the key/hmac/uri 
> encrypted with TLS to the BMC. There will be no replay attacks because 
> TLS prevents it.
> Maybe I am missing something?

We may be talking about slightly different contexts indeed. I meant to 
suggest something that doesn't require

1) any "out-of-band" channel for each file being sent (signature can be 
appended to the file itself, one-time/infrequent public key updates at 
the BMC side are discounted as arguably negligible). I use the 
"out-of-band" here as "channel different than the one where the file 
itself is being sent";
2) any symmetric secret generation/storage/distribution upon sending 
files (which is an additional burden and risk);

plus

1) provides replay protection;
2) doesn't require major infrastructure modifications (here I'm assuming 
that signing and checking the signature in this flow is less change than 
generating key, MAC and sending them over "out-of-band" channel);

I read your proposal as that the key will be a longer-term one + that 
the MAC will be sent along with the file (as it's not a secret and 
typically it's sent together with the data in such schemes). If you 
meant that both the key and MAC are generated on the fly, are one-time 
and are sent via that "out-of-band" TLS-protected channel, then I'd say 
you don't even need a keyed MAC at all and a plain cryptographic hash 
would suffice. You're sending the key along with the MAC and file 
channel itself is unprotected - so if the adversary can break the TLS, 
they can compute the MAC as easily as a plain hash and if they can't, 
they anyway can't substitute the hash value, so IMHO keyed MAC is 
surplus here.

Maybe outlining a flow (maybe as a sequence/flow diagram) in more detail 
and defining which specific adversaries (and their capabilities) we're 
talking about here would help? I agree we surely don't need a more 
complex scheme if a simpler one suffices, it's just that we may be 
talking about slightly different flows and adversaries right now :)

regards,
Alexander

Re: BMC update via TFTP

[Joseph Reynolds]

On 12/6/19 8:03 AM, Alexander Tereschenko wrote:
> On 05-Dec-19 23:37, Vernon Mauery wrote:
>> On 05-Dec-2019 12:05 PM, Alexander Tereschenko wrote:
>>> On 04-Dec-19 22:36, Vernon Mauery wrote:
>>>> Even if the BMC only accepts signed images, we want to make sure 
>>>> that the signed image the BMC fetches is the one that the 
>>>> administrator *wants* to be fetched. With tftp or http (or any 
>>>> insecure transport), one possible MiTM attack would be to 
>>>> substitute an alternate valid image. Let's say the admin wants to 
>>>> go from 1.18 to 1.20, bu the attacker wants to go to 1.16, which 
>>>> has a known vulnerability. The image would be authenticated by the 
>>>> signature, and will be accepted.
...snip...
>> I am not convinced that it needs to be this elaborate. I don't see 
>> what it adds over the simple case of sending the key/hmac/uri 
>> encrypted with TLS to the BMC. There will be no replay attacks 
>> because TLS prevents it.
>> Maybe I am missing something?
>
> We may be talking about slightly different contexts indeed. I meant to 
> suggest something that doesn't require
>
...snip...

I was thinking along the lines of adding [SFTP][] (or SCP) support and 
then migrating existing TFTP users to the new secure solution.

That is, the BMC admin performing [code update][] can currently get a 
firmware image via POST DownloadViaTFTP to URI 
/xyz/openbmc_project/software.
My idea is to offer a DownloadViaSFTP method (or preferably a Redfish 
API) for this.  Note that the TFTP download is disabled by default per 
[bmcweb config][].

Once OpenBMC supports downloading firmware via SFTP, we can encourage 
our users to set up their SFTP servers and take down their TFTP 
servers.  I realize that sounds easy, but I don't have a feeling how 
difficult that would be in practice.

Does that sound feasible?

- Joseph

[SFTP]: https://man.openbsd.org/sftp-server
[code update]: 
https://github.com/openbmc/docs/blob/master/code-update/code-update.md
[bmcweb config]: 
https://github.com/openbmc/bmcweb/blob/41d1d1833f476766f88cfb624e66eef7906bdf8c/CMakeLists.txt#L98

> regards,
> Alexander
>
>

Re: BMC update via TFTP

[Alexander Tereschenko]

On 06-Dec-19 23:52, Joseph Reynolds wrote:
> I was thinking along the lines of adding [SFTP][] (or SCP) support and 
> then migrating existing TFTP users to the new secure solution.
>
> That is, the BMC admin performing [code update][] can currently get a 
> firmware image via POST DownloadViaTFTP to URI 
> /xyz/openbmc_project/software.
> My idea is to offer a DownloadViaSFTP method (or preferably a Redfish 
> API) for this.  Note that the TFTP download is disabled by default per 
> [bmcweb config][].
>
> Once OpenBMC supports downloading firmware via SFTP, we can encourage 
> our users to set up their SFTP servers and take down their TFTP 
> servers.  I realize that sounds easy, but I don't have a feeling how 
> difficult that would be in practice.
>
> Does that sound feasible?
>
> - Joseph
>
> [SFTP]: https://man.openbsd.org/sftp-server
> [code update]: 
> https://github.com/openbmc/docs/blob/master/code-update/code-update.md
> [bmcweb config]: 
> https://github.com/openbmc/bmcweb/blob/41d1d1833f476766f88cfb624e66eef7906bdf8c/CMakeLists.txt#L98

Yes, that could be a solution for the problem we discuss, providing both 
integrity and confidentiality, without any major OpenBMC development 
necessary - but it would mean more operational burden for BMC admins. 
The problem with SCP/SFTP in this context is that for this to work in 
the same manner as TFTP, the BMC must be an SSH client - i.e. have some 
sort of identity/credentials for the SCP/SFTP server provisioned first. 
That might not be the easiest solution to setup, but it's of course 
possible and can be automated if OpenBMC provides respective config knobs.

Existing ways we have in code-update.md either don't require credentials 
(TFTP), so being a client is easy, or are not making a "client" from 
BMC, it's the admin who uploads stuff (SCP/REST).

regards,
Alexander

Re: BMC update via TFTP

[Joseph Reynolds]

On 12/9/19 10:06 AM, Alexander Tereschenko wrote:
> On 06-Dec-19 23:52, Joseph Reynolds wrote:
>> I was thinking along the lines of adding [SFTP][] (or SCP) support 
>> and then migrating existing TFTP users to the new secure solution.
>>
>> That is, the BMC admin performing [code update][] can currently get a 
>> firmware image via POST DownloadViaTFTP to URI 
>> /xyz/openbmc_project/software.
>> My idea is to offer a DownloadViaSFTP method (or preferably a Redfish 
>> API) for this.  Note that the TFTP download is disabled by default 
>> per [bmcweb config][].
>>
>> Once OpenBMC supports downloading firmware via SFTP, we can encourage 
>> our users to set up their SFTP servers and take down their TFTP 
>> servers.  I realize that sounds easy, but I don't have a feeling how 
>> difficult that would be in practice.
>>
>> Does that sound feasible?
>>
>> - Joseph
>>
>> [SFTP]: https://man.openbsd.org/sftp-server
>> [code update]: 
>> https://github.com/openbmc/docs/blob/master/code-update/code-update.md
>> [bmcweb config]: 
>> https://github.com/openbmc/bmcweb/blob/41d1d1833f476766f88cfb624e66eef7906bdf8c/CMakeLists.txt#L98
>
> Yes, that could be a solution for the problem we discuss, providing 
> both integrity and confidentiality, without any major OpenBMC 
> development necessary - but it would mean more operational burden for 
> BMC admins. The problem with SCP/SFTP in this context is that for this 
> to work in the same manner as TFTP, the BMC must be an SSH client - 
> i.e. have some sort of identity/credentials for the SCP/SFTP server 
> provisioned first. That might not be the easiest solution to setup, 
> but it's of course possible and can be automated if OpenBMC provides 
> respective config knobs.
>
> Existing ways we have in code-update.md either don't require 
> credentials (TFTP), so being a client is easy, or are not making a 
> "client" from BMC, it's the admin who uploads stuff (SCP/REST).

Yes, that's what I was thinking.  (And no, I am not going to recommend 
setting up a SCP or SFTP server that allows anonymous access.)

This highlight the need for OpenBMC to put together a guide to 
provisioning your BMC.    Such as guide would give us a place to talk 
about uploading to the BMC SSH client certificates needed to access and 
download the firmware images.

- Joseph

>
> regards,
> Alexander
>

RE: [EXTERNAL] Re: BMC update via TFTP

[Neeraj Ladkani]

Are there any thoughts to get rid of BMC reset to trigger FW update? I understand FW reset is required after the update. 


-----Original Message-----
From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> On Behalf Of Joseph Reynolds
Sent: Monday, December 9, 2019 5:25 PM
To: Alexander Tereschenko <aleksandr.v.tereschenko@linux.intel.com>; openbmc@lists.ozlabs.org
Subject: [EXTERNAL] Re: BMC update via TFTP

On 12/9/19 10:06 AM, Alexander Tereschenko wrote:
> On 06-Dec-19 23:52, Joseph Reynolds wrote:
>> I was thinking along the lines of adding [SFTP][] (or SCP) support 
>> and then migrating existing TFTP users to the new secure solution.
>>
>> That is, the BMC admin performing [code update][] can currently get a 
>> firmware image via POST DownloadViaTFTP to URI 
>> /xyz/openbmc_project/software.
>> My idea is to offer a DownloadViaSFTP method (or preferably a Redfish
>> API) for this.  Note that the TFTP download is disabled by default 
>> per [bmcweb config][].
>>
>> Once OpenBMC supports downloading firmware via SFTP, we can encourage 
>> our users to set up their SFTP servers and take down their TFTP 
>> servers.  I realize that sounds easy, but I don't have a feeling how 
>> difficult that would be in practice.
>>
>> Does that sound feasible?
>>
>> - Joseph
>>
>> [SFTP]: 
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fman
>> .openbsd.org%2Fsftp-server&amp;data=02%7C01%7Cneladk%40microsoft.com%
>> 7C9cc71f33a3014260e36f08d77d0fe11b%7C72f988bf86f141af91ab2d7cd011db47
>> %7C1%7C0%7C637115379469052876&amp;sdata=Zj%2BjAlaXlyeBkTsl7MvtbPoSeH7
>> az%2FAJS1UxXeCy0Pc%3D&amp;reserved=0
>> [code update]: 
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
>> hub.com%2Fopenbmc%2Fdocs%2Fblob%2Fmaster%2Fcode-update%2Fcode-update.
>> md&amp;data=02%7C01%7Cneladk%40microsoft.com%7C9cc71f33a3014260e36f08
>> d77d0fe11b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6371153794690
>> 52876&amp;sdata=ScV14ytcPCYn%2BlI%2B9lPgkgKY4yVh%2BrwMVgdbnB0h5z4%3D&
>> amp;reserved=0
>> [bmcweb config]: 
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
>> hub.com%2Fopenbmc%2Fbmcweb%2Fblob%2F41d1d1833f476766f88cfb624e66eef79
>> 06bdf8c%2FCMakeLists.txt%23L98&amp;data=02%7C01%7Cneladk%40microsoft.
>> com%7C9cc71f33a3014260e36f08d77d0fe11b%7C72f988bf86f141af91ab2d7cd011
>> db47%7C1%7C0%7C637115379469052876&amp;sdata=wkV8x6Ce1A0Wf%2FMN%2F%2B9
>> FDSYwQ9z47YI6bc6RwqrTcEg%3D&amp;reserved=0
>
> Yes, that could be a solution for the problem we discuss, providing 
> both integrity and confidentiality, without any major OpenBMC 
> development necessary - but it would mean more operational burden for 
> BMC admins. The problem with SCP/SFTP in this context is that for this 
> to work in the same manner as TFTP, the BMC must be an SSH client - 
> i.e. have some sort of identity/credentials for the SCP/SFTP server 
> provisioned first. That might not be the easiest solution to setup, 
> but it's of course possible and can be automated if OpenBMC provides 
> respective config knobs.
>
> Existing ways we have in code-update.md either don't require 
> credentials (TFTP), so being a client is easy, or are not making a 
> "client" from BMC, it's the admin who uploads stuff (SCP/REST).

Yes, that's what I was thinking.  (And no, I am not going to recommend setting up a SCP or SFTP server that allows anonymous access.)

This highlight the need for OpenBMC to put together a guide to provisioning your BMC.    Such as guide would give us a place to talk about uploading to the BMC SSH client certificates needed to access and download the firmware images.

- Joseph

>
> regards,
> Alexander
>

Re: [EXTERNAL] Re: BMC update via TFTP

[Joseph Reynolds]

On 12/10/19 12:58 PM, Neeraj Ladkani wrote:
> Are there any thoughts to get rid of BMC reset to trigger FW update? I understand FW reset is required after the update.

I'm not sure I understand the question.  I think the answer depends on 
the [Software.VersionPurpose][1].
For VersionPurpose=BMC or System, the BMC must be reset.
For VersionPurpose=Host, PSU, or Other, I don't know why the BMC would 
need to be reset.

Do you want to be able to update non-BMC firmware without having to 
reset the BMC?

- Joseph

[1]: 
https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/Version.interface.yaml

>
> -----Original Message-----
> From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> On Behalf Of Joseph Reynolds
> Sent: Monday, December 9, 2019 5:25 PM
> To: Alexander Tereschenko <aleksandr.v.tereschenko@linux.intel.com>; openbmc@lists.ozlabs.org
> Subject: [EXTERNAL] Re: BMC update via TFTP
>
> On 12/9/19 10:06 AM, Alexander Tereschenko wrote:
>> On 06-Dec-19 23:52, Joseph Reynolds wrote:
>>> I was thinking along the lines of adding [SFTP][] (or SCP) support
>>> and then migrating existing TFTP users to the new secure solution.

[...snip...]

>> Yes, that could be a solution for the problem we discuss, providing
>> both integrity and confidentiality, without any major OpenBMC
>> development necessary - but it would mean more operational burden for
>> BMC admins. The problem with SCP/SFTP in this context is that for this
>> to work in the same manner as TFTP, the BMC must be an SSH client -
>> i.e. have some sort of identity/credentials for the SCP/SFTP server
>> provisioned first. That might not be the easiest solution to setup,
>> but it's of course possible and can be automated if OpenBMC provides
>> respective config knobs.
>>
>> Existing ways we have in code-update.md either don't require
>> credentials (TFTP), so being a client is easy, or are not making a
>> "client" from BMC, it's the admin who uploads stuff (SCP/REST).
> Yes, that's what I was thinking.  (And no, I am not going to recommend setting up a SCP or SFTP server that allows anonymous access.)
>
> This highlight the need for OpenBMC to put together a guide to provisioning your BMC.    Such as guide would give us a place to talk about uploading to the BMC SSH client certificates needed to access and download the firmware images.
>
> - Joseph
>
>> regards,
>> Alexander
>>

RE: [EXTERNAL] Re: BMC update via TFTP

[Bonnie Lo/WYHQ/Wiwynn]

Dear Joseph,

In my understanding, the BMC firmware update flow is as below:
1. Trigger reboot 
2. Systemd stop all service
3. Unmount file system
4. image is in /run/initramfs 
5. Do the flashcp command to update the flash 

If there is any misunderstanding, please correct me.

Based on the discussion with Neeraj.
We want to be able to update BMC firmware without having to trigger the BMC reboot command before the system do flashcp command.
It means that we can do the flashcp first. If the flashcp command complete and success, then we do the reset manually.
Is it workable on current upstream code?
If not, why? I means is there any advantage to trigger the reboot before we do the flashcp.

Thanks,
Bonnie

-----Original Message-----
From: openbmc <openbmc-bounces+bonnie_lo=wiwynn.com@lists.ozlabs.org> On Behalf Of Joseph Reynolds
Sent: Wednesday, December 11, 2019 5:11 AM
To: Neeraj Ladkani <neladk@microsoft.com>; Alexander Tereschenko <aleksandr.v.tereschenko@linux.intel.com>; openbmc@lists.ozlabs.org; Patrick Venture <venture@google.com>
Subject: Re: [EXTERNAL] Re: BMC update via TFTP

On 12/10/19 12:58 PM, Neeraj Ladkani wrote:
> Are there any thoughts to get rid of BMC reset to trigger FW update? I understand FW reset is required after the update.

I'm not sure I understand the question.  I think the answer depends on the [Software.VersionPurpose][1].
For VersionPurpose=BMC or System, the BMC must be reset.
For VersionPurpose=Host, PSU, or Other, I don't know why the BMC would need to be reset.

Do you want to be able to update non-BMC firmware without having to reset the BMC?

- Joseph

[1]: 
https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenbmc%2Fphosphor-dbus-interfaces%2Fblob%2Fmaster%2Fxyz%2Fopenbmc_project%2FSoftware%2FVersion.interface.yaml&amp;data=02%7C01%7CBonnie_Lo%40wiwynn.com%7C9d56d702cea54e52e29808d77db630a1%7Cde0795e0d7c04eebb9bbbc94d8980d3b%7C1%7C0%7C637116093768947793&amp;sdata=vAh3kDz2onq8b0%2Flx1AyPNyFngyPLag7go%2Fruw9nEtU%3D&amp;reserved=0

>
> -----Original Message-----
> From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> 
> On Behalf Of Joseph Reynolds
> Sent: Monday, December 9, 2019 5:25 PM
> To: Alexander Tereschenko <aleksandr.v.tereschenko@linux.intel.com>; 
> openbmc@lists.ozlabs.org
> Subject: [EXTERNAL] Re: BMC update via TFTP
>
> On 12/9/19 10:06 AM, Alexander Tereschenko wrote:
>> On 06-Dec-19 23:52, Joseph Reynolds wrote:
>>> I was thinking along the lines of adding [SFTP][] (or SCP) support 
>>> and then migrating existing TFTP users to the new secure solution.

[...snip...]

>> Yes, that could be a solution for the problem we discuss, providing 
>> both integrity and confidentiality, without any major OpenBMC 
>> development necessary - but it would mean more operational burden for 
>> BMC admins. The problem with SCP/SFTP in this context is that for 
>> this to work in the same manner as TFTP, the BMC must be an SSH 
>> client - i.e. have some sort of identity/credentials for the SCP/SFTP 
>> server provisioned first. That might not be the easiest solution to 
>> setup, but it's of course possible and can be automated if OpenBMC 
>> provides respective config knobs.
>>
>> Existing ways we have in code-update.md either don't require 
>> credentials (TFTP), so being a client is easy, or are not making a 
>> "client" from BMC, it's the admin who uploads stuff (SCP/REST).
> Yes, that's what I was thinking.  (And no, I am not going to recommend 
> setting up a SCP or SFTP server that allows anonymous access.)
>
> This highlight the need for OpenBMC to put together a guide to provisioning your BMC.    Such as guide would give us a place to talk about uploading to the BMC SSH client certificates needed to access and download the firmware images.
>
> - Joseph
>
>> regards,
>> Alexander
>>

Re: [EXTERNAL] Re: BMC update via TFTP

[Lei YU]

On Wed, Dec 11, 2019 at 12:58 PM Bonnie Lo/WYHQ/Wiwynn
<Bonnie_Lo@wiwynn.com> wrote:
>
> Dear Joseph,
>
> In my understanding, the BMC firmware update flow is as below:
> 1. Trigger reboot
> 2. Systemd stop all service
> 3. Unmount file system
> 4. image is in /run/initramfs
> 5. Do the flashcp command to update the flash
>
> If there is any misunderstanding, please correct me.
>
> Based on the discussion with Neeraj.
> We want to be able to update BMC firmware without having to trigger the BMC reboot command before the system do flashcp command.
> It means that we can do the flashcp first. If the flashcp command complete and success, then we do the reset manually.
> Is it workable on current upstream code?
> If not, why? I means is there any advantage to trigger the reboot before we do the flashcp.

It is not safe to flashcp the flash chip while BMC is running on it,
because BMC's filesystem is mounted on the flash's RO and RW
partitions.

Re: BMC update via TFTP

[Alexander Tereschenko]

[-- Attachment #1: Type: text/plain, Size: 1735 bytes --]

>> Yes, that could be a solution for the problem we discuss, providing 
>> both integrity and confidentiality, without any major OpenBMC 
>> development necessary - but it would mean more operational burden for 
>> BMC admins. The problem with SCP/SFTP in this context is that for 
>> this to work in the same manner as TFTP, the BMC must be an SSH 
>> client - i.e. have some sort of identity/credentials for the SCP/SFTP 
>> server provisioned first. That might not be the easiest solution to 
>> setup, but it's of course possible and can be automated if OpenBMC 
>> provides respective config knobs.
>>
>> Existing ways we have in code-update.md either don't require 
>> credentials (TFTP), so being a client is easy, or are not making a 
>> "client" from BMC, it's the admin who uploads stuff (SCP/REST).
>
> Yes, that's what I was thinking.  (And no, I am not going to recommend 
> setting up a SCP or SFTP server that allows anonymous access.)
>
> This highlight the need for OpenBMC to put together a guide to 
> provisioning your BMC.    Such as guide would give us a place to talk 
> about uploading to the BMC SSH client certificates needed to access 
> and download the firmware images.
>
> - Joseph 

Agree, the provisioning guide could be a good point to have this 
discussion. However I beieve updates in general is a broader and more 
"operational" (i.e. "continuous" as opposed to provisioning being rather 
"one-time") topic, so the approach in the organization/of a given BMC 
admin can change and I believe whatever configuration mechanism we 
develop for this (if at all), should be available at any point during 
BMC lifetime, not only at provisioning, and be architected respectively.


regards,
Alexander



[-- Attachment #2: Type: text/html, Size: 2368 bytes --]

Re: BMC update via TFTP

[Joseph Reynolds]

On 12/11/19 5:02 AM, Alexander Tereschenko wrote:
>>> Yes, that could be a solution for the problem we discuss, providing 
>>> both integrity and confidentiality, without any major OpenBMC 
>>> development necessary - but it would mean more operational burden 
>>> for BMC admins. The problem with SCP/SFTP in this context is that 
>>> for this to work in the same manner as TFTP, the BMC must be an SSH 
>>> client - i.e. have some sort of identity/credentials for the 
>>> SCP/SFTP server provisioned first. That might not be the easiest 
>>> solution to setup, but it's of course possible and can be automated 
>>> if OpenBMC provides respective config knobs.
>>>
>>> Existing ways we have in code-update.md either don't require 
>>> credentials (TFTP), so being a client is easy, or are not making a 
>>> "client" from BMC, it's the admin who uploads stuff (SCP/REST).
>>
>> Yes, that's what I was thinking.  (And no, I am not going to 
>> recommend setting up a SCP or SFTP server that allows anonymous access.)
>>
>> This highlight the need for OpenBMC to put together a guide to 
>> provisioning your BMC.    Such as guide would give us a place to talk 
>> about uploading to the BMC SSH client certificates needed to access 
>> and download the firmware images.
>>
>> - Joseph 
>
> Agree, the provisioning guide could be a good point to have this 
> discussion. However I beieve updates in general is a broader and more 
> "operational" (i.e. "continuous" as opposed to provisioning being 
> rather "one-time") topic, so the approach in the organization/of a 
> given BMC admin can change and I believe whatever configuration 
> mechanism we develop for this (if at all), should be available at any 
> point during BMC lifetime, not only at provisioning, and be 
> architected respectively.
>

That makes sense to me.  Thanks!  I'll plan to add an item to both:
https://github.com/openbmc/docs/blob/master/security/network-security-considerations.md
and
to the threat model considerations (in review):
https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/22404

- Joseph

>
> regards,
> Alexander
>
>

RE: BMC update via TFTP

[Milton Miller II]

On 12/10/2019 around 10:57PM in some timezone, Bonnie Lo/WYHQ/Wiwynn  wrote:
>In my understanding, the BMC firmware update flow is as below:
>1. Trigger reboot 
>2. Systemd stop all service
>3. Unmount file system
>4. image is in /run/initramfs 
>5. Do the flashcp command to update the flash 
>
>If there is any misunderstanding, please correct me.
>
>Based on the discussion with Neeraj.
>We want to be able to update BMC firmware without having to trigger
>the BMC reboot command before the system do flashcp command.
>It means that we can do the flashcp first. If the flashcp command
>complete and success, then we do the reset manually.
>Is it workable on current upstream code?
>If not, why? I means is there any advantage to trigger the reboot
>before we do the flashcp.

The init script in the initramfs takes options from u-boot variables 
and the kernel command line.  Some of the options allow that the
read-only and files in the read-write partition are copied into RAM
before the file systems are mounted, which allows the BMC to be
operational while the new flash image is transfered and written into
the flash.  Note that if a reboot is triggered for some reason you 
can still end up needing additional recovery.

This state can be initiated by "prepare for update" mentioned in
the REST documentation for code-update.  I don't think it can be
initiated in the current Redfish.

The flow is:

1. Trigger "prepare for update" 

    - This uses the variable cleared every boot to copy 
       rw partition content to ram (tmpfs), and the mtd ro partition 
       ram (as a file then mounted by loop device).

2. Upload images as before

    - They will be signature checked as compiled then coped to /run/initramfs

3. run /run/initramfs/update

  - This script will look at /proc/mounts and see that the mtd-ro and mtdrw
   partitions are not present, and proceed to flashcp the files to the flash

  - WARNING: do not try to hide the mounts by unmounting the mtd ro and
   rw partitions.  The root file system is overlayfs and will retain write access
  to the flash, and your BMC will crash


4. Initiate reboot at leisure to move to new code level

  - the contents of the tmpfs with the overlay files will be copied back to 
    the rwfs file system in the mtd partitions

Note:  This is not the normal operation in part because the rwfs is
in tmpfs and therefore any panic or power cut will result in all modifications
to the rwfs being lost.


If needed I can lookup the exact variable names and contents to 
communicate the desired maintenance actions from the phosphor 
init script.  There are two variables, one for persistent settings, 
one for transient settings which a script clears each time.   One
can also customize or disable this features by adding additional
customization files into the initramfs recipe. 

milton


>>Thanks,
>Bonnie
>