* BMC update via TFTP @ 2019-12-03 12:02 rgrs 2019-12-03 16:12 ` Gunnar Mills 0 siblings, 1 reply; 19+ messages in thread From: rgrs @ 2019-12-03 12:02 UTC (permalink / raw) To: openbmc [-- Attachment #1: Type: text/plain, Size: 289 bytes --] Hi, In BMC WebUI under "Download image file from TFTP server" section, we have text fields "TFTP Server IP address" and "File name". "File name" doesn't take folders in path. Is this a bug or expected behavior? TFTP downloads work only if file is kept in root of tftp share. Thanks, Raj [-- Attachment #2: Type: text/html, Size: 652 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-03 12:02 BMC update via TFTP rgrs @ 2019-12-03 16:12 ` Gunnar Mills 2019-12-03 17:08 ` Gunnar Mills 0 siblings, 1 reply; 19+ messages in thread From: Gunnar Mills @ 2019-12-03 16:12 UTC (permalink / raw) To: rgrs, openbmc > In BMC WebUI under "Download image file from TFTP server" section, > we have text fields "TFTP Server IP address" and "File name". > "File name" doesn't take folders in path. Is this a bug or expected > behavior? > > TFTP downloads work only if file is kept in root of tftp share. > This is expected. https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-03 16:12 ` Gunnar Mills @ 2019-12-03 17:08 ` Gunnar Mills 2019-12-03 19:55 ` Adriana Kobylak 0 siblings, 1 reply; 19+ messages in thread From: Gunnar Mills @ 2019-12-03 17:08 UTC (permalink / raw) To: rgrs, openbmc; +Cc: anoo, patrick [-- Attachment #1: Type: text/plain, Size: 710 bytes --] On 12/3/2019 10:12 AM, Gunnar Mills wrote: > >> In BMC WebUI under "Download image file from TFTP server" section, >> we have text fields "TFTP Server IP address" and "File name". >> "File name" doesn't take folders in path. Is this a bug or expected >> behavior? >> >> TFTP downloads work only if file is kept in root of tftp share. >> > This is expected. > https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 > As long as we continue to sanitize the local filename, I don't see why we need to limit the source file name. Patrick, Adriana, Any objection? Remember why we wrote it this way? I assume for simplicity..? Thanks, Gunnar [-- Attachment #2: Type: text/html, Size: 1588 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-03 17:08 ` Gunnar Mills @ 2019-12-03 19:55 ` Adriana Kobylak 2019-12-04 15:47 ` Joseph Reynolds 0 siblings, 1 reply; 19+ messages in thread From: Adriana Kobylak @ 2019-12-03 19:55 UTC (permalink / raw) To: Gunnar Mills; +Cc: rgrs, openbmc, anoo On 2019-12-03 11:08, Gunnar Mills wrote: > On 12/3/2019 10:12 AM, Gunnar Mills wrote: > >>> In BMC WebUI under "Download image file from TFTP server" section, >>> >>> we have text fields "TFTP Server IP address" and "File name". >>> "File name" doesn't take folders in path. Is this a bug or >>> expected behavior? >>> >>> TFTP downloads work only if file is kept in root of tftp share. >> This is expected. >> > https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 >> [1] > > As long as we continue to sanitize the local filename, I don't see why > we need to limit the source file name. > Patrick, Adriana, Any objection? Remember why we wrote it this way? I > assume for simplicity..? It was done for security to prevent users from specifying a file outside the tftp directory, such as /mydir/../root/system-file. But seems the current file name handling breaks the ability to get files from a subdir like Raj pointed out, we should be able to fix that out, tftp supports having a file in subdirs such as /tftpboot/subdirectory/file, and passing /subdirectory/file to it as the path. > > Thanks, > Gunnar > > > Links: > ------ > [1] > https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-03 19:55 ` Adriana Kobylak @ 2019-12-04 15:47 ` Joseph Reynolds 2019-12-04 21:36 ` Vernon Mauery 0 siblings, 1 reply; 19+ messages in thread From: Joseph Reynolds @ 2019-12-04 15:47 UTC (permalink / raw) To: Adriana Kobylak, Gunnar Mills; +Cc: openbmc, rgrs, anoo On 12/3/19 1:55 PM, Adriana Kobylak wrote: > On 2019-12-03 11:08, Gunnar Mills wrote: >> On 12/3/2019 10:12 AM, Gunnar Mills wrote: >> >>>> In BMC WebUI under "Download image file from TFTP server" section, >>>> >>>> we have text fields "TFTP Server IP address" and "File name". >>>> "File name" doesn't take folders in path. Is this a bug or >>>> expected behavior? >>>> >>>> TFTP downloads work only if file is kept in root of tftp share. >>> This is expected. >>> >> https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 >> >>> [1] >> >> As long as we continue to sanitize the local filename, I don't see why >> we need to limit the source file name. >> Patrick, Adriana, Any objection? Remember why we wrote it this way? I >> assume for simplicity..? > > It was done for security to prevent users from specifying a file > outside the tftp directory, such as /mydir/../root/system-file. > But seems the current file name handling breaks the ability to get > files from a subdir like Raj pointed out, we should be able to fix > that out, tftp supports having a file in subdirs such as > /tftpboot/subdirectory/file, and passing /subdirectory/file to it as > the path. It seems to me that the burden must be on the TFTP server to protect itself. See the "Security Considerations" section of RFC 1350 - https://tools.ietf.org/html/rfc1350 Specifically, OpenBMC only needs read access (it does not need to write files to the TFTP server) and only needs to be able to access files the TFTP server intends to provide (typically: all the files under a specific directory). To be more clear: - OpenBMC does not provide a TFTP server, so this is guidance for someone setting a TFTP server for use with an OpenBMC system - The TFTP server only needs to offer READ access. - The TFTP server ought to prevent clients from using cleverly-constructed pathnames (for example, any file which begins with "/" or contains an ".." element) to access files outside of its sandbox. - That said, I am not a TFTP expert, and whichever TFTP server is used, its security guidance should be followed. What all of the above, I think it would be okay for OpenBMC to validate the pathname it sends to the TFTP server, but don't think it is necessary. Finally, I hope we can move away from TFTP and more toward a secure image distribution approach. Note that TFTP is disabled in OpenBMC by default. See https://github.com/openbmc/bmcweb/blob/master/CMakeLists.txt - Joseph >> >> Thanks, >> Gunnar >> >> >> Links: >> ------ >> [1] >> https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 >> ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-04 15:47 ` Joseph Reynolds @ 2019-12-04 21:36 ` Vernon Mauery 2019-12-05 11:05 ` Alexander Tereschenko 0 siblings, 1 reply; 19+ messages in thread From: Vernon Mauery @ 2019-12-04 21:36 UTC (permalink / raw) To: Joseph Reynolds; +Cc: Adriana Kobylak, Gunnar Mills, openbmc, rgrs, anoo On 04-Dec-2019 09:47 AM, Joseph Reynolds wrote: >On 12/3/19 1:55 PM, Adriana Kobylak wrote: >>On 2019-12-03 11:08, Gunnar Mills wrote: >>>On 12/3/2019 10:12 AM, Gunnar Mills wrote: >>> >>>>>In BMC WebUI under "Download image file from TFTP server" section, >>>>> >>>>>we have text fields "TFTP Server IP address" and "File name". >>>>>"File name" doesn't take folders in path. Is this a bug or >>>>>expected behavior? >>>>> >>>>>TFTP downloads work only if file is kept in root of tftp share. >>>>This is expected. >>>> >>>https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 >>> >>>>[1] >>> >>>As long as we continue to sanitize the local filename, I don't see why >>>we need to limit the source file name. >>> Patrick, Adriana, Any objection? Remember why we wrote it this way? I >>>assume for simplicity..? >> >>It was done for security to prevent users from specifying a file >>outside the tftp directory, such as /mydir/../root/system-file. >>But seems the current file name handling breaks the ability to get >>files from a subdir like Raj pointed out, we should be able to fix >>that out, tftp supports having a file in subdirs such as >>/tftpboot/subdirectory/file, and passing /subdirectory/file to it as >>the path. While we are on the topic of security, should we do something about unsecured protocols for image transfers? Adding an HMAC and key along with the IP/path would then make sure the file that was fetched contains the contents that were expected. Even if the BMC only accepts signed images, we want to make sure that the signed image the BMC fetches is the one that the administrator *wants* to be fetched. With tftp or http (or any insecure transport), one possible MiTM attack would be to substitute an alternate valid image. Let's say the admin wants to go from 1.18 to 1.20, bu the attacker wants to go to 1.16, which has a known vulnerability. The image would be authenticated by the signature, and will be accepted. With an added HMAC, the MiTM attack would be thwarted. The HMAC and key would be transmitted to the BMC over a secure channel (https) so the attacker would not be able create a suitable replacement for the image in the insecure channel. --Vernon >It seems to me that the burden must be on the TFTP server to protect >itself. See the "Security Considerations" section of RFC 1350 - >https://tools.ietf.org/html/rfc1350 >Specifically, OpenBMC only needs read access (it does not need to >write files to the TFTP server) and only needs to be able to access >files the TFTP server intends to provide (typically: all the files >under a specific directory). > >To be more clear: >- OpenBMC does not provide a TFTP server, so this is guidance for >someone setting a TFTP server for use with an OpenBMC system >- The TFTP server only needs to offer READ access. >- The TFTP server ought to prevent clients from using >cleverly-constructed pathnames (for example, any file which begins >with "/" or contains an ".." element) to access files outside of its >sandbox. >- That said, I am not a TFTP expert, and whichever TFTP server is >used, its security guidance should be followed. > >What all of the above, I think it would be okay for OpenBMC to >validate the pathname it sends to the TFTP server, but don't think it >is necessary. > >Finally, I hope we can move away from TFTP and more toward a secure >image distribution approach. >Note that TFTP is disabled in OpenBMC by default. See >https://github.com/openbmc/bmcweb/blob/master/CMakeLists.txt > >- Joseph > >>> >>> Thanks, >>> Gunnar >>> >>> >>>Links: >>>------ >>>[1] >>>https://github.com/openbmc/phosphor-bmc-code-mgmt/blob/b0ce996ac60cf80487d71c3cdb7165d065079377/download_manager.cpp#L33 >>> > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-04 21:36 ` Vernon Mauery @ 2019-12-05 11:05 ` Alexander Tereschenko 2019-12-05 22:37 ` Vernon Mauery 0 siblings, 1 reply; 19+ messages in thread From: Alexander Tereschenko @ 2019-12-05 11:05 UTC (permalink / raw) To: openbmc On 04-Dec-19 22:36, Vernon Mauery wrote: > Even if the BMC only accepts signed images, we want to make sure that > the signed image the BMC fetches is the one that the administrator > *wants* to be fetched. With tftp or http (or any insecure transport), > one possible MiTM attack would be to substitute an alternate valid > image. Let's say the admin wants to go from 1.18 to 1.20, bu the > attacker wants to go to 1.16, which has a known vulnerability. The > image would be authenticated by the signature, and will be accepted. Thanks Vernon for raising this one - I think this is indeed a valid concern. So there are essentially two things that we are talking about here, if we peel it a little bit: 1) an additional authorization by the BMC admin of specific images they want to run 2) plain integrity protection of the file being sent over insecure channel The first one is a bit different from the second one (and frequently solved by co-signing these days), but in thise specific use case I'd say these two indeed overlap. In this case, I'd rather suggest us to look into asymmetric crypto, so that no secret sharing/storage (with accompanying additional risk of compromise) is necessary. An admin would sign the image being sent + some context information (like time, to provide replay protection - also applicable to MAC case, otherwise the attacker can trivially replay the file/MAC) and the BMC would check the admin's signature using pre-provisioned public key (send over HTTPS, same as MAC key in your proposal - but only for integrity protection, con confidentiality is necessary) + verify that the context is "fresh" and then use the file. I'd be happy to help out with elaborating details of the scheme, if that's deemed interesting enough for wider audience. regards, Alexander ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-05 11:05 ` Alexander Tereschenko @ 2019-12-05 22:37 ` Vernon Mauery 2019-12-06 14:03 ` Alexander Tereschenko 0 siblings, 1 reply; 19+ messages in thread From: Vernon Mauery @ 2019-12-05 22:37 UTC (permalink / raw) To: Alexander Tereschenko; +Cc: openbmc On 05-Dec-2019 12:05 PM, Alexander Tereschenko wrote: >On 04-Dec-19 22:36, Vernon Mauery wrote: >>Even if the BMC only accepts signed images, we want to make sure >>that the signed image the BMC fetches is the one that the >>administrator *wants* to be fetched. With tftp or http (or any >>insecure transport), one possible MiTM attack would be to substitute >>an alternate valid image. Let's say the admin wants to go from 1.18 >>to 1.20, bu the attacker wants to go to 1.16, which has a known >>vulnerability. The image would be authenticated by the signature, >>and will be accepted. > >Thanks Vernon for raising this one - I think this is indeed a valid concern. > >So there are essentially two things that we are talking about here, if >we peel it a little bit: > >1) an additional authorization by the BMC admin of specific images >they want to run >2) plain integrity protection of the file being sent over insecure channel > >The first one is a bit different from the second one (and frequently >solved by co-signing these days), but in thise specific use case I'd >say these two indeed overlap. > >In this case, I'd rather suggest us to look into asymmetric crypto, so >that no secret sharing/storage (with accompanying additional risk of >compromise) is necessary. An admin would sign the image being sent + >some context information (like time, to provide replay protection - >also applicable to MAC case, otherwise the attacker can trivially >replay the file/MAC) and the BMC would check the admin's signature >using pre-provisioned public key (send over HTTPS, same as MAC key in >your proposal - but only for integrity protection, con confidentiality >is necessary) + verify that the context is "fresh" and then use the >file. I am not convinced that it needs to be this elaborate. I don't see what it adds over the simple case of sending the key/hmac/uri encrypted with TLS to the BMC. There will be no replay attacks because TLS prevents it. Maybe I am missing something? --Vernon >I'd be happy to help out with elaborating details of the scheme, if >that's deemed interesting enough for wider audience. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-05 22:37 ` Vernon Mauery @ 2019-12-06 14:03 ` Alexander Tereschenko 2019-12-06 22:52 ` Joseph Reynolds 0 siblings, 1 reply; 19+ messages in thread From: Alexander Tereschenko @ 2019-12-06 14:03 UTC (permalink / raw) To: openbmc On 05-Dec-19 23:37, Vernon Mauery wrote: > On 05-Dec-2019 12:05 PM, Alexander Tereschenko wrote: >> On 04-Dec-19 22:36, Vernon Mauery wrote: >>> Even if the BMC only accepts signed images, we want to make sure >>> that the signed image the BMC fetches is the one that the >>> administrator *wants* to be fetched. With tftp or http (or any >>> insecure transport), one possible MiTM attack would be to substitute >>> an alternate valid image. Let's say the admin wants to go from 1.18 >>> to 1.20, bu the attacker wants to go to 1.16, which has a known >>> vulnerability. The image would be authenticated by the signature, >>> and will be accepted. >> >> Thanks Vernon for raising this one - I think this is indeed a valid >> concern. >> >> So there are essentially two things that we are talking about here, >> if we peel it a little bit: >> >> 1) an additional authorization by the BMC admin of specific images >> they want to run >> 2) plain integrity protection of the file being sent over insecure >> channel >> >> The first one is a bit different from the second one (and frequently >> solved by co-signing these days), but in thise specific use case I'd >> say these two indeed overlap. >> >> In this case, I'd rather suggest us to look into asymmetric crypto, >> so that no secret sharing/storage (with accompanying additional risk >> of compromise) is necessary. An admin would sign the image being sent >> + some context information (like time, to provide replay protection - >> also applicable to MAC case, otherwise the attacker can trivially >> replay the file/MAC) and the BMC would check the admin's signature >> using pre-provisioned public key (send over HTTPS, same as MAC key in >> your proposal - but only for integrity protection, con >> confidentiality is necessary) + verify that the context is "fresh" >> and then use the file. > > I am not convinced that it needs to be this elaborate. I don't see > what it adds over the simple case of sending the key/hmac/uri > encrypted with TLS to the BMC. There will be no replay attacks because > TLS prevents it. > Maybe I am missing something? We may be talking about slightly different contexts indeed. I meant to suggest something that doesn't require 1) any "out-of-band" channel for each file being sent (signature can be appended to the file itself, one-time/infrequent public key updates at the BMC side are discounted as arguably negligible). I use the "out-of-band" here as "channel different than the one where the file itself is being sent"; 2) any symmetric secret generation/storage/distribution upon sending files (which is an additional burden and risk); plus 1) provides replay protection; 2) doesn't require major infrastructure modifications (here I'm assuming that signing and checking the signature in this flow is less change than generating key, MAC and sending them over "out-of-band" channel); I read your proposal as that the key will be a longer-term one + that the MAC will be sent along with the file (as it's not a secret and typically it's sent together with the data in such schemes). If you meant that both the key and MAC are generated on the fly, are one-time and are sent via that "out-of-band" TLS-protected channel, then I'd say you don't even need a keyed MAC at all and a plain cryptographic hash would suffice. You're sending the key along with the MAC and file channel itself is unprotected - so if the adversary can break the TLS, they can compute the MAC as easily as a plain hash and if they can't, they anyway can't substitute the hash value, so IMHO keyed MAC is surplus here. Maybe outlining a flow (maybe as a sequence/flow diagram) in more detail and defining which specific adversaries (and their capabilities) we're talking about here would help? I agree we surely don't need a more complex scheme if a simpler one suffices, it's just that we may be talking about slightly different flows and adversaries right now :) regards, Alexander ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-06 14:03 ` Alexander Tereschenko @ 2019-12-06 22:52 ` Joseph Reynolds 2019-12-09 16:06 ` Alexander Tereschenko 0 siblings, 1 reply; 19+ messages in thread From: Joseph Reynolds @ 2019-12-06 22:52 UTC (permalink / raw) To: Alexander Tereschenko, openbmc, vernon.mauery On 12/6/19 8:03 AM, Alexander Tereschenko wrote: > On 05-Dec-19 23:37, Vernon Mauery wrote: >> On 05-Dec-2019 12:05 PM, Alexander Tereschenko wrote: >>> On 04-Dec-19 22:36, Vernon Mauery wrote: >>>> Even if the BMC only accepts signed images, we want to make sure >>>> that the signed image the BMC fetches is the one that the >>>> administrator *wants* to be fetched. With tftp or http (or any >>>> insecure transport), one possible MiTM attack would be to >>>> substitute an alternate valid image. Let's say the admin wants to >>>> go from 1.18 to 1.20, bu the attacker wants to go to 1.16, which >>>> has a known vulnerability. The image would be authenticated by the >>>> signature, and will be accepted. ...snip... >> I am not convinced that it needs to be this elaborate. I don't see >> what it adds over the simple case of sending the key/hmac/uri >> encrypted with TLS to the BMC. There will be no replay attacks >> because TLS prevents it. >> Maybe I am missing something? > > We may be talking about slightly different contexts indeed. I meant to > suggest something that doesn't require > ...snip... I was thinking along the lines of adding [SFTP][] (or SCP) support and then migrating existing TFTP users to the new secure solution. That is, the BMC admin performing [code update][] can currently get a firmware image via POST DownloadViaTFTP to URI /xyz/openbmc_project/software. My idea is to offer a DownloadViaSFTP method (or preferably a Redfish API) for this. Note that the TFTP download is disabled by default per [bmcweb config][]. Once OpenBMC supports downloading firmware via SFTP, we can encourage our users to set up their SFTP servers and take down their TFTP servers. I realize that sounds easy, but I don't have a feeling how difficult that would be in practice. Does that sound feasible? - Joseph [SFTP]: https://man.openbsd.org/sftp-server [code update]: https://github.com/openbmc/docs/blob/master/code-update/code-update.md [bmcweb config]: https://github.com/openbmc/bmcweb/blob/41d1d1833f476766f88cfb624e66eef7906bdf8c/CMakeLists.txt#L98 > regards, > Alexander > > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-06 22:52 ` Joseph Reynolds @ 2019-12-09 16:06 ` Alexander Tereschenko 2019-12-10 1:25 ` Joseph Reynolds 0 siblings, 1 reply; 19+ messages in thread From: Alexander Tereschenko @ 2019-12-09 16:06 UTC (permalink / raw) To: openbmc On 06-Dec-19 23:52, Joseph Reynolds wrote: > I was thinking along the lines of adding [SFTP][] (or SCP) support and > then migrating existing TFTP users to the new secure solution. > > That is, the BMC admin performing [code update][] can currently get a > firmware image via POST DownloadViaTFTP to URI > /xyz/openbmc_project/software. > My idea is to offer a DownloadViaSFTP method (or preferably a Redfish > API) for this. Note that the TFTP download is disabled by default per > [bmcweb config][]. > > Once OpenBMC supports downloading firmware via SFTP, we can encourage > our users to set up their SFTP servers and take down their TFTP > servers. I realize that sounds easy, but I don't have a feeling how > difficult that would be in practice. > > Does that sound feasible? > > - Joseph > > [SFTP]: https://man.openbsd.org/sftp-server > [code update]: > https://github.com/openbmc/docs/blob/master/code-update/code-update.md > [bmcweb config]: > https://github.com/openbmc/bmcweb/blob/41d1d1833f476766f88cfb624e66eef7906bdf8c/CMakeLists.txt#L98 Yes, that could be a solution for the problem we discuss, providing both integrity and confidentiality, without any major OpenBMC development necessary - but it would mean more operational burden for BMC admins. The problem with SCP/SFTP in this context is that for this to work in the same manner as TFTP, the BMC must be an SSH client - i.e. have some sort of identity/credentials for the SCP/SFTP server provisioned first. That might not be the easiest solution to setup, but it's of course possible and can be automated if OpenBMC provides respective config knobs. Existing ways we have in code-update.md either don't require credentials (TFTP), so being a client is easy, or are not making a "client" from BMC, it's the admin who uploads stuff (SCP/REST). regards, Alexander ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-09 16:06 ` Alexander Tereschenko @ 2019-12-10 1:25 ` Joseph Reynolds 2019-12-10 18:58 ` [EXTERNAL] " Neeraj Ladkani 2019-12-11 11:02 ` Alexander Tereschenko 0 siblings, 2 replies; 19+ messages in thread From: Joseph Reynolds @ 2019-12-10 1:25 UTC (permalink / raw) To: Alexander Tereschenko, openbmc On 12/9/19 10:06 AM, Alexander Tereschenko wrote: > On 06-Dec-19 23:52, Joseph Reynolds wrote: >> I was thinking along the lines of adding [SFTP][] (or SCP) support >> and then migrating existing TFTP users to the new secure solution. >> >> That is, the BMC admin performing [code update][] can currently get a >> firmware image via POST DownloadViaTFTP to URI >> /xyz/openbmc_project/software. >> My idea is to offer a DownloadViaSFTP method (or preferably a Redfish >> API) for this. Note that the TFTP download is disabled by default >> per [bmcweb config][]. >> >> Once OpenBMC supports downloading firmware via SFTP, we can encourage >> our users to set up their SFTP servers and take down their TFTP >> servers. I realize that sounds easy, but I don't have a feeling how >> difficult that would be in practice. >> >> Does that sound feasible? >> >> - Joseph >> >> [SFTP]: https://man.openbsd.org/sftp-server >> [code update]: >> https://github.com/openbmc/docs/blob/master/code-update/code-update.md >> [bmcweb config]: >> https://github.com/openbmc/bmcweb/blob/41d1d1833f476766f88cfb624e66eef7906bdf8c/CMakeLists.txt#L98 > > Yes, that could be a solution for the problem we discuss, providing > both integrity and confidentiality, without any major OpenBMC > development necessary - but it would mean more operational burden for > BMC admins. The problem with SCP/SFTP in this context is that for this > to work in the same manner as TFTP, the BMC must be an SSH client - > i.e. have some sort of identity/credentials for the SCP/SFTP server > provisioned first. That might not be the easiest solution to setup, > but it's of course possible and can be automated if OpenBMC provides > respective config knobs. > > Existing ways we have in code-update.md either don't require > credentials (TFTP), so being a client is easy, or are not making a > "client" from BMC, it's the admin who uploads stuff (SCP/REST). Yes, that's what I was thinking. (And no, I am not going to recommend setting up a SCP or SFTP server that allows anonymous access.) This highlight the need for OpenBMC to put together a guide to provisioning your BMC. Such as guide would give us a place to talk about uploading to the BMC SSH client certificates needed to access and download the firmware images. - Joseph > > regards, > Alexander > ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [EXTERNAL] Re: BMC update via TFTP 2019-12-10 1:25 ` Joseph Reynolds @ 2019-12-10 18:58 ` Neeraj Ladkani 2019-12-10 21:10 ` Joseph Reynolds 2019-12-11 11:02 ` Alexander Tereschenko 1 sibling, 1 reply; 19+ messages in thread From: Neeraj Ladkani @ 2019-12-10 18:58 UTC (permalink / raw) To: Joseph Reynolds, Alexander Tereschenko, openbmc, Patrick Venture Are there any thoughts to get rid of BMC reset to trigger FW update? I understand FW reset is required after the update. -----Original Message----- From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> On Behalf Of Joseph Reynolds Sent: Monday, December 9, 2019 5:25 PM To: Alexander Tereschenko <aleksandr.v.tereschenko@linux.intel.com>; openbmc@lists.ozlabs.org Subject: [EXTERNAL] Re: BMC update via TFTP On 12/9/19 10:06 AM, Alexander Tereschenko wrote: > On 06-Dec-19 23:52, Joseph Reynolds wrote: >> I was thinking along the lines of adding [SFTP][] (or SCP) support >> and then migrating existing TFTP users to the new secure solution. >> >> That is, the BMC admin performing [code update][] can currently get a >> firmware image via POST DownloadViaTFTP to URI >> /xyz/openbmc_project/software. >> My idea is to offer a DownloadViaSFTP method (or preferably a Redfish >> API) for this. Note that the TFTP download is disabled by default >> per [bmcweb config][]. >> >> Once OpenBMC supports downloading firmware via SFTP, we can encourage >> our users to set up their SFTP servers and take down their TFTP >> servers. I realize that sounds easy, but I don't have a feeling how >> difficult that would be in practice. >> >> Does that sound feasible? >> >> - Joseph >> >> [SFTP]: >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fman >> .openbsd.org%2Fsftp-server&data=02%7C01%7Cneladk%40microsoft.com% >> 7C9cc71f33a3014260e36f08d77d0fe11b%7C72f988bf86f141af91ab2d7cd011db47 >> %7C1%7C0%7C637115379469052876&sdata=Zj%2BjAlaXlyeBkTsl7MvtbPoSeH7 >> az%2FAJS1UxXeCy0Pc%3D&reserved=0 >> [code update]: >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit >> hub.com%2Fopenbmc%2Fdocs%2Fblob%2Fmaster%2Fcode-update%2Fcode-update. >> md&data=02%7C01%7Cneladk%40microsoft.com%7C9cc71f33a3014260e36f08 >> d77d0fe11b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6371153794690 >> 52876&sdata=ScV14ytcPCYn%2BlI%2B9lPgkgKY4yVh%2BrwMVgdbnB0h5z4%3D& >> amp;reserved=0 >> [bmcweb config]: >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit >> hub.com%2Fopenbmc%2Fbmcweb%2Fblob%2F41d1d1833f476766f88cfb624e66eef79 >> 06bdf8c%2FCMakeLists.txt%23L98&data=02%7C01%7Cneladk%40microsoft. >> com%7C9cc71f33a3014260e36f08d77d0fe11b%7C72f988bf86f141af91ab2d7cd011 >> db47%7C1%7C0%7C637115379469052876&sdata=wkV8x6Ce1A0Wf%2FMN%2F%2B9 >> FDSYwQ9z47YI6bc6RwqrTcEg%3D&reserved=0 > > Yes, that could be a solution for the problem we discuss, providing > both integrity and confidentiality, without any major OpenBMC > development necessary - but it would mean more operational burden for > BMC admins. The problem with SCP/SFTP in this context is that for this > to work in the same manner as TFTP, the BMC must be an SSH client - > i.e. have some sort of identity/credentials for the SCP/SFTP server > provisioned first. That might not be the easiest solution to setup, > but it's of course possible and can be automated if OpenBMC provides > respective config knobs. > > Existing ways we have in code-update.md either don't require > credentials (TFTP), so being a client is easy, or are not making a > "client" from BMC, it's the admin who uploads stuff (SCP/REST). Yes, that's what I was thinking. (And no, I am not going to recommend setting up a SCP or SFTP server that allows anonymous access.) This highlight the need for OpenBMC to put together a guide to provisioning your BMC. Such as guide would give us a place to talk about uploading to the BMC SSH client certificates needed to access and download the firmware images. - Joseph > > regards, > Alexander > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [EXTERNAL] Re: BMC update via TFTP 2019-12-10 18:58 ` [EXTERNAL] " Neeraj Ladkani @ 2019-12-10 21:10 ` Joseph Reynolds 2019-12-11 4:56 ` Bonnie Lo/WYHQ/Wiwynn 2019-12-11 21:51 ` Milton Miller II 0 siblings, 2 replies; 19+ messages in thread From: Joseph Reynolds @ 2019-12-10 21:10 UTC (permalink / raw) To: Neeraj Ladkani, Alexander Tereschenko, openbmc, Patrick Venture On 12/10/19 12:58 PM, Neeraj Ladkani wrote: > Are there any thoughts to get rid of BMC reset to trigger FW update? I understand FW reset is required after the update. I'm not sure I understand the question. I think the answer depends on the [Software.VersionPurpose][1]. For VersionPurpose=BMC or System, the BMC must be reset. For VersionPurpose=Host, PSU, or Other, I don't know why the BMC would need to be reset. Do you want to be able to update non-BMC firmware without having to reset the BMC? - Joseph [1]: https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Software/Version.interface.yaml > > -----Original Message----- > From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> On Behalf Of Joseph Reynolds > Sent: Monday, December 9, 2019 5:25 PM > To: Alexander Tereschenko <aleksandr.v.tereschenko@linux.intel.com>; openbmc@lists.ozlabs.org > Subject: [EXTERNAL] Re: BMC update via TFTP > > On 12/9/19 10:06 AM, Alexander Tereschenko wrote: >> On 06-Dec-19 23:52, Joseph Reynolds wrote: >>> I was thinking along the lines of adding [SFTP][] (or SCP) support >>> and then migrating existing TFTP users to the new secure solution. [...snip...] >> Yes, that could be a solution for the problem we discuss, providing >> both integrity and confidentiality, without any major OpenBMC >> development necessary - but it would mean more operational burden for >> BMC admins. The problem with SCP/SFTP in this context is that for this >> to work in the same manner as TFTP, the BMC must be an SSH client - >> i.e. have some sort of identity/credentials for the SCP/SFTP server >> provisioned first. That might not be the easiest solution to setup, >> but it's of course possible and can be automated if OpenBMC provides >> respective config knobs. >> >> Existing ways we have in code-update.md either don't require >> credentials (TFTP), so being a client is easy, or are not making a >> "client" from BMC, it's the admin who uploads stuff (SCP/REST). > Yes, that's what I was thinking. (And no, I am not going to recommend setting up a SCP or SFTP server that allows anonymous access.) > > This highlight the need for OpenBMC to put together a guide to provisioning your BMC. Such as guide would give us a place to talk about uploading to the BMC SSH client certificates needed to access and download the firmware images. > > - Joseph > >> regards, >> Alexander >> ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [EXTERNAL] Re: BMC update via TFTP 2019-12-10 21:10 ` Joseph Reynolds @ 2019-12-11 4:56 ` Bonnie Lo/WYHQ/Wiwynn 2019-12-11 5:59 ` Lei YU 2019-12-11 21:51 ` Milton Miller II 1 sibling, 1 reply; 19+ messages in thread From: Bonnie Lo/WYHQ/Wiwynn @ 2019-12-11 4:56 UTC (permalink / raw) To: Joseph Reynolds, Neeraj Ladkani, Alexander Tereschenko, openbmc, Patrick Venture Cc: Delphine Chiu/WYHQ/Wiwynn, Aldofo Lin/WYHQ/Wiwynn Dear Joseph, In my understanding, the BMC firmware update flow is as below: 1. Trigger reboot 2. Systemd stop all service 3. Unmount file system 4. image is in /run/initramfs 5. Do the flashcp command to update the flash If there is any misunderstanding, please correct me. Based on the discussion with Neeraj. We want to be able to update BMC firmware without having to trigger the BMC reboot command before the system do flashcp command. It means that we can do the flashcp first. If the flashcp command complete and success, then we do the reset manually. Is it workable on current upstream code? If not, why? I means is there any advantage to trigger the reboot before we do the flashcp. Thanks, Bonnie -----Original Message----- From: openbmc <openbmc-bounces+bonnie_lo=wiwynn.com@lists.ozlabs.org> On Behalf Of Joseph Reynolds Sent: Wednesday, December 11, 2019 5:11 AM To: Neeraj Ladkani <neladk@microsoft.com>; Alexander Tereschenko <aleksandr.v.tereschenko@linux.intel.com>; openbmc@lists.ozlabs.org; Patrick Venture <venture@google.com> Subject: Re: [EXTERNAL] Re: BMC update via TFTP On 12/10/19 12:58 PM, Neeraj Ladkani wrote: > Are there any thoughts to get rid of BMC reset to trigger FW update? I understand FW reset is required after the update. I'm not sure I understand the question. I think the answer depends on the [Software.VersionPurpose][1]. For VersionPurpose=BMC or System, the BMC must be reset. For VersionPurpose=Host, PSU, or Other, I don't know why the BMC would need to be reset. Do you want to be able to update non-BMC firmware without having to reset the BMC? - Joseph [1]: https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenbmc%2Fphosphor-dbus-interfaces%2Fblob%2Fmaster%2Fxyz%2Fopenbmc_project%2FSoftware%2FVersion.interface.yaml&data=02%7C01%7CBonnie_Lo%40wiwynn.com%7C9d56d702cea54e52e29808d77db630a1%7Cde0795e0d7c04eebb9bbbc94d8980d3b%7C1%7C0%7C637116093768947793&sdata=vAh3kDz2onq8b0%2Flx1AyPNyFngyPLag7go%2Fruw9nEtU%3D&reserved=0 > > -----Original Message----- > From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> > On Behalf Of Joseph Reynolds > Sent: Monday, December 9, 2019 5:25 PM > To: Alexander Tereschenko <aleksandr.v.tereschenko@linux.intel.com>; > openbmc@lists.ozlabs.org > Subject: [EXTERNAL] Re: BMC update via TFTP > > On 12/9/19 10:06 AM, Alexander Tereschenko wrote: >> On 06-Dec-19 23:52, Joseph Reynolds wrote: >>> I was thinking along the lines of adding [SFTP][] (or SCP) support >>> and then migrating existing TFTP users to the new secure solution. [...snip...] >> Yes, that could be a solution for the problem we discuss, providing >> both integrity and confidentiality, without any major OpenBMC >> development necessary - but it would mean more operational burden for >> BMC admins. The problem with SCP/SFTP in this context is that for >> this to work in the same manner as TFTP, the BMC must be an SSH >> client - i.e. have some sort of identity/credentials for the SCP/SFTP >> server provisioned first. That might not be the easiest solution to >> setup, but it's of course possible and can be automated if OpenBMC >> provides respective config knobs. >> >> Existing ways we have in code-update.md either don't require >> credentials (TFTP), so being a client is easy, or are not making a >> "client" from BMC, it's the admin who uploads stuff (SCP/REST). > Yes, that's what I was thinking. (And no, I am not going to recommend > setting up a SCP or SFTP server that allows anonymous access.) > > This highlight the need for OpenBMC to put together a guide to provisioning your BMC. Such as guide would give us a place to talk about uploading to the BMC SSH client certificates needed to access and download the firmware images. > > - Joseph > >> regards, >> Alexander >> ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [EXTERNAL] Re: BMC update via TFTP 2019-12-11 4:56 ` Bonnie Lo/WYHQ/Wiwynn @ 2019-12-11 5:59 ` Lei YU 0 siblings, 0 replies; 19+ messages in thread From: Lei YU @ 2019-12-11 5:59 UTC (permalink / raw) To: Bonnie Lo/WYHQ/Wiwynn Cc: Joseph Reynolds, Neeraj Ladkani, Alexander Tereschenko, openbmc, Patrick Venture, Aldofo Lin/WYHQ/Wiwynn, Delphine Chiu/WYHQ/Wiwynn On Wed, Dec 11, 2019 at 12:58 PM Bonnie Lo/WYHQ/Wiwynn <Bonnie_Lo@wiwynn.com> wrote: > > Dear Joseph, > > In my understanding, the BMC firmware update flow is as below: > 1. Trigger reboot > 2. Systemd stop all service > 3. Unmount file system > 4. image is in /run/initramfs > 5. Do the flashcp command to update the flash > > If there is any misunderstanding, please correct me. > > Based on the discussion with Neeraj. > We want to be able to update BMC firmware without having to trigger the BMC reboot command before the system do flashcp command. > It means that we can do the flashcp first. If the flashcp command complete and success, then we do the reset manually. > Is it workable on current upstream code? > If not, why? I means is there any advantage to trigger the reboot before we do the flashcp. It is not safe to flashcp the flash chip while BMC is running on it, because BMC's filesystem is mounted on the flash's RO and RW partitions. ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: BMC update via TFTP 2019-12-10 21:10 ` Joseph Reynolds 2019-12-11 4:56 ` Bonnie Lo/WYHQ/Wiwynn @ 2019-12-11 21:51 ` Milton Miller II 1 sibling, 0 replies; 19+ messages in thread From: Milton Miller II @ 2019-12-11 21:51 UTC (permalink / raw) To: Bonnie Lo/WYHQ/Wiwynn Cc: Joseph Reynolds, Neeraj Ladkani, Alexander Tereschenko, openbmc, Patrick Venture, Aldofo Lin/WYHQ/Wiwynn, Delphine Chiu/WYHQ/Wiwynn, Lei YU On 12/10/2019 around 10:57PM in some timezone, Bonnie Lo/WYHQ/Wiwynn wrote: >In my understanding, the BMC firmware update flow is as below: >1. Trigger reboot >2. Systemd stop all service >3. Unmount file system >4. image is in /run/initramfs >5. Do the flashcp command to update the flash > >If there is any misunderstanding, please correct me. > >Based on the discussion with Neeraj. >We want to be able to update BMC firmware without having to trigger >the BMC reboot command before the system do flashcp command. >It means that we can do the flashcp first. If the flashcp command >complete and success, then we do the reset manually. >Is it workable on current upstream code? >If not, why? I means is there any advantage to trigger the reboot >before we do the flashcp. The init script in the initramfs takes options from u-boot variables and the kernel command line. Some of the options allow that the read-only and files in the read-write partition are copied into RAM before the file systems are mounted, which allows the BMC to be operational while the new flash image is transfered and written into the flash. Note that if a reboot is triggered for some reason you can still end up needing additional recovery. This state can be initiated by "prepare for update" mentioned in the REST documentation for code-update. I don't think it can be initiated in the current Redfish. The flow is: 1. Trigger "prepare for update" - This uses the variable cleared every boot to copy rw partition content to ram (tmpfs), and the mtd ro partition ram (as a file then mounted by loop device). 2. Upload images as before - They will be signature checked as compiled then coped to /run/initramfs 3. run /run/initramfs/update - This script will look at /proc/mounts and see that the mtd-ro and mtdrw partitions are not present, and proceed to flashcp the files to the flash - WARNING: do not try to hide the mounts by unmounting the mtd ro and rw partitions. The root file system is overlayfs and will retain write access to the flash, and your BMC will crash 4. Initiate reboot at leisure to move to new code level - the contents of the tmpfs with the overlay files will be copied back to the rwfs file system in the mtd partitions Note: This is not the normal operation in part because the rwfs is in tmpfs and therefore any panic or power cut will result in all modifications to the rwfs being lost. If needed I can lookup the exact variable names and contents to communicate the desired maintenance actions from the phosphor init script. There are two variables, one for persistent settings, one for transient settings which a script clears each time. One can also customize or disable this features by adding additional customization files into the initramfs recipe. milton >>Thanks, >Bonnie > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-10 1:25 ` Joseph Reynolds 2019-12-10 18:58 ` [EXTERNAL] " Neeraj Ladkani @ 2019-12-11 11:02 ` Alexander Tereschenko 2019-12-11 20:17 ` Joseph Reynolds 1 sibling, 1 reply; 19+ messages in thread From: Alexander Tereschenko @ 2019-12-11 11:02 UTC (permalink / raw) To: openbmc [-- Attachment #1: Type: text/plain, Size: 1735 bytes --] >> Yes, that could be a solution for the problem we discuss, providing >> both integrity and confidentiality, without any major OpenBMC >> development necessary - but it would mean more operational burden for >> BMC admins. The problem with SCP/SFTP in this context is that for >> this to work in the same manner as TFTP, the BMC must be an SSH >> client - i.e. have some sort of identity/credentials for the SCP/SFTP >> server provisioned first. That might not be the easiest solution to >> setup, but it's of course possible and can be automated if OpenBMC >> provides respective config knobs. >> >> Existing ways we have in code-update.md either don't require >> credentials (TFTP), so being a client is easy, or are not making a >> "client" from BMC, it's the admin who uploads stuff (SCP/REST). > > Yes, that's what I was thinking. (And no, I am not going to recommend > setting up a SCP or SFTP server that allows anonymous access.) > > This highlight the need for OpenBMC to put together a guide to > provisioning your BMC. Such as guide would give us a place to talk > about uploading to the BMC SSH client certificates needed to access > and download the firmware images. > > - Joseph Agree, the provisioning guide could be a good point to have this discussion. However I beieve updates in general is a broader and more "operational" (i.e. "continuous" as opposed to provisioning being rather "one-time") topic, so the approach in the organization/of a given BMC admin can change and I believe whatever configuration mechanism we develop for this (if at all), should be available at any point during BMC lifetime, not only at provisioning, and be architected respectively. regards, Alexander [-- Attachment #2: Type: text/html, Size: 2368 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: BMC update via TFTP 2019-12-11 11:02 ` Alexander Tereschenko @ 2019-12-11 20:17 ` Joseph Reynolds 0 siblings, 0 replies; 19+ messages in thread From: Joseph Reynolds @ 2019-12-11 20:17 UTC (permalink / raw) To: Alexander Tereschenko, openbmc On 12/11/19 5:02 AM, Alexander Tereschenko wrote: >>> Yes, that could be a solution for the problem we discuss, providing >>> both integrity and confidentiality, without any major OpenBMC >>> development necessary - but it would mean more operational burden >>> for BMC admins. The problem with SCP/SFTP in this context is that >>> for this to work in the same manner as TFTP, the BMC must be an SSH >>> client - i.e. have some sort of identity/credentials for the >>> SCP/SFTP server provisioned first. That might not be the easiest >>> solution to setup, but it's of course possible and can be automated >>> if OpenBMC provides respective config knobs. >>> >>> Existing ways we have in code-update.md either don't require >>> credentials (TFTP), so being a client is easy, or are not making a >>> "client" from BMC, it's the admin who uploads stuff (SCP/REST). >> >> Yes, that's what I was thinking. (And no, I am not going to >> recommend setting up a SCP or SFTP server that allows anonymous access.) >> >> This highlight the need for OpenBMC to put together a guide to >> provisioning your BMC. Such as guide would give us a place to talk >> about uploading to the BMC SSH client certificates needed to access >> and download the firmware images. >> >> - Joseph > > Agree, the provisioning guide could be a good point to have this > discussion. However I beieve updates in general is a broader and more > "operational" (i.e. "continuous" as opposed to provisioning being > rather "one-time") topic, so the approach in the organization/of a > given BMC admin can change and I believe whatever configuration > mechanism we develop for this (if at all), should be available at any > point during BMC lifetime, not only at provisioning, and be > architected respectively. > That makes sense to me. Thanks! I'll plan to add an item to both: https://github.com/openbmc/docs/blob/master/security/network-security-considerations.md and to the threat model considerations (in review): https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/22404 - Joseph > > regards, > Alexander > > ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2019-12-11 21:51 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-12-03 12:02 BMC update via TFTP rgrs 2019-12-03 16:12 ` Gunnar Mills 2019-12-03 17:08 ` Gunnar Mills 2019-12-03 19:55 ` Adriana Kobylak 2019-12-04 15:47 ` Joseph Reynolds 2019-12-04 21:36 ` Vernon Mauery 2019-12-05 11:05 ` Alexander Tereschenko 2019-12-05 22:37 ` Vernon Mauery 2019-12-06 14:03 ` Alexander Tereschenko 2019-12-06 22:52 ` Joseph Reynolds 2019-12-09 16:06 ` Alexander Tereschenko 2019-12-10 1:25 ` Joseph Reynolds 2019-12-10 18:58 ` [EXTERNAL] " Neeraj Ladkani 2019-12-10 21:10 ` Joseph Reynolds 2019-12-11 4:56 ` Bonnie Lo/WYHQ/Wiwynn 2019-12-11 5:59 ` Lei YU 2019-12-11 21:51 ` Milton Miller II 2019-12-11 11:02 ` Alexander Tereschenko 2019-12-11 20:17 ` Joseph Reynolds
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.