* ipmi password storage @ 2020-04-13 23:00 Vernon Mauery 2020-04-14 15:50 ` Patrick Williams ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Vernon Mauery @ 2020-04-13 23:00 UTC (permalink / raw) To: OpenBMC Development Internally, an issue was raised that basically says that the mechanism by which we are storing the IPMI passwords on the BMC is insufficiently obfuscated. I have come up with a patch set that resolves this at the expense of no downgrading the BMC without the side-effect of losing all IPMI passwords. I would like to know what the community thinks about usability vs. security in this scenario. Current Implementation ====================== 1) If the user is part of the ipmi group (/etc/group) then when the user changes their password, pam-ipmisave.so intercepts the password as a one of the PAM layers and saves it, encrypted, to /etc/ipmi_pass. 2) Encryption (obfuscation, because we don't really have a secure mechanism of storing secret keys), is done like this: a) read 8 bytes (S) from /etc/key_file (currently pre-loaded with "OPENBMC=") b) create a random value H (read from /dev/urandom) c) create the AES-CBC secret key K=HMAC-SHA256(S, H) d) encrypt the list of username:password data using K e) store H along with the encrypted data in /etc/ipmi_pass 3) reading the password (for establishing IPMI RMCP+ sessions) a) read 8 bytes (S) from /etc/key_file b) read H from /etc/ipmi_pass c) compute the AES-CBC secret key K=HMAC-SHA256(S, H) d) decrypt and verify the contents of /etc/key_file There are many issues with this mechanism, but we cannot fix all of them without some secure mechanism for storing secret keys. That is why really, at best, this is obfuscation, not encryption. The data is not in plain text, it takes some work to get to it. More than xor or rot13, but not so much that a person could do it with a bash script. 1) the default /etc/key_file is the same for every BMC built with the default settings (changing this requires a bbappend for pam-ipmi). This means the /etc/key_file could basically not exist; all you need is the algorithm and /etc/ipmi_pass. 2) the size of the /etc/key_file is also not really great. Even if it was different on every machine, computing only 2^64 possibilities is not so hard. Possible Solution ================= Migrate to a solution that uses a key that is longer that does not get written directly to the flash 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID) 2) S is longer (32 bytes instead of 8) 3) S is not written to flash, because it can be computed 4) S is different for every machine because it is a derivative of /etc/machine-id The migration from the old mechanism to the new could be done simply by using the new key on the next write to the /etc/ipmi_pass file. After a firmware update to this new code, a password change would trigger a decrypt of the /etc/ipmi_pass file, a modification of the plain text, and a re-encryption of the data. If it reads the 'legacy' key in and writes out the data using the new key mechanism and deletes the legacy key, it would use the new key mechanism from that point onward. However, this would cause any downgrades to prior versions to fail to decrypt the /etc/ipmi_pass file, thereby losing all the ipmi passwords. This is not ideal, but could possibly be mitigating by truncating the new machine-id derivative password to 8 bytes and storing it in the /etc/key_file instead of just deleting it. This might improve security only slightly at for the price of a better user experience. I know that some companies using OpenBMC have products with users out in the field, so it is not great to make changes like this. Also, it is not great to have low-grade security. So here I am, writing to ask for opinions and options. --Vernon ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-13 23:00 ipmi password storage Vernon Mauery @ 2020-04-14 15:50 ` Patrick Williams 2020-04-14 16:46 ` Vernon Mauery 2020-04-14 16:27 ` Alexander Tereschenko 2020-04-14 18:04 ` Milton Miller II 2 siblings, 1 reply; 21+ messages in thread From: Patrick Williams @ 2020-04-14 15:50 UTC (permalink / raw) To: Vernon Mauery; +Cc: OpenBMC Development [-- Attachment #1: Type: text/plain, Size: 4069 bytes --] On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote: Vernon, Is there some background pointers on why IPMI needs to store passwords in a reverable way? I never understood that. > Internally, an issue was raised that basically says that the mechanism > by which we are storing the IPMI passwords on the BMC is insufficiently > obfuscated. I have come up with a patch set that resolves this at the > expense of no downgrading the BMC without the side-effect of losing all > IPMI passwords. I would like to know what the community thinks about > usability vs. security in this scenario. > > Current Implementation > ====================== > 1) If the user is part of the ipmi group (/etc/group) then when the user > changes their password, pam-ipmisave.so intercepts the password as a one > of the PAM layers and saves it, encrypted, to /etc/ipmi_pass. > 2) Encryption (obfuscation, because we don't really have a secure > mechanism of storing secret keys), is done like this: > a) read 8 bytes (S) from /etc/key_file (currently pre-loaded with "OPENBMC=") > b) create a random value H (read from /dev/urandom) > c) create the AES-CBC secret key K=HMAC-SHA256(S, H) > d) encrypt the list of username:password data using K > e) store H along with the encrypted data in /etc/ipmi_pass > 3) reading the password (for establishing IPMI RMCP+ sessions) > a) read 8 bytes (S) from /etc/key_file > b) read H from /etc/ipmi_pass > c) compute the AES-CBC secret key K=HMAC-SHA256(S, H) > d) decrypt and verify the contents of /etc/key_file > > > There are many issues with this mechanism, but we cannot fix all of them > without some secure mechanism for storing secret keys. That is why > really, at best, this is obfuscation, not encryption. The data is not in > plain text, it takes some work to get to it. More than xor or rot13, but > not so much that a person could do it with a bash script. > 1) the default /etc/key_file is the same for every BMC built with the > default settings (changing this requires a bbappend for pam-ipmi). This > means the /etc/key_file could basically not exist; all you need is the > algorithm and /etc/ipmi_pass. > 2) the size of the /etc/key_file is also not really great. Even if it > was different on every machine, computing only 2^64 possibilities is not > so hard. > > > Possible Solution > ================= > Migrate to a solution that uses a key that is longer that does not > get written directly to the flash > 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID) > 2) S is longer (32 bytes instead of 8) > 3) S is not written to flash, because it can be computed > 4) S is different for every machine because it is a derivative of > /etc/machine-id > > The migration from the old mechanism to the new could be done simply by > using the new key on the next write to the /etc/ipmi_pass file. After a > firmware update to this new code, a password change would trigger a > decrypt of the /etc/ipmi_pass file, a modification of the plain text, > and a re-encryption of the data. If it reads the 'legacy' key in and > writes out the data using the new key mechanism and deletes the legacy > key, it would use the new key mechanism from that point onward. However, > this would cause any downgrades to prior versions to fail to decrypt the > /etc/ipmi_pass file, thereby losing all the ipmi passwords. This is not > ideal, but could possibly be mitigating by truncating the new machine-id > derivative password to 8 bytes and storing it in the /etc/key_file > instead of just deleting it. This might improve security only slightly > at for the price of a better user experience. > > I know that some companies using OpenBMC have products with users out in > the field, so it is not great to make changes like this. Also, it is not > great to have low-grade security. So here I am, writing to ask for > opinions and options. > > --Vernon -- Patrick Williams [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 15:50 ` Patrick Williams @ 2020-04-14 16:46 ` Vernon Mauery 2020-04-14 22:03 ` Joseph Reynolds 2020-04-22 11:38 ` Patrick Williams 0 siblings, 2 replies; 21+ messages in thread From: Vernon Mauery @ 2020-04-14 16:46 UTC (permalink / raw) To: Patrick Williams; +Cc: OpenBMC Development On 14-Apr-2020 10:50 AM, Patrick Williams wrote: >On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote: > >Vernon, > >Is there some background pointers on why IPMI needs to store passwords >in a reverable way? I never understood that. Sure. I think this is most clearly described in section 13.31 "RMCP+ Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec. Here is a high level overview though: It is baked into the RMCP+ key exchange (RAKP). The password never goes across the wire in plaintext, but it does get used during the session creation in RAKP messages as the key to integrity and authentication HMACs. Specifically, the RAKP3 message (User->BMC) contains an HMAC of the various parts of the exchanged session (User random number, session ID, Role, Username) using the password as the key for the HMAC. The BMC needs to compute this same HMAC to compare (this is the main authentication challenge). Then, the session key is generated using an HMAC of similar data (BMC random number, user random number, role, username) using either the user password or the channel password. Almost nobody uses the channel password, which is good because it allows for simple privilege escalation and session hijacking. Both sides use the same inputs and HMAC key, so the BMC needs to store the user passwords in a way that they can be used as the key for an HMAC. Ideally this would be stored in some sort of secure enclave or HSM, but that is not yet available. --Vernon >> Internally, an issue was raised that basically says that the mechanism >> by which we are storing the IPMI passwords on the BMC is insufficiently >> obfuscated. I have come up with a patch set that resolves this at the >> expense of no downgrading the BMC without the side-effect of losing all >> IPMI passwords. I would like to know what the community thinks about >> usability vs. security in this scenario. >> >> Current Implementation >> ====================== >> 1) If the user is part of the ipmi group (/etc/group) then when the user >> changes their password, pam-ipmisave.so intercepts the password as a one >> of the PAM layers and saves it, encrypted, to /etc/ipmi_pass. >> 2) Encryption (obfuscation, because we don't really have a secure >> mechanism of storing secret keys), is done like this: >> a) read 8 bytes (S) from /etc/key_file (currently pre-loaded with "OPENBMC=") >> b) create a random value H (read from /dev/urandom) >> c) create the AES-CBC secret key K=HMAC-SHA256(S, H) >> d) encrypt the list of username:password data using K >> e) store H along with the encrypted data in /etc/ipmi_pass >> 3) reading the password (for establishing IPMI RMCP+ sessions) >> a) read 8 bytes (S) from /etc/key_file >> b) read H from /etc/ipmi_pass >> c) compute the AES-CBC secret key K=HMAC-SHA256(S, H) >> d) decrypt and verify the contents of /etc/key_file >> >> >> There are many issues with this mechanism, but we cannot fix all of them >> without some secure mechanism for storing secret keys. That is why >> really, at best, this is obfuscation, not encryption. The data is not in >> plain text, it takes some work to get to it. More than xor or rot13, but >> not so much that a person could do it with a bash script. >> 1) the default /etc/key_file is the same for every BMC built with the >> default settings (changing this requires a bbappend for pam-ipmi). This >> means the /etc/key_file could basically not exist; all you need is the >> algorithm and /etc/ipmi_pass. >> 2) the size of the /etc/key_file is also not really great. Even if it >> was different on every machine, computing only 2^64 possibilities is not >> so hard. >> >> >> Possible Solution >> ================= >> Migrate to a solution that uses a key that is longer that does not >> get written directly to the flash >> 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID) >> 2) S is longer (32 bytes instead of 8) >> 3) S is not written to flash, because it can be computed >> 4) S is different for every machine because it is a derivative of >> /etc/machine-id >> >> The migration from the old mechanism to the new could be done simply by >> using the new key on the next write to the /etc/ipmi_pass file. After a >> firmware update to this new code, a password change would trigger a >> decrypt of the /etc/ipmi_pass file, a modification of the plain text, >> and a re-encryption of the data. If it reads the 'legacy' key in and >> writes out the data using the new key mechanism and deletes the legacy >> key, it would use the new key mechanism from that point onward. However, >> this would cause any downgrades to prior versions to fail to decrypt the >> /etc/ipmi_pass file, thereby losing all the ipmi passwords. This is not >> ideal, but could possibly be mitigating by truncating the new machine-id >> derivative password to 8 bytes and storing it in the /etc/key_file >> instead of just deleting it. This might improve security only slightly >> at for the price of a better user experience. >> >> I know that some companies using OpenBMC have products with users out in >> the field, so it is not great to make changes like this. Also, it is not >> great to have low-grade security. So here I am, writing to ask for >> opinions and options. >> >> --Vernon > >-- >Patrick Williams ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 16:46 ` Vernon Mauery @ 2020-04-14 22:03 ` Joseph Reynolds 2020-04-14 22:42 ` Vernon Mauery 2020-04-22 11:38 ` Patrick Williams 1 sibling, 1 reply; 21+ messages in thread From: Joseph Reynolds @ 2020-04-14 22:03 UTC (permalink / raw) To: Vernon Mauery, Patrick Williams; +Cc: OpenBMC Development On 4/14/20 11:46 AM, Vernon Mauery wrote: > On 14-Apr-2020 10:50 AM, Patrick Williams wrote: >> On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote: >> >> Vernon, >> >> Is there some background pointers on why IPMI needs to store passwords >> in a reverable way? I never understood that. > > Sure. I think this is most clearly described in section 13.31 "RMCP+ > Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec. This may be a bit naive.... Is it possible to expand the RCMP+ spec with a new cipher suite or similar, to use a mechanism more like HTTPS or SSH that does not require the server to be able to produce a clear text password? Given that IPMI will be used for many years, this seems worthwhile, and would solve the current problem. - Joseph > --Vernon > >>> >>> >>> --Vernon >> >> -- >> Patrick Williams > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 22:03 ` Joseph Reynolds @ 2020-04-14 22:42 ` Vernon Mauery 2020-04-16 6:04 ` Joel Stanley 0 siblings, 1 reply; 21+ messages in thread From: Vernon Mauery @ 2020-04-14 22:42 UTC (permalink / raw) To: Joseph Reynolds; +Cc: Patrick Williams, OpenBMC Development, Alexander Amelkin On 14-Apr-2020 05:03 PM, Joseph Reynolds wrote: > > >On 4/14/20 11:46 AM, Vernon Mauery wrote: >>On 14-Apr-2020 10:50 AM, Patrick Williams wrote: >>>On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote: >>> >>>Vernon, >>> >>>Is there some background pointers on why IPMI needs to store passwords >>>in a reverable way? I never understood that. >> >>Sure. I think this is most clearly described in section 13.31 "RMCP+ >>Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec. > >This may be a bit naive.... Is it possible to expand the RCMP+ spec >with a new cipher suite or similar, to use a mechanism more like HTTPS >or SSH that does not require the server to be able to produce a clear >text password? Given that IPMI will be used for many years, this >seems worthwhile, and would solve the current problem. While IPMI will not likely be abandoned for many years to come, there are not any plans to update the specification. Redfish is supposed to be the answer, but like IPv4 was supposed to be supplanted by IPv6 long ago, full adoption is still dragging its feet. That being said, I am not opposed to creating a new de-facto standard. In the name of security, I would not be opposed to using modern crypto protocols to establish secure IPMI sessions. This would likely cause the adoption of redfish to be even slower, because the biggest detractor of IPMI would be fixed. We have the maintainer of ipmitool as a member of the OpenBMC community, (Alexander Amelkin) so we could even implement both ends of this new de-facto standard. I would suggest a DTLS connection on UDP 623, fully replacing RCMP+. --Vernon ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 22:42 ` Vernon Mauery @ 2020-04-16 6:04 ` Joel Stanley 2020-04-20 14:29 ` Vernon Mauery 0 siblings, 1 reply; 21+ messages in thread From: Joel Stanley @ 2020-04-16 6:04 UTC (permalink / raw) To: Vernon Mauery; +Cc: Joseph Reynolds, OpenBMC Development, Alexander Amelkin On Tue, 14 Apr 2020 at 22:43, Vernon Mauery <vernon.mauery@linux.intel.com> wrote: > > On 14-Apr-2020 05:03 PM, Joseph Reynolds wrote: > > > > > >On 4/14/20 11:46 AM, Vernon Mauery wrote: > >>On 14-Apr-2020 10:50 AM, Patrick Williams wrote: > >>>On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote: > >>> > >>>Vernon, > >>> > >>>Is there some background pointers on why IPMI needs to store passwords > >>>in a reverable way? I never understood that. > >> > >>Sure. I think this is most clearly described in section 13.31 "RMCP+ > >>Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec. > > > >This may be a bit naive.... Is it possible to expand the RCMP+ spec > >with a new cipher suite or similar, to use a mechanism more like HTTPS > >or SSH that does not require the server to be able to produce a clear > >text password? Given that IPMI will be used for many years, this > >seems worthwhile, and would solve the current problem. > > While IPMI will not likely be abandoned for many years to come, there > are not any plans to update the specification. Redfish is supposed to be > the answer, but like IPv4 was supposed to be supplanted by IPv6 long > ago, full adoption is still dragging its feet. > > That being said, I am not opposed to creating a new de-facto standard. > In the name of security, I would not be opposed to using modern crypto > protocols to establish secure IPMI sessions. This would likely cause the > adoption of redfish to be even slower, because the biggest detractor of > IPMI would be fixed. This is a great suggestion. > We have the maintainer of ipmitool as a member of the OpenBMC community, > (Alexander Amelkin) so we could even implement both ends of this new > de-facto standard. I would suggest a DTLS connection on UDP 623, fully > replacing RCMP+. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-16 6:04 ` Joel Stanley @ 2020-04-20 14:29 ` Vernon Mauery 0 siblings, 0 replies; 21+ messages in thread From: Vernon Mauery @ 2020-04-20 14:29 UTC (permalink / raw) To: Joel Stanley; +Cc: Joseph Reynolds, OpenBMC Development, Alexander Amelkin On 16-Apr-2020 06:04 AM, Joel Stanley wrote: >On Tue, 14 Apr 2020 at 22:43, Vernon Mauery ><vernon.mauery@linux.intel.com> wrote: >> >> On 14-Apr-2020 05:03 PM, Joseph Reynolds wrote: >> > >> > >> >On 4/14/20 11:46 AM, Vernon Mauery wrote: >> >>On 14-Apr-2020 10:50 AM, Patrick Williams wrote: >> >>>On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote: >> >>> >> >>>Vernon, >> >>> >> >>>Is there some background pointers on why IPMI needs to store passwords >> >>>in a reverable way? I never understood that. >> >> >> >>Sure. I think this is most clearly described in section 13.31 "RMCP+ >> >>Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec. >> > >> >This may be a bit naive.... Is it possible to expand the RCMP+ spec >> >with a new cipher suite or similar, to use a mechanism more like HTTPS >> >or SSH that does not require the server to be able to produce a clear >> >text password? Given that IPMI will be used for many years, this >> >seems worthwhile, and would solve the current problem. >> >> While IPMI will not likely be abandoned for many years to come, there >> are not any plans to update the specification. Redfish is supposed to be >> the answer, but like IPv4 was supposed to be supplanted by IPv6 long >> ago, full adoption is still dragging its feet. >> >> That being said, I am not opposed to creating a new de-facto standard. >> In the name of security, I would not be opposed to using modern crypto >> protocols to establish secure IPMI sessions. This would likely cause the >> adoption of redfish to be even slower, because the biggest detractor of >> IPMI would be fixed. > >This is a great suggestion. Thanks! Here is a design doc I wrote up. https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/31548 --Vernon >> We have the maintainer of ipmitool as a member of the OpenBMC community, >> (Alexander Amelkin) so we could even implement both ends of this new >> de-facto standard. I would suggest a DTLS connection on UDP 623, fully >> replacing RCMP+. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 16:46 ` Vernon Mauery 2020-04-14 22:03 ` Joseph Reynolds @ 2020-04-22 11:38 ` Patrick Williams 2020-04-22 22:19 ` Vernon Mauery 1 sibling, 1 reply; 21+ messages in thread From: Patrick Williams @ 2020-04-22 11:38 UTC (permalink / raw) To: Vernon Mauery; +Cc: OpenBMC Development [-- Attachment #1: Type: text/plain, Size: 2180 bytes --] Vernon, Thanks for the info. It's taken a few days before I could find a time slot to think about this in detail. On Tue, Apr 14, 2020 at 09:46:10AM -0700, Vernon Mauery wrote: > On 14-Apr-2020 10:50 AM, Patrick Williams wrote: > >On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote: > Specifically, the RAKP3 message (User->BMC) contains an HMAC of the > various parts of the exchanged session (User random number, session ID, Role, > Username) using the password as the key for the HMAC. The BMC needs to > compute this same HMAC to compare (this is the main authentication > challenge). > > Then, the session key is generated using an HMAC of similar > data (BMC random number, user random number, role, username) using > either the user password or the channel password. Almost nobody uses the > channel password, which is good because it allows for simple privilege > escalation and session hijacking. If I understand this right, the algorithm uses this to create a symetric key for the session: HMAC(password, hash(pseudo_random_session_data)) The client gives some data to seed the session-data and the server gives some data to seed the session-data, so that part is unique per connection. The part that is constant, and that they both have, is the key to the HMAC. Do I have that right? The HMAC algorithm, at least for SHA-1/2, is a two-phase hash call, but the key is always used at the front of the first hash phase. The SHA hash algorithms allow you to do a partial hash, extract the hash state, and then resume the hash (See SHA1_Init / SHA1_Update functions in openssl/sha1.h as example). Rather than calling the OpenSSL HMAC directly, can't we rewrite it such that it uses the partial hash on the key as the starting point rather than the raw password? This approach would allow us to save a SHA-1 + SHA-2 hash'd version of the password rather than the raw (or even obfuscated) password. It isn't as secure as the normal crypt methods because we cannot salt them, but we could certainly obfuscate the hash-state in a similar method to what you already proposed. -- Patrick Williams [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-22 11:38 ` Patrick Williams @ 2020-04-22 22:19 ` Vernon Mauery 2020-04-24 15:15 ` Patrick Williams 0 siblings, 1 reply; 21+ messages in thread From: Vernon Mauery @ 2020-04-22 22:19 UTC (permalink / raw) To: Patrick Williams; +Cc: OpenBMC Development On 22-Apr-2020 06:38 AM, Patrick Williams wrote: >Vernon, > >Thanks for the info. It's taken a few days before I could find a time >slot to think about this in detail. > >On Tue, Apr 14, 2020 at 09:46:10AM -0700, Vernon Mauery wrote: >> On 14-Apr-2020 10:50 AM, Patrick Williams wrote: >> >On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote: >> Specifically, the RAKP3 message (User->BMC) contains an HMAC of the >> various parts of the exchanged session (User random number, session ID, Role, >> Username) using the password as the key for the HMAC. The BMC needs to >> compute this same HMAC to compare (this is the main authentication >> challenge). >> >> Then, the session key is generated using an HMAC of similar >> data (BMC random number, user random number, role, username) using >> either the user password or the channel password. Almost nobody uses the >> channel password, which is good because it allows for simple privilege >> escalation and session hijacking. > >If I understand this right, the algorithm uses this to create a symetric >key for the session: > HMAC(password, hash(pseudo_random_session_data)) > >The client gives some data to seed the session-data and the server gives >some data to seed the session-data, so that part is unique per >connection. The part that is constant, and that they both have, is the >key to the HMAC. Do I have that right? Yes. >The HMAC algorithm, at least for SHA-1/2, is a two-phase hash call, but >the key is always used at the front of the first hash phase. The SHA >hash algorithms allow you to do a partial hash, extract the hash state, >and then resume the hash (See SHA1_Init / SHA1_Update functions in >openssl/sha1.h as example). Rather than calling the OpenSSL HMAC >directly, can't we rewrite it such that it uses the partial hash on the >key as the starting point rather than the raw password? I had to code up a quick test to make sure, but yes, it is possible to do this. >This approach would allow us to save a SHA-1 + SHA-2 hash'd version of >the password rather than the raw (or even obfuscated) password. It >isn't as secure as the normal crypt methods because we cannot salt them, >but we could certainly obfuscate the hash-state in a similar method to >what you already proposed. Pros and cons of this new solution: Pros: 1) Obfuscates passwords more than currently. A hash is even harder to deal with than an encrypted file that has the key sitting right next to it. 2) Item one is a very strong plus Cons: 1) Would require us to write or hack up an HMAC that would work for us. This is not ideal; writing crypto stuff is generally best left to the experts. OpenSSL is trying harder and harder to hide its internal workings from its API, so extracting the hash state is hard. libTomCrypt allows you to do this easier, but still does not have an HMAC function that takes two partial hash states as inputs. 2) Is not backwards compatible with any of our current password storage mechanisms. This is workable, but would likely take some time to stage things so that they would work better without breaking everything. 3) We would need to store two hashes of each password for both SHA1 and SHA256, for a total of 4 states per password. This greatly reduces the effectiveness of the solution in the first place. Not that we are planning on adding MD5- or MD2-based cipher suites, but those would break the value of this completely. So pro-1 is good. but con-2 and con-3 are bad. Con-2 we can deal with, but con-3 brings down the value of pro-1. --Vernon ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-22 22:19 ` Vernon Mauery @ 2020-04-24 15:15 ` Patrick Williams 2020-04-29 16:02 ` Alexander Tereschenko 0 siblings, 1 reply; 21+ messages in thread From: Patrick Williams @ 2020-04-24 15:15 UTC (permalink / raw) To: Vernon Mauery; +Cc: OpenBMC Development [-- Attachment #1: Type: text/plain, Size: 2614 bytes --] On Wed, Apr 22, 2020 at 03:19:43PM -0700, Vernon Mauery wrote: > Pros and cons of this new solution: > Pros: > 1) Obfuscates passwords more than currently. A hash is even harder to > deal with than an encrypted file that has the key sitting right next to > it. > 2) Item one is a very strong plus > > Cons: > 1) Would require us to write or hack up an HMAC that would work for us. > This is not ideal; writing crypto stuff is generally best left to the > experts. OpenSSL is trying harder and harder to hide its internal > workings from its API, so extracting the hash state is hard. libTomCrypt > allows you to do this easier, but still does not have an HMAC function > that takes two partial hash states as inputs. I don't think that the interfaces we'd be needing to use here will ever be deprecated. The "hash-resume" functions are needed in order to hash very large files that cannot be fit in memory. They're just too useful to deprecate. You are correct that we would be effectively implementing HMAC ourselves using the SHA hash functions though. It _seems_ straight-forward enough though on the surface. > 2) Is not backwards compatible with any of our current password storage > mechanisms. This is workable, but would likely take some time to stage > things so that they would work better without breaking everything. Agreed, but it isn't any less backwards compatible than the obfuscation proposal. This could certainly be staged in with a compile flag that keeps the old file around. If a vendor is concerned about backwards compatiblity they could enable that flag for a period of time until their customers upgrade through the new support version. (Disabled by default in meta-phosphor). > 3) We would need to store two hashes of each password for both SHA1 and > SHA256, for a total of 4 states per password. This greatly reduces the > effectiveness of the solution in the first place. Not that we are > planning on adding MD5- or MD2-based cipher suites, but those would > break the value of this completely. Agree on keeping anything like MD5/MD2 out of it. I'm not a cryptographic expert, but I'm not sure it "greatly reduces" the effectiveness. The addition of the SHA-2 state probably doesn't help you reverse or collide the SHA-1 much at all, so this is only slightly weaker than SHA-1 itself. My reading on the SHA-1 algorithm would lead me to believe that SHA-1 state is the same as SHA-1(password) itself. Meaning, by doing this partial hash we're not weaking the algorithm at all. -- Patrick Williams [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-24 15:15 ` Patrick Williams @ 2020-04-29 16:02 ` Alexander Tereschenko 0 siblings, 0 replies; 21+ messages in thread From: Alexander Tereschenko @ 2020-04-29 16:02 UTC (permalink / raw) To: openbmc On 24-Apr-20 17:15, Patrick Williams wrote: > On Wed, Apr 22, 2020 at 03:19:43PM -0700, Vernon Mauery wrote: >> Pros and cons of this new solution: >> Pros: >> 1) Obfuscates passwords more than currently. A hash is even harder to >> deal with than an encrypted file that has the key sitting right next to >> it. >> 2) Item one is a very strong plus >> >> Cons: >> 1) Would require us to write or hack up an HMAC that would work for us. >> This is not ideal; writing crypto stuff is generally best left to the >> experts. OpenSSL is trying harder and harder to hide its internal >> workings from its API, so extracting the hash state is hard. libTomCrypt >> allows you to do this easier, but still does not have an HMAC function >> that takes two partial hash states as inputs. > I don't think that the interfaces we'd be needing to use here will ever > be deprecated. The "hash-resume" functions are needed in order to hash > very large files that cannot be fit in memory. They're just too useful > to deprecate. > > You are correct that we would be effectively implementing HMAC ourselves > using the SHA hash functions though. It _seems_ straight-forward enough > though on the surface. Yes, that should be doable and HMAC is generally not that prone to implementation problems, so this is reasonable to assume it could be done safely using just the plain hash primitives. The major problem with this one, from my standpoint, is that the unsalted hashes are these days a too easy target for attackers, as hash tables can be (and actually are, you can download those I believe) precomputed on password dictionaries and without salt it's a BORE-type risk (break once, run everywhere) - the hash will be the same on all BMCs if the password is the same, so compromising one means compromising others. >> 3) We would need to store two hashes of each password for both SHA1 and >> SHA256, for a total of 4 states per password. This greatly reduces the >> effectiveness of the solution in the first place. Not that we are >> planning on adding MD5- or MD2-based cipher suites, but those would >> break the value of this completely. > Agree on keeping anything like MD5/MD2 out of it. > > I'm not a cryptographic expert, but I'm not sure it "greatly reduces" > the effectiveness. The addition of the SHA-2 state probably doesn't > help you reverse or collide the SHA-1 much at all, so this is only > slightly weaker than SHA-1 itself. > > My reading on the SHA-1 algorithm would lead me to believe that SHA-1 state > is the same as SHA-1(password) itself. Meaning, by doing this partial > hash we're not weaking the algorithm at all. For SHA-1 and SHA2-256/512 the hashing output *is* the hash internal state at the end of calculation, so there's no additional compromise here from the crypto standpoint. Essentially, this proposal is a hash length extension attack, but used for good, not evil :) The problem here is with hashes being unsalted, but I agree this is still better than encrypting with a static key thay lies nearby. FWIW, that DTLS solution looks a bit better to me as it allows us to ditch that password thing completely, but if that one is not possible, this one could be the next best. BTW, Vernon, could you please elaborate why it's 4 values that we need to store here? One is SHA1, another one is SHA2-256, but what are the other two? regards, Alexander ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-13 23:00 ipmi password storage Vernon Mauery 2020-04-14 15:50 ` Patrick Williams @ 2020-04-14 16:27 ` Alexander Tereschenko 2020-04-14 19:11 ` Vernon Mauery 2020-04-14 18:04 ` Milton Miller II 2 siblings, 1 reply; 21+ messages in thread From: Alexander Tereschenko @ 2020-04-14 16:27 UTC (permalink / raw) To: openbmc To be more specific, I'm considering two attack scenarios in the below comments: 1) The attacker gets into BMC and is able to copy off the data files, including ipmi_pass. This IMHO is a more realistic scenario in this case. 2) The attacker gets ipmi_pass file/contents only, without being able to retrieve anything else. Which ones do *you* have in mind? For the sake of discussion, it would be helpful to specify them all and see how potential solutions address them. On 14-Apr-20 01:00, Vernon Mauery wrote: > Possible Solution > ================= > Migrate to a solution that uses a key that is longer that does not get > written directly to the flash > 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID) I like the direction and the guiding principle, however from "proper" cryptography standpoint, for producing keys, this is not noticeably better than the previous solution. The key material is still readily available on the filesystem (making points #3 and #4 irrelevant for attack scenario #1), so chances are that the adversary who can pull /etc/ipmi_pass file, can also pull /etc/machine-id one. In addition (but that's really a nitpick in this case with all these much bigger problems + easy to fix), plain HMAC is not a proper and recommended key derivation function, an HKDF or one of the NIST constructs from SP800-108 should be used instead. > 2) S is longer (32 bytes instead of 8) This (and #4) only helps the attack scenario #2, where attacker has to brute-force all possible values for S - and I believe that scenario is less realictic. But yes, case #2 gets a bit better than before, if we disregard the fact that HMAC is not a proper KDF and that MachineID is not a proper cryptographic key material [1, last paragraph in Description suggests that], which may provide additional advantage to the attacker. BTW, machine-id has at most 128 bits of entropy (if produced by a proper CSRNG, which I'm not sure about - e.g. I see they do some formatting), so practically it's not 32 bytes, but 16 [at most]. So while this new approach does provide some advantage for scenario #2, it doesn't address a more important case of #1 and to me it still looks like low security. The proper way here is indeed to get some capability for storing the keys securely, but I see how this is hard to impossible on AST2500 (if that's what we're talking about here), without TrustZone or anything similar, or some sort of ROM/bootloader-accessible-only part-unique secret, so OTMH I can't propose any viable alternatives for a given context (brownfield deployment, backward compatibility), but I'll think more about that. [1] https://www.freedesktop.org/software/systemd/man/machine-id.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 16:27 ` Alexander Tereschenko @ 2020-04-14 19:11 ` Vernon Mauery 2020-04-14 22:18 ` Joseph Reynolds 0 siblings, 1 reply; 21+ messages in thread From: Vernon Mauery @ 2020-04-14 19:11 UTC (permalink / raw) To: Alexander Tereschenko; +Cc: openbmc On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote: >To be more specific, I'm considering two attack scenarios in the below >comments: >1) The attacker gets into BMC and is able to copy off the data files, >including ipmi_pass. This IMHO is a more realistic scenario in this >case. >2) The attacker gets ipmi_pass file/contents only, without being able >to retrieve anything else. > >Which ones do *you* have in mind? For the sake of discussion, it would >be helpful to specify them all and see how potential solutions address >them. Attack one is the most likely, since if you can read one file, you can probably get any/all of them. >On 14-Apr-20 01:00, Vernon Mauery wrote: >>Possible Solution >>================= >>Migrate to a solution that uses a key that is longer that does not >>get written directly to the flash >>1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID) > >I like the direction and the guiding principle, however from "proper" >cryptography standpoint, for producing keys, this is not noticeably >better than the previous solution. The key material is still readily >available on the filesystem (making points #3 and #4 irrelevant for >attack scenario #1), so chances are that the adversary who can pull >/etc/ipmi_pass file, can also pull /etc/machine-id one. In addition >(but that's really a nitpick in this case with all these much bigger >problems + easy to fix), plain HMAC is not a proper and recommended >key derivation function, an HKDF or one of the NIST constructs from >SP800-108 should be used instead. Cool. I did not know about HKDF. I will look into that. >>2) S is longer (32 bytes instead of 8) >This (and #4) only helps the attack scenario #2, where attacker has to >brute-force all possible values for S - and I believe that scenario is >less realictic. But yes, case #2 gets a bit better than before, if we >disregard the fact that HMAC is not a proper KDF and that MachineID is >not a proper cryptographic key material [1, last paragraph in >Description suggests that], which may provide additional advantage to >the attacker. BTW, machine-id has at most 128 bits of entropy (if >produced by a proper CSRNG, which I'm not sure about - e.g. I see they >do some formatting), so practically it's not 32 bytes, but 16 [at >most]. Yes, this was mostly a minor point. And yes, the entropy in S is limited by machine-id, but either one is better than a hard-coded 8-byte ASCII string :) >So while this new approach does provide some advantage for scenario >#2, it doesn't address a more important case of #1 and to me it still >looks like low security. The proper way here is indeed to get some >capability for storing the keys securely, but I see how this is hard >to impossible on AST2500 (if that's what we're talking about here), >without TrustZone or anything similar, or some sort of >ROM/bootloader-accessible-only part-unique secret, so OTMH I can't >propose any viable alternatives for a given context (brownfield >deployment, backward compatibility), but I'll think more about that. > >[1] https://www.freedesktop.org/software/systemd/man/machine-id.html Really, I don't see a real solution without some kind of secure storage, which we don't have at this time on the ast2500. While that is not the only machine running OpenBMC, ideally the solution would be platform independent so that any machine could do it. Maybe that requirement would be that it supported some minimal cryptographic features or something (like trustzone support). But in the meantime, some sort of solution that security researchers will not rake us over the coals for would be nice. Maybe there isn't one. Some minimal configuration that is better than writing the passwords in plain text. --Vernon ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 19:11 ` Vernon Mauery @ 2020-04-14 22:18 ` Joseph Reynolds 2020-04-14 22:44 ` Vernon Mauery 0 siblings, 1 reply; 21+ messages in thread From: Joseph Reynolds @ 2020-04-14 22:18 UTC (permalink / raw) To: Vernon Mauery, Alexander Tereschenko; +Cc: openbmc On 4/14/20 2:11 PM, Vernon Mauery wrote: > On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote: >> To be more specific, I'm considering two attack scenarios in the >> below comments: >> 1) The attacker gets into BMC and is able to copy off the data files, >> including ipmi_pass. This IMHO is a more realistic scenario in this >> case. >> 2) The attacker gets ipmi_pass file/contents only, without being able >> to retrieve anything else. >> >> Which ones do *you* have in mind? For the sake of discussion, it >> would be helpful to specify them all and see how potential solutions >> address them. > > Attack one is the most likely, since if you can read one file, you can > probably get any/all of them. An alternate solution is to make those file readable only by root, restricting root logins, and restricting SSH access in general. See https://github.com/ibm-openbmc/dev/issues/1528 In this way, only an admin can get the files by enabling the SSH interface, logging in via SSH, and using su or sudo to access the files. All of these events should be audit-able. - Joseph > >> On 14-Apr-20 01:00, Vernon Mauery wrote: >>> Possible Solution >>> ================= >>> Migrate to a solution that uses a key that is longer that does not >>> get written directly to the flash >>> 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID) >> >> I like the direction and the guiding principle, however from "proper" >> cryptography standpoint, for producing keys, this is not noticeably >> better than the previous solution. The key material is still readily >> available on the filesystem (making points #3 and #4 irrelevant for >> attack scenario #1), so chances are that the adversary who can pull >> /etc/ipmi_pass file, can also pull /etc/machine-id one. In addition >> (but that's really a nitpick in this case with all these much bigger >> problems + easy to fix), plain HMAC is not a proper and recommended >> key derivation function, an HKDF or one of the NIST constructs from >> SP800-108 should be used instead. > > Cool. I did not know about HKDF. I will look into that. > >>> 2) S is longer (32 bytes instead of 8) >> This (and #4) only helps the attack scenario #2, where attacker has >> to brute-force all possible values for S - and I believe that >> scenario is less realictic. But yes, case #2 gets a bit better than >> before, if we disregard the fact that HMAC is not a proper KDF and >> that MachineID is not a proper cryptographic key material [1, last >> paragraph in Description suggests that], which may provide additional >> advantage to the attacker. BTW, machine-id has at most 128 bits of >> entropy (if produced by a proper CSRNG, which I'm not sure about - >> e.g. I see they do some formatting), so practically it's not 32 >> bytes, but 16 [at most]. > > Yes, this was mostly a minor point. And yes, the entropy in S is > limited by machine-id, but either one is better than a hard-coded > 8-byte ASCII string :) > >> So while this new approach does provide some advantage for scenario >> #2, it doesn't address a more important case of #1 and to me it still >> looks like low security. The proper way here is indeed to get some >> capability for storing the keys securely, but I see how this is hard >> to impossible on AST2500 (if that's what we're talking about here), >> without TrustZone or anything similar, or some sort of >> ROM/bootloader-accessible-only part-unique secret, so OTMH I can't >> propose any viable alternatives for a given context (brownfield >> deployment, backward compatibility), but I'll think more about that. >> >> [1] https://www.freedesktop.org/software/systemd/man/machine-id.html > > Really, I don't see a real solution without some kind of secure > storage, which we don't have at this time on the ast2500. While that > is not the only machine running OpenBMC, ideally the solution would be > platform independent so that any machine could do it. Maybe that > requirement would be that it supported some minimal cryptographic > features or something (like trustzone support). But in the meantime, > some sort of solution that security researchers will not rake us over > the coals for would be nice. Maybe there isn't one. Some minimal > configuration that is better than writing the passwords in plain text. > > --Vernon > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 22:18 ` Joseph Reynolds @ 2020-04-14 22:44 ` Vernon Mauery 2020-04-14 22:48 ` Joseph Reynolds 0 siblings, 1 reply; 21+ messages in thread From: Vernon Mauery @ 2020-04-14 22:44 UTC (permalink / raw) To: Joseph Reynolds; +Cc: Alexander Tereschenko, openbmc On 14-Apr-2020 05:18 PM, Joseph Reynolds wrote: >On 4/14/20 2:11 PM, Vernon Mauery wrote: >>On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote: >>>To be more specific, I'm considering two attack scenarios in the >>>below comments: >>>1) The attacker gets into BMC and is able to copy off the data >>>files, including ipmi_pass. This IMHO is a more realistic scenario >>>in this case. >>>2) The attacker gets ipmi_pass file/contents only, without being >>>able to retrieve anything else. >>> >>>Which ones do *you* have in mind? For the sake of discussion, it >>>would be helpful to specify them all and see how potential >>>solutions address them. >> >>Attack one is the most likely, since if you can read one file, you >>can probably get any/all of them. > >An alternate solution is to make those file readable only by root, >restricting root logins, and restricting SSH access in general. See >https://github.com/ibm-openbmc/dev/issues/1528 Changes have been made already to restrict permissions to 0600. The problem is that currently that is not much of protection at all because all the processes run as root right now anyway. --Vernon >In this way, only an admin can get the files by enabling the SSH >interface, logging in via SSH, and using su or sudo to access the >files. All of these events should be audit-able. > >- Joseph > >> >>>On 14-Apr-20 01:00, Vernon Mauery wrote: >>>>Possible Solution >>>>================= >>>>Migrate to a solution that uses a key that is longer that does >>>>not get written directly to the flash >>>>1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID) >>> >>>I like the direction and the guiding principle, however from >>>"proper" cryptography standpoint, for producing keys, this is not >>>noticeably better than the previous solution. The key material is >>>still readily available on the filesystem (making points #3 and #4 >>>irrelevant for attack scenario #1), so chances are that the >>>adversary who can pull /etc/ipmi_pass file, can also pull >>>/etc/machine-id one. In addition (but that's really a nitpick in >>>this case with all these much bigger problems + easy to fix), >>>plain HMAC is not a proper and recommended key derivation >>>function, an HKDF or one of the NIST constructs from SP800-108 >>>should be used instead. >> >>Cool. I did not know about HKDF. I will look into that. >> >>>>2) S is longer (32 bytes instead of 8) >>>This (and #4) only helps the attack scenario #2, where attacker >>>has to brute-force all possible values for S - and I believe that >>>scenario is less realictic. But yes, case #2 gets a bit better >>>than before, if we disregard the fact that HMAC is not a proper >>>KDF and that MachineID is not a proper cryptographic key material >>>[1, last paragraph in Description suggests that], which may >>>provide additional advantage to the attacker. BTW, machine-id has >>>at most 128 bits of entropy (if produced by a proper CSRNG, which >>>I'm not sure about - e.g. I see they do some formatting), so >>>practically it's not 32 bytes, but 16 [at most]. >> >>Yes, this was mostly a minor point. And yes, the entropy in S is >>limited by machine-id, but either one is better than a hard-coded >>8-byte ASCII string :) >> >>>So while this new approach does provide some advantage for >>>scenario #2, it doesn't address a more important case of #1 and to >>>me it still looks like low security. The proper way here is indeed >>>to get some capability for storing the keys securely, but I see >>>how this is hard to impossible on AST2500 (if that's what we're >>>talking about here), without TrustZone or anything similar, or >>>some sort of ROM/bootloader-accessible-only part-unique secret, so >>>OTMH I can't propose any viable alternatives for a given context >>>(brownfield deployment, backward compatibility), but I'll think >>>more about that. >>> >>>[1] https://www.freedesktop.org/software/systemd/man/machine-id.html >> >>Really, I don't see a real solution without some kind of secure >>storage, which we don't have at this time on the ast2500. While that >>is not the only machine running OpenBMC, ideally the solution would >>be platform independent so that any machine could do it. Maybe that >>requirement would be that it supported some minimal cryptographic >>features or something (like trustzone support). But in the meantime, >>some sort of solution that security researchers will not rake us >>over the coals for would be nice. Maybe there isn't one. Some >>minimal configuration that is better than writing the passwords in >>plain text. >> >>--Vernon >> >> > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 22:44 ` Vernon Mauery @ 2020-04-14 22:48 ` Joseph Reynolds 2020-04-15 20:32 ` Joseph Reynolds 0 siblings, 1 reply; 21+ messages in thread From: Joseph Reynolds @ 2020-04-14 22:48 UTC (permalink / raw) To: Vernon Mauery; +Cc: Alexander Tereschenko, openbmc On 4/14/20 5:44 PM, Vernon Mauery wrote: > On 14-Apr-2020 05:18 PM, Joseph Reynolds wrote: >> On 4/14/20 2:11 PM, Vernon Mauery wrote: >>> On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote: >>>> To be more specific, I'm considering two attack scenarios in the >>>> below comments: >>>> 1) The attacker gets into BMC and is able to copy off the data >>>> files, including ipmi_pass. This IMHO is a more realistic scenario >>>> in this case. >>>> 2) The attacker gets ipmi_pass file/contents only, without being >>>> able to retrieve anything else. >>>> >>>> Which ones do *you* have in mind? For the sake of discussion, it >>>> would be helpful to specify them all and see how potential >>>> solutions address them. >>> >>> Attack one is the most likely, since if you can read one file, you >>> can probably get any/all of them. >> >> An alternate solution is to make those file readable only by root, >> restricting root logins, and restricting SSH access in general. See >> https://github.com/ibm-openbmc/dev/issues/1528 > > Changes have been made already to restrict permissions to 0600. The > problem is that currently that is not much of protection at all > because all the processes run as root right now anyway. Understood. An attacker who gets control of any process will have read access to this file, etc. The issue that all processes run as root is documented here https://github.com/openbmc/openbmc/issues/3383 and was recently discussed in the security working group, 2020-04-01 - https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI - Joseph > > --Vernon > >> In this way, only an admin can get the files by enabling the SSH >> interface, logging in via SSH, and using su or sudo to access the >> files. All of these events should be audit-able. >> >> - Joseph >> >>> >>>> On 14-Apr-20 01:00, Vernon Mauery wrote: >>>>> Possible Solution >>>>> ================= >>>>> Migrate to a solution that uses a key that is longer that does not >>>>> get written directly to the flash >>>>> 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID) >>>> >>>> I like the direction and the guiding principle, however from >>>> "proper" cryptography standpoint, for producing keys, this is not >>>> noticeably better than the previous solution. The key material is >>>> still readily available on the filesystem (making points #3 and #4 >>>> irrelevant for attack scenario #1), so chances are that the >>>> adversary who can pull /etc/ipmi_pass file, can also pull >>>> /etc/machine-id one. In addition (but that's really a nitpick in >>>> this case with all these much bigger problems + easy to fix), plain >>>> HMAC is not a proper and recommended key derivation function, an >>>> HKDF or one of the NIST constructs from SP800-108 should be used >>>> instead. >>> >>> Cool. I did not know about HKDF. I will look into that. >>> >>>>> 2) S is longer (32 bytes instead of 8) >>>> This (and #4) only helps the attack scenario #2, where attacker has >>>> to brute-force all possible values for S - and I believe that >>>> scenario is less realictic. But yes, case #2 gets a bit better than >>>> before, if we disregard the fact that HMAC is not a proper KDF and >>>> that MachineID is not a proper cryptographic key material [1, last >>>> paragraph in Description suggests that], which may provide >>>> additional advantage to the attacker. BTW, machine-id has at most >>>> 128 bits of entropy (if produced by a proper CSRNG, which I'm not >>>> sure about - e.g. I see they do some formatting), so practically >>>> it's not 32 bytes, but 16 [at most]. >>> >>> Yes, this was mostly a minor point. And yes, the entropy in S is >>> limited by machine-id, but either one is better than a hard-coded >>> 8-byte ASCII string :) >>> >>>> So while this new approach does provide some advantage for scenario >>>> #2, it doesn't address a more important case of #1 and to me it >>>> still looks like low security. The proper way here is indeed to get >>>> some capability for storing the keys securely, but I see how this >>>> is hard to impossible on AST2500 (if that's what we're talking >>>> about here), without TrustZone or anything similar, or some sort of >>>> ROM/bootloader-accessible-only part-unique secret, so OTMH I can't >>>> propose any viable alternatives for a given context (brownfield >>>> deployment, backward compatibility), but I'll think more about that. >>>> >>>> [1] https://www.freedesktop.org/software/systemd/man/machine-id.html >>> >>> Really, I don't see a real solution without some kind of secure >>> storage, which we don't have at this time on the ast2500. While that >>> is not the only machine running OpenBMC, ideally the solution would >>> be platform independent so that any machine could do it. Maybe that >>> requirement would be that it supported some minimal cryptographic >>> features or something (like trustzone support). But in the meantime, >>> some sort of solution that security researchers will not rake us >>> over the coals for would be nice. Maybe there isn't one. Some >>> minimal configuration that is better than writing the passwords in >>> plain text. >>> >>> --Vernon >>> >>> >> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 22:48 ` Joseph Reynolds @ 2020-04-15 20:32 ` Joseph Reynolds 0 siblings, 0 replies; 21+ messages in thread From: Joseph Reynolds @ 2020-04-15 20:32 UTC (permalink / raw) To: Vernon Mauery; +Cc: openbmc, Thomaiyar, Richard Marian On 4/14/20 5:48 PM, Joseph Reynolds wrote: > > > On 4/14/20 5:44 PM, Vernon Mauery wrote: >> On 14-Apr-2020 05:18 PM, Joseph Reynolds wrote: >>> On 4/14/20 2:11 PM, Vernon Mauery wrote: >>>> On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote: >>>>> To be more specific, I'm considering two attack scenarios in the >>>>> below comments: >>>>> 1) The attacker gets into BMC and is able to copy off the data >>>>> files, including ipmi_pass. This IMHO is a more realistic scenario >>>>> in this case. >>>>> 2) The attacker gets ipmi_pass file/contents only, without being >>>>> able to retrieve anything else. >>>>> >>>>> Which ones do *you* have in mind? For the sake of discussion, it >>>>> would be helpful to specify them all and see how potential >>>>> solutions address them. >>>> >>>> Attack one is the most likely, since if you can read one file, you >>>> can probably get any/all of them. >>> >>> An alternate solution is to make those file readable only by root, >>> restricting root logins, and restricting SSH access in general. See >>> https://github.com/ibm-openbmc/dev/issues/1528 >> >> Changes have been made already to restrict permissions to 0600. The >> problem is that currently that is not much of protection at all >> because all the processes run as root right now anyway. > > Understood. An attacker who gets control of any process will have > read access to this file, etc. > The issue that all processes run as root is documented here > https://github.com/openbmc/openbmc/issues/3383 > and was recently discussed in the security working group, 2020-04-01 - > https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI > > > - Joseph We discussed this in the OpenBMC security working group meeting 2020-04-15 - https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI My summary: There is a patch to fix the file permissions so only root can read the files. It addresses the most severe aspect of the vulnerability. With the patch, only the root user can exploit the vulnerability. But even with the fix, the underlying problem remains that encrypted passwords are stored on the BMC's file system that the root user can decrypt. The root user should be trustworthy, but also should not have such access to user passwords. Also, some other user can become root, making it a priority to restrict root access (including access to SSH and to the su and sudo commands). Also, the fact that most OpenBMC processes run as the root user id (userid=0) makes escalation more likely. Additional attempts were made to secure the ipmi_pass file, but the cost/benefit is unclear: 1. Encrypting the file differently was considered, but provides little benefit because the decryption credentials are still available to root. And we don't have a TPM and are not using ARM TrustZone to help isolate access to the key. And changes like this cause upgrade and downgrade complications. 2. Abandoning RMCP+ would mean the ipmi_pass file is no longer needed, so the problem goes away. We would need to create a new way for ipmitool to authenticate with the ipmi network daemon, thus creating a de-facto standard. That would be a bunch of work and may extend the lifetime of IPMI. So the answer for now is to fix the file permissions, pending coming up with a better solution. - Joseph ... snip... ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-13 23:00 ipmi password storage Vernon Mauery 2020-04-14 15:50 ` Patrick Williams 2020-04-14 16:27 ` Alexander Tereschenko @ 2020-04-14 18:04 ` Milton Miller II 2020-04-14 19:14 ` Vernon Mauery 2 siblings, 1 reply; 21+ messages in thread From: Milton Miller II @ 2020-04-14 18:04 UTC (permalink / raw) To: Vernon Mauery; +Cc: OpenBMC Development On Apr 13, 2020 around 6:01PM in some time zone, Vernon Mauery wrote: > >Internally, an issue was raised that basically says that the >mechanism >by which we are storing the IPMI passwords on the BMC is >insufficiently >obfuscated. I have come up with a patch set that resolves this at the > >expense of no downgrading the BMC without the side-effect of losing >all >IPMI passwords. I would like to know what the community thinks about >usability vs. security in this scenario. ... >The migration from the old mechanism to the new could be done simply>by >using the new key on the next write to the /etc/ipmi_pass file. After >a >firmware update to this new code, a password change would trigger a >decrypt of the /etc/ipmi_pass file, a modification of the plain text, > >and a re-encryption of the data. If it reads the 'legacy' key in and >writes out the data using the new key mechanism and deletes the >legacy >key, it would use the new key mechanism from that point onward. >However, >this would cause any downgrades to prior versions to fail to decrypt >the >/etc/ipmi_pass file, thereby losing all the ipmi passwords. This is >not >ideal, but could possibly be mitigating by truncating the new >machine-id >derivative password to 8 bytes and storing it in the /etc/key_file >instead of just deleting it. This might improve security only >slightly >at for the price of a better user experience. > I'll point out the code to handle the new password could be added before the cdoe to use the new method, allowing test and revert until the users are upgraded to the new method. It does require both methods to be supported. I didn't follow why currently all openbmc systems end up with the same encryption^Wobsfucation for what that is worth. >I know that some companies using OpenBMC have products with users out>in >the field, so it is not great to make changes like this. Also, it is >not >great to have low-grade security. So here I am, writing to ask for >opinions and options. > >--Vernon Milton ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 18:04 ` Milton Miller II @ 2020-04-14 19:14 ` Vernon Mauery 2020-04-14 22:30 ` Joseph Reynolds 0 siblings, 1 reply; 21+ messages in thread From: Vernon Mauery @ 2020-04-14 19:14 UTC (permalink / raw) To: Milton Miller II; +Cc: OpenBMC Development On 14-Apr-2020 06:04 PM, Milton Miller II wrote: >On Apr 13, 2020 around 6:01PM in some time zone, Vernon Mauery wrote: >> >>Internally, an issue was raised that basically says that the >>mechanism >>by which we are storing the IPMI passwords on the BMC is >>insufficiently >>obfuscated. I have come up with a patch set that resolves this at the >> >>expense of no downgrading the BMC without the side-effect of losing >>all >>IPMI passwords. I would like to know what the community thinks about >>usability vs. security in this scenario. > >... > >>The migration from the old mechanism to the new could be done simply>by >>using the new key on the next write to the /etc/ipmi_pass file. After >>a >>firmware update to this new code, a password change would trigger a >>decrypt of the /etc/ipmi_pass file, a modification of the plain text, >> >>and a re-encryption of the data. If it reads the 'legacy' key in and >>writes out the data using the new key mechanism and deletes the >>legacy >>key, it would use the new key mechanism from that point onward. >>However, >>this would cause any downgrades to prior versions to fail to decrypt >>the >>/etc/ipmi_pass file, thereby losing all the ipmi passwords. This is >>not >>ideal, but could possibly be mitigating by truncating the new >>machine-id >>derivative password to 8 bytes and storing it in the /etc/key_file >>instead of just deleting it. This might improve security only >>slightly >>at for the price of a better user experience. >> > >I'll point out the code to handle the new password could be added >before the cdoe to use the new method, allowing test and revert >until the users are upgraded to the new method. It does require >both methods to be supported. Yes, it looks like any sort of change here would need to be a staged change to reduce the disruption. >I didn't follow why currently all openbmc systems end up with >the same encryption^Wobsfucation for what that is worth. Unless the build has a bbappend that changes the contents of the key_file that is a part of the pam-ipmi package, all of the builds will contain that same key_file. I can't say for sure how many builds have this already, but I did not see much documentation around that fact that would have spurred people to take action, so it is my assumption that most builds would use the default. --Vernon ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 19:14 ` Vernon Mauery @ 2020-04-14 22:30 ` Joseph Reynolds 2020-04-14 22:47 ` Vernon Mauery 0 siblings, 1 reply; 21+ messages in thread From: Joseph Reynolds @ 2020-04-14 22:30 UTC (permalink / raw) To: Vernon Mauery, Milton Miller II; +Cc: OpenBMC Development On 4/14/20 2:14 PM, Vernon Mauery wrote: > On 14-Apr-2020 06:04 PM, Milton Miller II wrote: >> On Apr 13, 2020 around 6:01PM in some time zone, Vernon Mauery wrote: >>> >>> Internally, an issue was raised that basically says that the >>> mechanism >>> by which we are storing the IPMI passwords on the BMC is >>> insufficiently >>> obfuscated. I have come up with a patch set that resolves this at the >>> >>> expense of no downgrading the BMC without the side-effect of losing >>> all >>> IPMI passwords. I would like to know what the community thinks about >>> usability vs. security in this scenario. >> >> ... >> >>> The migration from the old mechanism to the new could be done simply>by >>> using the new key on the next write to the /etc/ipmi_pass file. After >>> a >>> firmware update to this new code, a password change would trigger a >>> decrypt of the /etc/ipmi_pass file, a modification of the plain text, >>> >>> and a re-encryption of the data. If it reads the 'legacy' key in and >>> writes out the data using the new key mechanism and deletes the >>> legacy >>> key, it would use the new key mechanism from that point onward. >>> However, >>> this would cause any downgrades to prior versions to fail to decrypt >>> the >>> /etc/ipmi_pass file, thereby losing all the ipmi passwords. This is >>> not >>> ideal, but could possibly be mitigating by truncating the new >>> machine-id >>> derivative password to 8 bytes and storing it in the /etc/key_file >>> instead of just deleting it. This might improve security only >>> slightly >>> at for the price of a better user experience. >>> >> >> I'll point out the code to handle the new password could be added >> before the cdoe to use the new method, allowing test and revert >> until the users are upgraded to the new method. It does require >> both methods to be supported. > > Yes, it looks like any sort of change here would need to be a staged > change to reduce the disruption. Thanks for handling this issue -- I appreciate it. Don't take this the wrong way, but... If this change provides little value and causes upgrade issues, would it be better to avoid having an upgrade path? Instead, use this new approach for new major release that requires a fresh install and upgrading is not an option. > >> I didn't follow why currently all openbmc systems end up with >> the same encryption^Wobsfucation for what that is worth. > > Unless the build has a bbappend that changes the contents of the > key_file that is a part of the pam-ipmi package, all of the builds > will contain that same key_file. I can't say for sure how many builds > have this already, but I did not see much documentation around that > fact that would have spurred people to take action, so it is my > assumption that most builds would use the default. From previous emails in this thread, it doesn't seem like having each BMC having an unique key_file would help much. Nevertheless, I've added this to my notes for BMC build considerations: https://github.com/ibm-openbmc/dev/issues/1531#issuecomment-613676676 - Joseph > > --Vernon ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: ipmi password storage 2020-04-14 22:30 ` Joseph Reynolds @ 2020-04-14 22:47 ` Vernon Mauery 0 siblings, 0 replies; 21+ messages in thread From: Vernon Mauery @ 2020-04-14 22:47 UTC (permalink / raw) To: Joseph Reynolds; +Cc: Milton Miller II, OpenBMC Development On 14-Apr-2020 05:30 PM, Joseph Reynolds wrote: >>>>key, it would use the new key mechanism from that point onward. >>>>However, >>>>this would cause any downgrades to prior versions to fail to decrypt >>>>the >>>>/etc/ipmi_pass file, thereby losing all the ipmi passwords. This is >>>>not >>>>ideal, but could possibly be mitigating by truncating the new >>>>machine-id >>>>derivative password to 8 bytes and storing it in the /etc/key_file >>>>instead of just deleting it. This might improve security only >>>>slightly >>>>at for the price of a better user experience. >>>> >>> >>>I'll point out the code to handle the new password could be added >>>before the cdoe to use the new method, allowing test and revert >>>until the users are upgraded to the new method. It does require >>>both methods to be supported. >> >>Yes, it looks like any sort of change here would need to be a staged >>change to reduce the disruption. > >Thanks for handling this issue -- I appreciate it. Don't take this >the wrong way, but... >If this change provides little value and causes upgrade issues, would >it be better to avoid having an upgrade path? >Instead, use this new approach for new major release that requires a >fresh install and upgrading is not an option. > It doesn't cause upgrade issues, it would cause downgrade issues. While I am not personally opposed to not downgrading (always move forward), not all IT shops have the same opinion. I thought it might be nice to offer an opportunity to support downgrades. --Vernon >>>I didn't follow why currently all openbmc systems end up with >>>the same encryption^Wobsfucation for what that is worth. >> >>Unless the build has a bbappend that changes the contents of the >>key_file that is a part of the pam-ipmi package, all of the builds >>will contain that same key_file. I can't say for sure how many >>builds have this already, but I did not see much documentation >>around that fact that would have spurred people to take action, so >>it is my assumption that most builds would use the default. > From previous emails in this thread, it doesn't seem like having each >BMC having an unique key_file would help much. Nevertheless, I've >added this to my notes for BMC build considerations: >https://github.com/ibm-openbmc/dev/issues/1531#issuecomment-613676676 > >- Joseph > >> >>--Vernon > ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2020-04-29 16:02 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-04-13 23:00 ipmi password storage Vernon Mauery 2020-04-14 15:50 ` Patrick Williams 2020-04-14 16:46 ` Vernon Mauery 2020-04-14 22:03 ` Joseph Reynolds 2020-04-14 22:42 ` Vernon Mauery 2020-04-16 6:04 ` Joel Stanley 2020-04-20 14:29 ` Vernon Mauery 2020-04-22 11:38 ` Patrick Williams 2020-04-22 22:19 ` Vernon Mauery 2020-04-24 15:15 ` Patrick Williams 2020-04-29 16:02 ` Alexander Tereschenko 2020-04-14 16:27 ` Alexander Tereschenko 2020-04-14 19:11 ` Vernon Mauery 2020-04-14 22:18 ` Joseph Reynolds 2020-04-14 22:44 ` Vernon Mauery 2020-04-14 22:48 ` Joseph Reynolds 2020-04-15 20:32 ` Joseph Reynolds 2020-04-14 18:04 ` Milton Miller II 2020-04-14 19:14 ` Vernon Mauery 2020-04-14 22:30 ` Joseph Reynolds 2020-04-14 22:47 ` Vernon Mauery
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.