ipmi password storage

ipmi password storage

[Vernon Mauery]

Internally, an issue was raised that basically says that the mechanism 
by which we are storing the IPMI passwords on the BMC is insufficiently 
obfuscated. I have come up with a patch set that resolves this at the 
expense of no downgrading the BMC without the side-effect of losing all 
IPMI passwords. I would like to know what the community thinks about 
usability vs. security in this scenario.

Current Implementation
======================
1) If the user is part of the ipmi group (/etc/group) then when the user 
changes their password, pam-ipmisave.so intercepts the password as a one 
of the PAM layers and saves it, encrypted, to /etc/ipmi_pass.
2) Encryption (obfuscation, because we don't really have a secure 
mechanism of storing secret keys), is done like this:
   a) read 8 bytes (S) from /etc/key_file (currently pre-loaded with "OPENBMC=")
   b) create a random value H (read from /dev/urandom)
   c) create the AES-CBC secret key K=HMAC-SHA256(S, H)
   d) encrypt the list of username:password data using K
   e) store H along with the encrypted data in /etc/ipmi_pass
3) reading the password (for establishing IPMI RMCP+ sessions)
   a) read 8 bytes (S) from /etc/key_file
   b) read H from /etc/ipmi_pass
   c) compute the AES-CBC secret key K=HMAC-SHA256(S, H)
   d) decrypt and verify the contents of /etc/key_file


There are many issues with this mechanism, but we cannot fix all of them 
without some secure mechanism for storing secret keys. That is why 
really, at best, this is obfuscation, not encryption. The data is not in 
plain text, it takes some work to get to it. More than xor or rot13, but 
not so much that a person could do it with a bash script.
1) the default /etc/key_file is the same for every BMC built with the 
default settings (changing this requires a bbappend for pam-ipmi). This 
means the /etc/key_file could basically not exist; all you need is the 
algorithm and /etc/ipmi_pass.
2) the size of the /etc/key_file is also not really great. Even if it 
was different on every machine, computing only 2^64 possibilities is not 
so hard.


Possible Solution
=================
Migrate to a solution that uses a key that is longer that does not 
get written directly to the flash
1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID)
2) S is longer (32 bytes instead of 8)
3) S is not written to flash, because it can be computed
4) S is different for every machine because it is a derivative of 
/etc/machine-id

The migration from the old mechanism to the new could be done simply by 
using the new key on the next write to the /etc/ipmi_pass file. After a 
firmware update to this new code, a password change would trigger a 
decrypt of the /etc/ipmi_pass file, a modification of the plain text, 
and a re-encryption of the data. If it reads the 'legacy' key in and 
writes out the data using the new key mechanism and deletes the legacy 
key, it would use the new key mechanism from that point onward. However, 
this would cause any downgrades to prior versions to fail to decrypt the 
/etc/ipmi_pass file, thereby losing all the ipmi passwords. This is not 
ideal, but could possibly be mitigating by truncating the new machine-id 
derivative password to 8 bytes and storing it in the /etc/key_file 
instead of just deleting it. This might improve security only slightly 
at for the price of a better user experience.

I know that some companies using OpenBMC have products with users out in 
the field, so it is not great to make changes like this. Also, it is not 
great to have low-grade security. So here I am, writing to ask for 
opinions and options.

--Vernon

Re: ipmi password storage

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 4069 bytes --]

On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote:

Vernon,

Is there some background pointers on why IPMI needs to store passwords
in a reverable way?  I never understood that.

> Internally, an issue was raised that basically says that the mechanism 
> by which we are storing the IPMI passwords on the BMC is insufficiently 
> obfuscated. I have come up with a patch set that resolves this at the 
> expense of no downgrading the BMC without the side-effect of losing all 
> IPMI passwords. I would like to know what the community thinks about 
> usability vs. security in this scenario.
> 
> Current Implementation
> ======================
> 1) If the user is part of the ipmi group (/etc/group) then when the user 
> changes their password, pam-ipmisave.so intercepts the password as a one 
> of the PAM layers and saves it, encrypted, to /etc/ipmi_pass.
> 2) Encryption (obfuscation, because we don't really have a secure 
> mechanism of storing secret keys), is done like this:
>    a) read 8 bytes (S) from /etc/key_file (currently pre-loaded with "OPENBMC=")
>    b) create a random value H (read from /dev/urandom)
>    c) create the AES-CBC secret key K=HMAC-SHA256(S, H)
>    d) encrypt the list of username:password data using K
>    e) store H along with the encrypted data in /etc/ipmi_pass
> 3) reading the password (for establishing IPMI RMCP+ sessions)
>    a) read 8 bytes (S) from /etc/key_file
>    b) read H from /etc/ipmi_pass
>    c) compute the AES-CBC secret key K=HMAC-SHA256(S, H)
>    d) decrypt and verify the contents of /etc/key_file
> 
> 
> There are many issues with this mechanism, but we cannot fix all of them 
> without some secure mechanism for storing secret keys. That is why 
> really, at best, this is obfuscation, not encryption. The data is not in 
> plain text, it takes some work to get to it. More than xor or rot13, but 
> not so much that a person could do it with a bash script.
> 1) the default /etc/key_file is the same for every BMC built with the 
> default settings (changing this requires a bbappend for pam-ipmi). This 
> means the /etc/key_file could basically not exist; all you need is the 
> algorithm and /etc/ipmi_pass.
> 2) the size of the /etc/key_file is also not really great. Even if it 
> was different on every machine, computing only 2^64 possibilities is not 
> so hard.
> 
> 
> Possible Solution
> =================
> Migrate to a solution that uses a key that is longer that does not 
> get written directly to the flash
> 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID)
> 2) S is longer (32 bytes instead of 8)
> 3) S is not written to flash, because it can be computed
> 4) S is different for every machine because it is a derivative of 
> /etc/machine-id
> 
> The migration from the old mechanism to the new could be done simply by 
> using the new key on the next write to the /etc/ipmi_pass file. After a 
> firmware update to this new code, a password change would trigger a 
> decrypt of the /etc/ipmi_pass file, a modification of the plain text, 
> and a re-encryption of the data. If it reads the 'legacy' key in and 
> writes out the data using the new key mechanism and deletes the legacy 
> key, it would use the new key mechanism from that point onward. However, 
> this would cause any downgrades to prior versions to fail to decrypt the 
> /etc/ipmi_pass file, thereby losing all the ipmi passwords. This is not 
> ideal, but could possibly be mitigating by truncating the new machine-id 
> derivative password to 8 bytes and storing it in the /etc/key_file 
> instead of just deleting it. This might improve security only slightly 
> at for the price of a better user experience.
> 
> I know that some companies using OpenBMC have products with users out in 
> the field, so it is not great to make changes like this. Also, it is not 
> great to have low-grade security. So here I am, writing to ask for 
> opinions and options.
> 
> --Vernon

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: ipmi password storage

[Alexander Tereschenko]

To be more specific, I'm considering two attack scenarios in the below 
comments:
1) The attacker gets into BMC and is able to copy off the data files, 
including ipmi_pass. This IMHO is a more realistic scenario in this case.
2) The attacker gets ipmi_pass file/contents only, without being able to 
retrieve anything else.

Which ones do *you* have in mind? For the sake of discussion, it would 
be helpful to specify them all and see how potential solutions address them.

On 14-Apr-20 01:00, Vernon Mauery wrote:
> Possible Solution
> =================
> Migrate to a solution that uses a key that is longer that does not get 
> written directly to the flash
> 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID)

I like the direction and the guiding principle, however from "proper" 
cryptography standpoint, for producing keys, this is not noticeably 
better than the previous solution. The key material is still readily 
available on the filesystem (making points #3 and #4 irrelevant for 
attack scenario #1), so chances are that the adversary who can pull 
/etc/ipmi_pass file, can also pull /etc/machine-id one. In addition (but 
that's really a nitpick in this case with all these much bigger problems 
+ easy to fix), plain HMAC is not a proper and recommended key 
derivation function, an HKDF or one of the NIST constructs from 
SP800-108 should be used instead.

> 2) S is longer (32 bytes instead of 8)
This (and #4) only helps the attack scenario #2, where attacker has to 
brute-force all possible values for S - and I believe that scenario is 
less realictic. But yes, case #2 gets a bit better than before, if we 
disregard the fact that HMAC is not a proper KDF and that MachineID is 
not a proper cryptographic key material [1, last paragraph in 
Description suggests that], which may provide additional advantage to 
the attacker. BTW, machine-id has at most 128 bits of entropy (if 
produced by a proper CSRNG, which I'm not sure about - e.g. I see they 
do some formatting), so practically it's not 32 bytes, but 16 [at most].

So while this new approach does provide some advantage for scenario #2, 
it doesn't address a more important case of #1 and to me it still looks 
like low security. The proper way here is indeed to get some capability 
for storing the keys securely, but I see how this is hard to impossible 
on AST2500 (if that's what we're talking about here), without TrustZone 
or anything similar, or some sort of ROM/bootloader-accessible-only 
part-unique secret, so OTMH I can't propose any viable alternatives for 
a given context (brownfield deployment, backward compatibility), but 
I'll think more about that.

[1] https://www.freedesktop.org/software/systemd/man/machine-id.html

Re: ipmi password storage

[Vernon Mauery]

On 14-Apr-2020 10:50 AM, Patrick Williams wrote:
>On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote:
>
>Vernon,
>
>Is there some background pointers on why IPMI needs to store passwords
>in a reverable way?  I never understood that.

Sure. I think this is most clearly described in section 13.31 "RMCP+ 
Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec.

Here is a high level overview though: It is baked into the RMCP+ key 
exchange (RAKP). The password never goes across the wire in plaintext, 
but it does get used during the session creation in RAKP messages as the 
key to integrity and authentication HMACs.

Specifically, the RAKP3 message (User->BMC) contains an HMAC of the 
various parts of the exchanged session (User random number, session ID, Role, 
Username) using the password as the key for the HMAC. The BMC needs to 
compute this same HMAC to compare (this is the main authentication 
challenge).

Then, the session key is generated using an HMAC of similar 
data (BMC random number, user random number, role, username) using 
either the user password or the channel password. Almost nobody uses the 
channel password, which is good because it allows for simple privilege 
escalation and session hijacking.

Both sides use the same inputs and HMAC key, so the BMC needs to store 
the user passwords in a way that they can be used as the key for an 
HMAC. Ideally this would be stored in some sort of secure enclave or 
HSM, but that is not yet available.

--Vernon

>> Internally, an issue was raised that basically says that the mechanism
>> by which we are storing the IPMI passwords on the BMC is insufficiently
>> obfuscated. I have come up with a patch set that resolves this at the
>> expense of no downgrading the BMC without the side-effect of losing all
>> IPMI passwords. I would like to know what the community thinks about
>> usability vs. security in this scenario.
>>
>> Current Implementation
>> ======================
>> 1) If the user is part of the ipmi group (/etc/group) then when the user
>> changes their password, pam-ipmisave.so intercepts the password as a one
>> of the PAM layers and saves it, encrypted, to /etc/ipmi_pass.
>> 2) Encryption (obfuscation, because we don't really have a secure
>> mechanism of storing secret keys), is done like this:
>>    a) read 8 bytes (S) from /etc/key_file (currently pre-loaded with "OPENBMC=")
>>    b) create a random value H (read from /dev/urandom)
>>    c) create the AES-CBC secret key K=HMAC-SHA256(S, H)
>>    d) encrypt the list of username:password data using K
>>    e) store H along with the encrypted data in /etc/ipmi_pass
>> 3) reading the password (for establishing IPMI RMCP+ sessions)
>>    a) read 8 bytes (S) from /etc/key_file
>>    b) read H from /etc/ipmi_pass
>>    c) compute the AES-CBC secret key K=HMAC-SHA256(S, H)
>>    d) decrypt and verify the contents of /etc/key_file
>>
>>
>> There are many issues with this mechanism, but we cannot fix all of them
>> without some secure mechanism for storing secret keys. That is why
>> really, at best, this is obfuscation, not encryption. The data is not in
>> plain text, it takes some work to get to it. More than xor or rot13, but
>> not so much that a person could do it with a bash script.
>> 1) the default /etc/key_file is the same for every BMC built with the
>> default settings (changing this requires a bbappend for pam-ipmi). This
>> means the /etc/key_file could basically not exist; all you need is the
>> algorithm and /etc/ipmi_pass.
>> 2) the size of the /etc/key_file is also not really great. Even if it
>> was different on every machine, computing only 2^64 possibilities is not
>> so hard.
>>
>>
>> Possible Solution
>> =================
>> Migrate to a solution that uses a key that is longer that does not
>> get written directly to the flash
>> 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID)
>> 2) S is longer (32 bytes instead of 8)
>> 3) S is not written to flash, because it can be computed
>> 4) S is different for every machine because it is a derivative of
>> /etc/machine-id
>>
>> The migration from the old mechanism to the new could be done simply by
>> using the new key on the next write to the /etc/ipmi_pass file. After a
>> firmware update to this new code, a password change would trigger a
>> decrypt of the /etc/ipmi_pass file, a modification of the plain text,
>> and a re-encryption of the data. If it reads the 'legacy' key in and
>> writes out the data using the new key mechanism and deletes the legacy
>> key, it would use the new key mechanism from that point onward. However,
>> this would cause any downgrades to prior versions to fail to decrypt the
>> /etc/ipmi_pass file, thereby losing all the ipmi passwords. This is not
>> ideal, but could possibly be mitigating by truncating the new machine-id
>> derivative password to 8 bytes and storing it in the /etc/key_file
>> instead of just deleting it. This might improve security only slightly
>> at for the price of a better user experience.
>>
>> I know that some companies using OpenBMC have products with users out in
>> the field, so it is not great to make changes like this. Also, it is not
>> great to have low-grade security. So here I am, writing to ask for
>> opinions and options.
>>
>> --Vernon
>
>-- 
>Patrick Williams

Re: ipmi password storage

[Milton Miller II]

On Apr 13, 2020 around 6:01PM in some time zone, Vernon Mauery wrote:
>
>Internally, an issue was raised that basically says that the
>mechanism 
>by which we are storing the IPMI passwords on the BMC is
>insufficiently 
>obfuscated. I have come up with a patch set that resolves this at the
>
>expense of no downgrading the BMC without the side-effect of losing
>all 
>IPMI passwords. I would like to know what the community thinks about 
>usability vs. security in this scenario.

...

>The migration from the old mechanism to the new could be done simply>by 
>using the new key on the next write to the /etc/ipmi_pass file. After
>a 
>firmware update to this new code, a password change would trigger a 
>decrypt of the /etc/ipmi_pass file, a modification of the plain text,
>
>and a re-encryption of the data. If it reads the 'legacy' key in and 
>writes out the data using the new key mechanism and deletes the
>legacy 
>key, it would use the new key mechanism from that point onward.
>However, 
>this would cause any downgrades to prior versions to fail to decrypt
>the 
>/etc/ipmi_pass file, thereby losing all the ipmi passwords. This is
>not 
>ideal, but could possibly be mitigating by truncating the new
>machine-id 
>derivative password to 8 bytes and storing it in the /etc/key_file 
>instead of just deleting it. This might improve security only
>slightly 
>at for the price of a better user experience.
>

I'll point out the code to handle the new password could be added 
before the cdoe to use the new method, allowing test and revert 
until the users are upgraded to the new method.  It does require 
both methods to be supported.

I didn't follow why currently all openbmc systems end up with
the same encryption^Wobsfucation for what that is worth.

>I know that some companies using OpenBMC have products with users out>in 
>the field, so it is not great to make changes like this. Also, it is
>not 
>great to have low-grade security. So here I am, writing to ask for 
>opinions and options.
>
>--Vernon

Milton

Re: ipmi password storage

[Vernon Mauery]

On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote:
>To be more specific, I'm considering two attack scenarios in the below 
>comments:
>1) The attacker gets into BMC and is able to copy off the data files, 
>including ipmi_pass. This IMHO is a more realistic scenario in this 
>case.
>2) The attacker gets ipmi_pass file/contents only, without being able 
>to retrieve anything else.
>
>Which ones do *you* have in mind? For the sake of discussion, it would 
>be helpful to specify them all and see how potential solutions address 
>them.

Attack one is the most likely, since if you can read one file, you can 
probably get any/all of them.

>On 14-Apr-20 01:00, Vernon Mauery wrote:
>>Possible Solution
>>=================
>>Migrate to a solution that uses a key that is longer that does not 
>>get written directly to the flash
>>1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID)
>
>I like the direction and the guiding principle, however from "proper" 
>cryptography standpoint, for producing keys, this is not noticeably 
>better than the previous solution. The key material is still readily 
>available on the filesystem (making points #3 and #4 irrelevant for 
>attack scenario #1), so chances are that the adversary who can pull 
>/etc/ipmi_pass file, can also pull /etc/machine-id one. In addition 
>(but that's really a nitpick in this case with all these much bigger 
>problems + easy to fix), plain HMAC is not a proper and recommended 
>key derivation function, an HKDF or one of the NIST constructs from 
>SP800-108 should be used instead.

Cool. I did not know about HKDF. I will look into that.

>>2) S is longer (32 bytes instead of 8)
>This (and #4) only helps the attack scenario #2, where attacker has to 
>brute-force all possible values for S - and I believe that scenario is 
>less realictic. But yes, case #2 gets a bit better than before, if we 
>disregard the fact that HMAC is not a proper KDF and that MachineID is 
>not a proper cryptographic key material [1, last paragraph in 
>Description suggests that], which may provide additional advantage to 
>the attacker. BTW, machine-id has at most 128 bits of entropy (if 
>produced by a proper CSRNG, which I'm not sure about - e.g. I see they 
>do some formatting), so practically it's not 32 bytes, but 16 [at 
>most].

Yes, this was mostly a minor point. And yes, the entropy in S is 
limited by machine-id, but either one is better than a hard-coded 8-byte 
ASCII string :)

>So while this new approach does provide some advantage for scenario 
>#2, it doesn't address a more important case of #1 and to me it still 
>looks like low security. The proper way here is indeed to get some 
>capability for storing the keys securely, but I see how this is hard 
>to impossible on AST2500 (if that's what we're talking about here), 
>without TrustZone or anything similar, or some sort of 
>ROM/bootloader-accessible-only part-unique secret, so OTMH I can't 
>propose any viable alternatives for a given context (brownfield 
>deployment, backward compatibility), but I'll think more about that.
>
>[1] https://www.freedesktop.org/software/systemd/man/machine-id.html

Really, I don't see a real solution without some kind of secure storage, 
which we don't have at this time on the ast2500. While that is not the 
only machine running OpenBMC, ideally the solution would be platform 
independent so that any machine could do it. Maybe that requirement 
would be that it supported some minimal cryptographic features or 
something (like trustzone support). But in the meantime, some sort of 
solution that security researchers will not rake us over the coals for 
would be nice. Maybe there isn't one. Some minimal configuration that is 
better than writing the passwords in plain text.

--Vernon

Re: ipmi password storage

[Vernon Mauery]

On 14-Apr-2020 06:04 PM, Milton Miller II wrote:
>On Apr 13, 2020 around 6:01PM in some time zone, Vernon Mauery wrote:
>>
>>Internally, an issue was raised that basically says that the
>>mechanism
>>by which we are storing the IPMI passwords on the BMC is
>>insufficiently
>>obfuscated. I have come up with a patch set that resolves this at the
>>
>>expense of no downgrading the BMC without the side-effect of losing
>>all
>>IPMI passwords. I would like to know what the community thinks about
>>usability vs. security in this scenario.
>
>...
>
>>The migration from the old mechanism to the new could be done simply>by
>>using the new key on the next write to the /etc/ipmi_pass file. After
>>a
>>firmware update to this new code, a password change would trigger a
>>decrypt of the /etc/ipmi_pass file, a modification of the plain text,
>>
>>and a re-encryption of the data. If it reads the 'legacy' key in and
>>writes out the data using the new key mechanism and deletes the
>>legacy
>>key, it would use the new key mechanism from that point onward.
>>However,
>>this would cause any downgrades to prior versions to fail to decrypt
>>the
>>/etc/ipmi_pass file, thereby losing all the ipmi passwords. This is
>>not
>>ideal, but could possibly be mitigating by truncating the new
>>machine-id
>>derivative password to 8 bytes and storing it in the /etc/key_file
>>instead of just deleting it. This might improve security only
>>slightly
>>at for the price of a better user experience.
>>
>
>I'll point out the code to handle the new password could be added
>before the cdoe to use the new method, allowing test and revert
>until the users are upgraded to the new method.  It does require
>both methods to be supported.

Yes, it looks like any sort of change here would need to be a staged 
change to reduce the disruption.

>I didn't follow why currently all openbmc systems end up with
>the same encryption^Wobsfucation for what that is worth.

Unless the build has a bbappend that changes the contents of the 
key_file that is a part of the pam-ipmi package, all of the builds will 
contain that same key_file. I can't say for sure how many builds have 
this already, but I did not see much documentation around that fact that 
would have spurred people to take action, so it is my assumption that 
most builds would use the default.

--Vernon

Re: ipmi password storage

[Joseph Reynolds]

On 4/14/20 11:46 AM, Vernon Mauery wrote:
> On 14-Apr-2020 10:50 AM, Patrick Williams wrote:
>> On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote:
>>
>> Vernon,
>>
>> Is there some background pointers on why IPMI needs to store passwords
>> in a reverable way?  I never understood that.
>
> Sure. I think this is most clearly described in section 13.31 "RMCP+ 
> Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec.

This may be a bit naive....  Is it possible to expand the RCMP+ spec 
with a new cipher suite or similar, to use a mechanism more like HTTPS 
or SSH that does not require the server to be able to produce a clear 
text password?  Given that IPMI will be used for many years, this seems 
worthwhile, and would solve the current problem.

- Joseph

> --Vernon
>
>>>
>>>
>>> --Vernon
>>
>> -- 
>> Patrick Williams
>
>

Re: ipmi password storage

[Joseph Reynolds]

On 4/14/20 2:11 PM, Vernon Mauery wrote:
> On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote:
>> To be more specific, I'm considering two attack scenarios in the 
>> below comments:
>> 1) The attacker gets into BMC and is able to copy off the data files, 
>> including ipmi_pass. This IMHO is a more realistic scenario in this 
>> case.
>> 2) The attacker gets ipmi_pass file/contents only, without being able 
>> to retrieve anything else.
>>
>> Which ones do *you* have in mind? For the sake of discussion, it 
>> would be helpful to specify them all and see how potential solutions 
>> address them.
>
> Attack one is the most likely, since if you can read one file, you can 
> probably get any/all of them.

An alternate solution is to make those file readable only by root, 
restricting root logins, and restricting SSH access in general.  See 
https://github.com/ibm-openbmc/dev/issues/1528

In this way, only an admin can get the files by enabling the SSH 
interface, logging in via SSH, and using su or sudo to access the 
files.  All of these events should be audit-able.

- Joseph

>
>> On 14-Apr-20 01:00, Vernon Mauery wrote:
>>> Possible Solution
>>> =================
>>> Migrate to a solution that uses a key that is longer that does not 
>>> get written directly to the flash
>>> 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID)
>>
>> I like the direction and the guiding principle, however from "proper" 
>> cryptography standpoint, for producing keys, this is not noticeably 
>> better than the previous solution. The key material is still readily 
>> available on the filesystem (making points #3 and #4 irrelevant for 
>> attack scenario #1), so chances are that the adversary who can pull 
>> /etc/ipmi_pass file, can also pull /etc/machine-id one. In addition 
>> (but that's really a nitpick in this case with all these much bigger 
>> problems + easy to fix), plain HMAC is not a proper and recommended 
>> key derivation function, an HKDF or one of the NIST constructs from 
>> SP800-108 should be used instead.
>
> Cool. I did not know about HKDF. I will look into that.
>
>>> 2) S is longer (32 bytes instead of 8)
>> This (and #4) only helps the attack scenario #2, where attacker has 
>> to brute-force all possible values for S - and I believe that 
>> scenario is less realictic. But yes, case #2 gets a bit better than 
>> before, if we disregard the fact that HMAC is not a proper KDF and 
>> that MachineID is not a proper cryptographic key material [1, last 
>> paragraph in Description suggests that], which may provide additional 
>> advantage to the attacker. BTW, machine-id has at most 128 bits of 
>> entropy (if produced by a proper CSRNG, which I'm not sure about - 
>> e.g. I see they do some formatting), so practically it's not 32 
>> bytes, but 16 [at most].
>
> Yes, this was mostly a minor point. And yes, the entropy in S is 
> limited by machine-id, but either one is better than a hard-coded 
> 8-byte ASCII string :)
>
>> So while this new approach does provide some advantage for scenario 
>> #2, it doesn't address a more important case of #1 and to me it still 
>> looks like low security. The proper way here is indeed to get some 
>> capability for storing the keys securely, but I see how this is hard 
>> to impossible on AST2500 (if that's what we're talking about here), 
>> without TrustZone or anything similar, or some sort of 
>> ROM/bootloader-accessible-only part-unique secret, so OTMH I can't 
>> propose any viable alternatives for a given context (brownfield 
>> deployment, backward compatibility), but I'll think more about that.
>>
>> [1] https://www.freedesktop.org/software/systemd/man/machine-id.html
>
> Really, I don't see a real solution without some kind of secure 
> storage, which we don't have at this time on the ast2500. While that 
> is not the only machine running OpenBMC, ideally the solution would be 
> platform independent so that any machine could do it. Maybe that 
> requirement would be that it supported some minimal cryptographic 
> features or something (like trustzone support). But in the meantime, 
> some sort of solution that security researchers will not rake us over 
> the coals for would be nice. Maybe there isn't one. Some minimal 
> configuration that is better than writing the passwords in plain text.
>
> --Vernon
>
>

Re: ipmi password storage

[Joseph Reynolds]

On 4/14/20 2:14 PM, Vernon Mauery wrote:
> On 14-Apr-2020 06:04 PM, Milton Miller II wrote:
>> On Apr 13, 2020 around 6:01PM in some time zone, Vernon Mauery wrote:
>>>
>>> Internally, an issue was raised that basically says that the
>>> mechanism
>>> by which we are storing the IPMI passwords on the BMC is
>>> insufficiently
>>> obfuscated. I have come up with a patch set that resolves this at the
>>>
>>> expense of no downgrading the BMC without the side-effect of losing
>>> all
>>> IPMI passwords. I would like to know what the community thinks about
>>> usability vs. security in this scenario.
>>
>> ...
>>
>>> The migration from the old mechanism to the new could be done simply>by
>>> using the new key on the next write to the /etc/ipmi_pass file. After
>>> a
>>> firmware update to this new code, a password change would trigger a
>>> decrypt of the /etc/ipmi_pass file, a modification of the plain text,
>>>
>>> and a re-encryption of the data. If it reads the 'legacy' key in and
>>> writes out the data using the new key mechanism and deletes the
>>> legacy
>>> key, it would use the new key mechanism from that point onward.
>>> However,
>>> this would cause any downgrades to prior versions to fail to decrypt
>>> the
>>> /etc/ipmi_pass file, thereby losing all the ipmi passwords. This is
>>> not
>>> ideal, but could possibly be mitigating by truncating the new
>>> machine-id
>>> derivative password to 8 bytes and storing it in the /etc/key_file
>>> instead of just deleting it. This might improve security only
>>> slightly
>>> at for the price of a better user experience.
>>>
>>
>> I'll point out the code to handle the new password could be added
>> before the cdoe to use the new method, allowing test and revert
>> until the users are upgraded to the new method.  It does require
>> both methods to be supported.
>
> Yes, it looks like any sort of change here would need to be a staged 
> change to reduce the disruption.

Thanks for handling this issue -- I appreciate it.  Don't take this the 
wrong way, but...
If this change provides little value and causes upgrade issues, would it 
be better to avoid having an upgrade path?
Instead, use this new approach for new major release that requires a 
fresh install and upgrading is not an option.

>
>> I didn't follow why currently all openbmc systems end up with
>> the same encryption^Wobsfucation for what that is worth.
>
> Unless the build has a bbappend that changes the contents of the 
> key_file that is a part of the pam-ipmi package, all of the builds 
> will contain that same key_file. I can't say for sure how many builds 
> have this already, but I did not see much documentation around that 
> fact that would have spurred people to take action, so it is my 
> assumption that most builds would use the default.

 From previous emails in this thread, it doesn't seem like having each 
BMC having an unique key_file would help much.  Nevertheless, I've added 
this to my notes for BMC build considerations: 
https://github.com/ibm-openbmc/dev/issues/1531#issuecomment-613676676

- Joseph

>
> --Vernon

Re: ipmi password storage

[Vernon Mauery]

On 14-Apr-2020 05:03 PM, Joseph Reynolds wrote:
>
>
>On 4/14/20 11:46 AM, Vernon Mauery wrote:
>>On 14-Apr-2020 10:50 AM, Patrick Williams wrote:
>>>On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote:
>>>
>>>Vernon,
>>>
>>>Is there some background pointers on why IPMI needs to store passwords
>>>in a reverable way?  I never understood that.
>>
>>Sure. I think this is most clearly described in section 13.31 "RMCP+ 
>>Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec.
>
>This may be a bit naive....  Is it possible to expand the RCMP+ spec 
>with a new cipher suite or similar, to use a mechanism more like HTTPS 
>or SSH that does not require the server to be able to produce a clear 
>text password?  Given that IPMI will be used for many years, this 
>seems worthwhile, and would solve the current problem.

While IPMI will not likely be abandoned for many years to come, there 
are not any plans to update the specification. Redfish is supposed to be 
the answer, but like IPv4 was supposed to be supplanted by IPv6 long 
ago, full adoption is still dragging its feet.

That being said, I am not opposed to creating a new de-facto standard. 
In the name of security, I would not be opposed to using modern crypto 
protocols to establish secure IPMI sessions. This would likely cause the 
adoption of redfish to be even slower, because the biggest detractor of 
IPMI would be fixed.

We have the maintainer of ipmitool as a member of the OpenBMC community, 
(Alexander Amelkin) so we could even implement both ends of this new 
de-facto standard. I would suggest a DTLS connection on UDP 623, fully 
replacing RCMP+.

--Vernon

Re: ipmi password storage

[Vernon Mauery]

On 14-Apr-2020 05:18 PM, Joseph Reynolds wrote:
>On 4/14/20 2:11 PM, Vernon Mauery wrote:
>>On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote:
>>>To be more specific, I'm considering two attack scenarios in the 
>>>below comments:
>>>1) The attacker gets into BMC and is able to copy off the data 
>>>files, including ipmi_pass. This IMHO is a more realistic scenario 
>>>in this case.
>>>2) The attacker gets ipmi_pass file/contents only, without being 
>>>able to retrieve anything else.
>>>
>>>Which ones do *you* have in mind? For the sake of discussion, it 
>>>would be helpful to specify them all and see how potential 
>>>solutions address them.
>>
>>Attack one is the most likely, since if you can read one file, you 
>>can probably get any/all of them.
>
>An alternate solution is to make those file readable only by root, 
>restricting root logins, and restricting SSH access in general.  See 
>https://github.com/ibm-openbmc/dev/issues/1528

Changes have been made already to restrict permissions to 0600. The 
problem is that currently that is not much of protection at all because 
all the processes run as root right now anyway.

--Vernon

>In this way, only an admin can get the files by enabling the SSH 
>interface, logging in via SSH, and using su or sudo to access the 
>files.  All of these events should be audit-able.
>
>- Joseph
>
>>
>>>On 14-Apr-20 01:00, Vernon Mauery wrote:
>>>>Possible Solution
>>>>=================
>>>>Migrate to a solution that uses a key that is longer that does 
>>>>not get written directly to the flash
>>>>1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID)
>>>
>>>I like the direction and the guiding principle, however from 
>>>"proper" cryptography standpoint, for producing keys, this is not 
>>>noticeably better than the previous solution. The key material is 
>>>still readily available on the filesystem (making points #3 and #4 
>>>irrelevant for attack scenario #1), so chances are that the 
>>>adversary who can pull /etc/ipmi_pass file, can also pull 
>>>/etc/machine-id one. In addition (but that's really a nitpick in 
>>>this case with all these much bigger problems + easy to fix), 
>>>plain HMAC is not a proper and recommended key derivation 
>>>function, an HKDF or one of the NIST constructs from SP800-108 
>>>should be used instead.
>>
>>Cool. I did not know about HKDF. I will look into that.
>>
>>>>2) S is longer (32 bytes instead of 8)
>>>This (and #4) only helps the attack scenario #2, where attacker 
>>>has to brute-force all possible values for S - and I believe that 
>>>scenario is less realictic. But yes, case #2 gets a bit better 
>>>than before, if we disregard the fact that HMAC is not a proper 
>>>KDF and that MachineID is not a proper cryptographic key material 
>>>[1, last paragraph in Description suggests that], which may 
>>>provide additional advantage to the attacker. BTW, machine-id has 
>>>at most 128 bits of entropy (if produced by a proper CSRNG, which 
>>>I'm not sure about - e.g. I see they do some formatting), so 
>>>practically it's not 32 bytes, but 16 [at most].
>>
>>Yes, this was mostly a minor point. And yes, the entropy in S is 
>>limited by machine-id, but either one is better than a hard-coded 
>>8-byte ASCII string :)
>>
>>>So while this new approach does provide some advantage for 
>>>scenario #2, it doesn't address a more important case of #1 and to 
>>>me it still looks like low security. The proper way here is indeed 
>>>to get some capability for storing the keys securely, but I see 
>>>how this is hard to impossible on AST2500 (if that's what we're 
>>>talking about here), without TrustZone or anything similar, or 
>>>some sort of ROM/bootloader-accessible-only part-unique secret, so 
>>>OTMH I can't propose any viable alternatives for a given context 
>>>(brownfield deployment, backward compatibility), but I'll think 
>>>more about that.
>>>
>>>[1] https://www.freedesktop.org/software/systemd/man/machine-id.html
>>
>>Really, I don't see a real solution without some kind of secure 
>>storage, which we don't have at this time on the ast2500. While that 
>>is not the only machine running OpenBMC, ideally the solution would 
>>be platform independent so that any machine could do it. Maybe that 
>>requirement would be that it supported some minimal cryptographic 
>>features or something (like trustzone support). But in the meantime, 
>>some sort of solution that security researchers will not rake us 
>>over the coals for would be nice. Maybe there isn't one. Some 
>>minimal configuration that is better than writing the passwords in 
>>plain text.
>>
>>--Vernon
>>
>>
>

Re: ipmi password storage

[Vernon Mauery]

On 14-Apr-2020 05:30 PM, Joseph Reynolds wrote:
>>>>key, it would use the new key mechanism from that point onward.
>>>>However,
>>>>this would cause any downgrades to prior versions to fail to decrypt
>>>>the
>>>>/etc/ipmi_pass file, thereby losing all the ipmi passwords. This is
>>>>not
>>>>ideal, but could possibly be mitigating by truncating the new
>>>>machine-id
>>>>derivative password to 8 bytes and storing it in the /etc/key_file
>>>>instead of just deleting it. This might improve security only
>>>>slightly
>>>>at for the price of a better user experience.
>>>>
>>>
>>>I'll point out the code to handle the new password could be added
>>>before the cdoe to use the new method, allowing test and revert
>>>until the users are upgraded to the new method.  It does require
>>>both methods to be supported.
>>
>>Yes, it looks like any sort of change here would need to be a staged 
>>change to reduce the disruption.
>
>Thanks for handling this issue -- I appreciate it.  Don't take this 
>the wrong way, but...
>If this change provides little value and causes upgrade issues, would 
>it be better to avoid having an upgrade path?
>Instead, use this new approach for new major release that requires a 
>fresh install and upgrading is not an option.
>

It doesn't cause upgrade issues, it would cause downgrade issues. While 
I am not personally opposed to not downgrading (always move forward), 
not all IT shops have the same opinion. I thought it might be nice to 
offer an opportunity to support downgrades.

--Vernon

>>>I didn't follow why currently all openbmc systems end up with
>>>the same encryption^Wobsfucation for what that is worth.
>>
>>Unless the build has a bbappend that changes the contents of the 
>>key_file that is a part of the pam-ipmi package, all of the builds 
>>will contain that same key_file. I can't say for sure how many 
>>builds have this already, but I did not see much documentation 
>>around that fact that would have spurred people to take action, so 
>>it is my assumption that most builds would use the default.
>
From previous emails in this thread, it doesn't seem like having each 
>BMC having an unique key_file would help much.  Nevertheless, I've 
>added this to my notes for BMC build considerations: 
>https://github.com/ibm-openbmc/dev/issues/1531#issuecomment-613676676
>
>- Joseph
>
>>
>>--Vernon
>

Re: ipmi password storage

[Joseph Reynolds]

On 4/14/20 5:44 PM, Vernon Mauery wrote:
> On 14-Apr-2020 05:18 PM, Joseph Reynolds wrote:
>> On 4/14/20 2:11 PM, Vernon Mauery wrote:
>>> On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote:
>>>> To be more specific, I'm considering two attack scenarios in the 
>>>> below comments:
>>>> 1) The attacker gets into BMC and is able to copy off the data 
>>>> files, including ipmi_pass. This IMHO is a more realistic scenario 
>>>> in this case.
>>>> 2) The attacker gets ipmi_pass file/contents only, without being 
>>>> able to retrieve anything else.
>>>>
>>>> Which ones do *you* have in mind? For the sake of discussion, it 
>>>> would be helpful to specify them all and see how potential 
>>>> solutions address them.
>>>
>>> Attack one is the most likely, since if you can read one file, you 
>>> can probably get any/all of them.
>>
>> An alternate solution is to make those file readable only by root, 
>> restricting root logins, and restricting SSH access in general.  See 
>> https://github.com/ibm-openbmc/dev/issues/1528
>
> Changes have been made already to restrict permissions to 0600. The 
> problem is that currently that is not much of protection at all 
> because all the processes run as root right now anyway.

Understood.  An attacker who gets control of any process will have read 
access to this file, etc.
The issue that all processes run as root is documented here 
https://github.com/openbmc/openbmc/issues/3383
and was recently discussed in the security working group, 2020-04-01 - 
https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI 


- Joseph

>
> --Vernon
>
>> In this way, only an admin can get the files by enabling the SSH 
>> interface, logging in via SSH, and using su or sudo to access the 
>> files.  All of these events should be audit-able.
>>
>> - Joseph
>>
>>>
>>>> On 14-Apr-20 01:00, Vernon Mauery wrote:
>>>>> Possible Solution
>>>>> =================
>>>>> Migrate to a solution that uses a key that is longer that does not 
>>>>> get written directly to the flash
>>>>> 1) S is now computed instead of hard-coded. S=HMAC(MachineId, AppID)
>>>>
>>>> I like the direction and the guiding principle, however from 
>>>> "proper" cryptography standpoint, for producing keys, this is not 
>>>> noticeably better than the previous solution. The key material is 
>>>> still readily available on the filesystem (making points #3 and #4 
>>>> irrelevant for attack scenario #1), so chances are that the 
>>>> adversary who can pull /etc/ipmi_pass file, can also pull 
>>>> /etc/machine-id one. In addition (but that's really a nitpick in 
>>>> this case with all these much bigger problems + easy to fix), plain 
>>>> HMAC is not a proper and recommended key derivation function, an 
>>>> HKDF or one of the NIST constructs from SP800-108 should be used 
>>>> instead.
>>>
>>> Cool. I did not know about HKDF. I will look into that.
>>>
>>>>> 2) S is longer (32 bytes instead of 8)
>>>> This (and #4) only helps the attack scenario #2, where attacker has 
>>>> to brute-force all possible values for S - and I believe that 
>>>> scenario is less realictic. But yes, case #2 gets a bit better than 
>>>> before, if we disregard the fact that HMAC is not a proper KDF and 
>>>> that MachineID is not a proper cryptographic key material [1, last 
>>>> paragraph in Description suggests that], which may provide 
>>>> additional advantage to the attacker. BTW, machine-id has at most 
>>>> 128 bits of entropy (if produced by a proper CSRNG, which I'm not 
>>>> sure about - e.g. I see they do some formatting), so practically 
>>>> it's not 32 bytes, but 16 [at most].
>>>
>>> Yes, this was mostly a minor point. And yes, the entropy in S is 
>>> limited by machine-id, but either one is better than a hard-coded 
>>> 8-byte ASCII string :)
>>>
>>>> So while this new approach does provide some advantage for scenario 
>>>> #2, it doesn't address a more important case of #1 and to me it 
>>>> still looks like low security. The proper way here is indeed to get 
>>>> some capability for storing the keys securely, but I see how this 
>>>> is hard to impossible on AST2500 (if that's what we're talking 
>>>> about here), without TrustZone or anything similar, or some sort of 
>>>> ROM/bootloader-accessible-only part-unique secret, so OTMH I can't 
>>>> propose any viable alternatives for a given context (brownfield 
>>>> deployment, backward compatibility), but I'll think more about that.
>>>>
>>>> [1] https://www.freedesktop.org/software/systemd/man/machine-id.html
>>>
>>> Really, I don't see a real solution without some kind of secure 
>>> storage, which we don't have at this time on the ast2500. While that 
>>> is not the only machine running OpenBMC, ideally the solution would 
>>> be platform independent so that any machine could do it. Maybe that 
>>> requirement would be that it supported some minimal cryptographic 
>>> features or something (like trustzone support). But in the meantime, 
>>> some sort of solution that security researchers will not rake us 
>>> over the coals for would be nice. Maybe there isn't one. Some 
>>> minimal configuration that is better than writing the passwords in 
>>> plain text.
>>>
>>> --Vernon
>>>
>>>
>>

Re: ipmi password storage

[Joseph Reynolds]

On 4/14/20 5:48 PM, Joseph Reynolds wrote:
>
>
> On 4/14/20 5:44 PM, Vernon Mauery wrote:
>> On 14-Apr-2020 05:18 PM, Joseph Reynolds wrote:
>>> On 4/14/20 2:11 PM, Vernon Mauery wrote:
>>>> On 14-Apr-2020 06:27 PM, Alexander Tereschenko wrote:
>>>>> To be more specific, I'm considering two attack scenarios in the 
>>>>> below comments:
>>>>> 1) The attacker gets into BMC and is able to copy off the data 
>>>>> files, including ipmi_pass. This IMHO is a more realistic scenario 
>>>>> in this case.
>>>>> 2) The attacker gets ipmi_pass file/contents only, without being 
>>>>> able to retrieve anything else.
>>>>>
>>>>> Which ones do *you* have in mind? For the sake of discussion, it 
>>>>> would be helpful to specify them all and see how potential 
>>>>> solutions address them.
>>>>
>>>> Attack one is the most likely, since if you can read one file, you 
>>>> can probably get any/all of them.
>>>
>>> An alternate solution is to make those file readable only by root, 
>>> restricting root logins, and restricting SSH access in general.  See 
>>> https://github.com/ibm-openbmc/dev/issues/1528
>>
>> Changes have been made already to restrict permissions to 0600. The 
>> problem is that currently that is not much of protection at all 
>> because all the processes run as root right now anyway.
>
> Understood.  An attacker who gets control of any process will have 
> read access to this file, etc.
> The issue that all processes run as root is documented here 
> https://github.com/openbmc/openbmc/issues/3383
> and was recently discussed in the security working group, 2020-04-01 - 
> https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI 
>
>
> - Joseph

We discussed this in the OpenBMC security working group meeting 
2020-04-15 - 
https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI

My summary:

There is a patch to fix the file permissions so only root can read the 
files.  It addresses the most severe aspect of the vulnerability.  With 
the patch, only the root user can exploit the vulnerability.

But even with the fix, the underlying problem remains that encrypted 
passwords are stored on the BMC's file system that the root user can 
decrypt.  The root user should be trustworthy, but also should not have 
such access to user passwords.  Also, some other user can become root, 
making it a priority to restrict root access (including access to SSH 
and to the su and sudo commands).  Also, the fact that most OpenBMC 
processes run as the root user id (userid=0) makes escalation more likely.

Additional attempts were made to secure the ipmi_pass file, but the 
cost/benefit is unclear:

1. Encrypting the file differently was considered, but provides little 
benefit because the decryption credentials are still available to root.  
And we don't have a TPM and are not using ARM TrustZone to help isolate 
access to the key.  And changes like this cause upgrade and downgrade 
complications.

2. Abandoning RMCP+ would mean the ipmi_pass file is no longer needed, 
so the problem goes away.  We would need to create a new way for 
ipmitool to authenticate with the ipmi network daemon, thus creating a 
de-facto standard.  That would be a bunch of work and may extend the 
lifetime of IPMI.

So the answer for now is to fix the file permissions, pending coming up 
with a better solution.

- Joseph

... snip...

Re: ipmi password storage

[Joel Stanley]

On Tue, 14 Apr 2020 at 22:43, Vernon Mauery
<vernon.mauery@linux.intel.com> wrote:
>
> On 14-Apr-2020 05:03 PM, Joseph Reynolds wrote:
> >
> >
> >On 4/14/20 11:46 AM, Vernon Mauery wrote:
> >>On 14-Apr-2020 10:50 AM, Patrick Williams wrote:
> >>>On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote:
> >>>
> >>>Vernon,
> >>>
> >>>Is there some background pointers on why IPMI needs to store passwords
> >>>in a reverable way?  I never understood that.
> >>
> >>Sure. I think this is most clearly described in section 13.31 "RMCP+
> >>Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec.
> >
> >This may be a bit naive....  Is it possible to expand the RCMP+ spec
> >with a new cipher suite or similar, to use a mechanism more like HTTPS
> >or SSH that does not require the server to be able to produce a clear
> >text password?  Given that IPMI will be used for many years, this
> >seems worthwhile, and would solve the current problem.
>
> While IPMI will not likely be abandoned for many years to come, there
> are not any plans to update the specification. Redfish is supposed to be
> the answer, but like IPv4 was supposed to be supplanted by IPv6 long
> ago, full adoption is still dragging its feet.
>
> That being said, I am not opposed to creating a new de-facto standard.
> In the name of security, I would not be opposed to using modern crypto
> protocols to establish secure IPMI sessions. This would likely cause the
> adoption of redfish to be even slower, because the biggest detractor of
> IPMI would be fixed.

This is a great suggestion.

> We have the maintainer of ipmitool as a member of the OpenBMC community,
> (Alexander Amelkin) so we could even implement both ends of this new
> de-facto standard. I would suggest a DTLS connection on UDP 623, fully
> replacing RCMP+.

Re: ipmi password storage

[Vernon Mauery]

On 16-Apr-2020 06:04 AM, Joel Stanley wrote:
>On Tue, 14 Apr 2020 at 22:43, Vernon Mauery
><vernon.mauery@linux.intel.com> wrote:
>>
>> On 14-Apr-2020 05:03 PM, Joseph Reynolds wrote:
>> >
>> >
>> >On 4/14/20 11:46 AM, Vernon Mauery wrote:
>> >>On 14-Apr-2020 10:50 AM, Patrick Williams wrote:
>> >>>On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote:
>> >>>
>> >>>Vernon,
>> >>>
>> >>>Is there some background pointers on why IPMI needs to store passwords
>> >>>in a reverable way?  I never understood that.
>> >>
>> >>Sure. I think this is most clearly described in section 13.31 "RMCP+
>> >>Authenticated Key-Exchange Protocol (RAKP)" in the IPMI v2 1.1 spec.
>> >
>> >This may be a bit naive....  Is it possible to expand the RCMP+ spec
>> >with a new cipher suite or similar, to use a mechanism more like HTTPS
>> >or SSH that does not require the server to be able to produce a clear
>> >text password?  Given that IPMI will be used for many years, this
>> >seems worthwhile, and would solve the current problem.
>>
>> While IPMI will not likely be abandoned for many years to come, there
>> are not any plans to update the specification. Redfish is supposed to be
>> the answer, but like IPv4 was supposed to be supplanted by IPv6 long
>> ago, full adoption is still dragging its feet.
>>
>> That being said, I am not opposed to creating a new de-facto standard.
>> In the name of security, I would not be opposed to using modern crypto
>> protocols to establish secure IPMI sessions. This would likely cause the
>> adoption of redfish to be even slower, because the biggest detractor of
>> IPMI would be fixed.
>
>This is a great suggestion.

Thanks! Here is a design doc I wrote up.
https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/31548

--Vernon

>> We have the maintainer of ipmitool as a member of the OpenBMC community,
>> (Alexander Amelkin) so we could even implement both ends of this new
>> de-facto standard. I would suggest a DTLS connection on UDP 623, fully
>> replacing RCMP+.

Re: ipmi password storage

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2180 bytes --]

Vernon,

Thanks for the info.  It's taken a few days before I could find a time
slot to think about this in detail.

On Tue, Apr 14, 2020 at 09:46:10AM -0700, Vernon Mauery wrote:
> On 14-Apr-2020 10:50 AM, Patrick Williams wrote:
> >On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote:
> Specifically, the RAKP3 message (User->BMC) contains an HMAC of the 
> various parts of the exchanged session (User random number, session ID, Role, 
> Username) using the password as the key for the HMAC. The BMC needs to 
> compute this same HMAC to compare (this is the main authentication 
> challenge).
> 
> Then, the session key is generated using an HMAC of similar 
> data (BMC random number, user random number, role, username) using 
> either the user password or the channel password. Almost nobody uses the 
> channel password, which is good because it allows for simple privilege 
> escalation and session hijacking.

If I understand this right, the algorithm uses this to create a symetric
key for the session:
    HMAC(password, hash(pseudo_random_session_data))

The client gives some data to seed the session-data and the server gives
some data to seed the session-data, so that part is unique per
connection.  The part that is constant, and that they both have, is the
key to the HMAC.  Do I have that right?

The HMAC algorithm, at least for SHA-1/2, is a two-phase hash call, but
the key is always used at the front of the first hash phase.  The SHA
hash algorithms allow you to do a partial hash, extract the hash state,
and then resume the hash (See SHA1_Init / SHA1_Update functions in
openssl/sha1.h as example).  Rather than calling the OpenSSL HMAC
directly, can't we rewrite it such that it uses the partial hash on the
key as the starting point rather than the raw password?

This approach would allow us to save a SHA-1 + SHA-2 hash'd version of
the password rather than the raw (or even obfuscated) password.  It
isn't as secure as the normal crypt methods because we cannot salt them,
but we could certainly obfuscate the hash-state in a similar method to
what you already proposed.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: ipmi password storage

[Vernon Mauery]

On 22-Apr-2020 06:38 AM, Patrick Williams wrote:
>Vernon,
>
>Thanks for the info.  It's taken a few days before I could find a time
>slot to think about this in detail.
>
>On Tue, Apr 14, 2020 at 09:46:10AM -0700, Vernon Mauery wrote:
>> On 14-Apr-2020 10:50 AM, Patrick Williams wrote:
>> >On Mon, Apr 13, 2020 at 04:00:15PM -0700, Vernon Mauery wrote:
>> Specifically, the RAKP3 message (User->BMC) contains an HMAC of the
>> various parts of the exchanged session (User random number, session ID, Role,
>> Username) using the password as the key for the HMAC. The BMC needs to
>> compute this same HMAC to compare (this is the main authentication
>> challenge).
>>
>> Then, the session key is generated using an HMAC of similar
>> data (BMC random number, user random number, role, username) using
>> either the user password or the channel password. Almost nobody uses the
>> channel password, which is good because it allows for simple privilege
>> escalation and session hijacking.
>
>If I understand this right, the algorithm uses this to create a symetric
>key for the session:
>    HMAC(password, hash(pseudo_random_session_data))
>
>The client gives some data to seed the session-data and the server gives
>some data to seed the session-data, so that part is unique per
>connection.  The part that is constant, and that they both have, is the
>key to the HMAC.  Do I have that right?

Yes.

>The HMAC algorithm, at least for SHA-1/2, is a two-phase hash call, but
>the key is always used at the front of the first hash phase.  The SHA
>hash algorithms allow you to do a partial hash, extract the hash state,
>and then resume the hash (See SHA1_Init / SHA1_Update functions in
>openssl/sha1.h as example).  Rather than calling the OpenSSL HMAC
>directly, can't we rewrite it such that it uses the partial hash on the
>key as the starting point rather than the raw password?

I had to code up a quick test to make sure, but yes, it is possible to 
do this.

>This approach would allow us to save a SHA-1 + SHA-2 hash'd version of
>the password rather than the raw (or even obfuscated) password.  It
>isn't as secure as the normal crypt methods because we cannot salt them,
>but we could certainly obfuscate the hash-state in a similar method to
>what you already proposed.

Pros and cons of this new solution:
Pros:
1) Obfuscates passwords more than currently. A hash is even harder to 
deal with than an encrypted file that has the key sitting right next to 
it.
2) Item one is a very strong plus

Cons:
1) Would require us to write or hack up an HMAC that would work for us. 
This is not ideal; writing crypto stuff is generally best left to the 
experts. OpenSSL is trying harder and harder to hide its internal 
workings from its API, so extracting the hash state is hard. libTomCrypt 
allows you to do this easier, but still does not have an HMAC function 
that takes two partial hash states as inputs.
2) Is not backwards compatible with any of our current password storage 
mechanisms. This is workable, but would likely take some time to stage 
things so that they would work better without breaking everything.
3) We would need to store two hashes of each password for both SHA1 and 
SHA256, for a total of 4 states per password. This greatly reduces the 
effectiveness of the solution in the first place. Not that we are 
planning on adding MD5- or MD2-based cipher suites, but those would 
break the value of this completely.

So pro-1 is good. but con-2 and con-3 are bad. Con-2 we can deal with, 
but con-3 brings down the value of pro-1.

--Vernon

Re: ipmi password storage

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2614 bytes --]

On Wed, Apr 22, 2020 at 03:19:43PM -0700, Vernon Mauery wrote:
> Pros and cons of this new solution:
> Pros:
> 1) Obfuscates passwords more than currently. A hash is even harder to 
> deal with than an encrypted file that has the key sitting right next to 
> it.
> 2) Item one is a very strong plus
> 
> Cons:
> 1) Would require us to write or hack up an HMAC that would work for us. 
> This is not ideal; writing crypto stuff is generally best left to the 
> experts. OpenSSL is trying harder and harder to hide its internal 
> workings from its API, so extracting the hash state is hard. libTomCrypt 
> allows you to do this easier, but still does not have an HMAC function 
> that takes two partial hash states as inputs.

I don't think that the interfaces we'd be needing to use here will ever
be deprecated.  The "hash-resume" functions are needed in order to hash
very large files that cannot be fit in memory.  They're just too useful
to deprecate.

You are correct that we would be effectively implementing HMAC ourselves
using the SHA hash functions though.  It _seems_ straight-forward enough
though on the surface.

> 2) Is not backwards compatible with any of our current password storage 
> mechanisms. This is workable, but would likely take some time to stage 
> things so that they would work better without breaking everything.

Agreed, but it isn't any less backwards compatible than the obfuscation
proposal.  This could certainly be staged in with a compile flag that
keeps the old file around.  If a vendor is concerned about backwards
compatiblity they could enable that flag for a period of time until
their customers upgrade through the new support version.  (Disabled by
default in meta-phosphor).

> 3) We would need to store two hashes of each password for both SHA1 and 
> SHA256, for a total of 4 states per password. This greatly reduces the 
> effectiveness of the solution in the first place. Not that we are 
> planning on adding MD5- or MD2-based cipher suites, but those would 
> break the value of this completely.

Agree on keeping anything like MD5/MD2 out of it.

I'm not a cryptographic expert, but I'm not sure it "greatly reduces"
the effectiveness.  The addition of the SHA-2 state probably doesn't
help you reverse or collide the SHA-1 much at all, so this is only
slightly weaker than SHA-1 itself.

My reading on the SHA-1 algorithm would lead me to believe that SHA-1 state
is the same as SHA-1(password) itself.  Meaning, by doing this partial
hash we're not weaking the algorithm at all.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: ipmi password storage

[Alexander Tereschenko]

On 24-Apr-20 17:15, Patrick Williams wrote:
> On Wed, Apr 22, 2020 at 03:19:43PM -0700, Vernon Mauery wrote:
>> Pros and cons of this new solution:
>> Pros:
>> 1) Obfuscates passwords more than currently. A hash is even harder to
>> deal with than an encrypted file that has the key sitting right next to
>> it.
>> 2) Item one is a very strong plus
>>
>> Cons:
>> 1) Would require us to write or hack up an HMAC that would work for us.
>> This is not ideal; writing crypto stuff is generally best left to the
>> experts. OpenSSL is trying harder and harder to hide its internal
>> workings from its API, so extracting the hash state is hard. libTomCrypt
>> allows you to do this easier, but still does not have an HMAC function
>> that takes two partial hash states as inputs.
> I don't think that the interfaces we'd be needing to use here will ever
> be deprecated.  The "hash-resume" functions are needed in order to hash
> very large files that cannot be fit in memory.  They're just too useful
> to deprecate.
>
> You are correct that we would be effectively implementing HMAC ourselves
> using the SHA hash functions though.  It _seems_ straight-forward enough
> though on the surface.

Yes, that should be doable and HMAC is generally not that prone to 
implementation problems, so this is reasonable to assume it could be 
done safely using just the plain hash primitives. The major problem with 
this one, from my standpoint, is that the unsalted hashes are these days 
a too easy target for attackers, as hash tables can be (and actually 
are, you can download those I believe) precomputed on password 
dictionaries and without salt it's a BORE-type risk (break once, run 
everywhere) - the hash will be the same on all BMCs if the password is 
the same, so compromising one means compromising others.

>> 3) We would need to store two hashes of each password for both SHA1 and
>> SHA256, for a total of 4 states per password. This greatly reduces the
>> effectiveness of the solution in the first place. Not that we are
>> planning on adding MD5- or MD2-based cipher suites, but those would
>> break the value of this completely.
> Agree on keeping anything like MD5/MD2 out of it.
>
> I'm not a cryptographic expert, but I'm not sure it "greatly reduces"
> the effectiveness.  The addition of the SHA-2 state probably doesn't
> help you reverse or collide the SHA-1 much at all, so this is only
> slightly weaker than SHA-1 itself.
>
> My reading on the SHA-1 algorithm would lead me to believe that SHA-1 state
> is the same as SHA-1(password) itself.  Meaning, by doing this partial
> hash we're not weaking the algorithm at all.

For SHA-1 and SHA2-256/512 the hashing output *is* the hash internal 
state at the end of calculation, so there's no additional compromise 
here from the crypto standpoint. Essentially, this proposal is a hash 
length extension attack, but used for good, not evil :) The problem here 
is with hashes being unsalted, but I agree this is still better than 
encrypting with a static key thay lies nearby. FWIW, that DTLS solution 
looks a bit better to me as it allows us to ditch that password thing 
completely, but if that one is not possible, this one could be the next 
best.

BTW, Vernon, could you please elaborate why it's 4 values that we need 
to store here? One is SHA1, another one is SHA2-256, but what are the 
other two?

regards,
Alexander