Expired password + disabled power button design

Expired password + disabled power button design

[Joseph Reynolds]

I am working to reduce the time window when the BMC offers a default 
password to its network users.  See the [expired password design][].  
The idea is to force users to change the BMC's password as soon as possible.

[expired password design]: 
https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/23849

Note: This design seeks to reduce the time window when the BMC is on the 
network with a default password, not to completely remove this 
vulnerability.  The time window begins when the system is plugged into 
power, and ends when the BMC's password is changed.

When complete, I believe this will offer significant protection for the 
BMC and can be used to help comply with laws such as CA Law [SB-327][], 
specifically section 1798.91.04 (b) (2) which states "The device 
contains a security feature that requires a user to generate a new means 
of authentication before access is granted to the device for the first 
time."  Where "the device" is the BMC, and "access" means access past 
the BMC's authentication mechanism to a session which can operate the 
BMC's controls.  The section cited above seems to accept the time window 
as a reasonable security exposure and to use the term "access" as I have 
done.

[SB-327]: 
https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=201720180SB327

HOWEVER, there is a hole in this design which extends the time window 
indefinitely.  The scenario begins when the installer takes possession 
of a new system (BMC + host) and plugs it into power.  At this point, 
the BMC starts running and offering its services to network users.  The 
host remains powered off.  The installer then disregards the BMC and 
uses the power button to boot the host system, then continues to 
disregard the BMC when provisioning the host, either using physical 
access to the host (not via the BMC), or a pre-configured host.  This 
results in a fully-functional host and a BMC which still has its default 
password.

THEREFORE, I am proposing a new "disabled power button" image feature.  
Normally, pressing the power button tells the BMC to power on and boot 
the host.  With this design, if the BMC still has its default expired 
password, it will ignore a power button press, and will instead indicate 
to the operator to configure the BMC's password, and try again.  Options 
for the BMC to indicate this are machine-specific and include: messages 
to an operator panel, or LED blink codes.  The recovery procedure is for 
the installer to access the BMC, change its password, and try again to 
power on the server.

The implementation might possibly be in the [state manager][] so it can 
perform this check if the server is powered on for any reason. The test 
if the BMC still has its default expired password could be performed by 
invoking Linux PAM APIs, effectively trying to authenticate using the 
default credentials, and checking if the credentials and valid and 
expired.  An alternate approach could instead check if the BMC is in 
provisioning mode.

[state manager]: https://github.com/openbmc/phosphor-state-manager

Note: The "disabled power button" design feature would be off default.  
Platforms wanting to use it would have enable the image feature.

- Joseph

Re: Expired password + disabled power button design

[Ed Tanous]

On 9/5/19 1:19 PM, Joseph Reynolds wrote:
> HOWEVER, there is a hole in this design which extends the time window
> indefinitely.  The scenario begins when the installer takes possession
> of a new system (BMC + host) and plugs it into power.  At this point,
> the BMC starts running and offering its services to network users. 
> The host remains powered off.  The installer then disregards the BMC
> and uses the power button to boot the host system, then continues to
> disregard the BMC when provisioning the host, either using physical
> access to the host (not via the BMC), or a pre-configured host.  This
> results in a fully-functional host and a BMC which still has its
> default password.
> THEREFORE, I am proposing a new "disabled power button" image
> feature.  Normally, pressing the power button tells the BMC to power
> on and boot the host.  With this design, if the BMC still has its
> default expired password, it will ignore a power button press, and
> will instead indicate to the operator to configure the BMC's password,
> and try again.  Options for the BMC to indicate this are
> machine-specific and include: messages to an operator panel, or LED
> blink codes.  The recovery procedure is for the installer to access
> the BMC, change its password, and try again to power on the server.
>

This goes completely against one of the principals that some commercial
products hold dear:  The BMC should NEVER be able to brick the host.

In a perfect world, the BMC would never crash, get loaded with bad
firmware or brick itself.  In practice, this is far from the case.  In
general the primary avenue used to resolve these cases is a force
firmware update jumper, and a download of a new (fixed) firmware via KCS
or block transfer.  For that flow to function, the power button needs to
work, and the host needs to boot, otherwise the board is considered
bricked, and needs to be returned to the factory to be reflashed.

With all of the above said, if it's an option that can be disabled (and
hopefully disabled by default) I have no objection to it, but I don't
think it's an acceptable solution for most of the BMCs out on the market
today.


Another option for your scenario would also be to default the BMC to a
static IP of 0.0.0.0, and disable the NIC(s) by default.  To set up the
network, a user would need to log into the BMC the first time over an
in-band channel, set their password, then set up either a valid static
IP, or DHCP.  This is how some of the more security conscious BMCs I
know of get around this problem today.

Re: Expired password + disabled power button design

[Joseph Reynolds]

On 9/5/19 4:02 PM, Ed Tanous wrote:
> On 9/5/19 1:19 PM, Joseph Reynolds wrote:
>> HOWEVER, there is a hole in this design which extends the time window
>> indefinitely.  The scenario begins when the installer takes possession
>> of a new system (BMC + host) and plugs it into power.  At this point,
>> the BMC starts running and offering its services to network users.
>> The host remains powered off.  The installer then disregards the BMC
>> and uses the power button to boot the host system, then continues to
>> disregard the BMC when provisioning the host, either using physical
>> access to the host (not via the BMC), or a pre-configured host.  This
>> results in a fully-functional host and a BMC which still has its
>> default password.
>> THEREFORE, I am proposing a new "disabled power button" image
>> feature.  Normally, pressing the power button tells the BMC to power
>> on and boot the host.  With this design, if the BMC still has its
>> default expired password, it will ignore a power button press, and
>> will instead indicate to the operator to configure the BMC's password,
>> and try again.  Options for the BMC to indicate this are
>> machine-specific and include: messages to an operator panel, or LED
>> blink codes.  The recovery procedure is for the installer to access
>> the BMC, change its password, and try again to power on the server.
>>
> This goes completely against one of the principals that some commercial
> products hold dear:  The BMC should NEVER be able to brick the host.
>
> In a perfect world, the BMC would never crash, get loaded with bad
> firmware or brick itself.  In practice, this is far from the case.  In
> general the primary avenue used to resolve these cases is a force
> firmware update jumper, and a download of a new (fixed) firmware via KCS
> or block transfer.  For that flow to function, the power button needs to
> work, and the host needs to boot, otherwise the board is considered
> bricked, and needs to be returned to the factory to be reflashed.
>
> With all of the above said, if it's an option that can be disabled (and
> hopefully disabled by default) I have no objection to it, but I don't
> think it's an acceptable solution for most of the BMCs out on the market
> today.
>
>
> Another option for your scenario would also be to default the BMC to a
> static IP of 0.0.0.0, and disable the NIC(s) by default.  To set up the
> network, a user would need to log into the BMC the first time over an
> in-band channel, set their password, then set up either a valid static
> IP, or DHCP.  This is how some of the more security conscious BMCs I
> know of get around this problem today.

Thanks for your reply.  That helps.  We also don't want to brick our 
systems. :-)

How likely is the scenario described in the HOWEVER clause above? To 
create this security exposure, the installer would have to connect a 
network cable to the BMC and then not access the BMC via its network 
connection.  Is that something that happens in practice?  Could this be 
mitigated by installation instructions? The installation instructions 
could state, for example, not to connect a network cable to the BMC 
unless you then change the BMC's default password.  Then to emphasize 
the point, the instructions could also state, for example, that you must 
change the BMC's default password if you connect a network cable to it.

However, this approach doesn't feel secure by default.  I understand the 
more general problem is how to securely provision your BMC. Does anyone 
know where these discussions are happening?

- Joseph

Re: Expired password + disabled power button design

[vishwa]

Hi Joseph,

Not sure if this info helps much.. But, I know we did provision the FSP 
on the SoftLayer environment which had constraints.
Possibly we can look into how that was done ?

!! Vishwa !!

On 9/8/19 3:46 AM, Joseph Reynolds wrote:
>
> On 9/5/19 4:02 PM, Ed Tanous wrote:
>> On 9/5/19 1:19 PM, Joseph Reynolds wrote:
>>> HOWEVER, there is a hole in this design which extends the time window
>>> indefinitely.  The scenario begins when the installer takes possession
>>> of a new system (BMC + host) and plugs it into power.  At this point,
>>> the BMC starts running and offering its services to network users.
>>> The host remains powered off.  The installer then disregards the BMC
>>> and uses the power button to boot the host system, then continues to
>>> disregard the BMC when provisioning the host, either using physical
>>> access to the host (not via the BMC), or a pre-configured host.  This
>>> results in a fully-functional host and a BMC which still has its
>>> default password.
>>> THEREFORE, I am proposing a new "disabled power button" image
>>> feature.  Normally, pressing the power button tells the BMC to power
>>> on and boot the host.  With this design, if the BMC still has its
>>> default expired password, it will ignore a power button press, and
>>> will instead indicate to the operator to configure the BMC's password,
>>> and try again.  Options for the BMC to indicate this are
>>> machine-specific and include: messages to an operator panel, or LED
>>> blink codes.  The recovery procedure is for the installer to access
>>> the BMC, change its password, and try again to power on the server.
>>>
>> This goes completely against one of the principals that some commercial
>> products hold dear:  The BMC should NEVER be able to brick the host.
>>
>> In a perfect world, the BMC would never crash, get loaded with bad
>> firmware or brick itself.  In practice, this is far from the case.  In
>> general the primary avenue used to resolve these cases is a force
>> firmware update jumper, and a download of a new (fixed) firmware via KCS
>> or block transfer.  For that flow to function, the power button needs to
>> work, and the host needs to boot, otherwise the board is considered
>> bricked, and needs to be returned to the factory to be reflashed.
>>
>> With all of the above said, if it's an option that can be disabled (and
>> hopefully disabled by default) I have no objection to it, but I don't
>> think it's an acceptable solution for most of the BMCs out on the market
>> today.
>>
>>
>> Another option for your scenario would also be to default the BMC to a
>> static IP of 0.0.0.0, and disable the NIC(s) by default.  To set up the
>> network, a user would need to log into the BMC the first time over an
>> in-band channel, set their password, then set up either a valid static
>> IP, or DHCP.  This is how some of the more security conscious BMCs I
>> know of get around this problem today.
>
> Thanks for your reply.  That helps.  We also don't want to brick our 
> systems. :-)
>
> How likely is the scenario described in the HOWEVER clause above? To 
> create this security exposure, the installer would have to connect a 
> network cable to the BMC and then not access the BMC via its network 
> connection.  Is that something that happens in practice?  Could this 
> be mitigated by installation instructions? The installation 
> instructions could state, for example, not to connect a network cable 
> to the BMC unless you then change the BMC's default password.  Then to 
> emphasize the point, the instructions could also state, for example, 
> that you must change the BMC's default password if you connect a 
> network cable to it.
>
> However, this approach doesn't feel secure by default.  I understand 
> the more general problem is how to securely provision your BMC. Does 
> anyone know where these discussions are happening?
>
> - Joseph
>
>