overlayFS security concern

overlayFS security concern

[Kun Zhao]

[-- Attachment #1: Type: text/plain, Size: 490 bytes --]

Hi Team,

Have the following case ever been discussed before?,
Anyone knows the root password will be able to let bmc run their own code by scp the code into bmc with the same file path as any services in rootfs. It will make the secure boot totally useless.

So besides,
1. disable scp (but scp is one of the firmware upload way)
2. don’t use overlayFS (but it’s really useful for debugging during develop, and configuration management)
Any other solutions?



Thanks.
Kun


[-- Attachment #2: Type: text/html, Size: 4859 bytes --]

Re: overlayFS security concern

[chunhui.jia]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

Maintaining 2 different build configurations would be possible solution:  dev build and release build. 
1. enable debugging tech in dev build. 
2. when using openbmc for product, disable all potential ways that could harm security.


2021-02-20 

chunhui.jia 



发件人：Kun Zhao <zkxz@hotmail.com>
发送时间：2021-02-20 08:31
主题：overlayFS security concern
收件人："openbmc@lists.ozlabs.org"<openbmc@lists.ozlabs.org>
抄送：

Hi Team,
 
Have the following case ever been discussed before?,
Anyone knows the root password will be able to let bmc run their own code by scp the code into bmc with the same file path as any services in rootfs. It will make the secure boot totally useless.
 
So besides,
1. disable scp (but scp is one of the firmware upload way)
2. don’t use overlayFS (but it’s really useful for debugging during develop, and configuration management)
Any other solutions?
 
 
 
Thanks.
Kun
 

[-- Attachment #2: Type: text/html, Size: 6467 bytes --]

RE: overlayFS security concern

[Kun Zhao]

[-- Attachment #1.1: Type: text/plain, Size: 1408 bytes --]

Thank you, Chunhui. But you mean to disable scp, right? Firmware upload through scp function will be lost in this way. Maybe not a good choice for us.
BTW, is scp still a recommended way for OpenBMC firmware update?



Thanks.
Kun

From: chunhui.jia<mailto:chunhui.jia@linux.intel.com>
Sent: Friday, February 19, 2021 4:53 PM
To: Kun Zhao<mailto:zkxz@hotmail.com>; openbmc@lists.ozlabs.org<mailto:openbmc@lists.ozlabs.org>
Subject: Re: overlayFS security concern

Maintaining 2 different build configurations would be possible solution:  dev build and release build.
1. enable debugging tech in dev build.
2. when using openbmc for product, disable all potential ways that could harm security.


2021-02-20

chunhui.jia

发件人：Kun Zhao <zkxz@hotmail.com>
发送时间：2021-02-20 08:31
主题：overlayFS security concern
收件人："openbmc@lists.ozlabs.org"<openbmc@lists.ozlabs.org>
抄送：

Hi Team,

Have the following case ever been discussed before?,
Anyone knows the root password will be able to let bmc run their own code by scp the code into bmc with the same file path as any services in rootfs. It will make the secure boot totally useless.

So besides,
1. disable scp (but scp is one of the firmware upload way)
2. don’t use overlayFS (but it’s really useful for debugging during develop, and configuration management)
Any other solutions?



Thanks.
Kun



[-- Attachment #1.2: Type: text/html, Size: 8888 bytes --]

[-- Attachment #2: A24FB62FC7144662BA1C9A0C79685324.png --]
[-- Type: image/png, Size: 122 bytes --]

[-- Attachment #3: 82282195F0154A20AF6CCE387F3ED633.png --]
[-- Type: image/png, Size: 133 bytes --]

Re: overlayFS security concern

[Andrew Jeffery]

On Sat, 20 Feb 2021, at 11:01, Kun Zhao wrote:
>  
> Hi Team,
> 
>  
> 
> Have the following case ever been discussed before?,
> 
> Anyone knows the root password will be able to let bmc run their own 
> code by scp the code into bmc with the same file path as any services 
> in rootfs. It will make the secure boot totally useless.
> 
>  
> 
> So besides,
> 
> 1. disable scp (but scp is one of the firmware upload way)

This isn't really a solution as there are other ways to upload files.

> 
> 2. don’t use overlayFS (but it’s really useful for debugging during 
> develop, and configuration management)

Possibly, but it's probably worth looking at IMA instead:

https://sourceforge.net/p/linux-ima/wiki/Home/

Andrew

Re: RE: overlayFS security concern

[chunhui.jia]

[-- Attachment #1.1: Type: text/plain, Size: 1674 bytes --]

You could use redfish firmware update instead.

2021-02-20 

chunhui.jia 



发件人：Kun Zhao <zkxz@hotmail.com>
发送时间：2021-02-20 09:13
主题：RE: overlayFS security concern
收件人："chunhui.jia"<chunhui.jia@linux.intel.com>,"openbmc@lists.ozlabs.org"<openbmc@lists.ozlabs.org>
抄送：

Thank you, Chunhui. But you mean to disable scp, right? Firmware upload through scp function will be lost in this way. Maybe not a good choice for us.
BTW, is scp still a recommended way for OpenBMC firmware update?
 
 
 
Thanks.
Kun
 
From: chunhui.jia
Sent: Friday, February 19, 2021 4:53 PM
To: Kun Zhao; openbmc@lists.ozlabs.org
Subject: Re: overlayFS security concern
 
Maintaining 2 different build configurations would be possible solution:  dev build and release build. 
1. enable debugging tech in dev build. 
2. when using openbmc for product, disable all potential ways that could harm security.
 
 
2021-02-20 

chunhui.jia 

发件人：Kun Zhao <zkxz@hotmail.com>
发送时间：2021-02-20 08:31
主题：overlayFS security concern
收件人："openbmc@lists.ozlabs.org"<openbmc@lists.ozlabs.org>
抄送：
 
Hi Team,
 
Have the following case ever been discussed before?,
Anyone knows the root password will be able to let bmc run their own code by scp the code into bmc with the same file path as any services in rootfs. It will make the secure boot totally useless.
 
So besides,
1. disable scp (but scp is one of the firmware upload way)
2. don’t use overlayFS (but it’s really useful for debugging during develop, and configuration management)
Any other solutions?
 
 
 
Thanks.
Kun
 
 

[-- Attachment #1.2: Type: text/html, Size: 10596 bytes --]

[-- Attachment #2: A24FB62FC7144662BA1C9A0C79685324.png --]
[-- Type: image/png, Size: 122 bytes --]

[-- Attachment #3: 82282195F0154A20AF6CCE387F3ED633.png --]
[-- Type: image/png, Size: 133 bytes --]

Re: overlayFS security concern

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Sat, Feb 20, 2021 at 11:46:08AM +1030, Andrew Jeffery wrote:
> On Sat, 20 Feb 2021, at 11:01, Kun Zhao wrote:
> > 2. don’t use overlayFS (but it’s really useful for debugging during 
> > develop, and configuration management)
> 
> Possibly, but it's probably worth looking at IMA instead:

IMA (or similar) is likely a good option.

There is also work going on to remove 'root' from many users and
daemons so it should be harder to overwrite executables.  If you
have root I'm pretty sure you can always subvert even something like
IMA.

A protection we could do which would make attacks slightly harder
than they are today would be to change how we mount OverlayFS.  Right
now we mount it on top of root, but we could be more explicit about
mounting it only on top of places we expect to be read-write. `/etc`
and `/var` are the two that come to mind but I'm sure there are others.
This shouldn't be very difficult to implement for someone wanting to
take the initiative.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: overlayFS security concern

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]


Patrick Williams <patrick@stwcx.xyz> wrote:
    > A protection we could do which would make attacks slightly harder than
    > they are today would be to change how we mount OverlayFS.  Right now we
    > mount it on top of root, but we could be more explicit about mounting

I was going to ask about that.  Could we just overlay less?
The second question is: would a non-persistent overlay be useful?

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: overlayFS security concern - threat model

[Joseph Reynolds]

On 2/19/21 6:31 PM, Kun Zhao wrote:
> Hi Team, Have the following case ever been discussed before?,...
> This Message Is From an External Sender
> This message came from outside your organization.
>
> Hi Team,
>
> Have the following case ever been discussed before?,
>
> Anyone knows the root password will be able to let bmc run their own 
> code by scp the code into bmc with the same file path as any services 
> in rootfs. It will make the secure boot totally useless.
>
> So besides,
>
> 1. disable scp (but scp is one of the firmware upload way)
>
> 2. don’t use overlayFS (but it’s really useful for debugging during 
> develop, and configuration management)
>
> Any other solutions?
>

Kun,

Thanks for starting this thread.  Good discussion everyone!  Note 
OpenBMC's SSH interfaces are described in 
[network-security-considerations][].

I would like to improve the project's threat model to cover these ideas 
and so propose a new document, threat-model-considerations.md, to go 
under [OpenBMC security docs][] with topics:

- OpenBMC has a well-known password.
     - Explanation: OpenBMC has a well-known root password.  There is no 
mechanism which forces anyone to change it.
     - Risk: OpenBMC's default configuration produce firmware which has 
a well-known password which allows an attacker to gain full access to 
BMC internals.
     - Recommendation: Three independent recommendations:
         1. Provide a [configuration guide][] for firmware builders 
which instructs them to change the default password for their distro, 
and similar for BMC administrators which instructs them to change the 
default password when they first access the BMC.
         2. Implement an image option to cause the BMC's initial 
password to be expired so it must be changed before access is granted.
         3. Move away from password-based authentication.
     - Caveat: The expired password solution leaves a time window when 
the BMC is available to attackers and has well known password.

- Root account login to the BMC's command shell.
     - Explanation: Root access to the BMC's command shell is required 
for developers and service personnel to do their job. (They need to 
access the BMC internal tools and interfaces.)  But root access is not 
needed for any administrators or users performing normal BMC functions.
     - Risk: Users will have more authority than they need to do their 
job.  (The BMC should be designed so the owner can give each user the 
authority they nee, but no more.)  Additional authority may allow a user 
or attacker to damage BMC internals or gain information about the host 
they are not intended to have.
     - Recommendation: Disable root access in production images. (Should 
we have an OpenBMC image feature to control this?)  If root access is no 
disabled, it must be strictly limited to authorized users.

- SSH access to the BMC (including scp firmware update).
     - Explanation: The BMC provides an SSH server to allow secure shell 
(ssh) sessions to the BMC's command shell, ssh sessions to the host 
console, and scp-based firmware update.
     - Risk: See the "Root account login to the BMC's command shell" 
topic.  Note that ssh access to the host console is intended.
     - Recommendation: As above, disable SSH access for production 
images.  Disable SCP and instead use Redfish firmware update.

- Running unintended code on the BMC.
     - Explanation: The BMC is intended as a closed appliance but runs 
Linux.
     - Risk: The BMC's Linux capabilities allow an attacker who gains 
access to the BMC to attack the BMC's network and attack the BMC's host.
     - Recommendation: Use technologies like SecureBoot and Integrity 
Measurement Architecture (IMA).
     - Caveats: Powerful scripting tools such as the command shell (sh 
or bash) would be allowed by IMA to run on the BMC but still give an 
attacker the ability to run harmful code.

- Having writable portions of the BMC's file system
     - Explanation: The BMC is intended as a closed appliance with a 
read-only file system but has a writable portions of the file system to 
store BMC admin-configurable items, firmware updates, internal and host 
logs, etc.
     - Risk: The BMC's writable file system allows an attacker who gains 
access to the BMC to make their access permanent by writing to the BMC's 
file system.
     - Recommendation: Strictly limit which portions of the BMC's file 
system are writable.  For example, to /var, /run, and a strict subset of 
/etc.  Use a non-persistent overlay for data that does not need to 
persist beyond a BMC reboot.

I've added this topic to the [OpenBMC security working group agenda][].  
But please keep the discussion going in the email list.

Thanks!
-Joseph

[network-security-considerations]: 
https://github.com/openbmc/docs/blob/master/security/network-security-considerations.md
[OpenBMC security docs]: 
https://github.com/openbmc/docs/blob/master/security
[configuration guide]: 
https://github.com/openbmc/openbmc/wiki/Configuration-guide
[OpenBMC security working group agenda]: 
https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI 


>
> Thanks.
>
> Kun
>

Re: overlayFS security concern

[Lei Yu]

On Sun, Feb 21, 2021 at 12:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Sat, Feb 20, 2021 at 11:46:08AM +1030, Andrew Jeffery wrote:
> > On Sat, 20 Feb 2021, at 11:01, Kun Zhao wrote:
> > > 2. don’t use overlayFS (but it’s really useful for debugging during
> > > develop, and configuration management)
> >
> > Possibly, but it's probably worth looking at IMA instead:
>
> IMA (or similar) is likely a good option.
>
> There is also work going on to remove 'root' from many users and
> daemons so it should be harder to overwrite executables.  If you
> have root I'm pretty sure you can always subvert even something like
> IMA.
>
> A protection we could do which would make attacks slightly harder
> than they are today would be to change how we mount OverlayFS.  Right
> now we mount it on top of root, but we could be more explicit about
> mounting it only on top of places we expect to be read-write. `/etc`
> and `/var` are the two that come to mind but I'm sure there are others.
> This shouldn't be very difficult to implement for someone wanting to
> take the initiative.

Yup, as far as I remember, the "ubi layout" distro feature only mount
specific directories instead of root.
Checking the code, it enables the `read-only-rootfs`
IMAGE_FEATURES[1], and use a different init script to mount only /etc
by `preinit-mounts.bb`[2]
The same for `phosphor-mmc` as well.

@anoo should know this well :)

[1]: https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb#L35
[2]: https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-phosphor/preinit-mounts/preinit-mounts/init

-- 
BRs,
Lei YU

RE: overlayFS security concern

[Milton Miller II]

On Feb 22, Lei Yu wrote:
>On Sun, Feb 21, 2021 at 12:56 AM Patrick Williams <patrick@stwcx.xyz>
>wrote:
>> On Sat, Feb 20, 2021 at 11:46:08AM +1030, Andrew Jeffery wrote:
>> > On Sat, 20 Feb 2021, at 11:01, Kun Zhao wrote:
>> > > 2. don’t use overlayFS (but it’s really useful for debugging
>during
>> > > develop, and configuration management)
>> >
>> > Possibly, but it's probably worth looking at IMA instead:
>>
>> IMA (or similar) is likely a good option.
>>
>> There is also work going on to remove 'root' from many users and
>> daemons so it should be harder to overwrite executables.  If you
>> have root I'm pretty sure you can always subvert even something
>like
>> IMA.
>>


>> A protection we could do which would make attacks slightly harder
>> than they are today would be to change how we mount OverlayFS.
>Right
>> now we mount it on top of root, but we could be more explicit about
>> mounting it only on top of places we expect to be read-write.
>`/etc`
>> and `/var` are the two that come to mind but I'm sure there are
>others.
>> This shouldn't be very difficult to implement for someone wanting
>to
>> take the initiative.
>

I've offered before and the offer still stands.

As the author of the original system layout including the init 
and update scripts in the base layout and havng provided design 
input to all 3 of the base, ubi, and mmc layouts I'm happy to 
work on migrating the base layout to also transition from full 
filesystem overlay to the direct mount of var with etc overlay 
that exists on the other two layouts.

However, I don't have the access needed to test and regress the 
transition from the current layout.  I need the assistance of 
someone that is using the current layout and willing to test and 
provide feedback on the transition.

[ Openbmc developent is not my primary work and I don't have 
access to a system using the static layout that I can get 
reflashed for recovery ]


Once this is done we can work as a community to seperate out the 
overload of defaults and configuration that is in etc, probably 
by a combination of moving openbmc content to /var/lib, and 
perhaps by making /etc distributed empty via the system empty 
init support (where /etc would be a plain writable filesystem 
of pure configuration vs distribution defaults).

Only after that can overlayfs be removed from the kernel.

>Yup, as far as I remember, the "ubi layout" distro feature only mount
>specific directories instead of root.
>Checking the code, it enables the `read-only-rootfs`
>IMAGE_FEATURES[1], and use a different init script to mount only /etc
>by `preinit-mounts.bb`[2]
>The same for `phosphor-mmc` as well.
>
>@anoo should know this well :)
>

And I also know it, having been involved in all three layouts.

>[1]:
>https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openb
>mc_openbmc_blob_master_meta-2Dphosphor_recipes-2Dphosphor_images_obmc
>-2Dphosphor-2Dimage.bb-23L35&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=bvv7
>AJEECoRKBU02rcu4F5DWd-EwX8As2xrXeO9ZSo4&m=Lykl2abBxWlXUeD9IOsaSRujrlt
>BLI3LARBleKpfHMA&s=R_DHDXjMbd3D6V1ycREvdpSYQpPPGmYQdRctW3JRnHU&e= 
>[2]:
>https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openb
>mc_openbmc_blob_master_meta-2Dphosphor_recipes-2Dphosphor_preinit-2Dm
>ounts_preinit-2Dmounts_init&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=bvv7A
>JEECoRKBU02rcu4F5DWd-EwX8As2xrXeO9ZSo4&m=Lykl2abBxWlXUeD9IOsaSRujrltB
>LI3LARBleKpfHMA&s=DSsCadWHqoLFHZ2JIx0c6psN1joBzjxI-je9q6is13I&e= 
>
>-- 
>BRs,
>Lei YU


milton

Re: overlayFS security concern - threat model

[Kun Zhao]

Thanks, Joseph. Seems we have several options to solve the problem.

Kun

On 2/22/2021 9:36 AM, Joseph Reynolds wrote:
> On 2/19/21 6:31 PM, Kun Zhao wrote:
>> Hi Team, Have the following case ever been discussed before?,...
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> Hi Team,
>>
>> Have the following case ever been discussed before?,
>>
>> Anyone knows the root password will be able to let bmc run their own 
>> code by scp the code into bmc with the same file path as any services 
>> in rootfs. It will make the secure boot totally useless.
>>
>> So besides,
>>
>> 1. disable scp (but scp is one of the firmware upload way)
>>
>> 2. don’t use overlayFS (but it’s really useful for debugging during 
>> develop, and configuration management)
>>
>> Any other solutions?
>>
>
> Kun,
>
> Thanks for starting this thread.  Good discussion everyone!  Note 
> OpenBMC's SSH interfaces are described in 
> [network-security-considerations][].
>
> I would like to improve the project's threat model to cover these 
> ideas and so propose a new document, threat-model-considerations.md, 
> to go under [OpenBMC security docs][] with topics:
>
> - OpenBMC has a well-known password.
>     - Explanation: OpenBMC has a well-known root password.  There is 
> no mechanism which forces anyone to change it.
>     - Risk: OpenBMC's default configuration produce firmware which has 
> a well-known password which allows an attacker to gain full access to 
> BMC internals.
>     - Recommendation: Three independent recommendations:
>         1. Provide a [configuration guide][] for firmware builders 
> which instructs them to change the default password for their distro, 
> and similar for BMC administrators which instructs them to change the 
> default password when they first access the BMC.
>         2. Implement an image option to cause the BMC's initial 
> password to be expired so it must be changed before access is granted.
>         3. Move away from password-based authentication.
>     - Caveat: The expired password solution leaves a time window when 
> the BMC is available to attackers and has well known password.
>
> - Root account login to the BMC's command shell.
>     - Explanation: Root access to the BMC's command shell is required 
> for developers and service personnel to do their job. (They need to 
> access the BMC internal tools and interfaces.)  But root access is not 
> needed for any administrators or users performing normal BMC functions.
>     - Risk: Users will have more authority than they need to do their 
> job.  (The BMC should be designed so the owner can give each user the 
> authority they nee, but no more.)  Additional authority may allow a 
> user or attacker to damage BMC internals or gain information about the 
> host they are not intended to have.
>     - Recommendation: Disable root access in production images. 
> (Should we have an OpenBMC image feature to control this?)  If root 
> access is no disabled, it must be strictly limited to authorized users.
>
> - SSH access to the BMC (including scp firmware update).
>     - Explanation: The BMC provides an SSH server to allow secure 
> shell (ssh) sessions to the BMC's command shell, ssh sessions to the 
> host console, and scp-based firmware update.
>     - Risk: See the "Root account login to the BMC's command shell" 
> topic.  Note that ssh access to the host console is intended.
>     - Recommendation: As above, disable SSH access for production 
> images.  Disable SCP and instead use Redfish firmware update.
>
> - Running unintended code on the BMC.
>     - Explanation: The BMC is intended as a closed appliance but runs 
> Linux.
>     - Risk: The BMC's Linux capabilities allow an attacker who gains 
> access to the BMC to attack the BMC's network and attack the BMC's host.
>     - Recommendation: Use technologies like SecureBoot and Integrity 
> Measurement Architecture (IMA).
>     - Caveats: Powerful scripting tools such as the command shell (sh 
> or bash) would be allowed by IMA to run on the BMC but still give an 
> attacker the ability to run harmful code.
>
> - Having writable portions of the BMC's file system
>     - Explanation: The BMC is intended as a closed appliance with a 
> read-only file system but has a writable portions of the file system 
> to store BMC admin-configurable items, firmware updates, internal and 
> host logs, etc.
>     - Risk: The BMC's writable file system allows an attacker who 
> gains access to the BMC to make their access permanent by writing to 
> the BMC's file system.
>     - Recommendation: Strictly limit which portions of the BMC's file 
> system are writable.  For example, to /var, /run, and a strict subset 
> of /etc.  Use a non-persistent overlay for data that does not need to 
> persist beyond a BMC reboot.
>
> I've added this topic to the [OpenBMC security working group 
> agenda][].  But please keep the discussion going in the email list.
>
> Thanks!
> -Joseph
>
> [network-security-considerations]: 
> https://github.com/openbmc/docs/blob/master/security/network-security-considerations.md
> [OpenBMC security docs]: 
> https://github.com/openbmc/docs/blob/master/security
> [configuration guide]: 
> https://github.com/openbmc/openbmc/wiki/Configuration-guide
> [OpenBMC security working group agenda]: 
> https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI 
>
>
>>
>> Thanks.
>>
>> Kun
>>
>

Re: overlayFS security concern - threat model

[Joseph Reynolds]

On 3/3/21 11:55 AM, Kun Zhao wrote:
> Thanks, Joseph. Seems we have several options to solve the problem.

I wrote an issue to address the /etc readwrite overlayfs issue.
https://github.com/openbmc/openbmc/issues/3766

- joseph

>
> Kun
>
...snip...