LXC+overlayfs in unprivileged mode

LXC+overlayfs in unprivileged mode

[Linas Vepstas]

Hi,

I tripped across an LXC bug that actually appears to be an overlayfs
security feature (maybe) and was wondering how to clarify the status
of the code. Apparently, Ubuntu is carrying patches that enable this
function, and so the question is if these or something more appropriate
can be pulled into the mainline kernel or into overlayfs (or perhaps they
have been already; the situation is confusing).

The issue is that LXC+overlayfs seems to not work in an unprivileged
container. A more detailed description, with a particularly simple test
case is given in https://github.com/lxc/lxc/issues/1370#issuecomment269845311
Based on searches through google, it seems likely that the reason it
does not work is due to one or more privilege-escalation exploits,
except that these may or may not be patched already... thus this email.

Any advice on how to proceed?

--linas

Re: LXC+overlayfs in unprivileged mode

[Amir Goldstein]

On Sat, Dec 31, 2016 at 7:42 PM, Linas Vepstas <linasvepstas@gmail.com> wrote:
> Hi,
>
> I tripped across an LXC bug that actually appears to be an overlayfs
> security feature (maybe) and was wondering how to clarify the status
> of the code. Apparently, Ubuntu is carrying patches that enable this
> function, and so the question is if these or something more appropriate
> can be pulled into the mainline kernel or into overlayfs (or perhaps they
> have been already; the situation is confusing).
>

The foundations for mount from unprivileged user namespace were merged
to kernel v4.8, you can read more about it in Eric's pull request:
https://lkml.org/lkml/2016/7/26/297

Seth Forshee was working on these vfs changes to allow fuse mount in
unprivileged user namespace:
https://lwn.net/Articles/685239/

Not sure about the status of the work on fuse mount?

> The issue is that LXC+overlayfs seems to not work in an unprivileged
> container. A more detailed description, with a particularly simple test
> case is given in https://github.com/lxc/lxc/issues/1370#issuecomment269845311
> Based on searches through google, it seems likely that the reason it
> does not work is due to one or more privilege-escalation exploits,
> except that these may or may not be patched already... thus this email.
>
> Any advice on how to proceed?

Not sure what you want to achieve?
If you just want to enable overlayfs mount in unprivileged container on
your own kernel at your own risk (i.e. of exposure to exploits), you should
probably use latest kernel and carry this (untested!) patch:

static struct file_system_type ovl_fs_type = {
        .owner          = THIS_MODULE,
        .name           = "overlay",
+       .fs_flags       = FS_USERNS_MOUNT,
        .mount          = ovl_mount,
        .kill_sb        = kill_anon_super,
};

If you want to get this patch up-streamed, you "only" need to prove
there are no more privilege-escalation exploits in overlayfs, like the one
you referenced above...

Because it is going be quite hard to provide that proof, it is much more
likely to see this patch in a distribution (e.g. Ubuntu) that is willing to
compromise security for a feature, then to see this feature enabled upstream.

On the bright side, sanitizing overlayfs for unprivileged container mount
is probably a much closer goal then sanitizing a blockdev filesystem for
unprivileged container (loop) mount.

I hope my brief was accurate. Seth/Eric/Miklos, please correct me if it wasn't.

Amir.

Re: LXC+overlayfs in unprivileged mode

[Linas Vepstas]

Ah, thanks!

On Sun, Jan 1, 2017 at 2:51 AM, Amir Goldstein <amir73il@gmail.com> wrote:
> On Sat, Dec 31, 2016 at 7:42 PM, Linas Vepstas <linasvepstas@gmail.com> wrote:

>> I tripped across an LXC bug that actually appears to be an overlayfs
...
>> the situation is confusing.
>>
>
> The foundations for mount from unprivileged user namespace were merged
> to kernel v4.8, you can read more about it in Eric's pull request:
> https://lkml.org/lkml/2016/7/26/297
>
> Seth Forshee was working on these vfs changes to allow fuse mount in
> unprivileged user namespace:
> https://lwn.net/Articles/685239/
>
> Not sure about the status of the work on fuse mount?
>>
>> Any advice on how to proceed?
>
> Not sure what you want to achieve?

I wanted to understand why things half-work, and figure out where to
report a bug, if that's what needed to be done.   But based on what
you wrote, I conclude that its "work in progress", and from a user
perspective, I can kick back and wait.  I guess some LXC wiki
somewhere should be updated to state that user-space containers
require kernel version xyz or patch pqr.

It's somehow ironic that the push for user-space mounts and containers
comes from this general fuzzy sensation that they are somehow "safer",
yet the changes to enable this provide a new attack surface for
privilege escalation. Funny world we live in. :-)  Happy New Year!

--linas

Re: LXC+overlayfs in unprivileged mode

[Vivek Goyal]

On Sun, Jan 01, 2017 at 02:32:20PM -0600, Linas Vepstas wrote:

[..]
> It's somehow ironic that the push for user-space mounts and containers
> comes from this general fuzzy sensation that they are somehow "safer",
> yet the changes to enable this provide a new attack surface for
> privilege escalation. Funny world we live in. :-)  Happy New Year!

Only if unprivileged users want to be able to mount overlayfs. Otherwise, a
privileged user can just mount overlayfs on host and bind mount that
inside container (this is what docker does). And then you don't have
to worry about allowing unprivileged users to be able to allow mounting.

Vivek

Re: LXC+overlayfs in unprivileged mode

[Linas Vepstas]

On Tue, Jan 3, 2017 at 7:48 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> On Sun, Jan 01, 2017 at 02:32:20PM -0600, Linas Vepstas wrote:
>
> [..]
>> It's somehow ironic that the push for user-space mounts and containers
>> comes from this general fuzzy sensation that they are somehow "safer",
>> yet the changes to enable this provide a new attack surface for
>> privilege escalation. Funny world we live in. :-)  Happy New Year!
>
> Only if unprivileged users want to be able to mount overlayfs. Otherwise, a
> privileged user can just mount overlayfs on host and bind mount that
> inside container (this is what docker does). And then you don't have
> to worry about allowing unprivileged users to be able to allow mounting.

:-(   The way that Ubuntu solves this is to carry patches to allow user-space
mounts.  Debian doesn't, which is how I tripped across this.  Anyway, Docker
and LXC are very different beasts: Docker makes for great demos, and
can get the occasional newbie going, but is kind of klunky and awkward
in real-life deployments.  It certainly fails to provide the ease-of-use and
flexibility that LXC offers.   (Docker tries to solve two unrelated problems,
and it handles both of them poorly: one problem is containerization, the
other problem is container build. LXC solves the first problem much more
elegantly, and completely ignores the second problem, which, in general,
is easily solved with shell scripts, so what was the point of Docker
reinventing a new kind of shell, badly?)

-- linas

Re: LXC+overlayfs in unprivileged mode

[Eric W. Biederman]

Vivek Goyal <vgoyal@redhat.com> writes:

> On Sun, Jan 01, 2017 at 02:32:20PM -0600, Linas Vepstas wrote:
>
> [..]
>> It's somehow ironic that the push for user-space mounts and containers
>> comes from this general fuzzy sensation that they are somehow "safer",
>> yet the changes to enable this provide a new attack surface for
>> privilege escalation. Funny world we live in. :-)  Happy New Year!
>
> Only if unprivileged users want to be able to mount overlayfs. Otherwise, a
> privileged user can just mount overlayfs on host and bind mount that
> inside container (this is what docker does). And then you don't have
> to worry about allowing unprivileged users to be able to allow
> mounting.

Which is a blame shifting game and we should not be playing.

A kernel feature useful for ordinary users for ordinary workloads
should not require special permissions to use.

Kernel features are built to be used.  User space programs will use
them.  If those features can not be used safely and we blame that on
userspace we are just not accepting the responsibility of making them
work ourselves.  A really jerk move.

In a change in what permissions are needed to use a kernel feature we
need to make the kernel change in small responsible steps.  Many kernel
features are not coded defensively so before we enable them for ordinary
users we need to do our best to ensure they are safe to use and ensure
their maintainers are on board with what is going on.

I can't see any policy that only allows root to mount things as healthy
or maintainable in the long term.  We have long exceeded the point where
root is the only user that wants to mount things.  So the best long term
strategy is to build a sufficient policy in the kernel that does not
require mediation by userspace.


Furthermore docker that does not need root privileges is currently being
debugged.  So we are long past any point where it even makes sense
to consider a policy of only allowing root to mount things.  That ship
has sailed.

Eric

Re: LXC+overlayfs in unprivileged mode

[Vivek Goyal]

On Tue, Jan 03, 2017 at 10:08:25AM -0600, Linas Vepstas wrote:
> On Tue, Jan 3, 2017 at 7:48 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> > On Sun, Jan 01, 2017 at 02:32:20PM -0600, Linas Vepstas wrote:
> >
> > [..]
> >> It's somehow ironic that the push for user-space mounts and containers
> >> comes from this general fuzzy sensation that they are somehow "safer",
> >> yet the changes to enable this provide a new attack surface for
> >> privilege escalation. Funny world we live in. :-)  Happy New Year!
> >
> > Only if unprivileged users want to be able to mount overlayfs. Otherwise, a
> > privileged user can just mount overlayfs on host and bind mount that
> > inside container (this is what docker does). And then you don't have
> > to worry about allowing unprivileged users to be able to allow mounting.
> 
> :-(   The way that Ubuntu solves this is to carry patches to allow user-space
> mounts.  Debian doesn't, which is how I tripped across this.  Anyway, Docker
> and LXC are very different beasts: Docker makes for great demos, and
> can get the occasional newbie going, but is kind of klunky and awkward
> in real-life deployments.  It certainly fails to provide the ease-of-use and
> flexibility that LXC offers.   (Docker tries to solve two unrelated problems,
> and it handles both of them poorly: one problem is containerization, the
> other problem is container build. LXC solves the first problem much more
> elegantly, and completely ignores the second problem, which, in general,
> is easily solved with shell scripts, so what was the point of Docker
> reinventing a new kind of shell, badly?)

I will not go into comparing LXC and Docker. For me, I do think that they
handled the ease of use case very well. I just had to run two commands
to get a container running.

- yum install docker
- docker run -ti fedora bash

I think LXC vs Docker conversation is besides the point for this thread.

Vivek