[Virtio-fs] [RFC] About non-root virtiofsd(1) process

[Virtio-fs] [RFC] About non-root virtiofsd(1) process

[P J P]

   Hello,

* Recently I realised that virtiofsd(1) process does not drop its 'root'
   privileges while sharing host directory tree with a guest VM.

   Libvirtd(8) generally starts a guest VM with non-root system user (ex. qemu)
   privileges. If virtiofsd(1) has 'root' privileges, that makes it an
   accomplice in a potential guest-to-host privilege escalation scenario. Which
   is not good.

* IMHO, ideally virtiofsd(1) should not run with 'root' privileges at all.

* But If it has to, then atleast all default configuration settings must be
   utmost strict and restrictive as possible. Ex. By default offer only read
   access to guest VM.

* Another option is for root virtiofsd(1) process to fork a sub-process which
   will run with non-root (ex. qemu) system user privileges.

    - All file I/O operations for sharing a host directory with a guest are
      performed by the sub-process with non-root system user privileges.

    - Sub-process shall talk to the parent virtiofsd(1) process only when
      privileged operation/assistance is required.

   Ex. https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/

...wdyt?

Thank you.
--
Prasad J Pandit / Red Hat Product Security Team
8685 545E B54C 486B C6EB 271E E285 8B5A F050 DE8D

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[Dr. David Alan Gilbert]

* P J P (ppandit@redhat.com) wrote:
>   Hello,
> 
> * Recently I realised that virtiofsd(1) process does not drop its 'root'
>   privileges while sharing host directory tree with a guest VM.

Correct.

>   Libvirtd(8) generally starts a guest VM with non-root system user (ex. qemu)
>   privileges. If virtiofsd(1) has 'root' privileges, that makes it an
>   accomplice in a potential guest-to-host privilege escalation scenario. Which
>   is not good.
> 
> * IMHO, ideally virtiofsd(1) should not run with 'root' privileges at all.
> 
> * But If it has to, then atleast all default configuration settings must be
>   utmost strict and restrictive as possible. Ex. By default offer only read
>   access to guest VM.
> 
> * Another option is for root virtiofsd(1) process to fork a sub-process which
>   will run with non-root (ex. qemu) system user privileges.
> 
>    - All file I/O operations for sharing a host directory with a guest are
>      performed by the sub-process with non-root system user privileges.
> 
>    - Sub-process shall talk to the parent virtiofsd(1) process only when
>      privileged operation/assistance is required.
> 
>   Ex. https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/
> 
> ...wdyt?

virtiofsd does a lot to sandbox itself after startup; and it has to be
able to provide access to a filesystem that on the host might want to
have files with root ownership, and xattr's and the like - i.e. to allow
the guest to do rpm installs for example.

The intent is that whoever starts virtiofsd passes it a directory
to be used only by the guest or that has appropriate permissions for the
guest to access.

The default sandboxing gives the virtiofsd it's own mount, pid and net
namespaces; so hopefully it can't escape to any other filetree other
than the one it's explicitly been told to give to the guest.
(That's -o sandbox=namespace which is the default)

It's seccomp'd to disallow as many syscalls as possible.

It also drops a lot of capabilities; although it is left
with a bunch of powerful ones (e.g. CAP_DAC_OVERRIDE) - but
you can also reduce those with the use of the -o modcaps= option.

Dave

> 
> Thank you.
> --
> Prasad J Pandit / Red Hat Product Security Team
> 8685 545E B54C 486B C6EB 271E E285 8B5A F050 DE8D
> 
> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://www.redhat.com/mailman/listinfo/virtio-fs
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[P J P]

+-- On Thu, 14 Jan 2021, Dr. David Alan Gilbert wrote --+
| virtiofsd does a lot to sandbox itself after startup; and it has to be able 
| to provide access to a filesystem that on the host might want to have files 
| with root ownership, and xattr's and the like - i.e. to allow the guest to 
| do rpm installs for example.

  ie. For guest to install RPMs under the shared directory?
 
| The intent is that whoever starts virtiofsd passes it a directory to be used 
| only by the guest or that has appropriate permissions for the guest to 
| access.
| 
| The default sandboxing gives the virtiofsd it's own mount, pid and net 
| namespaces; so hopefully it can't escape to any other filetree other than 
| the one it's explicitly been told to give to the guest. (That's -o 
| sandbox=namespace which is the default)
| 
| It's seccomp'd to disallow as many syscalls as possible.
| 
| It also drops a lot of capabilities; although it is left with a bunch of 
| powerful ones (e.g. CAP_DAC_OVERRIDE) - but you can also reduce those with 
| the use of the -o modcaps= option.

* True; But with these numerous options/parameters, it is possible to miss one 
  while starting virtiofsd(1) daemon, which may prove fatal.

  Hence having the most restrictive default values for them is better.

* Maybe having a command line switch similar to 'qemu -runas <user>' would be 
  helpful?

     $ ./virtiofsd -runas test -o source=...

  A user who wants to run virtiofsd(1) with non-root privileges, can do so.
 

Thank you.
--
Prasad J Pandit / Red Hat Product Security Team
8685 545E B54C 486B C6EB 271E E285 8B5A F050 DE8D

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2241 bytes --]

On Thu, Jan 14, 2021 at 02:11:28PM +0530, P J P wrote:
>   Hello,
> 
> * Recently I realised that virtiofsd(1) process does not drop its 'root'
>   privileges while sharing host directory tree with a guest VM.
> 
>   Libvirtd(8) generally starts a guest VM with non-root system user (ex. qemu)
>   privileges. If virtiofsd(1) has 'root' privileges, that makes it an
>   accomplice in a potential guest-to-host privilege escalation scenario. Which
>   is not good.
> 
> * IMHO, ideally virtiofsd(1) should not run with 'root' privileges at all.
> 
> * But If it has to, then atleast all default configuration settings must be
>   utmost strict and restrictive as possible. Ex. By default offer only read
>   access to guest VM.

That's not useful. Most users require read-write.

> * Another option is for root virtiofsd(1) process to fork a sub-process which
>   will run with non-root (ex. qemu) system user privileges.
> 
>    - All file I/O operations for sharing a host directory with a guest are
>      performed by the sub-process with non-root system user privileges.
> 
>    - Sub-process shall talk to the parent virtiofsd(1) process only when
>      privileged operation/assistance is required.
> 
>   Ex. https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/
> 
> ...wdyt?

Please take a look at the current sandboxing implementation in
virtiofsd. It greatly restricts the process. virtiofsd is not just a
process running freely with uid root.

Unlike nginx, virtiofsd needs to perform privileged operations as part
of serving requests. The fundamental issue is that the server must be
able to create, access, and modify files with arbitrary uids/gids.

If you have specific ideas, let's discuss them. We have been following
user namespace and uid/gid mapping/shifting features. They are not
widely available in Linux yet, but once they are it should be possible
to run with root only inside the user namespace (i.e. actually
unprivileged).

There is also an unfinished security document here that we could discuss
further and publish:
https://gitlab.com/virtio-fs/qemu/-/merge_requests/6/diffs?commit_id=718c71fa44f6b92ac27558c903d27935236b08ef

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[P J P]

  Hello,

+-- On Thu, 14 Jan 2021, Stefan Hajnoczi wrote --+
| On Thu, Jan 14, 2021 at 02:11:28PM +0530, P J P wrote:
| > Ex. By default offer only read access to guest VM.
| 
| That's not useful. Most users require read-write.

* Agreed. I meant let 'rw' access be user's choice than the default for 
  virtiofsd(1).
 
| The fundamental issue is that the server must be able to create, access, and 
| modify files with arbitrary uids/gids.

* Why arbitrary uids/gids? Once a directory is shared with a guest, its 
  uids/gids would stay same, no?

* We also start separate virtiofsd(1) process for each share/guest too. ie.  
  single virtiofsd(1) daemon is not catering to multiple guests and their
  respective shared directories, right?

| If you have specific ideas, let's discuss them.

* One was to have a command line switch similar to 'qemu -runas <user>'

     $ ./virtiofsd -runas test -o source=...

  If a user wants to run virtiofsd(1) with non-root privileges, it'll be 
  handy.

| https://gitlab.com/virtio-fs/qemu/-/merge_requests/6/diffs?commit_id=718c71fa44f6b92ac27558c903d27935236b08ef

* I'll go through it.


Thank you.
--
Prasad J Pandit / Red Hat Product Security Team
8685 545E B54C 486B C6EB 271E E285 8B5A F050 DE8D

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2412 bytes --]

On Fri, Jan 15, 2021 at 12:39:13PM +0530, P J P wrote:
> +-- On Thu, 14 Jan 2021, Stefan Hajnoczi wrote --+
> | On Thu, Jan 14, 2021 at 02:11:28PM +0530, P J P wrote:
> | > Ex. By default offer only read access to guest VM.
> | 
> | That's not useful. Most users require read-write.
> 
> * Agreed. I meant let 'rw' access be user's choice than the default for 
>   virtiofsd(1).

I'm not sure I understand. It's the user's choice to run virtiofsd. And
if they are running it then most of the time they want read-write
access.

Requiring extra syntax for read-write doesn't help. Safe defaults work
well for rarely-used features that can really be disabled most of the
time to improve security. This isn't that case.

> | The fundamental issue is that the server must be able to create, access, and 
> | modify files with arbitrary uids/gids.
> 
> * Why arbitrary uids/gids? Once a directory is shared with a guest, its 
>   uids/gids would stay same, no?

Guest applications may run with different uids/gids. The host has no
control over that.

Imagine booting a guest form a virtio-fs root file system and installing
packages. The guest must be able to control uids/gids for that to work.

There are specific use cases where restrictions are acceptable, but they
have trade-offs (e.g. the file system on the host does not have the same
uid/gids as inside the guest). The current approach supports all guest
applications, whereas other approaches do not. It is possible to add
uid/gid restrictions to tighten security for some use cases (like the
unprivileged virtiofsd that you suggested below with -runas).

> * We also start separate virtiofsd(1) process for each share/guest too. ie.  
>   single virtiofsd(1) daemon is not catering to multiple guests and their
>   respective shared directories, right?

Yes. Each virtio-fs device has a separate virtiofsd process.

> | If you have specific ideas, let's discuss them.
> 
> * One was to have a command line switch similar to 'qemu -runas <user>'
> 
>      $ ./virtiofsd -runas test -o source=...
> 
>   If a user wants to run virtiofsd(1) with non-root privileges, it'll be 
>   handy.

Patches for this are welcome. It can't be used for Kata Containers or
other virtiofsd use cases that where uids/gids matter, but it's a good
choice to share an unprivileged user's directory with the guest.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[P J P]

+-- On Mon, 18 Jan 2021, Stefan Hajnoczi wrote --+
| Guest applications may run with different uids/gids. The host has no control 
| over that.
| 
| Imagine booting a guest form a virtio-fs root file system and installing 
| packages. The guest must be able to control uids/gids for that to work.

* I see; I'll try to better understand how it's done.

* With UID namespaces, I thought virtiofsd(1) would be able to operate files 
  with arbitrary uid/gid, even after dropping its root privileges to acquire 
  non-root privileges on the host; Because it has 'root' privileges under the 
  shared directory & UID namespace.

| > $ ./virtiofsd -runas test -o source=...
| 
| Patches for this are welcome.

* Okay, will try.

Thank you.
--
Prasad J Pandit / Red Hat Product Security Team
8685 545E B54C 486B C6EB 271E E285 8B5A F050 DE8D

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[Chirantan Ekbote]

On Tue, Jan 19, 2021 at 11:34 PM P J P <ppandit@redhat.com> wrote:
>
> +-- On Mon, 18 Jan 2021, Stefan Hajnoczi wrote --+
> | Guest applications may run with different uids/gids. The host has no control
> | over that.
> |
> | Imagine booting a guest form a virtio-fs root file system and installing
> | packages. The guest must be able to control uids/gids for that to work.
>
> * I see; I'll try to better understand how it's done.
>
> * With UID namespaces, I thought virtiofsd(1) would be able to operate files
>   with arbitrary uid/gid, even after dropping its root privileges to acquire
>   non-root privileges on the host; Because it has 'root' privileges under the
>   shared directory & UID namespace.
>

This is actually what we do in crosvm.  It works pretty well but has a
few limitations:

- Setting up the uidmap and gidmap requires CAP_SETUID and CAP_SETGID
(unless you're only mapping in the current euid/egid) so a user
without sudo access cannot do it.
- The daemon cannot do anything that requires caps in the init
namespace.  For example, it cannot set any security.* or trusted.*
xattrs but that can be worked around by re-mapping them into the
user.* xattr namespace.  Another example is anything to do with
filesystem quotas.  We deal with this by forwarding them to another
daemon running in the init namespace but it's not ideal.

Overall I think we're pretty happy with the user namespace sandbox.  I
think the benefit of dropping all privileges in the init namespace has
been worth the cost of dealing with the limitations we've run into so
far.

Chirantan

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

On Tue, Jan 19, 2021 at 08:04:29PM +0530, P J P wrote:
> +-- On Mon, 18 Jan 2021, Stefan Hajnoczi wrote --+
> | Guest applications may run with different uids/gids. The host has no control 
> | over that.
> | 
> | Imagine booting a guest form a virtio-fs root file system and installing 
> | packages. The guest must be able to control uids/gids for that to work.
> 
> * I see; I'll try to better understand how it's done.
> 
> * With UID namespaces, I thought virtiofsd(1) would be able to operate files 
>   with arbitrary uid/gid, even after dropping its root privileges to acquire 
>   non-root privileges on the host; Because it has 'root' privileges under the 
>   shared directory & UID namespace.

Excatly, this is the most promising direction but it has a limitation:
the uid/gid on the host file system are in the uid/gid range that was
assigned to the user namespace. If anything other than the VM needs to
access those files then you need to either ensure it runs with the
currect uid/gid on the host or chown the files. That complicates some
use cases. It does seem worth offering as a feature but only works for
some use cases.

When Linux gets uid/gid mapping functionality this issue will be solved.
virtiofsd will run unprivileged but the host files can have the correct
uid/gids.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

On Wed, Jan 20, 2021 at 02:49:52PM +0900, Chirantan Ekbote wrote:
> On Tue, Jan 19, 2021 at 11:34 PM P J P <ppandit@redhat.com> wrote:
> >
> > +-- On Mon, 18 Jan 2021, Stefan Hajnoczi wrote --+
> > | Guest applications may run with different uids/gids. The host has no control
> > | over that.
> > |
> > | Imagine booting a guest form a virtio-fs root file system and installing
> > | packages. The guest must be able to control uids/gids for that to work.
> >
> > * I see; I'll try to better understand how it's done.
> >
> > * With UID namespaces, I thought virtiofsd(1) would be able to operate files
> >   with arbitrary uid/gid, even after dropping its root privileges to acquire
> >   non-root privileges on the host; Because it has 'root' privileges under the
> >   shared directory & UID namespace.
> >
> 
> This is actually what we do in crosvm.  It works pretty well but has a
> few limitations:
> 
> - Setting up the uidmap and gidmap requires CAP_SETUID and CAP_SETGID
> (unless you're only mapping in the current euid/egid) so a user
> without sudo access cannot do it.
> - The daemon cannot do anything that requires caps in the init
> namespace.  For example, it cannot set any security.* or trusted.*
> xattrs but that can be worked around by re-mapping them into the
> user.* xattr namespace.  Another example is anything to do with
> filesystem quotas.  We deal with this by forwarding them to another
> daemon running in the init namespace but it's not ideal.
> 
> Overall I think we're pretty happy with the user namespace sandbox.  I
> think the benefit of dropping all privileges in the init namespace has
> been worth the cost of dealing with the limitations we've run into so
> far.

Cool, thanks for sharing those details!

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[Vivek Goyal]

On Tue, Jan 19, 2021 at 08:04:29PM +0530, P J P wrote:
> +-- On Mon, 18 Jan 2021, Stefan Hajnoczi wrote --+
> | Guest applications may run with different uids/gids. The host has no control 
> | over that.
> | 
> | Imagine booting a guest form a virtio-fs root file system and installing 
> | packages. The guest must be able to control uids/gids for that to work.
> 
> * I see; I'll try to better understand how it's done.
> 
> * With UID namespaces, I thought virtiofsd(1) would be able to operate files 
>   with arbitrary uid/gid, even after dropping its root privileges to acquire 
>   non-root privileges on the host; Because it has 'root' privileges under the 
>   shared directory & UID namespace.
> 
> | > $ ./virtiofsd -runas test -o source=...
> | 
> | Patches for this are welcome.
> 
> * Okay, will try.

Catching up with this thread now.

I had posted minimal patches to allow running virtiofsd unpriviliged.
They did not make further progress though.

https://patchew.org/QEMU/20200730194736.173994-1-vgoyal@redhat.com/

While being able to run virtiofsd in a user namespace is certainly
valuable, I feel being able to run virtiofsd unpriviliged has it
use cases as well. For example, if a user wants to share just its
home directory on host with guest. In that case, we probably don't
require lot of priviliged operations to be performed by virtiofsd.

Vivek

Re: [Virtio-fs] [RFC] About non-root virtiofsd(1) process

[Vivek Goyal]

On Wed, Jan 20, 2021 at 02:49:52PM +0900, Chirantan Ekbote wrote:
> On Tue, Jan 19, 2021 at 11:34 PM P J P <ppandit@redhat.com> wrote:
> >
> > +-- On Mon, 18 Jan 2021, Stefan Hajnoczi wrote --+
> > | Guest applications may run with different uids/gids. The host has no control
> > | over that.
> > |
> > | Imagine booting a guest form a virtio-fs root file system and installing
> > | packages. The guest must be able to control uids/gids for that to work.
> >
> > * I see; I'll try to better understand how it's done.
> >
> > * With UID namespaces, I thought virtiofsd(1) would be able to operate files
> >   with arbitrary uid/gid, even after dropping its root privileges to acquire
> >   non-root privileges on the host; Because it has 'root' privileges under the
> >   shared directory & UID namespace.
> >
> 
> This is actually what we do in crosvm.  It works pretty well but has a
> few limitations:
> 
> - Setting up the uidmap and gidmap requires CAP_SETUID and CAP_SETGID
> (unless you're only mapping in the current euid/egid) so a user
> without sudo access cannot do it.
> - The daemon cannot do anything that requires caps in the init
> namespace.  For example, it cannot set any security.* or trusted.*
> xattrs but that can be worked around by re-mapping them into the
> user.* xattr namespace.  Another example is anything to do with
> filesystem quotas.  We deal with this by forwarding them to another
> daemon running in the init namespace but it's not ideal.

As of now probably "chown" is another limitation. One has to chown
all uid/gid to appropriate mapping. Also if directory is shared
between two VMs, then they have to be using same uid/gid map.

Vivek

> 
> Overall I think we're pretty happy with the user namespace sandbox.  I
> think the benefit of dropping all privileges in the init namespace has
> been worth the cost of dealing with the limitations we've run into so
> far.
> 
> Chirantan
> 
> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://www.redhat.com/mailman/listinfo/virtio-fs