Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Connor Kuehl]

On 5/21/21 11:59 AM, Link Dupont wrote:

Adding the virtio-fs mailing list.

> I am mounting a filesystem into a domain using the virtiofs driver.
> 
> <filesystem accessmode="passthrough" type="mount">
>       <source dir="/home"/>
>       <target dir="/home"/>
>       <driver type="virtiofs"/>
> </filesystem>
> 
> Both my host (Fedora 34) and guest (CentOS 8.4) are running with SELinux 
> enforcing. From my host, I can see that the SELinux context type is set to 
> user_home_dir_t.
> 
> $ ls -ldZ /home/link
> drwxr-xr-x. 61 link link system_u:object_r:user_home_dir_t:s0 8192 May 21 
> 12:41 /home/link
> 
>>From within the guest however, the volume is unlabeled_t
> 
> $ ls -lZd /home/link
> drwxr-xr-x. 61 link link system_u:object_r:unlabeled_t:s0 8192 May 21 12:53 /
> home/link
> 
> Is there a way to pass the SELinux context through to the guest? Or mount the 
> volume with the correct options to map SELinux contexts?
> 
> 

Hi,

I'm afraid I actually don't know that much about SELinux but I read
that it relies on using extended attributes in the file system to
accomplish its labeling.

Do you still experience this issue when you enable extended attribute
support[1] in virtiofsd? The example in the optional parameters snippet
enables extended attributes with the xattr='on' element.

Connor

[1] https://libvirt.org/kbase/virtiofs.html#optional-parameters

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Harry G. Coin]

On 6/2/21 3:55 PM, Connor Kuehl wrote:
> On 5/21/21 11:59 AM, Link Dupont wrote:
>
> Adding the virtio-fs mailing list.
>
>> I am mounting a filesystem into a domain using the virtiofs driver.
>>
>> <filesystem accessmode="passthrough" type="mount">
>>       <source dir="/home"/>
>>       <target dir="/home"/>
>>       <driver type="virtiofs"/>
>> </filesystem>
>>
>> Both my host (Fedora 34) and guest (CentOS 8.4) are running with SELinux 
>> enforcing. From my host, I can see that the SELinux context type is set to 
>> user_home_dir_t.
>>
>> $ ls -ldZ /home/link
>> drwxr-xr-x. 61 link link system_u:object_r:user_home_dir_t:s0 8192 May 21 
>> 12:41 /home/link
>>
>> >From within the guest however, the volume is unlabeled_t
>>
>> $ ls -lZd /home/link
>> drwxr-xr-x. 61 link link system_u:object_r:unlabeled_t:s0 8192 May 21 12:53 /
>> home/link
>>
>> Is there a way to pass the SELinux context through to the guest? Or mount the 
>> volume with the correct options to map SELinux contexts?
>>
>>
> Hi,
>
> I'm afraid I actually don't know that much about SELinux but I read
> that it relies on using extended attributes in the file system to
> accomplish its labeling.
>
> Do you still experience this issue when you enable extended attribute
> support[1] in virtiofsd? The example in the optional parameters snippet
> enables extended attributes with the xattr='on' element.
>
> Connor
>
> [1] https://libvirt.org/kbase/virtiofs.html#optional-parameters


Take a look at this thread. There you will find commands that will allow
what you want until they become standard.

https://listman.redhat.com/archives/virtio-fs/2020-November/msg00110.html

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Link Dupont]

Adding the <binary xattr='on'> element to the <filesystem> device does 
seem to spawn virtiofsd with the option string "source=/home,xattr". My 
guest can no longer mount the device though.

It errors with:

[ 170.225553] 9pnet_virtio: no channels available
mount: mount(2) failed: No such file or directory

I think what this is doing is causing libvirt to create the device as a 
virtiofs device instead of a 9p device. The EL7 kernel doesn't have a 
virtiofs driver, so it can't mount virtiofs devices.

My knowledge is unfortunately limited about the nuances between 9p and 
virtiofs. So I'm mostly experimenting by trial-and-error here.

On Wed, Jun 2 2021 at 03:55:40 PM -0500, Connor Kuehl 
<ckuehl@redhat.com> wrote:
> On 5/21/21 11:59 AM, Link Dupont wrote:
> 
> Adding the virtio-fs mailing list.
> 
>>  I am mounting a filesystem into a domain using the virtiofs driver.
>> 
>>  <filesystem accessmode="passthrough" type="mount">
>>        <source dir="/home"/>
>>        <target dir="/home"/>
>>        <driver type="virtiofs"/>
>>  </filesystem>
>> 
>>  Both my host (Fedora 34) and guest (CentOS 8.4) are running with 
>> SELinux
>>  enforcing. From my host, I can see that the SELinux context type is 
>> set to
>>  user_home_dir_t.
>> 
>>  $ ls -ldZ /home/link
>>  drwxr-xr-x. 61 link link system_u:object_r:user_home_dir_t:s0 8192 
>> May 21
>>  12:41 /home/link
>> 
>>> From within the guest however, the volume is unlabeled_t
>> 
>>  $ ls -lZd /home/link
>>  drwxr-xr-x. 61 link link system_u:object_r:unlabeled_t:s0 8192 May 
>> 21 12:53 /
>>  home/link
>> 
>>  Is there a way to pass the SELinux context through to the guest? Or 
>> mount the
>>  volume with the correct options to map SELinux contexts?
>> 
>> 
> 
> Hi,
> 
> I'm afraid I actually don't know that much about SELinux but I read
> that it relies on using extended attributes in the file system to
> accomplish its labeling.
> 
> Do you still experience this issue when you enable extended attribute
> support[1] in virtiofsd? The example in the optional parameters 
> snippet
> enables extended attributes with the xattr='on' element.
> 
> Connor
> 
> [1] https://libvirt.org/kbase/virtiofs.html#optional-parameters
> 

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Dr. David Alan Gilbert]

* Link Dupont (link@sub-pop.net) wrote:
> Adding the <binary xattr='on'> element to the <filesystem> device does seem
> to spawn virtiofsd with the option string "source=/home,xattr". My guest can
> no longer mount the device though.
> 
> It errors with:
> 
> [ 170.225553] 9pnet_virtio: no channels available
> mount: mount(2) failed: No such file or directory
> 
> I think what this is doing is causing libvirt to create the device as a
> virtiofs device instead of a 9p device. The EL7 kernel doesn't have a
> virtiofs driver, so it can't mount virtiofs devices.
> 
> My knowledge is unfortunately limited about the nuances between 9p and
> virtiofs. So I'm mostly experimenting by trial-and-error here.

They're almost entirely different implementations; if you have a
virtiofsd then you're running virtiofs, not 9p, and yes RHEL7 won't like
that.

(I'm not sure el7 had 9p either??)

Dave


> On Wed, Jun 2 2021 at 03:55:40 PM -0500, Connor Kuehl <ckuehl@redhat.com>
> wrote:
> > On 5/21/21 11:59 AM, Link Dupont wrote:
> > 
> > Adding the virtio-fs mailing list.
> > 
> > >  I am mounting a filesystem into a domain using the virtiofs driver.
> > > 
> > >  <filesystem accessmode="passthrough" type="mount">
> > >        <source dir="/home"/>
> > >        <target dir="/home"/>
> > >        <driver type="virtiofs"/>
> > >  </filesystem>
> > > 
> > >  Both my host (Fedora 34) and guest (CentOS 8.4) are running with
> > > SELinux
> > >  enforcing. From my host, I can see that the SELinux context type is
> > > set to
> > >  user_home_dir_t.
> > > 
> > >  $ ls -ldZ /home/link
> > >  drwxr-xr-x. 61 link link system_u:object_r:user_home_dir_t:s0 8192
> > > May 21
> > >  12:41 /home/link
> > > 
> > > > From within the guest however, the volume is unlabeled_t
> > > 
> > >  $ ls -lZd /home/link
> > >  drwxr-xr-x. 61 link link system_u:object_r:unlabeled_t:s0 8192 May
> > > 21 12:53 /
> > >  home/link
> > > 
> > >  Is there a way to pass the SELinux context through to the guest? Or
> > > mount the
> > >  volume with the correct options to map SELinux contexts?
> > > 
> > > 
> > 
> > Hi,
> > 
> > I'm afraid I actually don't know that much about SELinux but I read
> > that it relies on using extended attributes in the file system to
> > accomplish its labeling.
> > 
> > Do you still experience this issue when you enable extended attribute
> > support[1] in virtiofsd? The example in the optional parameters snippet
> > enables extended attributes with the xattr='on' element.
> > 
> > Connor
> > 
> > [1] https://libvirt.org/kbase/virtiofs.html#optional-parameters
> > 
> 
> 
> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://listman.redhat.com/mailman/listinfo/virtio-fs
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Link Dupont]

On Thu, Jun 3 2021 at 08:24:02 PM +0100, Dr. David Alan Gilbert 
<dgilbert@redhat.com> wrote:
> (I'm not sure el7 had 9p either??)

Oh drat. I'm conflating my issues. I have an EL7 guest that is having a 
similar problem, except the guest sees the files as type nfs_t. I guess 
the symptoms are the same, but the solutions won't necessarily. Let me 
try and scale these down to reproducible scenarios.

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Link Dupont]

On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont <link@sub-pop.net> 
wrote:
>  reproducible scenarios

Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a 
virtiofs filesystem and with xattr on), the type of files in the 
mounted hierarchy are unlabeled_t. I can work around that by switching 
SELinux in the guest to permissive or disabled.

With a CentOS 7 guest, things get less usable. I digested this to a 
reproducible scenario.

Build a disk image with `virt-builder`, configuring the CentOS Plus 
kernel to get 9p support.

virt-builder centos-7.8 \
 --root-password password:centos \
 --output centos-7.8.qcow2 \
 --install yum-utils \
 --run-command 'yum-config-manager --enable centosplus' \
 --run-command 'sed -ie 
"s/DEFAULTKERNEL=kernel/DEFAULTKERNEL=kernel-plus/" 
/etc/sysconfig/kernel' \
 --append-line 
'/etc/dracut.conf.d/virtio.conf:add_drivers+="virtio_scsi virtio_pci 
virtio_console"' \
 --append-line '/etc/modules-load.d/9pnet_virtio.conf:9pnet_virtio' \
 --install kernel-plus \
 --append-line '/etc/fstab:home /home 9p trans=virtio,version=9p2000.L 
0 0'

Install the volume into the `default` pool.

sudo install -m644 centos-7.8.qcow2 /var/lib/libvirt/images

Next, define a domain using the disk image (using `virt-install` here 
for "easy mode").

virt-install \
 --import \
 --os-variant centos7.0 \
 --name centos \
 --ram 2048 \
 --disk path=/var/lib/libvirt/images/centos-7.8.qcow2 \
 --memorybacking access.mode=shared \
 --filesystem source=/home,target=home,accessmode=passthrough \
 --autoconsole none

Now with SELinux enforcing, I cannot list the contents of the 
directories in the mounted hierarchy.

[root@localhost ~]# ls -lZ /home/link
ls: cannot open directory /home/link: Permission denied

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Harry G. Coin]

On 6/3/21 9:14 PM, Link Dupont wrote:
> On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont
> <link@sub-pop.net> wrote:
>>  reproducible scenarios
>
> Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> virtiofs filesystem and with xattr on), the type of files in the
> mounted hierarchy are unlabeled_t. I can work around that by switching
> SELinux in the guest to permissive or disabled.
>
> With a CentOS 7 guest, things get less usable. I digested this to a
> reproducible scenario.
>
> Build a disk image with `virt-builder`, configuring the CentOS Plus
> kernel to get 9p support.
>
> virt-builder centos-7.8 \
> --root-password password:centos \
> --output centos-7.8.qcow2 \
> --install yum-utils \
> --run-command 'yum-config-manager --enable centosplus' \
> --run-command 'sed -ie
> "s/DEFAULTKERNEL=kernel/DEFAULTKERNEL=kernel-plus/"
> /etc/sysconfig/kernel' \
> --append-line
> '/etc/dracut.conf.d/virtio.conf:add_drivers+="virtio_scsi virtio_pci
> virtio_console"' \
> --append-line '/etc/modules-load.d/9pnet_virtio.conf:9pnet_virtio' \
> --install kernel-plus \
> --append-line '/etc/fstab:home /home 9p trans=virtio,version=9p2000.L
> 0 0'
>
> Install the volume into the `default` pool.
>
> sudo install -m644 centos-7.8.qcow2 /var/lib/libvirt/images
>
> Next, define a domain using the disk image (using `virt-install` here
> for "easy mode").
>
> virt-install \
> --import \
> --os-variant centos7.0 \
> --name centos \
> --ram 2048 \
> --disk path=/var/lib/libvirt/images/centos-7.8.qcow2 \
> --memorybacking access.mode=shared \
> --filesystem source=/home,target=home,accessmode=passthrough \
> --autoconsole none
>
> Now with SELinux enforcing, I cannot list the contents of the
> directories in the mounted hierarchy.
>
> [root@localhost ~]# ls -lZ /home/link
> ls: cannot open directory /home/link: Permission denied
>
>
Link, You have four choices right now:

1) Stick with rhel/centos/fedora type distros and ignore virtiofs for at
least 6 months, more like a year, for virtiofs to be implemented
properly by the typical GUI enabled virt-manager/qemu/kvm resources you
know about, and also not just wedge (worse than crash) your system by
breaking things like 'dnf' and 'rpm' randomly when upgrades happen --
blocking upgrades (and that's with 'permissive').   The big speed
advantage to virtiofs comes with a feature termed 'dax', and what it
takes to make that work is a whole lot of 'by hand' right now.  It's
going to be so fast I bet nearly everyone will migrate to it in due
course, but it's not ready for non-experts yet.

2) Switch to debian type distros (ubuntu, mint, debian, etc.) which do
not have selinux, instead apparmor, and enjoy virtiofs without dax in a
fairly stable way so long as you're ok with the promise of 'dax' coming
and it will all 'just work and get super fast almost automagically' with
typical upgrades in due course.  Just know that good operations will
only happen if the shared directory on the host is not used by anything
whatsoever of importance on the host while the guest is running.  You
can take 'snapshots' on the host that 'back up' the (not then running)
guest very easily.

3) Make virtiofs an area you want to develop deep knowledge about, be
willing to accept your systems won't be stable enough for anything
outside a lab or workshop, at least one thing you care about will break
in some way at least  one time per every four 'dnf upgrades' and switch
to fedora.  You'll have to learn a fair few quite obscure selinux
commands to get it to comprehend virtiofs is a filesystem and not flood
your logs, use the 'xml' features of virtual machine manager to get the
bits of virtiofs you want enabled properly (like 'numa', learn what
'hugepages' are and do the math on each host to set enough of those
aside with in sysctl.d (e.g. vm.nr_hugepages = 11776) , and also accept
some versions of hosts and guests will develop incompatibilities so
severe you'll have to restore to backups and downgrades from time to
time.  And you won't be able to use virtiofs as a root filesystem
without patching dracut config directories (search this list) or
compiling your own custom kernel.  But you will learn a whole lot.

4)  Stick with rhel/fedora/centos, create a file on the host (check on
fallocate and chattr +C) use that as a block disk image for the guest,
and watch and wait until virtiofs becomes 'real' on stable systems, then
bring the guest down, unpack the file into a directory tree, use the
then available 'all good' drop down list to 'pick' virtiofs and have a
rocket -- about a year from now or so.

That's pretty much where we are today I think.

Harry

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Vivek Goyal]

On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
> On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont <link@sub-pop.net>
> wrote:
> >  reproducible scenarios
> 
> Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> virtiofs filesystem and with xattr on), the type of files in the mounted
> hierarchy are unlabeled_t. I can work around that by switching SELinux in
> the guest to permissive or disabled.

cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.

In general, if we want to enable SELinux both on host and guest, then
both host and guest should have same SELinux policy. Otherwise there
will be lot of different kind of conflicts because both host and
guest will try to work with same selinux label. I guess that in
practice this will be very hard to achieve as people will run
different host and guest flavors and these might have different
policies.

So another option is to rename selinux xattr in virtiofs so that
any selinux xattr coming from guest is saved as
user.virtiofs.security.selinux xattr on host. That way host and guest
can have their separate labels without interfering with each other.
David Gilbert already has added support for this. I can't remember
the exact syntax but you can figure it out from documentation here
in xattr remappig section.

https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst

But I have question with selinux xattr remapping. What will happen
to initial labels when fs is exported. I mean until and unless
some process in guest labels all the exported files, they all
with either be unlabeled or pick some generic label for all the
files.

Another option is, can we use a single label for whole of the
virtiofs (using context=<label>) option in guest. That way nothing
is saved in files as such. But this means that processes in guest
can't have different selinux labels on different virtiofs dir/files.

Dan, what do you think?

Thanks
Vivek


> 
> With a CentOS 7 guest, things get less usable. I digested this to a
> reproducible scenario.
> 
> Build a disk image with `virt-builder`, configuring the CentOS Plus kernel
> to get 9p support.
> 
> virt-builder centos-7.8 \
> --root-password password:centos \
> --output centos-7.8.qcow2 \
> --install yum-utils \
> --run-command 'yum-config-manager --enable centosplus' \
> --run-command 'sed -ie "s/DEFAULTKERNEL=kernel/DEFAULTKERNEL=kernel-plus/"
> /etc/sysconfig/kernel' \
> --append-line '/etc/dracut.conf.d/virtio.conf:add_drivers+="virtio_scsi
> virtio_pci virtio_console"' \
> --append-line '/etc/modules-load.d/9pnet_virtio.conf:9pnet_virtio' \
> --install kernel-plus \
> --append-line '/etc/fstab:home /home 9p trans=virtio,version=9p2000.L 0 0'
> 
> Install the volume into the `default` pool.
> 
> sudo install -m644 centos-7.8.qcow2 /var/lib/libvirt/images
> 
> Next, define a domain using the disk image (using `virt-install` here for
> "easy mode").
> 
> virt-install \
> --import \
> --os-variant centos7.0 \
> --name centos \
> --ram 2048 \
> --disk path=/var/lib/libvirt/images/centos-7.8.qcow2 \
> --memorybacking access.mode=shared \
> --filesystem source=/home,target=home,accessmode=passthrough \
> --autoconsole none
> 
> Now with SELinux enforcing, I cannot list the contents of the directories in
> the mounted hierarchy.
> 
> [root@localhost ~]# ls -lZ /home/link
> ls: cannot open directory /home/link: Permission denied
> 
> 
> 
> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://listman.redhat.com/mailman/listinfo/virtio-fs
> 

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Daniel P. Berrangé]

On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
> On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
> > On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont <link@sub-pop.net>
> > wrote:
> > >  reproducible scenarios
> > 
> > Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> > virtiofs filesystem and with xattr on), the type of files in the mounted
> > hierarchy are unlabeled_t. I can work around that by switching SELinux in
> > the guest to permissive or disabled.
> 
> cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
> 
> In general, if we want to enable SELinux both on host and guest, then
> both host and guest should have same SELinux policy. Otherwise there
> will be lot of different kind of conflicts because both host and
> guest will try to work with same selinux label. I guess that in
> practice this will be very hard to achieve as people will run
> different host and guest flavors and these might have different
> policies.

Yeah, I think there's little to no chance of people keeping the
same SELinux policy in host/guest, except in very tightly controlled
narrow use cases where the host admin exerts direct control over
the precise guest config.


> So another option is to rename selinux xattr in virtiofs so that
> any selinux xattr coming from guest is saved as
> user.virtiofs.security.selinux xattr on host. That way host and guest
> can have their separate labels without interfering with each other.
> David Gilbert already has added support for this. I can't remember
> the exact syntax but you can figure it out from documentation here
> in xattr remappig section.

For general purpose virt usage, I think remapping in some way is
likely to be needed as the default strategy.

> https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
> 
> But I have question with selinux xattr remapping. What will happen
> to initial labels when fs is exported. I mean until and unless
> some process in guest labels all the exported files, they all
> with either be unlabeled or pick some generic label for all the
> files.

I'd say you need some mechanism to force a re-label inside the
guest. Normally a relabel will be done in /.autorelabel file
is present, or in certain other scenarios like selinux policy
RPM updates.

We wouldn't want to force a relabel neccesarily for the entire
FS if we're just hotplugging a new virtiofs export though. So
perhaps there's scope for supporting usage of a per-mount
point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
and whenever the guest sees a new virtiofs export arriving, it
can look for $VIRTIOFS-MOUNT-POINT/.autorelabel

> Another option is, can we use a single label for whole of the
> virtiofs (using context=<label>) option in guest. That way nothing
> is saved in files as such. But this means that processes in guest
> can't have different selinux labels on different virtiofs dir/files.

Forcing a single label for the entire export is passable as a
fallback plan. This is what people have done for years with
NFS v3 mounts. It has annoying usage limitations though, so
if at all possible remapping is a preferrable approach.

> 
> Dan, what do you think?
> 
> Thanks
> Vivek
> 
> 
> > 
> > With a CentOS 7 guest, things get less usable. I digested this to a
> > reproducible scenario.
> > 
> > Build a disk image with `virt-builder`, configuring the CentOS Plus kernel
> > to get 9p support.
> > 
> > virt-builder centos-7.8 \
> > --root-password password:centos \
> > --output centos-7.8.qcow2 \
> > --install yum-utils \
> > --run-command 'yum-config-manager --enable centosplus' \
> > --run-command 'sed -ie "s/DEFAULTKERNEL=kernel/DEFAULTKERNEL=kernel-plus/"
> > /etc/sysconfig/kernel' \
> > --append-line '/etc/dracut.conf.d/virtio.conf:add_drivers+="virtio_scsi
> > virtio_pci virtio_console"' \
> > --append-line '/etc/modules-load.d/9pnet_virtio.conf:9pnet_virtio' \
> > --install kernel-plus \
> > --append-line '/etc/fstab:home /home 9p trans=virtio,version=9p2000.L 0 0'
> > 
> > Install the volume into the `default` pool.
> > 
> > sudo install -m644 centos-7.8.qcow2 /var/lib/libvirt/images
> > 
> > Next, define a domain using the disk image (using `virt-install` here for
> > "easy mode").
> > 
> > virt-install \
> > --import \
> > --os-variant centos7.0 \
> > --name centos \
> > --ram 2048 \
> > --disk path=/var/lib/libvirt/images/centos-7.8.qcow2 \
> > --memorybacking access.mode=shared \
> > --filesystem source=/home,target=home,accessmode=passthrough \
> > --autoconsole none
> > 
> > Now with SELinux enforcing, I cannot list the contents of the directories in
> > the mounted hierarchy.
> > 
> > [root@localhost ~]# ls -lZ /home/link
> > ls: cannot open directory /home/link: Permission denied
> > 
> > 
> > 
> > _______________________________________________
> > Virtio-fs mailing list
> > Virtio-fs@redhat.com
> > https://listman.redhat.com/mailman/listinfo/virtio-fs
> > 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Vivek Goyal]

On Fri, Jun 04, 2021 at 08:30:26AM -0500, Harry G. Coin wrote:
> 
> On 6/3/21 9:14 PM, Link Dupont wrote:
> > On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont
> > <link@sub-pop.net> wrote:
> >>  reproducible scenarios
> >
> > Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> > virtiofs filesystem and with xattr on), the type of files in the
> > mounted hierarchy are unlabeled_t. I can work around that by switching
> > SELinux in the guest to permissive or disabled.
> >
> > With a CentOS 7 guest, things get less usable. I digested this to a
> > reproducible scenario.
> >
> > Build a disk image with `virt-builder`, configuring the CentOS Plus
> > kernel to get 9p support.
> >
> > virt-builder centos-7.8 \
> > --root-password password:centos \
> > --output centos-7.8.qcow2 \
> > --install yum-utils \
> > --run-command 'yum-config-manager --enable centosplus' \
> > --run-command 'sed -ie
> > "s/DEFAULTKERNEL=kernel/DEFAULTKERNEL=kernel-plus/"
> > /etc/sysconfig/kernel' \
> > --append-line
> > '/etc/dracut.conf.d/virtio.conf:add_drivers+="virtio_scsi virtio_pci
> > virtio_console"' \
> > --append-line '/etc/modules-load.d/9pnet_virtio.conf:9pnet_virtio' \
> > --install kernel-plus \
> > --append-line '/etc/fstab:home /home 9p trans=virtio,version=9p2000.L
> > 0 0'
> >
> > Install the volume into the `default` pool.
> >
> > sudo install -m644 centos-7.8.qcow2 /var/lib/libvirt/images
> >
> > Next, define a domain using the disk image (using `virt-install` here
> > for "easy mode").
> >
> > virt-install \
> > --import \
> > --os-variant centos7.0 \
> > --name centos \
> > --ram 2048 \
> > --disk path=/var/lib/libvirt/images/centos-7.8.qcow2 \
> > --memorybacking access.mode=shared \
> > --filesystem source=/home,target=home,accessmode=passthrough \
> > --autoconsole none
> >
> > Now with SELinux enforcing, I cannot list the contents of the
> > directories in the mounted hierarchy.
> >
> > [root@localhost ~]# ls -lZ /home/link
> > ls: cannot open directory /home/link: Permission denied
> >
> >
> Link, You have four choices right now:
> 
> 1) Stick with rhel/centos/fedora type distros and ignore virtiofs for at
> least 6 months, more like a year, for virtiofs to be implemented
> properly by the typical GUI enabled virt-manager/qemu/kvm resources you
> know about, and also not just wedge (worse than crash) your system by
> breaking things like 'dnf' and 'rpm' randomly when upgrades happen --
> blocking upgrades (and that's with 'permissive').   The big speed
> advantage to virtiofs comes with a feature termed 'dax', and what it
> takes to make that work is a whole lot of 'by hand' right now.  It's
> going to be so fast I bet nearly everyone will migrate to it in due
> course, but it's not ready for non-experts yet.
> 
> 2) Switch to debian type distros (ubuntu, mint, debian, etc.) which do
> not have selinux, instead apparmor, and enjoy virtiofs without dax in a
> fairly stable way so long as you're ok with the promise of 'dax' coming
> and it will all 'just work and get super fast almost automagically' with
> typical upgrades in due course.  Just know that good operations will
> only happen if the shared directory on the host is not used by anything
> whatsoever of importance on the host while the guest is running.  You
> can take 'snapshots' on the host that 'back up' the (not then running)
> guest very easily.
> 
> 3) Make virtiofs an area you want to develop deep knowledge about, be
> willing to accept your systems won't be stable enough for anything
> outside a lab or workshop, at least one thing you care about will break
> in some way at least  one time per every four 'dnf upgrades' and switch
> to fedora.

You are using virtiofs as rootfs and upgrade breaks? If virtiofs is
being used only as volumes, then its outside the upgrade path. Will
like to know what broke for you. Was it all SELinux related or something
else.

> You'll have to learn a fair few quite obscure selinux
> commands to get it to comprehend virtiofs is a filesystem and not flood
> your logs, use the 'xml' features of virtual machine manager to get the
> bits of virtiofs you want enabled properly (like 'numa',

I am assumig libvirt takes care of specifying "numa" stuff? Is that
not the case?

> learn what
> 'hugepages' are and do the math on each host to set enough of those
> aside with in sysctl.d (e.g. vm.nr_hugepages = 11776) ,

What's the virtiofs dependency on vm.nr_hugepages? I think I am not
aware of this aspect.


> and also accept
> some versions of hosts and guests will develop incompatibilities so
> severe you'll have to restore to backups and downgrades from time to
> time.  And you won't be able to use virtiofs as a root filesystem
> without patching dracut config directories (search this list) or
> compiling your own custom kernel.  But you will learn a whole lot.

Agreed that dracut does not seem to have support to mount virtiofs
as rootfs. That's one TODO item. 

I am assuming you prefer using virtiofs as rootfs for your VMs and
that's why running into these dracut related issues.

Thanks
Vivek

> 
> 4)  Stick with rhel/fedora/centos, create a file on the host (check on
> fallocate and chattr +C) use that as a block disk image for the guest,
> and watch and wait until virtiofs becomes 'real' on stable systems, then
> bring the guest down, unpack the file into a directory tree, use the
> then available 'all good' drop down list to 'pick' virtiofs and have a
> rocket -- about a year from now or so.
> 
> That's pretty much where we are today I think.
> 
> Harry
> 
> 
> 
> 
> 
> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://listman.redhat.com/mailman/listinfo/virtio-fs

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Vivek Goyal]

On Fri, Jun 04, 2021 at 02:59:29PM +0100, Daniel P. Berrangé wrote:
> On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
> > On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
> > > On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont <link@sub-pop.net>
> > > wrote:
> > > >  reproducible scenarios
> > > 
> > > Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> > > virtiofs filesystem and with xattr on), the type of files in the mounted
> > > hierarchy are unlabeled_t. I can work around that by switching SELinux in
> > > the guest to permissive or disabled.
> > 
> > cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
> > 
> > In general, if we want to enable SELinux both on host and guest, then
> > both host and guest should have same SELinux policy. Otherwise there
> > will be lot of different kind of conflicts because both host and
> > guest will try to work with same selinux label. I guess that in
> > practice this will be very hard to achieve as people will run
> > different host and guest flavors and these might have different
> > policies.
> 
> Yeah, I think there's little to no chance of people keeping the
> same SELinux policy in host/guest, except in very tightly controlled
> narrow use cases where the host admin exerts direct control over
> the precise guest config.
> 
> 
> > So another option is to rename selinux xattr in virtiofs so that
> > any selinux xattr coming from guest is saved as
> > user.virtiofs.security.selinux xattr on host. That way host and guest
> > can have their separate labels without interfering with each other.
> > David Gilbert already has added support for this. I can't remember
> > the exact syntax but you can figure it out from documentation here
> > in xattr remappig section.
> 
> For general purpose virt usage, I think remapping in some way is
> likely to be needed as the default strategy.
> 
> > https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
> > 
> > But I have question with selinux xattr remapping. What will happen
> > to initial labels when fs is exported. I mean until and unless
> > some process in guest labels all the exported files, they all
> > with either be unlabeled or pick some generic label for all the
> > files.
> 
> I'd say you need some mechanism to force a re-label inside the
> guest. Normally a relabel will be done in /.autorelabel file
> is present, or in certain other scenarios like selinux policy
> RPM updates.
> 
> We wouldn't want to force a relabel neccesarily for the entire
> FS if we're just hotplugging a new virtiofs export though. So
> perhaps there's scope for supporting usage of a per-mount
> point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
> and whenever the guest sees a new virtiofs export arriving, it
> can look for $VIRTIOFS-MOUNT-POINT/.autorelabel

Per mount point auto relabel seems interesting. Will it relabel
everytime virtiofs export shows up. Or it will intelligence to
figure out exported fs is already labeled (say from previous boot)
and no need to relabel again.

> 
> > Another option is, can we use a single label for whole of the
> > virtiofs (using context=<label>) option in guest. That way nothing
> > is saved in files as such. But this means that processes in guest
> > can't have different selinux labels on different virtiofs dir/files.
> 
> Forcing a single label for the entire export is passable as a
> fallback plan. This is what people have done for years with
> NFS v3 mounts.

Aha, good to know.

> It has annoying usage limitations though, so
> if at all possible remapping is a preferrable approach.

Agreed that this appraoch will have more limitations as comapred to
selinux xattr remapping + autorelabel. At the same time, this is the
simplest approach to begin with. And one does not pay the penalty
of relabeling files.

Maybe this can be the default and we document that how to enable
proper selinux support by using xattr remapping + autorelabel.

Thanks
Vivek

> 
> > 
> > Dan, what do you think?
> > 
> > Thanks
> > Vivek
> > 
> > 
> > > 
> > > With a CentOS 7 guest, things get less usable. I digested this to a
> > > reproducible scenario.
> > > 
> > > Build a disk image with `virt-builder`, configuring the CentOS Plus kernel
> > > to get 9p support.
> > > 
> > > virt-builder centos-7.8 \
> > > --root-password password:centos \
> > > --output centos-7.8.qcow2 \
> > > --install yum-utils \
> > > --run-command 'yum-config-manager --enable centosplus' \
> > > --run-command 'sed -ie "s/DEFAULTKERNEL=kernel/DEFAULTKERNEL=kernel-plus/"
> > > /etc/sysconfig/kernel' \
> > > --append-line '/etc/dracut.conf.d/virtio.conf:add_drivers+="virtio_scsi
> > > virtio_pci virtio_console"' \
> > > --append-line '/etc/modules-load.d/9pnet_virtio.conf:9pnet_virtio' \
> > > --install kernel-plus \
> > > --append-line '/etc/fstab:home /home 9p trans=virtio,version=9p2000.L 0 0'
> > > 
> > > Install the volume into the `default` pool.
> > > 
> > > sudo install -m644 centos-7.8.qcow2 /var/lib/libvirt/images
> > > 
> > > Next, define a domain using the disk image (using `virt-install` here for
> > > "easy mode").
> > > 
> > > virt-install \
> > > --import \
> > > --os-variant centos7.0 \
> > > --name centos \
> > > --ram 2048 \
> > > --disk path=/var/lib/libvirt/images/centos-7.8.qcow2 \
> > > --memorybacking access.mode=shared \
> > > --filesystem source=/home,target=home,accessmode=passthrough \
> > > --autoconsole none
> > > 
> > > Now with SELinux enforcing, I cannot list the contents of the directories in
> > > the mounted hierarchy.
> > > 
> > > [root@localhost ~]# ls -lZ /home/link
> > > ls: cannot open directory /home/link: Permission denied
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > Virtio-fs mailing list
> > > Virtio-fs@redhat.com
> > > https://listman.redhat.com/mailman/listinfo/virtio-fs
> > > 
> > 
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Harry G. Coin]

Hi Vivek,

I've cut the o.p.'s text as your questions and my answers are off-topic
to his interest.

On 6/4/21 9:11 AM, Vivek Goyal wrote:

> You are using virtiofs as rootfs and upgrade breaks? If virtiofs is
> being used only as volumes, then its outside the upgrade path. Will
> like to know what broke for you. Was it all SELinux related or something
> else.

Seems 100% selinux related -- even in permissive mode on the guest. 
Here's your short-and-sweet 100% reproducer:  
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1930756  There's a
similar bug report on redhat.

Even with selinux set to permissive in the guest, no selinux installed
in the host, but with xattrs on, no and no xattr mapping, lsetfilecon
returns with a permission error.  The files on the host are entirely
dedicated to the guest -- no use of those files is made on the host. 
Underlying fs on the host is btrfs.
>> You'll have to learn a fair few quite obscure selinux
>> commands to get it to comprehend virtiofs is a filesystem and not flood
>> your logs, use the 'xml' features of virtual machine manager to get the
>> bits of virtiofs you want enabled properly (like 'numa',
> I am assumig libvirt takes care of specifying "numa" stuff? Is that
> not the case?

Not outside the lab.  There is no virt-manager GUI way to coordinate
memory allocation with hugepages/numa.  Hand-editing the xml is required
to manually match the numa set-aside with the memory allocation.  If
reliability is important, unless you use systctl.d to set aside the
hugepages the system can randomly not launch the guests owing to lack of
sufficient contiguous memory at guest launch time.

>> learn what
>> 'hugepages' are and do the math on each host to set enough of those
>> aside with in sysctl.d (e.g. vm.nr_hugepages = 11776) ,
> What's the virtiofs dependency on vm.nr_hugepages? I think I am not
> aware of this aspect.

Detailed here https://libvirt.org/kbase/virtiofs.html


>
>> and also accept
>> some versions of hosts and guests will develop incompatibilities so
>> severe you'll have to restore to backups and downgrades from time to
>> time.  And you won't be able to use virtiofs as a root filesystem
>> without patching dracut config directories (search this list) or
>> compiling your own custom kernel.  But you will learn a whole lot.
> Agreed that dracut does not seem to have support to mount virtiofs
> as rootfs. That's one TODO item. 

I've done it. It's been working for months, automatically maintained and
updated through both host and guest kernel upgrades.  I haven't posted
the bit that updates the kernel / initrd links for direct kernel launch
(using initrds) but I have posted what's needed for dracut to make
rootfs virtiofs work.

All that's required for stability -- for now -- is for the host to treat
the files shared on the guest as 'unusable' while the guest is running
-- or at least 'read only'.

> I am assuming you prefer using virtiofs as rootfs for your VMs and
> that's why running into these dracut related issues.

Let me restate -- I am NOT having a dracut issue.  I solved that long
ago and it works very well.   In fact the whole virtiofs thing was
working very well (without dax as main-distro kernels still lack the pci
bar in the host) -- until a regression in the last few weeks that broke
rpm / dnf / yum (noted above, even with permissive on the guest
lsetfilecon returns a permission error blocking all dnf operations). 
The only 'maintainable' way forward I can see is to reduce the number of
core packages involved.  The disk space required for a basic workstation
or server image is 'small' by today's storage standards.  I see no
reason multiply complexities by implementing one host-guest relationship
for 'booting' a guest and another for 'the files on the guest'.  Making
it possible to just snapshot one directory tree on the host and know
everything important to the guest is there -- that's a big win.

Harry

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Daniel P. Berrangé]

On Fri, Jun 04, 2021 at 10:43:13AM -0400, Vivek Goyal wrote:
> On Fri, Jun 04, 2021 at 02:59:29PM +0100, Daniel P. Berrangé wrote:
> > On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
> > > On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
> > > > On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont <link@sub-pop.net>
> > > > wrote:
> > > > >  reproducible scenarios
> > > > 
> > > > Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> > > > virtiofs filesystem and with xattr on), the type of files in the mounted
> > > > hierarchy are unlabeled_t. I can work around that by switching SELinux in
> > > > the guest to permissive or disabled.
> > > 
> > > cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
> > > 
> > > In general, if we want to enable SELinux both on host and guest, then
> > > both host and guest should have same SELinux policy. Otherwise there
> > > will be lot of different kind of conflicts because both host and
> > > guest will try to work with same selinux label. I guess that in
> > > practice this will be very hard to achieve as people will run
> > > different host and guest flavors and these might have different
> > > policies.
> > 
> > Yeah, I think there's little to no chance of people keeping the
> > same SELinux policy in host/guest, except in very tightly controlled
> > narrow use cases where the host admin exerts direct control over
> > the precise guest config.
> > 
> > 
> > > So another option is to rename selinux xattr in virtiofs so that
> > > any selinux xattr coming from guest is saved as
> > > user.virtiofs.security.selinux xattr on host. That way host and guest
> > > can have their separate labels without interfering with each other.
> > > David Gilbert already has added support for this. I can't remember
> > > the exact syntax but you can figure it out from documentation here
> > > in xattr remappig section.
> > 
> > For general purpose virt usage, I think remapping in some way is
> > likely to be needed as the default strategy.
> > 
> > > https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
> > > 
> > > But I have question with selinux xattr remapping. What will happen
> > > to initial labels when fs is exported. I mean until and unless
> > > some process in guest labels all the exported files, they all
> > > with either be unlabeled or pick some generic label for all the
> > > files.
> > 
> > I'd say you need some mechanism to force a re-label inside the
> > guest. Normally a relabel will be done in /.autorelabel file
> > is present, or in certain other scenarios like selinux policy
> > RPM updates.
> > 
> > We wouldn't want to force a relabel neccesarily for the entire
> > FS if we're just hotplugging a new virtiofs export though. So
> > perhaps there's scope for supporting usage of a per-mount
> > point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
> > and whenever the guest sees a new virtiofs export arriving, it
> > can look for $VIRTIOFS-MOUNT-POINT/.autorelabel
> 
> Per mount point auto relabel seems interesting. Will it relabel
> everytime virtiofs export shows up. Or it will intelligence to
> figure out exported fs is already labeled (say from previous boot)
> and no need to relabel again.

Normal behaviour with the existing global /.autorelabel is that
selinux deletes the file once the relabel is complete, so it is
a one-time thing.

Essentially anytime you need to force[1] a one-time only relabel you
just do

   "touch /.autorelabel && reboot"


DanW does however recommend that you configure your system such
that the labels are always correct and thus don't need the global
relabel. https://danwalsh.livejournal.com/38157.html

None the less I think the virtiofs situation is slightly different
and justifies a way to force relabel, provided it can be scoped
to just that one filesystem mount.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Daniel Walsh]

On 6/4/21 09:59, Daniel P. Berrangé wrote:
> On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
>> On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
>>> On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont <link@sub-pop.net>
>>> wrote:
>>>>   reproducible scenarios
>>> Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
>>> virtiofs filesystem and with xattr on), the type of files in the mounted
>>> hierarchy are unlabeled_t. I can work around that by switching SELinux in
>>> the guest to permissive or disabled.
>> cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
>>
>> In general, if we want to enable SELinux both on host and guest, then
>> both host and guest should have same SELinux policy. Otherwise there
>> will be lot of different kind of conflicts because both host and
>> guest will try to work with same selinux label. I guess that in
>> practice this will be very hard to achieve as people will run
>> different host and guest flavors and these might have different
>> policies.
> Yeah, I think there's little to no chance of people keeping the
> same SELinux policy in host/guest, except in very tightly controlled
> narrow use cases where the host admin exerts direct control over
> the precise guest config.
>
>
>> So another option is to rename selinux xattr in virtiofs so that
>> any selinux xattr coming from guest is saved as
>> user.virtiofs.security.selinux xattr on host. That way host and guest
>> can have their separate labels without interfering with each other.
>> David Gilbert already has added support for this. I can't remember
>> the exact syntax but you can figure it out from documentation here
>> in xattr remappig section.
> For general purpose virt usage, I think remapping in some way is
> likely to be needed as the default strategy.
>
>> https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
>>
>> But I have question with selinux xattr remapping. What will happen
>> to initial labels when fs is exported. I mean until and unless
>> some process in guest labels all the exported files, they all
>> with either be unlabeled or pick some generic label for all the
>> files.
> I'd say you need some mechanism to force a re-label inside the
> guest. Normally a relabel will be done in /.autorelabel file
> is present, or in certain other scenarios like selinux policy
> RPM updates.
>
> We wouldn't want to force a relabel neccesarily for the entire
> FS if we're just hotplugging a new virtiofs export though. So
> perhaps there's scope for supporting usage of a per-mount
> point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
> and whenever the guest sees a new virtiofs export arriving, it
> can look for $VIRTIOFS-MOUNT-POINT/.autorelabel
>
>> Another option is, can we use a single label for whole of the
>> virtiofs (using context=<label>) option in guest. That way nothing
>> is saved in files as such. But this means that processes in guest
>> can't have different selinux labels on different virtiofs dir/files.
> Forcing a single label for the entire export is passable as a
> fallback plan. This is what people have done for years with
> NFS v3 mounts. It has annoying usage limitations though, so
> if at all possible remapping is a preferrable approach.
>
>> Dan, what do you think?
>>
>> Thanks
>> Vivek
>>
>>
>>> With a CentOS 7 guest, things get less usable. I digested this to a
>>> reproducible scenario.
>>>
>>> Build a disk image with `virt-builder`, configuring the CentOS Plus kernel
>>> to get 9p support.
>>>
>>> virt-builder centos-7.8 \
>>> --root-password password:centos \
>>> --output centos-7.8.qcow2 \
>>> --install yum-utils \
>>> --run-command 'yum-config-manager --enable centosplus' \
>>> --run-command 'sed -ie "s/DEFAULTKERNEL=kernel/DEFAULTKERNEL=kernel-plus/"
>>> /etc/sysconfig/kernel' \
>>> --append-line '/etc/dracut.conf.d/virtio.conf:add_drivers+="virtio_scsi
>>> virtio_pci virtio_console"' \
>>> --append-line '/etc/modules-load.d/9pnet_virtio.conf:9pnet_virtio' \
>>> --install kernel-plus \
>>> --append-line '/etc/fstab:home /home 9p trans=virtio,version=9p2000.L 0 0'
>>>
>>> Install the volume into the `default` pool.
>>>
>>> sudo install -m644 centos-7.8.qcow2 /var/lib/libvirt/images
>>>
>>> Next, define a domain using the disk image (using `virt-install` here for
>>> "easy mode").
>>>
>>> virt-install \
>>> --import \
>>> --os-variant centos7.0 \
>>> --name centos \
>>> --ram 2048 \
>>> --disk path=/var/lib/libvirt/images/centos-7.8.qcow2 \
>>> --memorybacking access.mode=shared \
>>> --filesystem source=/home,target=home,accessmode=passthrough \
>>> --autoconsole none
>>>
>>> Now with SELinux enforcing, I cannot list the contents of the directories in
>>> the mounted hierarchy.
>>>
>>> [root@localhost ~]# ls -lZ /home/link
>>> ls: cannot open directory /home/link: Permission denied
>>>
>>>
>>>
>>> _______________________________________________
>>> Virtio-fs mailing list
>>> Virtio-fs@redhat.com
>>> https://listman.redhat.com/mailman/listinfo/virtio-fs
>>>
> Regards,
> Daniel

The separate XAttr support would also allow virtiofsd to work with 
labels inside of the container when the virtiofsd is confined on the host.


I would guess/hope virtiofsd on the host will be confined and only able 
to write certain labels like container_file_t or svirt_image_t.  If you 
want to allow virt file systems within the "VM" to be able to set 
labels, these labels will have to be something other then SELinux labels 
on the host, or SELinux will prevent them from being set.

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Link Dupont]

On Mon, Jun 7 2021 at 09:01:08 AM -0400, Daniel Walsh 
<dwalsh@redhat.com> wrote:
> On 6/4/21 09:59, Daniel P. Berrangé wrote:
>> On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
>>> On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
>>>> On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont 
>>>> <link@sub-pop.net>
>>>> wrote:
>>>>>   reproducible scenarios
>>>> Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with 
>>>> a
>>>> virtiofs filesystem and with xattr on), the type of files in the 
>>>> mounted
>>>> hierarchy are unlabeled_t. I can work around that by switching 
>>>> SELinux in
>>>> the guest to permissive or disabled.
>>> cc Dan Walsh. I was discussing this with Dan Walsh yesterday in 
>>> general.
>>> 
>>> In general, if we want to enable SELinux both on host and guest, 
>>> then
>>> both host and guest should have same SELinux policy. Otherwise there
>>> will be lot of different kind of conflicts because both host and
>>> guest will try to work with same selinux label. I guess that in
>>> practice this will be very hard to achieve as people will run
>>> different host and guest flavors and these might have different
>>> policies.
>> Yeah, I think there's little to no chance of people keeping the
>> same SELinux policy in host/guest, except in very tightly controlled
>> narrow use cases where the host admin exerts direct control over
>> the precise guest config.
>> 
>> 
>>> So another option is to rename selinux xattr in virtiofs so that
>>> any selinux xattr coming from guest is saved as
>>> user.virtiofs.security.selinux xattr on host. That way host and 
>>> guest
>>> can have their separate labels without interfering with each other.
>>> David Gilbert already has added support for this. I can't remember
>>> the exact syntax but you can figure it out from documentation here
>>> in xattr remappig section.
>> For general purpose virt usage, I think remapping in some way is
>> likely to be needed as the default strategy.
>> 
>>> https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
>>> 
>>> But I have question with selinux xattr remapping. What will happen
>>> to initial labels when fs is exported. I mean until and unless
>>> some process in guest labels all the exported files, they all
>>> with either be unlabeled or pick some generic label for all the
>>> files.
>> I'd say you need some mechanism to force a re-label inside the
>> guest. Normally a relabel will be done in /.autorelabel file
>> is present, or in certain other scenarios like selinux policy
>> RPM updates.
>> 
>> We wouldn't want to force a relabel neccesarily for the entire
>> FS if we're just hotplugging a new virtiofs export though. So
>> perhaps there's scope for supporting usage of a per-mount
>> point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
>> and whenever the guest sees a new virtiofs export arriving, it
>> can look for $VIRTIOFS-MOUNT-POINT/.autorelabel
>> 
>>> Another option is, can we use a single label for whole of the
>>> virtiofs (using context=<label>) option in guest. That way nothing
>>> is saved in files as such. But this means that processes in guest
>>> can't have different selinux labels on different virtiofs dir/files.
>> Forcing a single label for the entire export is passable as a
>> fallback plan. This is what people have done for years with
>> NFS v3 mounts. It has annoying usage limitations though, so
>> if at all possible remapping is a preferrable approach.
>> 
>>> Dan, what do you think?
>>> 
>>> Thanks
>>> Vivek
>>> 
>>> 
>>>> With a CentOS 7 guest, things get less usable. I digested this to a
>>>> reproducible scenario.
>>>> 
>>>> Build a disk image with `virt-builder`, configuring the CentOS 
>>>> Plus kernel
>>>> to get 9p support.
>>>> 
>>>> virt-builder centos-7.8 \
>>>> --root-password password:centos \
>>>> --output centos-7.8.qcow2 \
>>>> --install yum-utils \
>>>> --run-command 'yum-config-manager --enable centosplus' \
>>>> --run-command 'sed -ie 
>>>> "s/DEFAULTKERNEL=kernel/DEFAULTKERNEL=kernel-plus/"
>>>> /etc/sysconfig/kernel' \
>>>> --append-line 
>>>> '/etc/dracut.conf.d/virtio.conf:add_drivers+="virtio_scsi
>>>> virtio_pci virtio_console"' \
>>>> --append-line '/etc/modules-load.d/9pnet_virtio.conf:9pnet_virtio' 
>>>> \
>>>> --install kernel-plus \
>>>> --append-line '/etc/fstab:home /home 9p 
>>>> trans=virtio,version=9p2000.L 0 0'
>>>> 
>>>> Install the volume into the `default` pool.
>>>> 
>>>> sudo install -m644 centos-7.8.qcow2 /var/lib/libvirt/images
>>>> 
>>>> Next, define a domain using the disk image (using `virt-install` 
>>>> here for
>>>> "easy mode").
>>>> 
>>>> virt-install \
>>>> --import \
>>>> --os-variant centos7.0 \
>>>> --name centos \
>>>> --ram 2048 \
>>>> --disk path=/var/lib/libvirt/images/centos-7.8.qcow2 \
>>>> --memorybacking access.mode=shared \
>>>> --filesystem source=/home,target=home,accessmode=passthrough \
>>>> --autoconsole none
>>>> 
>>>> Now with SELinux enforcing, I cannot list the contents of the 
>>>> directories in
>>>> the mounted hierarchy.
>>>> 
>>>> [root@localhost ~]# ls -lZ /home/link
>>>> ls: cannot open directory /home/link: Permission denied
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Virtio-fs mailing list
>>>> Virtio-fs@redhat.com
>>>> https://listman.redhat.com/mailman/listinfo/virtio-fs
>>>> 
>> Regards,
>> Daniel
> 
> The separate XAttr support would also allow virtiofsd to work with 
> labels inside of the container when the virtiofsd is confined on the 
> host.
> 
> 
> I would guess/hope virtiofsd on the host will be confined and only 
> able to write certain labels like container_file_t or svirt_image_t.  
> If you want to allow virt file systems within the "VM" to be able to 
> set labels, these labels will have to be something other then SELinux 
> labels on the host, or SELinux will prevent them from being set.
> 

I don't think the guest needs to be able to set labels on the virtiofs 
filesystems; as long as it can read labels that grant read and write 
access as the guest user, I'd call it a success. The way I use this 
feature, I'm treating my host filesystem as the "source", and really 
only mounting it into the guest as a zero-delay synchronization. (I 
previously did this using sync solutions like rsync or lsyncd, but 
those have a slight delay before files sync over.)

If the labels as seen by the guest allow for reading and writing files 
from the guest to the host filesystem, I'd be happy. Doing any 
relabeling from the host is totally fine.

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Daniel P. Berrangé]

On Fri, Jun 04, 2021 at 02:59:29PM +0100, Daniel P. Berrangé wrote:
> On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
> > On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
> > > On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont <link@sub-pop.net>
> > > wrote:
> > > >  reproducible scenarios
> > > 
> > > Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> > > virtiofs filesystem and with xattr on), the type of files in the mounted
> > > hierarchy are unlabeled_t. I can work around that by switching SELinux in
> > > the guest to permissive or disabled.
> > 
> > cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
> > 
> > In general, if we want to enable SELinux both on host and guest, then
> > both host and guest should have same SELinux policy. Otherwise there
> > will be lot of different kind of conflicts because both host and
> > guest will try to work with same selinux label. I guess that in
> > practice this will be very hard to achieve as people will run
> > different host and guest flavors and these might have different
> > policies.
> 
> Yeah, I think there's little to no chance of people keeping the
> same SELinux policy in host/guest, except in very tightly controlled
> narrow use cases where the host admin exerts direct control over
> the precise guest config.
> 
> 
> > So another option is to rename selinux xattr in virtiofs so that
> > any selinux xattr coming from guest is saved as
> > user.virtiofs.security.selinux xattr on host. That way host and guest
> > can have their separate labels without interfering with each other.
> > David Gilbert already has added support for this. I can't remember
> > the exact syntax but you can figure it out from documentation here
> > in xattr remappig section.
> 
> For general purpose virt usage, I think remapping in some way is
> likely to be needed as the default strategy.
> 
> > https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
> > 
> > But I have question with selinux xattr remapping. What will happen
> > to initial labels when fs is exported. I mean until and unless
> > some process in guest labels all the exported files, they all
> > with either be unlabeled or pick some generic label for all the
> > files.
> 
> I'd say you need some mechanism to force a re-label inside the
> guest. Normally a relabel will be done in /.autorelabel file
> is present, or in certain other scenarios like selinux policy
> RPM updates.
> 
> We wouldn't want to force a relabel neccesarily for the entire
> FS if we're just hotplugging a new virtiofs export though. So
> perhaps there's scope for supporting usage of a per-mount
> point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
> and whenever the guest sees a new virtiofs export arriving, it
> can look for $VIRTIOFS-MOUNT-POINT/.autorelabel
> 
> > Another option is, can we use a single label for whole of the
> > virtiofs (using context=<label>) option in guest. That way nothing
> > is saved in files as such. But this means that processes in guest
> > can't have different selinux labels on different virtiofs dir/files.
> 
> Forcing a single label for the entire export is passable as a
> fallback plan. This is what people have done for years with
> NFS v3 mounts. It has annoying usage limitations though, so
> if at all possible remapping is a preferrable approach.

Another thing we should bear in mind is that it is entirely plausible for
users to want SELinux in their guest, wihle the host is not using SELinux
at all. eg running a RHEL guest, on a cloud running Ubuntu hosts.

In such a case SELinux essentially won't exist from the POV of the
host OS / admin using. Anything related to labelling will have to be
exclusively done by the guest. If we provide a general purpose way
to remap XAttrs, then this ought to be viable by having a generic
remap rule for anything the guest does.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Harry G. Coin]

On 6/7/21 9:15 AM, Daniel P. Berrangé wrote:
> On Fri, Jun 04, 2021 at 02:59:29PM +0100, Daniel P. Berrangé wrote:
>> On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
>>> On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
>>>> On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont <link@sub-pop.net>
>>>> wrote:
>>>>>  reproducible scenarios
>>>> Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
>>>> virtiofs filesystem and with xattr on), the type of files in the mounted
>>>> hierarchy are unlabeled_t. I can work around that by switching SELinux in
>>>> the guest to permissive or disabled.
>>> cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
>>>
>>> In general, if we want to enable SELinux both on host and guest, then
>>> both host and guest should have same SELinux policy. Otherwise there
>>> will be lot of different kind of conflicts because both host and
>>> guest will try to work with same selinux label. I guess that in
>>> practice this will be very hard to achieve as people will run
>>> different host and guest flavors and these might have different
>>> policies.
>> Yeah, I think there's little to no chance of people keeping the
>> same SELinux policy in host/guest, except in very tightly controlled
>> narrow use cases where the host admin exerts direct control over
>> the precise guest config.
>>
>>
>>> So another option is to rename selinux xattr in virtiofs so that
>>> any selinux xattr coming from guest is saved as
>>> user.virtiofs.security.selinux xattr on host. That way host and guest
>>> can have their separate labels without interfering with each other.
>>> David Gilbert already has added support for this. I can't remember
>>> the exact syntax but you can figure it out from documentation here
>>> in xattr remappig section.
>> For general purpose virt usage, I think remapping in some way is
>> likely to be needed as the default strategy.
>>
>>> https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
>>>
>>> But I have question with selinux xattr remapping. What will happen
>>> to initial labels when fs is exported. I mean until and unless
>>> some process in guest labels all the exported files, they all
>>> with either be unlabeled or pick some generic label for all the
>>> files.
>> I'd say you need some mechanism to force a re-label inside the
>> guest. Normally a relabel will be done in /.autorelabel file
>> is present, or in certain other scenarios like selinux policy
>> RPM updates.
>>
>> We wouldn't want to force a relabel neccesarily for the entire
>> FS if we're just hotplugging a new virtiofs export though. So
>> perhaps there's scope for supporting usage of a per-mount
>> point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
>> and whenever the guest sees a new virtiofs export arriving, it
>> can look for $VIRTIOFS-MOUNT-POINT/.autorelabel
>>
>>> Another option is, can we use a single label for whole of the
>>> virtiofs (using context=<label>) option in guest. That way nothing
>>> is saved in files as such. But this means that processes in guest
>>> can't have different selinux labels on different virtiofs dir/files.
>> Forcing a single label for the entire export is passable as a
>> fallback plan. This is what people have done for years with
>> NFS v3 mounts. It has annoying usage limitations though, so
>> if at all possible remapping is a preferrable approach.
> Another thing we should bear in mind is that it is entirely plausible for
> users to want SELinux in their guest, wihle the host is not using SELinux
> at all. eg running a RHEL guest, on a cloud running Ubuntu hosts.
>
> In such a case SELinux essentially won't exist from the POV of the
> host OS / admin using. Anything related to labelling will have to be
> exclusively done by the guest. If we provide a general purpose way
> to remap XAttrs, then this ought to be viable by having a generic
> remap rule for anything the guest does.
>
> Regards,
> Daniel

Three cheers -- As someone who has to deal with various host/guest
distros, including specifically rhel/centos/fedora guests on
debian/ubuntu/etc hosts:  Notice I think all concerned want a 'host
snapshot' of a rhel running guest directory tree to do the right thing
if 'sent' to another server which is selinux enabled.  The ability
virtiofsd has to map xattrs is pretty good, and the virtiofs team should
make sure that mapping system works on both selinux and non-selinux
hosts.  Whether the host enforces those can be left to the host.

Regards,

Harry

Re: [Virtio-fs] virtiofs mounted filesystems & SELinux

[Dr. David Alan Gilbert]

* Harry G. Coin (hgcoin@gmail.com) wrote:
> 
> On 6/7/21 9:15 AM, Daniel P. Berrangé wrote:
> > On Fri, Jun 04, 2021 at 02:59:29PM +0100, Daniel P. Berrangé wrote:
> >> On Fri, Jun 04, 2021 at 09:44:39AM -0400, Vivek Goyal wrote:
> >>> On Thu, Jun 03, 2021 at 10:14:24PM -0400, Link Dupont wrote:
> >>>> On Thu, Jun 3 2021 at 08:56:46 PM -0400, Link Dupont <link@sub-pop.net>
> >>>> wrote:
> >>>>>  reproducible scenarios
> >>>> Alright. I reran my tests with a CentOS 8 guest. On CentOS 8 (with a
> >>>> virtiofs filesystem and with xattr on), the type of files in the mounted
> >>>> hierarchy are unlabeled_t. I can work around that by switching SELinux in
> >>>> the guest to permissive or disabled.
> >>> cc Dan Walsh. I was discussing this with Dan Walsh yesterday in general.
> >>>
> >>> In general, if we want to enable SELinux both on host and guest, then
> >>> both host and guest should have same SELinux policy. Otherwise there
> >>> will be lot of different kind of conflicts because both host and
> >>> guest will try to work with same selinux label. I guess that in
> >>> practice this will be very hard to achieve as people will run
> >>> different host and guest flavors and these might have different
> >>> policies.
> >> Yeah, I think there's little to no chance of people keeping the
> >> same SELinux policy in host/guest, except in very tightly controlled
> >> narrow use cases where the host admin exerts direct control over
> >> the precise guest config.
> >>
> >>
> >>> So another option is to rename selinux xattr in virtiofs so that
> >>> any selinux xattr coming from guest is saved as
> >>> user.virtiofs.security.selinux xattr on host. That way host and guest
> >>> can have their separate labels without interfering with each other.
> >>> David Gilbert already has added support for this. I can't remember
> >>> the exact syntax but you can figure it out from documentation here
> >>> in xattr remappig section.
> >> For general purpose virt usage, I think remapping in some way is
> >> likely to be needed as the default strategy.
> >>
> >>> https://github.com/qemu/qemu/blob/master/docs/tools/virtiofsd.rst
> >>>
> >>> But I have question with selinux xattr remapping. What will happen
> >>> to initial labels when fs is exported. I mean until and unless
> >>> some process in guest labels all the exported files, they all
> >>> with either be unlabeled or pick some generic label for all the
> >>> files.
> >> I'd say you need some mechanism to force a re-label inside the
> >> guest. Normally a relabel will be done in /.autorelabel file
> >> is present, or in certain other scenarios like selinux policy
> >> RPM updates.
> >>
> >> We wouldn't want to force a relabel neccesarily for the entire
> >> FS if we're just hotplugging a new virtiofs export though. So
> >> perhaps there's scope for supporting usage of a per-mount
> >> point relabel trigger. eg Host creates $VIRTIOFS-ROOT/.autorelabel
> >> and whenever the guest sees a new virtiofs export arriving, it
> >> can look for $VIRTIOFS-MOUNT-POINT/.autorelabel
> >>
> >>> Another option is, can we use a single label for whole of the
> >>> virtiofs (using context=<label>) option in guest. That way nothing
> >>> is saved in files as such. But this means that processes in guest
> >>> can't have different selinux labels on different virtiofs dir/files.
> >> Forcing a single label for the entire export is passable as a
> >> fallback plan. This is what people have done for years with
> >> NFS v3 mounts. It has annoying usage limitations though, so
> >> if at all possible remapping is a preferrable approach.
> > Another thing we should bear in mind is that it is entirely plausible for
> > users to want SELinux in their guest, wihle the host is not using SELinux
> > at all. eg running a RHEL guest, on a cloud running Ubuntu hosts.
> >
> > In such a case SELinux essentially won't exist from the POV of the
> > host OS / admin using. Anything related to labelling will have to be
> > exclusively done by the guest. If we provide a general purpose way
> > to remap XAttrs, then this ought to be viable by having a generic
> > remap rule for anything the guest does.
> >
> > Regards,
> > Daniel
> 
> Three cheers -- As someone who has to deal with various host/guest
> distros, including specifically rhel/centos/fedora guests on
> debian/ubuntu/etc hosts:  Notice I think all concerned want a 'host
> snapshot' of a rhel running guest directory tree to do the right thing
> if 'sent' to another server which is selinux enabled.  The ability
> virtiofsd has to map xattrs is pretty good, and the virtiofs team should
> make sure that mapping system works on both selinux and non-selinux
> hosts.  Whether the host enforces those can be left to the host.

As long as they map to user.something (e.g. user.virtiofsd.) then
my understanding is the host should allow the virtiofsd to do what it
likes with those; it wouldn't be able to see the host labels though.

Note that there are weird mappings I'd probably caution against; for
example you can probably get the mapping engine to write any
security.something as user.virtiofsd.security.something but then
to see the visibility of both an original and a remapped name
as security.something inside the guest;  that might let you see
both host's labels and your own - which could be both convenient and
also very confusing.

Dave

> Regards,
> 
> Harry
> 
> 
> 
> 
> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://listman.redhat.com/mailman/listinfo/virtio-fs
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK