KVM / virtual networking access control

KVM / virtual networking access control

[Karl MacMillan]

Hi,

I'm trying to do some access control over the networking between KVM
virtual machines using Secmark and things are not working as I expect.
The key is that this is _between_ virtual machines, not just ingress /
egress to the virtual machines from external address.

Quick summary: I'm connecting two VMs together using veth pairs and a
virtual bridge and I only ever see:

    allow unlabeled_t bare_packet_a_t:packet { forward_in forward_out };

What I want is packet { send recv } because that would be against the
label of the KVM process rather than the peer label (which on the same
system would always be unlabeled - I see no way to use CIPSO/CALIPSO
to get peer labels to work here).

I can see that the packet send / recv LSM hook is in netfilter
postrouting code, but I simply cannot trigger this with the setup that
I have. Here are things that I have tried:

1. br_netfilter - I thought initially that my problem was that the
traffic was at layer 2 and was not traversing the netfilter hooks.
This was, in fact, a problem with my initial setup (both VMs on the
same subnet) and br_netfilter got traffic to hit netfilter hooks. But
no matter what hook I chose, the best I could get was forward_in /
forward_out.

2. various traffic selectors - I've used mac address, ip address, and
physical interface (all of the different interfaces) to set the
secmark labels.

3. ebtables - secmark support seems totally broken in ebtables, both
what is in RHEL 8.6 and upstream. It's in the options parsing.

4. nftables - to work around ebtables I used nftables bridge family to
label layer 2 traffic. None of the hooks would yield anything other
than forward_in / forward_out.

5. routing - I placed the VMs on separate subnets and added multiple
addresses to the bridge. My thought was that forcing the traffic to
route would push it up to layer 3 and would make it hit the
postrouting hook (this may be a completely broken mental model of
things).

You can see how I'm doing this with some very rough / poorly
documented scripts at
https://github.com/BigBadWolfSecurity/kvm-access-control. VMs are
created / managed through libvirt / virt-manager.

I would appreciate any insight into what might be going on. This is
mainly tested on a RHEL 8.6 box, but some testing has been done on a
Fedora 36 box as well.

Thanks - Karl

Re: KVM / virtual networking access control

[Paul Moore]

On Fri, Jul 29, 2022 at 11:36 PM Karl MacMillan
<karl@bigbadwolfsecurity.com> wrote:
>
> Hi,
>
> I'm trying to do some access control over the networking between KVM
> virtual machines using Secmark and things are not working as I expect.
> The key is that this is _between_ virtual machines, not just ingress /
> egress to the virtual machines from external address.
>
> Quick summary: I'm connecting two VMs together using veth pairs and a
> virtual bridge and I only ever see:
>
>     allow unlabeled_t bare_packet_a_t:packet { forward_in forward_out };

Just to make sure I'm understanding this correctly, we are talking
about the SELinux controls from the perspective of the
host/hypervisor, yes?

Assuming that is the case, the "packet/{ forward_in forward_out }"
permissions look correct to me.  In the case of the secmark forwarding
rules you basically have this:

  allow <peer> <secmark>:packet { forward_in };
  allow <peer> <secmark>:packet { forward_out };

... and I'm guessing you haven't setup any peer labeling between the
two guests so you are seeing unlabeled_t for "<peer>".

> What I want is packet { send recv } because that would be against the
> label of the KVM process rather than the peer label (which on the same
> system would always be unlabeled - I see no way to use CIPSO/CALIPSO
> to get peer labels to work here).

The "packet/{ send recv }" controls are for packets that originate and
terminate on the node applying the policy; think of your typical
client/server nodes.  The "packet/{ forward_in forward_out }" controls
are for packets that are passing through the node applying the policy,
e.g. intermediate nodes such as bridges and switches ... even software
bridges like that found in hypervisors ;)

> I would appreciate any insight into what might be going on. This is
> mainly tested on a RHEL 8.6 box, but some testing has been done on a
> Fedora 36 box as well.

I don't know all the details of your setup, or the security goals you
are trying to enforce, but since it looks like you are bridging the
network traffic between the guests, you might want to try setting up
the NetLabel static/fallback mechanism to assign a peer label to
traffic on those interfaces (no more unlabeled_t in the secmark
forwarding rules).  Adding the peer labeling information will also
trigger the network peer label controls in the kernel so you will see
additional "netif/{ ingress egress}" and "node/{ recvfrom sendto }"
checks.

Here are few slides that might be helpful (they are older, but still relevant):

https://www.paul-moore.com/docs/selinux_network_controls-pmoore-122012-r1.pdf
https://www.paul-moore.com/docs/labeled_networking-lfjapan-07092008.pdf

-- 
paul-moore.com

Re: KVM / virtual networking access control

[Karl MacMillan]

On Sun, Jul 31, 2022 at 12:45 PM Paul Moore <paul@paul-moore.com> wrote:
>
> On Fri, Jul 29, 2022 at 11:36 PM Karl MacMillan
> <karl@bigbadwolfsecurity.com> wrote:
> >
> > Hi,
> >
> > I'm trying to do some access control over the networking between KVM
> > virtual machines using Secmark and things are not working as I expect.
> > The key is that this is _between_ virtual machines, not just ingress /
> > egress to the virtual machines from external address.
> >
> > Quick summary: I'm connecting two VMs together using veth pairs and a
> > virtual bridge and I only ever see:
> >
> >     allow unlabeled_t bare_packet_a_t:packet { forward_in forward_out };
>
> Just to make sure I'm understanding this correctly, we are talking
> about the SELinux controls from the perspective of the
> host/hypervisor, yes?
>

Yes

> Assuming that is the case, the "packet/{ forward_in forward_out }"
> permissions look correct to me.  In the case of the secmark forwarding
> rules you basically have this:
>
>   allow <peer> <secmark>:packet { forward_in };
>   allow <peer> <secmark>:packet { forward_out };
>

So I get this - that the host is basically forwarding packets
generated by the virtual machines. And I'll say that was really the
piece I was missing, so thanks for clearing that up.

I'm curious under what other circumstances processes can generate
network traffic that are treated in this way. Packet sockets? Tun /
tap devices?

I'm sure I'm preaching to the choir, but some more documentation on
this would be great. I know you've done so much (and those pdfs
helped). But I was aware of much of that content, I just didn't know
how it all applied to this specific area. See - part of why I was
confused here was that here we have these KVM processes where we
already _know_ the process context. It just seemed unimaginable that
what we have to do is a) enable some sort of labeled networking and b)
tell the kernel all over what the label of the peer is using network
information rather than process information it already knows! I know
you've added special casing for the loopback devices so that this is
not required there. But I kept thinking that surely there must be a
way to trigger something more like the packet { send recv }
permissions because we have a process generating network traffic on
the same host that is trying to enforce those controls.

> ... and I'm guessing you haven't setup any peer labeling between the
> two guests so you are seeing unlabeled_t for "<peer>".
>

I see your suggestion below (which worked), but I will say that I
tried CIPSO / CALIPSO before sending this email. I just did not know
how to get the fallback labeling to do what I needed.

> > What I want is packet { send recv } because that would be against the
> > label of the KVM process rather than the peer label (which on the same
> > system would always be unlabeled - I see no way to use CIPSO/CALIPSO
> > to get peer labels to work here).
>
> The "packet/{ send recv }" controls are for packets that originate and
> terminate on the node applying the policy; think of your typical
> client/server nodes.  The "packet/{ forward_in forward_out }" controls
> are for packets that are passing through the node applying the policy,
> e.g. intermediate nodes such as bridges and switches ... even software
> bridges like that found in hypervisors ;)
>
> > I would appreciate any insight into what might be going on. This is
> > mainly tested on a RHEL 8.6 box, but some testing has been done on a
> > Fedora 36 box as well.
>
> I don't know all the details of your setup, or the security goals you
> are trying to enforce, but since it looks like you are bridging the
> network traffic between the guests, you might want to try setting up
> the NetLabel static/fallback mechanism to assign a peer label to
> traffic on those interfaces (no more unlabeled_t in the secmark
> forwarding rules).  Adding the peer labeling information will also
> trigger the network peer label controls in the kernel so you will see
> additional "netif/{ ingress egress}" and "node/{ recvfrom sendto }"
> checks.
>

So this works . . . with some interesting caveats. What works for me is:

1. Setting CIPSO fallback labels on the bridge interface and using IP
selectors to set labels per-VM.
2. Using physdev-in in SECMARK rules to set packet labels on packets
based on which veth pair is connected to the bridge (and I can use
other selectors for SECMARK - but this one is nice).

So at this point I can do what I want (except the namespace issue
below) - which is to control how my VMs talk on a single box and build
pipelines of VMs or other interesting topologies. I can also do some
coarse-grained enforcement using netif {ingress egress} which is also
wnice.

What is not working:

1. I cannot do CIPSO fallback labeling based on either the macvtap or
the veth pairs. It will accept the fallback rules, but I still get
unlabeled for the peer. This is not great because it means that I have
to use labeling based upon IP addresses to differentiate VMs (vs
interfaces), which means that my labeling is dependent on the
configuration inside of my VMs. I don't like having to trust my VMs at
all. Or I could add a lot more bridges I guess, but I've not tried
that.
2. I cannot get netlabelct to run inside of a network namespace, so I
can't do fallback labeling of bridges in a namespace. I get the error
"netlabelctl: error, failed to initialize the NetLabel library". This
makes many of the topologies I want to do much harder.

I have an overall concern, which is back to what I was saying above
about process labels. At that point, it doesn't feel like there is a
lot of advantage of doing SELinux enforcement of the networking vs
just the iptables network enforcement. Much of the power of the
SELinux enforcement, to me, is to tie iptables to process labels or,
with labeled networking, to transmit remote process labels. Here . . .
well, we are generating peer labels on box based on network properties
and then writing rules that are based solely on information that
iptables already had (ip addresses and interface names). Am I missing
something?

Thanks so much for all of your help,

Karl

> Here are few slides that might be helpful (they are older, but still relevant):
>
> https://www.paul-moore.com/docs/selinux_network_controls-pmoore-122012-r1.pdf
> https://www.paul-moore.com/docs/labeled_networking-lfjapan-07092008.pdf
>
> --
> paul-moore.com

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_virtualization/configuring-virtual-machine-network-connections_configuring-and-managing-virtualization#understanding-virtual-networking-overview_configuring-virtual-machine-network-connections

Re: KVM / virtual networking access control

[Paul Moore]

On Tue, Aug 9, 2022 at 2:45 PM Karl MacMillan
<karl@bigbadwolfsecurity.com> wrote:
> On Sun, Jul 31, 2022 at 12:45 PM Paul Moore <paul@paul-moore.com> wrote:
> > On Fri, Jul 29, 2022 at 11:36 PM Karl MacMillan
> > <karl@bigbadwolfsecurity.com> wrote:

...

> > > Quick summary: I'm connecting two VMs together using veth pairs and a
> > > virtual bridge and I only ever see:
> > >
> > >     allow unlabeled_t bare_packet_a_t:packet { forward_in forward_out };
> >
> > Just to make sure I'm understanding this correctly, we are talking
> > about the SELinux controls from the perspective of the
> > host/hypervisor, yes?
> >
>
> Yes
>
> > Assuming that is the case, the "packet/{ forward_in forward_out }"
> > permissions look correct to me.  In the case of the secmark forwarding
> > rules you basically have this:
> >
> >   allow <peer> <secmark>:packet { forward_in };
> >   allow <peer> <secmark>:packet { forward_out };
> >
>
> So I get this - that the host is basically forwarding packets
> generated by the virtual machines. And I'll say that was really the
> piece I was missing, so thanks for clearing that up.
>
> I'm curious under what other circumstances processes can generate
> network traffic that are treated in this way. Packet sockets? Tun /
> tap devices?

I've never thought to catalog all of the different ways a local
process could send network traffic into the stack in such a way that
it isn't treated as a local connection so I can't give you a complete
list off the top of my head.  However, I can tell you that the
packet/forward_in check will be triggered via the netfilter
NF_INET_FORWARD hook and the packet/forward_out check will be
triggered via the netfilter NF_INET_POST_ROUTING hook (there is logic
in the selinux_ip_postroute() function to determine if they packet was
locally generated or forwarded through the stack).  I realize that
probably isn't very helpful, but it's the best I have without digging
through a bunch of code right now - sorry.

> I'm sure I'm preaching to the choir, but some more documentation on
> this would be great. I know you've done so much (and those pdfs
> helped). But I was aware of much of that content, I just didn't know
> how it all applied to this specific area.

Yes, we can definitely do better here.  The SELinux Notebook has some
information on labeled networking, but I think it could be better.

I think part of what makes this so challenging is that in order to
configure this properly you need to have a good understanding of the
networking concepts, their implementation in the Linux Kernel, and
SELinux's various networking controls.  As you likely know at this
point, that's a lot of stuff :)

Oh, and don't forget the unique nature of any given solution's network
topology.  That dooms any hope of creating a useful template, recipe,
or some other general solution that could be used by admins to help
them deal with some of the complexity.

> See - part of why I was
> confused here was that here we have these KVM processes where we
> already _know_ the process context.

The important detail is that because of ... well, networking ... all
we really can rely on when making access control decisions at the
packet level is the packet itself.  If that packet doesn't have a
on-the-wire security label, or some other security hint (see labeled
IPsec, but don't use it), we need to fallback to other mechanisms if
we want to assign that packet a security label.  If the packet
originated from a local socket that the kernel knows about, it can
obtain the security label from the socket, but that's a special case.

> It just seemed unimaginable that
> what we have to do is a) enable some sort of labeled networking and b)
> tell the kernel all over what the label of the peer is using network
> information rather than process information it already knows!

Ah, but the part you are forgetting is that in your configuration the
kernel can't match the packets with the originating process.  I know
it sounds silly, but that's the wacky world of networking for you :)

> I know
> you've added special casing for the loopback devices so that this is
> not required there.

I think you are confusing the special full-context NetLabel trick that
we can do over loopback with connecting bridged network packets to a
local originating process.  They are *very* different things.

> But I kept thinking that surely there must be a
> way to trigger something more like the packet { send recv }
> permissions because we have a process generating network traffic on
> the same host that is trying to enforce those controls.

Sadly that is not how it works, or is likely to ever work.  Perhaps
thinking of it this way may help: what is actually generating those
packets that you want to control?  Is is the QEMU/KVM process on the
host system, or a virtualized process running inside of that QEMU/KVM
process?

(SPOILER: it's the latter, which makes all the difference)

> > I don't know all the details of your setup, or the security goals you
> > are trying to enforce, but since it looks like you are bridging the
> > network traffic between the guests, you might want to try setting up
> > the NetLabel static/fallback mechanism to assign a peer label to
> > traffic on those interfaces (no more unlabeled_t in the secmark
> > forwarding rules).  Adding the peer labeling information will also
> > trigger the network peer label controls in the kernel so you will see
> > additional "netif/{ ingress egress}" and "node/{ recvfrom sendto }"
> > checks.
> >
>
> So this works . . . with some interesting caveats. What works for me is:
>
> 1. Setting CIPSO fallback labels on the bridge interface and using IP
> selectors to set labels per-VM.
> 2. Using physdev-in in SECMARK rules to set packet labels on packets
> based on which veth pair is connected to the bridge (and I can use
> other selectors for SECMARK - but this one is nice).
>
> So at this point I can do what I want (except the namespace issue
> below) - which is to control how my VMs talk on a single box and build
> pipelines of VMs or other interesting topologies. I can also do some
> coarse-grained enforcement using netif {ingress egress} which is also
> wnice.

I'm glad you got it working.

> What is not working:
>
> 1. I cannot do CIPSO fallback labeling based on either the macvtap or
> the veth pairs.

Could you create an issue on the SELinux kernel repo on GH with as
much detail as you can gather?  I know macvtap is a bit "special" due
to how it is implemented, and veth pairs might have a similar issue
(or simply lack the netfilter hooks).

* https://github.com/SELinuxProject/selinux-kernel

> 2. I cannot get netlabelct to run inside of a network namespace, so I
> can't do fallback labeling of bridges in a namespace. I get the error
> "netlabelctl: error, failed to initialize the NetLabel library". This
> makes many of the topologies I want to do much harder.

That is likely due to netlabelctl being unable to talk to the NetLabel
subsystem in the kernel, which I'm guessing is due to the NetLabel
subsystem only registering itself in the initial network namespace.
That's something we will need to change at some point, but there are
bigger issues around namespacing that would likely need to be
addressed first.

> I have an overall concern, which is back to what I was saying above
> about process labels. At that point, it doesn't feel like there is a
> lot of advantage of doing SELinux enforcement of the networking vs
> just the iptables network enforcement.

Arguably I believe there is some value in having the allowed access
defined in the policy, although admittedly the labeling is still done
outside of policy much like we do with a context mount.

> Much of the power of the
> SELinux enforcement, to me, is to tie iptables to process labels or,
> with labeled networking, to transmit remote process labels.

Which works when the kernel has visibility to the process where the
network traffic originated.  In the VM case, the kernel's visibility
stops at the QEMU (or similar) process, it can not see the virtualized
process which created the network traffic.

> Here . . .
> well, we are generating peer labels on box based on network properties
> and then writing rules that are based solely on information that
> iptables already had (ip addresses and interface names). Am I missing
> something?

I believe my comments above have provided a bit more background on why
things are the way they are, but if things still don't make sense let
us know :)

-- 
paul-moore.com