KVM / virtual networking access control

[Karl MacMillan <karl@bigbadwolfsecurity.com>]

Hi,

I'm trying to do some access control over the networking between KVM
virtual machines using Secmark and things are not working as I expect.
The key is that this is _between_ virtual machines, not just ingress /
egress to the virtual machines from external address.

Quick summary: I'm connecting two VMs together using veth pairs and a
virtual bridge and I only ever see:

    allow unlabeled_t bare_packet_a_t:packet { forward_in forward_out };

What I want is packet { send recv } because that would be against the
label of the KVM process rather than the peer label (which on the same
system would always be unlabeled - I see no way to use CIPSO/CALIPSO
to get peer labels to work here).

I can see that the packet send / recv LSM hook is in netfilter
postrouting code, but I simply cannot trigger this with the setup that
I have. Here are things that I have tried:

1. br_netfilter - I thought initially that my problem was that the
traffic was at layer 2 and was not traversing the netfilter hooks.
This was, in fact, a problem with my initial setup (both VMs on the
same subnet) and br_netfilter got traffic to hit netfilter hooks. But
no matter what hook I chose, the best I could get was forward_in /
forward_out.

2. various traffic selectors - I've used mac address, ip address, and
physical interface (all of the different interfaces) to set the
secmark labels.

3. ebtables - secmark support seems totally broken in ebtables, both
what is in RHEL 8.6 and upstream. It's in the options parsing.

4. nftables - to work around ebtables I used nftables bridge family to
label layer 2 traffic. None of the hooks would yield anything other
than forward_in / forward_out.

5. routing - I placed the VMs on separate subnets and added multiple
addresses to the bridge. My thought was that forcing the traffic to
route would push it up to layer 3 and would make it hit the
postrouting hook (this may be a completely broken mental model of
things).

You can see how I'm doing this with some very rough / poorly
documented scripts at
https://github.com/BigBadWolfSecurity/kvm-access-control. VMs are
created / managed through libvirt / virt-manager.

I would appreciate any insight into what might be going on. This is
mainly tested on a RHEL 8.6 box, but some testing has been done on a
Fedora 36 box as well.

Thanks - Karl