Re: KVM / virtual networking access control

[Karl MacMillan <karl@bigbadwolfsecurity.com>]

On Sun, Jul 31, 2022 at 12:45 PM Paul Moore <paul@paul-moore.com> wrote:
>
> On Fri, Jul 29, 2022 at 11:36 PM Karl MacMillan
> <karl@bigbadwolfsecurity.com> wrote:
> >
> > Hi,
> >
> > I'm trying to do some access control over the networking between KVM
> > virtual machines using Secmark and things are not working as I expect.
> > The key is that this is _between_ virtual machines, not just ingress /
> > egress to the virtual machines from external address.
> >
> > Quick summary: I'm connecting two VMs together using veth pairs and a
> > virtual bridge and I only ever see:
> >
> >     allow unlabeled_t bare_packet_a_t:packet { forward_in forward_out };
>
> Just to make sure I'm understanding this correctly, we are talking
> about the SELinux controls from the perspective of the
> host/hypervisor, yes?
>

Yes

> Assuming that is the case, the "packet/{ forward_in forward_out }"
> permissions look correct to me.  In the case of the secmark forwarding
> rules you basically have this:
>
>   allow <peer> <secmark>:packet { forward_in };
>   allow <peer> <secmark>:packet { forward_out };
>

So I get this - that the host is basically forwarding packets
generated by the virtual machines. And I'll say that was really the
piece I was missing, so thanks for clearing that up.

I'm curious under what other circumstances processes can generate
network traffic that are treated in this way. Packet sockets? Tun /
tap devices?

I'm sure I'm preaching to the choir, but some more documentation on
this would be great. I know you've done so much (and those pdfs
helped). But I was aware of much of that content, I just didn't know
how it all applied to this specific area. See - part of why I was
confused here was that here we have these KVM processes where we
already _know_ the process context. It just seemed unimaginable that
what we have to do is a) enable some sort of labeled networking and b)
tell the kernel all over what the label of the peer is using network
information rather than process information it already knows! I know
you've added special casing for the loopback devices so that this is
not required there. But I kept thinking that surely there must be a
way to trigger something more like the packet { send recv }
permissions because we have a process generating network traffic on
the same host that is trying to enforce those controls.

> ... and I'm guessing you haven't setup any peer labeling between the
> two guests so you are seeing unlabeled_t for "<peer>".
>

I see your suggestion below (which worked), but I will say that I
tried CIPSO / CALIPSO before sending this email. I just did not know
how to get the fallback labeling to do what I needed.

> > What I want is packet { send recv } because that would be against the
> > label of the KVM process rather than the peer label (which on the same
> > system would always be unlabeled - I see no way to use CIPSO/CALIPSO
> > to get peer labels to work here).
>
> The "packet/{ send recv }" controls are for packets that originate and
> terminate on the node applying the policy; think of your typical
> client/server nodes.  The "packet/{ forward_in forward_out }" controls
> are for packets that are passing through the node applying the policy,
> e.g. intermediate nodes such as bridges and switches ... even software
> bridges like that found in hypervisors ;)
>
> > I would appreciate any insight into what might be going on. This is
> > mainly tested on a RHEL 8.6 box, but some testing has been done on a
> > Fedora 36 box as well.
>
> I don't know all the details of your setup, or the security goals you
> are trying to enforce, but since it looks like you are bridging the
> network traffic between the guests, you might want to try setting up
> the NetLabel static/fallback mechanism to assign a peer label to
> traffic on those interfaces (no more unlabeled_t in the secmark
> forwarding rules).  Adding the peer labeling information will also
> trigger the network peer label controls in the kernel so you will see
> additional "netif/{ ingress egress}" and "node/{ recvfrom sendto }"
> checks.
>

So this works . . . with some interesting caveats. What works for me is:

1. Setting CIPSO fallback labels on the bridge interface and using IP
selectors to set labels per-VM.
2. Using physdev-in in SECMARK rules to set packet labels on packets
based on which veth pair is connected to the bridge (and I can use
other selectors for SECMARK - but this one is nice).

So at this point I can do what I want (except the namespace issue
below) - which is to control how my VMs talk on a single box and build
pipelines of VMs or other interesting topologies. I can also do some
coarse-grained enforcement using netif {ingress egress} which is also
wnice.

What is not working:

1. I cannot do CIPSO fallback labeling based on either the macvtap or
the veth pairs. It will accept the fallback rules, but I still get
unlabeled for the peer. This is not great because it means that I have
to use labeling based upon IP addresses to differentiate VMs (vs
interfaces), which means that my labeling is dependent on the
configuration inside of my VMs. I don't like having to trust my VMs at
all. Or I could add a lot more bridges I guess, but I've not tried
that.
2. I cannot get netlabelct to run inside of a network namespace, so I
can't do fallback labeling of bridges in a namespace. I get the error
"netlabelctl: error, failed to initialize the NetLabel library". This
makes many of the topologies I want to do much harder.

I have an overall concern, which is back to what I was saying above
about process labels. At that point, it doesn't feel like there is a
lot of advantage of doing SELinux enforcement of the networking vs
just the iptables network enforcement. Much of the power of the
SELinux enforcement, to me, is to tie iptables to process labels or,
with labeled networking, to transmit remote process labels. Here . . .
well, we are generating peer labels on box based on network properties
and then writing rules that are based solely on information that
iptables already had (ip addresses and interface names). Am I missing
something?

Thanks so much for all of your help,

Karl

> Here are few slides that might be helpful (they are older, but still relevant):
>
> https://www.paul-moore.com/docs/selinux_network_controls-pmoore-122012-r1.pdf
> https://www.paul-moore.com/docs/labeled_networking-lfjapan-07092008.pdf
>
> --
> paul-moore.com

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_virtualization/configuring-virtual-machine-network-connections_configuring-and-managing-virtualization#understanding-virtual-networking-overview_configuring-virtual-machine-network-connections