Networking issue with l3s when using systemd

Networking issue with l3s when using systemd

[Matt Spencer]

[-- Attachment #1: Type: text/plain, Size: 665 bytes --]

Hi all

There seems to be a networking problem with k3s when using systemd.  The problem manifests in that none of the kube-system management containers are able to communicate with eachother.

The root cause seems to be that systemd-networking is actively managing veth interfaces created by k3s/flannel.  This happens because of '/lib/systemd/network/80-wired.network' added by the systemd recipe, which is matching on Type=ether.

My fix is to modify the 80-wired.network to add 'Name=eth*'.  With this in place, k3s works as expected.

I am not sure what the correct upstream solution should be for Yocto?  Your help would be appreciated.

/Matt

[-- Attachment #2: Type: text/html, Size: 721 bytes --]

Re: [meta-virtualization] Networking issue with l3s when using systemd

[Bruce Ashfield]

On Mon, Jun 14, 2021 at 6:27 AM Matt Spencer <matt.spencer@arm.com> wrote:
>
> Hi all
>
> There seems to be a networking problem with k3s when using systemd.  The problem manifests in that none of the kube-system management containers are able to communicate with eachother.
>
> The root cause seems to be that systemd-networking is actively managing veth interfaces created by k3s/flannel.  This happens because of '/lib/systemd/network/80-wired.network' added by the systemd recipe, which is matching on Type=ether.
>
> My fix is to modify the 80-wired.network to add 'Name=eth*'.  With this in place, k3s works as expected.
>
> I am not sure what the correct upstream solution should be for Yocto?  Your help would be appreciated.
>

This particular issue is known, in the sense that we have run into it before.

At a minimum, I need to warn about it in the k3s README files.

I created the cni bbclass to manage potentially conflicting networking
configs on that front, but systemd-networking is yet another variable.

We don't want to globally make it conflict, since someone might have a
working networkd config that they want to use, and exactly how
networking is set up, tends to be more of a distro feature. So the
recipes need to tread carefully.

Which takes me back to the README, and an enhancement to the
cni-networking bbclass to be more generic and pick up / warn on
configuration issues like this.

bruce


> /Matt
> 
>


--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] Networking issue with l3s when using systemd

[Matt Spencer]

[-- Attachment #1: Type: text/plain, Size: 3121 bytes --]

Hey Bruce,

I understand that changing things can be difficult once launched.

But looking at the systemd documentation here https://wiki.archlinux.org/title/systemd-networkd and in the note that says 'Devices can also be matched by their type. E.g. Type=ether for Ethernet, Type=wlan for Wi-Fi and Type=wwan for WWAN. Note that Type=ether will also match virtual Ethernet interfaces (veth*), which may be undesirable.'

I have created a patch that modifies poky's meta/recipes-core/systemd/systemd-conf/wired.network to add '+Name=!veth*' to the Match clause which the team at Arm are reviewing with a view to upstream.

I feel that this should have been the default behaviour anyway?

/Matt
________________________________
From: meta-virtualization@lists.yoctoproject.org <meta-virtualization@lists.yoctoproject.org> on behalf of Bruce Ashfield via lists.yoctoproject.org <bruce.ashfield=gmail.com@lists.yoctoproject.org>
Sent: 14 June 2021 15:24
To: Matt Spencer <Matt.Spencer@arm.com>
Cc: meta-virtualization@lists.yoctoproject.org <meta-virtualization@lists.yoctoproject.org>
Subject: Re: [meta-virtualization] Networking issue with l3s when using systemd

On Mon, Jun 14, 2021 at 6:27 AM Matt Spencer <matt.spencer@arm.com> wrote:
>
> Hi all
>
> There seems to be a networking problem with k3s when using systemd.  The problem manifests in that none of the kube-system management containers are able to communicate with eachother.
>
> The root cause seems to be that systemd-networking is actively managing veth interfaces created by k3s/flannel.  This happens because of '/lib/systemd/network/80-wired.network' added by the systemd recipe, which is matching on Type=ether.
>
> My fix is to modify the 80-wired.network to add 'Name=eth*'.  With this in place, k3s works as expected.
>
> I am not sure what the correct upstream solution should be for Yocto?  Your help would be appreciated.
>

This particular issue is known, in the sense that we have run into it before.

At a minimum, I need to warn about it in the k3s README files.

I created the cni bbclass to manage potentially conflicting networking
configs on that front, but systemd-networking is yet another variable.

We don't want to globally make it conflict, since someone might have a
working networkd config that they want to use, and exactly how
networking is set up, tends to be more of a distro feature. So the
recipes need to tread carefully.

Which takes me back to the README, and an enhancement to the
cni-networking bbclass to be more generic and pick up / warn on
configuration issues like this.

bruce


> /Matt
>
>


--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[-- Attachment #2: Type: text/html, Size: 6265 bytes --]

Re: [meta-virtualization] Networking issue with l3s when using systemd

[Bruce Ashfield]

On Mon, Jun 14, 2021 at 10:59 AM Matt Spencer <Matt.Spencer@arm.com> wrote:
>
> Hey Bruce,
>
> I understand that changing things can be difficult once launched.
>
> But looking at the systemd documentation here https://wiki.archlinux.org/title/systemd-networkd and in the note that says 'Devices can also be matched by their type. E.g. Type=ether for Ethernet, Type=wlan for Wi-Fi and Type=wwan for WWAN. Note that Type=ether will also match virtual Ethernet interfaces (veth*), which may be undesirable.'
>
> I have created a patch that modifies poky's meta/recipes-core/systemd/systemd-conf/wired.network to add '+Name=!veth*' to the Match clause which the team at Arm are reviewing with a view to upstream.
>
> I feel that this should have been the default behaviour anyway?

Perhaps, but it is probably a good idea to have the default as broad
as possible.  If someone comes up with an alternate requirement, you'd
end up having a harder time keeping them both working in oe-core.

We could also carry a bbappend in meta-virtualization that triggers
when "virtualization" is in the distro features. That way we are
providing an override that makes sense within the context of the layer
that is providing it, and the distro features that are enabled.

Bruce

>
> /Matt
> ________________________________
> From: meta-virtualization@lists.yoctoproject.org <meta-virtualization@lists.yoctoproject.org> on behalf of Bruce Ashfield via lists.yoctoproject.org <bruce.ashfield=gmail.com@lists.yoctoproject.org>
> Sent: 14 June 2021 15:24
> To: Matt Spencer <Matt.Spencer@arm.com>
> Cc: meta-virtualization@lists.yoctoproject.org <meta-virtualization@lists.yoctoproject.org>
> Subject: Re: [meta-virtualization] Networking issue with l3s when using systemd
>
> On Mon, Jun 14, 2021 at 6:27 AM Matt Spencer <matt.spencer@arm.com> wrote:
> >
> > Hi all
> >
> > There seems to be a networking problem with k3s when using systemd.  The problem manifests in that none of the kube-system management containers are able to communicate with eachother.
> >
> > The root cause seems to be that systemd-networking is actively managing veth interfaces created by k3s/flannel.  This happens because of '/lib/systemd/network/80-wired.network' added by the systemd recipe, which is matching on Type=ether.
> >
> > My fix is to modify the 80-wired.network to add 'Name=eth*'.  With this in place, k3s works as expected.
> >
> > I am not sure what the correct upstream solution should be for Yocto?  Your help would be appreciated.
> >
>
> This particular issue is known, in the sense that we have run into it before.
>
> At a minimum, I need to warn about it in the k3s README files.
>
> I created the cni bbclass to manage potentially conflicting networking
> configs on that front, but systemd-networking is yet another variable.
>
> We don't want to globally make it conflict, since someone might have a
> working networkd config that they want to use, and exactly how
> networking is set up, tends to be more of a distro feature. So the
> recipes need to tread carefully.
>
> Which takes me back to the README, and an enhancement to the
> cni-networking bbclass to be more generic and pick up / warn on
> configuration issues like this.
>
> bruce
>
>
> > /Matt
> >
> >
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.



--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] Networking issue with l3s when using systemd

[Richard Purdie]

On Mon, 2021-06-14 at 13:59 -0400, Bruce Ashfield wrote:
> On Mon, Jun 14, 2021 at 10:59 AM Matt Spencer <Matt.Spencer@arm.com> wrote:
> > 
> > Hey Bruce,
> > 
> > I understand that changing things can be difficult once launched.
> > 
> > But looking at the systemd documentation here https://wiki.archlinux.org/title/systemd-networkd and in the note that says 'Devices can also be matched by their type. E.g. Type=ether for Ethernet, Type=wlan for Wi-Fi and Type=wwan for WWAN. Note that Type=ether will also match virtual Ethernet interfaces (veth*), which may be undesirable.'
> > 
> > I have created a patch that modifies poky's meta/recipes-core/systemd/systemd-conf/wired.network to add '+Name=!veth*' to the Match clause which the team at Arm are reviewing with a view to upstream.
> > 
> > I feel that this should have been the default behaviour anyway?
> 
> Perhaps, but it is probably a good idea to have the default as broad
> as possible.  If someone comes up with an alternate requirement, you'd
> end up having a harder time keeping them both working in oe-core.
> 
> We could also carry a bbappend in meta-virtualization that triggers
> when "virtualization" is in the distro features. That way we are
> providing an override that makes sense within the context of the layer
> that is providing it, and the distro features that are enabled.

FWIW I suspect the veth change should be ok for OE-Core if I understand 
things correctly.

Cheers,

Richard

Re: [meta-virtualization] Networking issue with l3s when using systemd

[Bruce Ashfield]

On Mon, Jun 14, 2021 at 4:33 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2021-06-14 at 13:59 -0400, Bruce Ashfield wrote:
> > On Mon, Jun 14, 2021 at 10:59 AM Matt Spencer <Matt.Spencer@arm.com> wrote:
> > >
> > > Hey Bruce,
> > >
> > > I understand that changing things can be difficult once launched.
> > >
> > > But looking at the systemd documentation here https://wiki.archlinux.org/title/systemd-networkd and in the note that says 'Devices can also be matched by their type. E.g. Type=ether for Ethernet, Type=wlan for Wi-Fi and Type=wwan for WWAN. Note that Type=ether will also match virtual Ethernet interfaces (veth*), which may be undesirable.'
> > >
> > > I have created a patch that modifies poky's meta/recipes-core/systemd/systemd-conf/wired.network to add '+Name=!veth*' to the Match clause which the team at Arm are reviewing with a view to upstream.
> > >
> > > I feel that this should have been the default behaviour anyway?
> >
> > Perhaps, but it is probably a good idea to have the default as broad
> > as possible.  If someone comes up with an alternate requirement, you'd
> > end up having a harder time keeping them both working in oe-core.
> >
> > We could also carry a bbappend in meta-virtualization that triggers
> > when "virtualization" is in the distro features. That way we are
> > providing an override that makes sense within the context of the layer
> > that is providing it, and the distro features that are enabled.
>
> FWIW I suspect the veth change should be ok for OE-Core if I understand
> things correctly.

I'm sure we'll hear if someone is expecting networkd to manage their
veth devices :D

Either way works for me, I just lean towards keeping core very general
and not tailored to any particular use case.

If you are doing virtualization, and in particular kubernetes based,
you need to have a close look at your networking regardless.

Cheers,

Bruce

>
> Cheers,
>
> Richard
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II