* Networking issue with l3s when using systemd @ 2021-06-14 10:27 Matt Spencer 2021-06-14 14:24 ` [meta-virtualization] " Bruce Ashfield 0 siblings, 1 reply; 6+ messages in thread From: Matt Spencer @ 2021-06-14 10:27 UTC (permalink / raw) To: meta-virtualization [-- Attachment #1: Type: text/plain, Size: 665 bytes --] Hi all There seems to be a networking problem with k3s when using systemd. The problem manifests in that none of the kube-system management containers are able to communicate with eachother. The root cause seems to be that systemd-networking is actively managing veth interfaces created by k3s/flannel. This happens because of '/lib/systemd/network/80-wired.network' added by the systemd recipe, which is matching on Type=ether. My fix is to modify the 80-wired.network to add 'Name=eth*'. With this in place, k3s works as expected. I am not sure what the correct upstream solution should be for Yocto? Your help would be appreciated. /Matt [-- Attachment #2: Type: text/html, Size: 721 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [meta-virtualization] Networking issue with l3s when using systemd 2021-06-14 10:27 Networking issue with l3s when using systemd Matt Spencer @ 2021-06-14 14:24 ` Bruce Ashfield 2021-06-14 14:59 ` Matt Spencer 0 siblings, 1 reply; 6+ messages in thread From: Bruce Ashfield @ 2021-06-14 14:24 UTC (permalink / raw) To: Matt Spencer; +Cc: meta-virtualization On Mon, Jun 14, 2021 at 6:27 AM Matt Spencer <matt.spencer@arm.com> wrote: > > Hi all > > There seems to be a networking problem with k3s when using systemd. The problem manifests in that none of the kube-system management containers are able to communicate with eachother. > > The root cause seems to be that systemd-networking is actively managing veth interfaces created by k3s/flannel. This happens because of '/lib/systemd/network/80-wired.network' added by the systemd recipe, which is matching on Type=ether. > > My fix is to modify the 80-wired.network to add 'Name=eth*'. With this in place, k3s works as expected. > > I am not sure what the correct upstream solution should be for Yocto? Your help would be appreciated. > This particular issue is known, in the sense that we have run into it before. At a minimum, I need to warn about it in the k3s README files. I created the cni bbclass to manage potentially conflicting networking configs on that front, but systemd-networking is yet another variable. We don't want to globally make it conflict, since someone might have a working networkd config that they want to use, and exactly how networking is set up, tends to be more of a distro feature. So the recipes need to tread carefully. Which takes me back to the README, and an enhancement to the cni-networking bbclass to be more generic and pick up / warn on configuration issues like this. bruce > /Matt > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [meta-virtualization] Networking issue with l3s when using systemd 2021-06-14 14:24 ` [meta-virtualization] " Bruce Ashfield @ 2021-06-14 14:59 ` Matt Spencer 2021-06-14 17:59 ` Bruce Ashfield 0 siblings, 1 reply; 6+ messages in thread From: Matt Spencer @ 2021-06-14 14:59 UTC (permalink / raw) To: bruce.ashfield; +Cc: meta-virtualization [-- Attachment #1: Type: text/plain, Size: 3121 bytes --] Hey Bruce, I understand that changing things can be difficult once launched. But looking at the systemd documentation here https://wiki.archlinux.org/title/systemd-networkd and in the note that says 'Devices can also be matched by their type. E.g. Type=ether for Ethernet, Type=wlan for Wi-Fi and Type=wwan for WWAN. Note that Type=ether will also match virtual Ethernet interfaces (veth*), which may be undesirable.' I have created a patch that modifies poky's meta/recipes-core/systemd/systemd-conf/wired.network to add '+Name=!veth*' to the Match clause which the team at Arm are reviewing with a view to upstream. I feel that this should have been the default behaviour anyway? /Matt ________________________________ From: meta-virtualization@lists.yoctoproject.org <meta-virtualization@lists.yoctoproject.org> on behalf of Bruce Ashfield via lists.yoctoproject.org <bruce.ashfield=gmail.com@lists.yoctoproject.org> Sent: 14 June 2021 15:24 To: Matt Spencer <Matt.Spencer@arm.com> Cc: meta-virtualization@lists.yoctoproject.org <meta-virtualization@lists.yoctoproject.org> Subject: Re: [meta-virtualization] Networking issue with l3s when using systemd On Mon, Jun 14, 2021 at 6:27 AM Matt Spencer <matt.spencer@arm.com> wrote: > > Hi all > > There seems to be a networking problem with k3s when using systemd. The problem manifests in that none of the kube-system management containers are able to communicate with eachother. > > The root cause seems to be that systemd-networking is actively managing veth interfaces created by k3s/flannel. This happens because of '/lib/systemd/network/80-wired.network' added by the systemd recipe, which is matching on Type=ether. > > My fix is to modify the 80-wired.network to add 'Name=eth*'. With this in place, k3s works as expected. > > I am not sure what the correct upstream solution should be for Yocto? Your help would be appreciated. > This particular issue is known, in the sense that we have run into it before. At a minimum, I need to warn about it in the k3s README files. I created the cni bbclass to manage potentially conflicting networking configs on that front, but systemd-networking is yet another variable. We don't want to globally make it conflict, since someone might have a working networkd config that they want to use, and exactly how networking is set up, tends to be more of a distro feature. So the recipes need to tread carefully. Which takes me back to the README, and an enhancement to the cni-networking bbclass to be more generic and pick up / warn on configuration issues like this. bruce > /Matt > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. [-- Attachment #2: Type: text/html, Size: 6265 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [meta-virtualization] Networking issue with l3s when using systemd 2021-06-14 14:59 ` Matt Spencer @ 2021-06-14 17:59 ` Bruce Ashfield 2021-06-14 20:33 ` Richard Purdie 0 siblings, 1 reply; 6+ messages in thread From: Bruce Ashfield @ 2021-06-14 17:59 UTC (permalink / raw) To: Matt Spencer; +Cc: meta-virtualization On Mon, Jun 14, 2021 at 10:59 AM Matt Spencer <Matt.Spencer@arm.com> wrote: > > Hey Bruce, > > I understand that changing things can be difficult once launched. > > But looking at the systemd documentation here https://wiki.archlinux.org/title/systemd-networkd and in the note that says 'Devices can also be matched by their type. E.g. Type=ether for Ethernet, Type=wlan for Wi-Fi and Type=wwan for WWAN. Note that Type=ether will also match virtual Ethernet interfaces (veth*), which may be undesirable.' > > I have created a patch that modifies poky's meta/recipes-core/systemd/systemd-conf/wired.network to add '+Name=!veth*' to the Match clause which the team at Arm are reviewing with a view to upstream. > > I feel that this should have been the default behaviour anyway? Perhaps, but it is probably a good idea to have the default as broad as possible. If someone comes up with an alternate requirement, you'd end up having a harder time keeping them both working in oe-core. We could also carry a bbappend in meta-virtualization that triggers when "virtualization" is in the distro features. That way we are providing an override that makes sense within the context of the layer that is providing it, and the distro features that are enabled. Bruce > > /Matt > ________________________________ > From: meta-virtualization@lists.yoctoproject.org <meta-virtualization@lists.yoctoproject.org> on behalf of Bruce Ashfield via lists.yoctoproject.org <bruce.ashfield=gmail.com@lists.yoctoproject.org> > Sent: 14 June 2021 15:24 > To: Matt Spencer <Matt.Spencer@arm.com> > Cc: meta-virtualization@lists.yoctoproject.org <meta-virtualization@lists.yoctoproject.org> > Subject: Re: [meta-virtualization] Networking issue with l3s when using systemd > > On Mon, Jun 14, 2021 at 6:27 AM Matt Spencer <matt.spencer@arm.com> wrote: > > > > Hi all > > > > There seems to be a networking problem with k3s when using systemd. The problem manifests in that none of the kube-system management containers are able to communicate with eachother. > > > > The root cause seems to be that systemd-networking is actively managing veth interfaces created by k3s/flannel. This happens because of '/lib/systemd/network/80-wired.network' added by the systemd recipe, which is matching on Type=ether. > > > > My fix is to modify the 80-wired.network to add 'Name=eth*'. With this in place, k3s works as expected. > > > > I am not sure what the correct upstream solution should be for Yocto? Your help would be appreciated. > > > > This particular issue is known, in the sense that we have run into it before. > > At a minimum, I need to warn about it in the k3s README files. > > I created the cni bbclass to manage potentially conflicting networking > configs on that front, but systemd-networking is yet another variable. > > We don't want to globally make it conflict, since someone might have a > working networkd config that they want to use, and exactly how > networking is set up, tends to be more of a distro feature. So the > recipes need to tread carefully. > > Which takes me back to the README, and an enhancement to the > cni-networking bbclass to be more generic and pick up / warn on > configuration issues like this. > > bruce > > > > /Matt > > > > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [meta-virtualization] Networking issue with l3s when using systemd 2021-06-14 17:59 ` Bruce Ashfield @ 2021-06-14 20:33 ` Richard Purdie 2021-06-15 2:45 ` Bruce Ashfield 0 siblings, 1 reply; 6+ messages in thread From: Richard Purdie @ 2021-06-14 20:33 UTC (permalink / raw) To: Bruce Ashfield, Matt Spencer; +Cc: meta-virtualization On Mon, 2021-06-14 at 13:59 -0400, Bruce Ashfield wrote: > On Mon, Jun 14, 2021 at 10:59 AM Matt Spencer <Matt.Spencer@arm.com> wrote: > > > > Hey Bruce, > > > > I understand that changing things can be difficult once launched. > > > > But looking at the systemd documentation here https://wiki.archlinux.org/title/systemd-networkd and in the note that says 'Devices can also be matched by their type. E.g. Type=ether for Ethernet, Type=wlan for Wi-Fi and Type=wwan for WWAN. Note that Type=ether will also match virtual Ethernet interfaces (veth*), which may be undesirable.' > > > > I have created a patch that modifies poky's meta/recipes-core/systemd/systemd-conf/wired.network to add '+Name=!veth*' to the Match clause which the team at Arm are reviewing with a view to upstream. > > > > I feel that this should have been the default behaviour anyway? > > Perhaps, but it is probably a good idea to have the default as broad > as possible. If someone comes up with an alternate requirement, you'd > end up having a harder time keeping them both working in oe-core. > > We could also carry a bbappend in meta-virtualization that triggers > when "virtualization" is in the distro features. That way we are > providing an override that makes sense within the context of the layer > that is providing it, and the distro features that are enabled. FWIW I suspect the veth change should be ok for OE-Core if I understand things correctly. Cheers, Richard ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [meta-virtualization] Networking issue with l3s when using systemd 2021-06-14 20:33 ` Richard Purdie @ 2021-06-15 2:45 ` Bruce Ashfield 0 siblings, 0 replies; 6+ messages in thread From: Bruce Ashfield @ 2021-06-15 2:45 UTC (permalink / raw) To: Richard Purdie; +Cc: Matt Spencer, meta-virtualization On Mon, Jun 14, 2021 at 4:33 PM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Mon, 2021-06-14 at 13:59 -0400, Bruce Ashfield wrote: > > On Mon, Jun 14, 2021 at 10:59 AM Matt Spencer <Matt.Spencer@arm.com> wrote: > > > > > > Hey Bruce, > > > > > > I understand that changing things can be difficult once launched. > > > > > > But looking at the systemd documentation here https://wiki.archlinux.org/title/systemd-networkd and in the note that says 'Devices can also be matched by their type. E.g. Type=ether for Ethernet, Type=wlan for Wi-Fi and Type=wwan for WWAN. Note that Type=ether will also match virtual Ethernet interfaces (veth*), which may be undesirable.' > > > > > > I have created a patch that modifies poky's meta/recipes-core/systemd/systemd-conf/wired.network to add '+Name=!veth*' to the Match clause which the team at Arm are reviewing with a view to upstream. > > > > > > I feel that this should have been the default behaviour anyway? > > > > Perhaps, but it is probably a good idea to have the default as broad > > as possible. If someone comes up with an alternate requirement, you'd > > end up having a harder time keeping them both working in oe-core. > > > > We could also carry a bbappend in meta-virtualization that triggers > > when "virtualization" is in the distro features. That way we are > > providing an override that makes sense within the context of the layer > > that is providing it, and the distro features that are enabled. > > FWIW I suspect the veth change should be ok for OE-Core if I understand > things correctly. I'm sure we'll hear if someone is expecting networkd to manage their veth devices :D Either way works for me, I just lean towards keeping core very general and not tailored to any particular use case. If you are doing virtualization, and in particular kubernetes based, you need to have a close look at your networking regardless. Cheers, Bruce > > Cheers, > > Richard > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-06-15 2:46 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-06-14 10:27 Networking issue with l3s when using systemd Matt Spencer 2021-06-14 14:24 ` [meta-virtualization] " Bruce Ashfield 2021-06-14 14:59 ` Matt Spencer 2021-06-14 17:59 ` Bruce Ashfield 2021-06-14 20:33 ` Richard Purdie 2021-06-15 2:45 ` Bruce Ashfield
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.