* Docker 20.10.3 breaks due to iptables v1.8.7 (legacy) incompatibility #meta-virtualization @ 2021-09-30 14:41 Stephen 2021-09-30 15:29 ` [meta-virtualization] " Bruce Ashfield 0 siblings, 1 reply; 7+ messages in thread From: Stephen @ 2021-09-30 14:41 UTC (permalink / raw) To: meta-virtualization [-- Attachment #1: Type: text/plain, Size: 1699 bytes --] Hello all! The current meta-virtualisation docker is incompatible with the legacy v1.8.7 iptables. Docker version 20.10.3, build 41b3ea7e47 http://layers.openembedded.org/layerindex/recipe/176817/ iptables v1.8.7 (legacy) https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-extended/iptables/ level=info time=2021-09-30T08:58:56Z msg="TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGra viton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623551d39c6ab -> STOPPED, Known Sent: NONE, PullStartedAt: 2021-09-30 08:58:55.809460935 +0000 UTC m=+5 2315.765706001, PullStoppedAt: 2021-09-30 08:58:55.919351717 +0000 UTC m=+52315.875596782, ExecutionStoppedAt: 2021-09-30 08:58:56.159356552 +0000 UTC m=+52316.115601617, container change: arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGraviton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623 551d39c6ab web -> STOPPED, Reason CannotStartContainerError: Error response from daemon: driver failed programming external connectivity on endpoint e cs-GravitonIDecsTaskDefA2CA7A76-4-web-9eb9aba094eccadb1300 (db13dc1931d5be70284cac4de6899246035db8e5f9e0cf9ee3773000801a70b0): ( *iptables failed: ipta bles --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:3000 ! -i docker0: iptables v1.8.7 (legacy): unknown optio n \"--to-destination\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2))* , Known Sent: NONE] sent: false" module=task_ha ndler_types.go Possibly linked to this issue and nftables support? https://github.com/moby/moby/issues/38099 Any ideas for workarounds would be very much appreciated! [-- Attachment #2: Type: text/html, Size: 2159 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [meta-virtualization] Docker 20.10.3 breaks due to iptables v1.8.7 (legacy) incompatibility #meta-virtualization 2021-09-30 14:41 Docker 20.10.3 breaks due to iptables v1.8.7 (legacy) incompatibility #meta-virtualization Stephen @ 2021-09-30 15:29 ` Bruce Ashfield 2021-09-30 15:39 ` Stephen 0 siblings, 1 reply; 7+ messages in thread From: Bruce Ashfield @ 2021-09-30 15:29 UTC (permalink / raw) To: stephibb; +Cc: meta-virtualization On Thu, Sep 30, 2021 at 10:41 AM Stephen via lists.yoctoproject.org <stephibb=amazon.co.uk@lists.yoctoproject.org> wrote: > > Hello all! > > The current meta-virtualisation docker is incompatible with the legacy v1.8.7 iptables. > > Docker version 20.10.3, build 41b3ea7e47 http://layers.openembedded.org/layerindex/recipe/176817/ > > iptables v1.8.7 (legacy) https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-extended/iptables/ > > level=info time=2021-09-30T08:58:56Z msg="TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGra viton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623551d39c6ab -> STOPPED, Known Sent: NONE, PullStartedAt: 2021-09-30 08:58:55.809460935 +0000 UTC m=+5 2315.765706001, PullStoppedAt: 2021-09-30 08:58:55.919351717 +0000 UTC m=+52315.875596782, ExecutionStoppedAt: 2021-09-30 08:58:56.159356552 +0000 UTC m=+52316.115601617, container change: arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGraviton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623 551d39c6ab web -> STOPPED, Reason CannotStartContainerError: Error response from daemon: driver failed programming external connectivity on endpoint e cs-GravitonIDecsTaskDefA2CA7A76-4-web-9eb9aba094eccadb1300 (db13dc1931d5be70284cac4de6899246035db8e5f9e0cf9ee3773000801a70b0): (iptables failed: ipta bles --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:3000 ! -i docker0: iptables v1.8.7 (legacy): unknown optio n \"--to-destination\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)), Known Sent: NONE] sent: false" module=task_ha ndler_types.go > > Possibly linked to this issue and nftables support? https://github.com/moby/moby/issues/38099 > > Any ideas for workarounds would be very much appreciated! It's your kernel configuration, coupled with the iptables modules available .. but most often, it is a missing kernel module. So without knowing exactly what kernel and hardware you are running, it is hard to say more. Bruce > > > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [meta-virtualization] Docker 20.10.3 breaks due to iptables v1.8.7 (legacy) incompatibility #meta-virtualization 2021-09-30 15:29 ` [meta-virtualization] " Bruce Ashfield @ 2021-09-30 15:39 ` Stephen 2021-09-30 16:43 ` Bruce Ashfield 0 siblings, 1 reply; 7+ messages in thread From: Stephen @ 2021-09-30 15:39 UTC (permalink / raw) To: Bruce Ashfield; +Cc: meta-virtualization Thanks for the reply Bruce __ Let me know if these details help? root@generic-arm64:~# uname -r 5.10.46-yocto-standard Only setting these two kernel configs at the moment: CONFIG_ENA_ETHERNET=y CONFIG_BLK_DEV_NVME=y And these are the layers, running harknott... drwxrwxr-x 12 ubuntu ubuntu 4096 Sep 29 14:02 meta-arm/ drwxrwxr-x 8 ubuntu ubuntu 4096 Sep 29 14:00 meta-ewaol/ drwxrwxr-x 11 ubuntu ubuntu 4096 Sep 29 15:09 meta-openembedded/ drwxrwxr-x 24 ubuntu ubuntu 4096 Sep 29 14:02 meta-security/ drwxrwxr-x 17 ubuntu ubuntu 4096 Sep 29 14:02 meta-virtualization/ On 30/09/2021, 16:32, "Bruce Ashfield" <bruce.ashfield@gmail.com> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On Thu, Sep 30, 2021 at 10:41 AM Stephen via lists.yoctoproject.org <stephibb=amazon.co.uk@lists.yoctoproject.org> wrote: > > Hello all! > > The current meta-virtualisation docker is incompatible with the legacy v1.8.7 iptables. > > Docker version 20.10.3, build 41b3ea7e47 http://layers.openembedded.org/layerindex/recipe/176817/ > > iptables v1.8.7 (legacy) https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-extended/iptables/ > > level=info time=2021-09-30T08:58:56Z msg="TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGra viton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623551d39c6ab -> STOPPED, Known Sent: NONE, PullStartedAt: 2021-09-30 08:58:55.809460935 +0000 UTC m=+5 2315.765706001, PullStoppedAt: 2021-09-30 08:58:55.919351717 +0000 UTC m=+52315.875596782, ExecutionStoppedAt: 2021-09-30 08:58:56.159356552 +0000 UTC m=+52316.115601617, container change: arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGraviton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623 551d39c6ab web -> STOPPED, Reason CannotStartContainerError: Error response from daemon: driver failed programming external connectivity on endpoint e cs-GravitonIDecsTaskDefA2CA7A76-4-web-9eb9aba094eccadb1300 (db13dc1931d5be70284cac4de6899246035db8e5f9e0cf9ee3773000801a70b0): (iptables failed: ipta bles --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:3000 ! -i docker0: iptables v1.8.7 (legacy): unknown optio n \"--to-destination\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)), Known Sent: NONE] sent: false" module=task_ha ndler_types.go > > Possibly linked to this issue and nftables support? https://github.com/moby/moby/issues/38099 > > Any ideas for workarounds would be very much appreciated! It's your kernel configuration, coupled with the iptables modules available .. but most often, it is a missing kernel module. So without knowing exactly what kernel and hardware you are running, it is hard to say more. Bruce > > > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, UK Establishment No. BR019315 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [meta-virtualization] Docker 20.10.3 breaks due to iptables v1.8.7 (legacy) incompatibility #meta-virtualization 2021-09-30 15:39 ` Stephen @ 2021-09-30 16:43 ` Bruce Ashfield 2021-10-01 8:34 ` Stephen 0 siblings, 1 reply; 7+ messages in thread From: Bruce Ashfield @ 2021-09-30 16:43 UTC (permalink / raw) To: Hibbert, Stephen; +Cc: meta-virtualization On Thu, Sep 30, 2021 at 11:40 AM Hibbert, Stephen <stephibb@amazon.co.uk> wrote: > > Thanks for the reply Bruce __ Let me know if these details help? > > root@generic-arm64:~# uname -r > 5.10.46-yocto-standard > > Only setting these two kernel configs at the moment: > CONFIG_ENA_ETHERNET=y > CONFIG_BLK_DEV_NVME=y It'll be the iptables and cgroups options that are causing issues. The standard layers and kernel are extensively tested with meta-virt, so there really shouldn't be something missing. You can also install the docker-contrib package to your image, and run the check-config.sh script to see if it reports any issues. Bruce > > And these are the layers, running harknott... > drwxrwxr-x 12 ubuntu ubuntu 4096 Sep 29 14:02 meta-arm/ > drwxrwxr-x 8 ubuntu ubuntu 4096 Sep 29 14:00 meta-ewaol/ > drwxrwxr-x 11 ubuntu ubuntu 4096 Sep 29 15:09 meta-openembedded/ > drwxrwxr-x 24 ubuntu ubuntu 4096 Sep 29 14:02 meta-security/ > drwxrwxr-x 17 ubuntu ubuntu 4096 Sep 29 14:02 meta-virtualization/ > > > On 30/09/2021, 16:32, "Bruce Ashfield" <bruce.ashfield@gmail.com> wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On Thu, Sep 30, 2021 at 10:41 AM Stephen via lists.yoctoproject.org > <stephibb=amazon.co.uk@lists.yoctoproject.org> wrote: > > > > Hello all! > > > > The current meta-virtualisation docker is incompatible with the legacy v1.8.7 iptables. > > > > Docker version 20.10.3, build 41b3ea7e47 http://layers.openembedded.org/layerindex/recipe/176817/ > > > > iptables v1.8.7 (legacy) https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-extended/iptables/ > > > > level=info time=2021-09-30T08:58:56Z msg="TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGra viton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623551d39c6ab -> STOPPED, Known Sent: NONE, PullStartedAt: 2021-09-30 08:58:55.809460935 +0000 UTC m=+5 2315.765706001, PullStoppedAt: 2021-09-30 08:58:55.919351717 +0000 UTC m=+52315.875596782, ExecutionStoppedAt: 2021-09-30 08:58:56.159356552 +0000 UTC m=+52316.115601617, container change: arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGraviton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623 551d39c6ab web -> STOPPED, Reason CannotStartContainerError: Error response from daemon: driver failed programming external connectivity on endpoint e cs-GravitonIDecsTaskDefA2CA7A76-4-web-9eb9aba094eccadb1300 (db13dc1931d5be70284cac4de6899246035db8e5f9e0cf9ee3773000801a70b0): (iptables failed: ipta bles --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:3000 ! -i docker0: iptables v1.8.7 (legacy): unknown optio n \"--to-destination\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)), Known Sent: NONE] sent: false" module=task_ha ndler_types.go > > > > Possibly linked to this issue and nftables support? https://github.com/moby/moby/issues/38099 > > > > Any ideas for workarounds would be very much appreciated! > > It's your kernel configuration, coupled with the iptables modules > available .. but most often, it is a missing kernel module. > > So without knowing exactly what kernel and hardware you are running, > it is hard to say more. > > Bruce > > > > > > > > > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 > > Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, UK Establishment No. BR019315 > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [meta-virtualization] Docker 20.10.3 breaks due to iptables v1.8.7 (legacy) incompatibility #meta-virtualization 2021-09-30 16:43 ` Bruce Ashfield @ 2021-10-01 8:34 ` Stephen 2021-10-01 12:49 ` Bruce Ashfield 0 siblings, 1 reply; 7+ messages in thread From: Stephen @ 2021-10-01 8:34 UTC (permalink / raw) To: Bruce Ashfield; +Cc: meta-virtualization Yes, you're spot on! Running the script reviled the following. The issue I'm having now is finding the correct way of including the configs, I tried setting them in my myKernelConfigs.cfg and IMAGE_INSTALL_append = "docker-moby-contrib kernel-module-nf-nat kernel-module-xt-conntrack kernel-module-xt-addrtype" But running the config script still shows the output below: root@generic-arm64:/usr/share/docker# ./check-config.sh info: reading kernel config from /proc/config.gz ... Generally Necessary: - cgroup hierarchy: properly mounted [/sys/fs/cgroup] - CONFIG_NAMESPACES: enabled - CONFIG_NET_NS: enabled - CONFIG_PID_NS: enabled - CONFIG_IPC_NS: enabled - CONFIG_UTS_NS: enabled - CONFIG_CGROUPS: enabled - CONFIG_CGROUP_CPUACCT: enabled - CONFIG_CGROUP_DEVICE: enabled - CONFIG_CGROUP_FREEZER: missing - CONFIG_CGROUP_SCHED: enabled - CONFIG_CPUSETS: enabled - CONFIG_MEMCG: enabled - CONFIG_KEYS: enabled - CONFIG_VETH: enabled - CONFIG_BRIDGE: enabled (as module) - CONFIG_BRIDGE_NETFILTER: missing - CONFIG_NF_NAT_IPV4: missing - CONFIG_IP_NF_FILTER: enabled (as module) - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_IPVS: missing - CONFIG_IP_NF_NAT: enabled (as module) - CONFIG_NF_NAT: enabled (as module) - CONFIG_NF_NAT_NEEDED: missing - CONFIG_POSIX_MQUEUE: enabled On 30/09/2021, 17:48, "Bruce Ashfield" <bruce.ashfield@gmail.com> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On Thu, Sep 30, 2021 at 11:40 AM Hibbert, Stephen <stephibb@amazon.co.uk> wrote: > > Thanks for the reply Bruce __ Let me know if these details help? > > root@generic-arm64:~# uname -r > 5.10.46-yocto-standard > > Only setting these two kernel configs at the moment: > CONFIG_ENA_ETHERNET=y > CONFIG_BLK_DEV_NVME=y It'll be the iptables and cgroups options that are causing issues. The standard layers and kernel are extensively tested with meta-virt, so there really shouldn't be something missing. You can also install the docker-contrib package to your image, and run the check-config.sh script to see if it reports any issues. Bruce > > And these are the layers, running harknott... > drwxrwxr-x 12 ubuntu ubuntu 4096 Sep 29 14:02 meta-arm/ > drwxrwxr-x 8 ubuntu ubuntu 4096 Sep 29 14:00 meta-ewaol/ > drwxrwxr-x 11 ubuntu ubuntu 4096 Sep 29 15:09 meta-openembedded/ > drwxrwxr-x 24 ubuntu ubuntu 4096 Sep 29 14:02 meta-security/ > drwxrwxr-x 17 ubuntu ubuntu 4096 Sep 29 14:02 meta-virtualization/ > > > On 30/09/2021, 16:32, "Bruce Ashfield" <bruce.ashfield@gmail.com> wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On Thu, Sep 30, 2021 at 10:41 AM Stephen via lists.yoctoproject.org > <stephibb=amazon.co.uk@lists.yoctoproject.org> wrote: > > > > Hello all! > > > > The current meta-virtualisation docker is incompatible with the legacy v1.8.7 iptables. > > > > Docker version 20.10.3, build 41b3ea7e47 http://layers.openembedded.org/layerindex/recipe/176817/ > > > > iptables v1.8.7 (legacy) https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-extended/iptables/ > > > > level=info time=2021-09-30T08:58:56Z msg="TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGra viton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623551d39c6ab -> STOPPED, Known Sent: NONE, PullStartedAt: 2021-09-30 08:58:55.809460935 +0000 UTC m=+5 2315.765706001, PullStoppedAt: 2021-09-30 08:58:55.919351717 +0000 UTC m=+52315.875596782, ExecutionStoppedAt: 2021-09-30 08:58:56.159356552 +0000 UTC m=+52316.115601617, container change: arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGraviton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623 551d39c6ab web -> STOPPED, Reason CannotStartContainerError: Error response from daemon: driver failed programming external connectivity on endpoint e cs-GravitonIDecsTaskDefA2CA7A76-4-web-9eb9aba094eccadb1300 (db13dc1931d5be70284cac4de6899246035db8e5f9e0cf9ee3773000801a70b0): (iptables failed: ipta bles --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:3000 ! -i docker0: iptables v1.8.7 (legacy): unknown optio n \"--to-destination\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)), Known Sent: NONE] sent: false" module=task_ha ndler_types.go > > > > Possibly linked to this issue and nftables support? https://github.com/moby/moby/issues/38099 > > > > Any ideas for workarounds would be very much appreciated! > > It's your kernel configuration, coupled with the iptables modules > available .. but most often, it is a missing kernel module. > > So without knowing exactly what kernel and hardware you are running, > it is hard to say more. > > Bruce > > > > > > > > > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 > > Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, UK Establishment No. BR019315 > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, UK Establishment No. BR019315 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [meta-virtualization] Docker 20.10.3 breaks due to iptables v1.8.7 (legacy) incompatibility #meta-virtualization 2021-10-01 8:34 ` Stephen @ 2021-10-01 12:49 ` Bruce Ashfield 2021-10-01 13:42 ` Stephen 0 siblings, 1 reply; 7+ messages in thread From: Bruce Ashfield @ 2021-10-01 12:49 UTC (permalink / raw) To: Hibbert, Stephen; +Cc: meta-virtualization On Fri, Oct 1, 2021 at 4:35 AM Hibbert, Stephen <stephibb@amazon.co.uk> wrote: > > Yes, you're spot on! > > Running the script reviled the following. The issue I'm having now is finding the correct way of including the configs, I tried setting them in my myKernelConfigs.cfg and IMAGE_INSTALL_append = "docker-moby-contrib kernel-module-nf-nat kernel-module-xt-conntrack kernel-module-xt-addrtype" > In the K3S recipe, we actually have finer grained RRECOMMENDS than the docker recipes (due to the way k3s was developed and integrated). In K3S, I'm currently tracking: RRECOMMENDS:${PN} = "\ kernel-module-xt-addrtype \ kernel-module-xt-nat \ kernel-module-xt-multiport \ kernel-module-xt-conntrack \ kernel-module-xt-comment \ kernel-module-xt-mark \ kernel-module-xt-connmark \ kernel-module-vxlan \ kernel-module-xt-masquerade \ " So you could try that list, or do what I normally recommend .. use the meta package "kernel-modules" and get everything that was built. Since if you are using a linux-yocto variant, you'll already be getting fragments to build the right modules as part of the kernel build. I do have a new set of tested planned for the fall that do barebones testing to ensure that we've fully listed the rdepends/rrcommends for many of the recipes in meta-virt. But for now, I'd recommend that larger package, or you can do what I did for k3s. Build a package-feed enabled image, start docker, look at the error messages, install the required module, and then repeat to get the minimum list (if a kernel module wasn't being built at all, you may need to do some rebuilding in the middle). Bruce > But running the config script still shows the output below: > > root@generic-arm64:/usr/share/docker# ./check-config.sh > info: reading kernel config from /proc/config.gz ... > Generally Necessary: > - cgroup hierarchy: properly mounted [/sys/fs/cgroup] > - CONFIG_NAMESPACES: enabled > - CONFIG_NET_NS: enabled > - CONFIG_PID_NS: enabled > - CONFIG_IPC_NS: enabled > - CONFIG_UTS_NS: enabled > - CONFIG_CGROUPS: enabled > - CONFIG_CGROUP_CPUACCT: enabled > - CONFIG_CGROUP_DEVICE: enabled > - CONFIG_CGROUP_FREEZER: missing > - CONFIG_CGROUP_SCHED: enabled > - CONFIG_CPUSETS: enabled > - CONFIG_MEMCG: enabled > - CONFIG_KEYS: enabled > - CONFIG_VETH: enabled > - CONFIG_BRIDGE: enabled (as module) > - CONFIG_BRIDGE_NETFILTER: missing > - CONFIG_NF_NAT_IPV4: missing > - CONFIG_IP_NF_FILTER: enabled (as module) > - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module) > - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module) > - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module) > - CONFIG_NETFILTER_XT_MATCH_IPVS: missing > - CONFIG_IP_NF_NAT: enabled (as module) > - CONFIG_NF_NAT: enabled (as module) > - CONFIG_NF_NAT_NEEDED: missing > - CONFIG_POSIX_MQUEUE: enabled > > On 30/09/2021, 17:48, "Bruce Ashfield" <bruce.ashfield@gmail.com> wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On Thu, Sep 30, 2021 at 11:40 AM Hibbert, Stephen <stephibb@amazon.co.uk> wrote: > > > > Thanks for the reply Bruce __ Let me know if these details help? > > > > root@generic-arm64:~# uname -r > > 5.10.46-yocto-standard > > > > Only setting these two kernel configs at the moment: > > CONFIG_ENA_ETHERNET=y > > CONFIG_BLK_DEV_NVME=y > > It'll be the iptables and cgroups options that are causing issues. > > The standard layers and kernel are extensively tested with meta-virt, > so there really shouldn't be something missing. > > You can also install the docker-contrib package to your image, and run > the check-config.sh script to see if it reports any issues. > > Bruce > > > > > And these are the layers, running harknott... > > drwxrwxr-x 12 ubuntu ubuntu 4096 Sep 29 14:02 meta-arm/ > > drwxrwxr-x 8 ubuntu ubuntu 4096 Sep 29 14:00 meta-ewaol/ > > drwxrwxr-x 11 ubuntu ubuntu 4096 Sep 29 15:09 meta-openembedded/ > > drwxrwxr-x 24 ubuntu ubuntu 4096 Sep 29 14:02 meta-security/ > > drwxrwxr-x 17 ubuntu ubuntu 4096 Sep 29 14:02 meta-virtualization/ > > > > > > On 30/09/2021, 16:32, "Bruce Ashfield" <bruce.ashfield@gmail.com> wrote: > > > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > > > > > On Thu, Sep 30, 2021 at 10:41 AM Stephen via lists.yoctoproject.org > > <stephibb=amazon.co.uk@lists.yoctoproject.org> wrote: > > > > > > Hello all! > > > > > > The current meta-virtualisation docker is incompatible with the legacy v1.8.7 iptables. > > > > > > Docker version 20.10.3, build 41b3ea7e47 http://layers.openembedded.org/layerindex/recipe/176817/ > > > > > > iptables v1.8.7 (legacy) https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-extended/iptables/ > > > > > > level=info time=2021-09-30T08:58:56Z msg="TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGra viton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623551d39c6ab -> STOPPED, Known Sent: NONE, PullStartedAt: 2021-09-30 08:58:55.809460935 +0000 UTC m=+5 2315.765706001, PullStoppedAt: 2021-09-30 08:58:55.919351717 +0000 UTC m=+52315.875596782, ExecutionStoppedAt: 2021-09-30 08:58:56.159356552 +0000 UTC m=+52316.115601617, container change: arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGraviton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623 551d39c6ab web -> STOPPED, Reason CannotStartContainerError: Error response from daemon: driver failed programming external connectivity on endpoint e cs-GravitonIDecsTaskDefA2CA7A76-4-web-9eb9aba094eccadb1300 (db13dc1931d5be70284cac4de6899246035db8e5f9e0cf9ee3773000801a70b0): (iptables failed: ipta bles --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:3000 ! -i docker0: iptables v1.8.7 (legacy): unknown optio n \"--to-destination\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)), Known Sent: NONE] sent: false" module=task_ha ndler_types.go > > > > > > Possibly linked to this issue and nftables support? https://github.com/moby/moby/issues/38099 > > > > > > Any ideas for workarounds would be very much appreciated! > > > > It's your kernel configuration, coupled with the iptables modules > > available .. but most often, it is a missing kernel module. > > > > So without knowing exactly what kernel and hardware you are running, > > it is hard to say more. > > > > Bruce > > > > > > > > > > > > > > > > > > > > -- > > - Thou shalt not follow the NULL pointer, for chaos and madness await > > thee at its end > > - "Use the force Harry" - Gandalf, Star Trek II > > > > > > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 > > > > Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, UK Establishment No. BR019315 > > > > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 > > Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, UK Establishment No. BR019315 > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [meta-virtualization] Docker 20.10.3 breaks due to iptables v1.8.7 (legacy) incompatibility #meta-virtualization 2021-10-01 12:49 ` Bruce Ashfield @ 2021-10-01 13:42 ` Stephen 0 siblings, 0 replies; 7+ messages in thread From: Stephen @ 2021-10-01 13:42 UTC (permalink / raw) To: Bruce Ashfield; +Cc: meta-virtualization Adding kernel-modules did the trick, the docker daemon issued iptables command now works and supports --to-destination for dnat. Thank-you Bruce for taking the time to explain, I've learned a ton in the process! On 01/10/2021, 13:50, "Bruce Ashfield" <bruce.ashfield@gmail.com> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On Fri, Oct 1, 2021 at 4:35 AM Hibbert, Stephen <stephibb@amazon.co.uk> wrote: > > Yes, you're spot on! > > Running the script reviled the following. The issue I'm having now is finding the correct way of including the configs, I tried setting them in my myKernelConfigs.cfg and IMAGE_INSTALL_append = "docker-moby-contrib kernel-module-nf-nat kernel-module-xt-conntrack kernel-module-xt-addrtype" > In the K3S recipe, we actually have finer grained RRECOMMENDS than the docker recipes (due to the way k3s was developed and integrated). In K3S, I'm currently tracking: RRECOMMENDS:${PN} = "\ kernel-module-xt-addrtype \ kernel-module-xt-nat \ kernel-module-xt-multiport \ kernel-module-xt-conntrack \ kernel-module-xt-comment \ kernel-module-xt-mark \ kernel-module-xt-connmark \ kernel-module-vxlan \ kernel-module-xt-masquerade \ " So you could try that list, or do what I normally recommend .. use the meta package "kernel-modules" and get everything that was built. Since if you are using a linux-yocto variant, you'll already be getting fragments to build the right modules as part of the kernel build. I do have a new set of tested planned for the fall that do barebones testing to ensure that we've fully listed the rdepends/rrcommends for many of the recipes in meta-virt. But for now, I'd recommend that larger package, or you can do what I did for k3s. Build a package-feed enabled image, start docker, look at the error messages, install the required module, and then repeat to get the minimum list (if a kernel module wasn't being built at all, you may need to do some rebuilding in the middle). Bruce > But running the config script still shows the output below: > > root@generic-arm64:/usr/share/docker# ./check-config.sh > info: reading kernel config from /proc/config.gz ... > Generally Necessary: > - cgroup hierarchy: properly mounted [/sys/fs/cgroup] > - CONFIG_NAMESPACES: enabled > - CONFIG_NET_NS: enabled > - CONFIG_PID_NS: enabled > - CONFIG_IPC_NS: enabled > - CONFIG_UTS_NS: enabled > - CONFIG_CGROUPS: enabled > - CONFIG_CGROUP_CPUACCT: enabled > - CONFIG_CGROUP_DEVICE: enabled > - CONFIG_CGROUP_FREEZER: missing > - CONFIG_CGROUP_SCHED: enabled > - CONFIG_CPUSETS: enabled > - CONFIG_MEMCG: enabled > - CONFIG_KEYS: enabled > - CONFIG_VETH: enabled > - CONFIG_BRIDGE: enabled (as module) > - CONFIG_BRIDGE_NETFILTER: missing > - CONFIG_NF_NAT_IPV4: missing > - CONFIG_IP_NF_FILTER: enabled (as module) > - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module) > - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module) > - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module) > - CONFIG_NETFILTER_XT_MATCH_IPVS: missing > - CONFIG_IP_NF_NAT: enabled (as module) > - CONFIG_NF_NAT: enabled (as module) > - CONFIG_NF_NAT_NEEDED: missing > - CONFIG_POSIX_MQUEUE: enabled > > On 30/09/2021, 17:48, "Bruce Ashfield" <bruce.ashfield@gmail.com> wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On Thu, Sep 30, 2021 at 11:40 AM Hibbert, Stephen <stephibb@amazon.co.uk> wrote: > > > > Thanks for the reply Bruce __ Let me know if these details help? > > > > root@generic-arm64:~# uname -r > > 5.10.46-yocto-standard > > > > Only setting these two kernel configs at the moment: > > CONFIG_ENA_ETHERNET=y > > CONFIG_BLK_DEV_NVME=y > > It'll be the iptables and cgroups options that are causing issues. > > The standard layers and kernel are extensively tested with meta-virt, > so there really shouldn't be something missing. > > You can also install the docker-contrib package to your image, and run > the check-config.sh script to see if it reports any issues. > > Bruce > > > > > And these are the layers, running harknott... > > drwxrwxr-x 12 ubuntu ubuntu 4096 Sep 29 14:02 meta-arm/ > > drwxrwxr-x 8 ubuntu ubuntu 4096 Sep 29 14:00 meta-ewaol/ > > drwxrwxr-x 11 ubuntu ubuntu 4096 Sep 29 15:09 meta-openembedded/ > > drwxrwxr-x 24 ubuntu ubuntu 4096 Sep 29 14:02 meta-security/ > > drwxrwxr-x 17 ubuntu ubuntu 4096 Sep 29 14:02 meta-virtualization/ > > > > > > On 30/09/2021, 16:32, "Bruce Ashfield" <bruce.ashfield@gmail.com> wrote: > > > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > > > > > On Thu, Sep 30, 2021 at 10:41 AM Stephen via lists.yoctoproject.org > > <stephibb=amazon.co.uk@lists.yoctoproject.org> wrote: > > > > > > Hello all! > > > > > > The current meta-virtualisation docker is incompatible with the legacy v1.8.7 iptables. > > > > > > Docker version 20.10.3, build 41b3ea7e47 http://layers.openembedded.org/layerindex/recipe/176817/ > > > > > > iptables v1.8.7 (legacy) https://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-extended/iptables/ > > > > > > level=info time=2021-09-30T08:58:56Z msg="TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGra viton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623551d39c6ab -> STOPPED, Known Sent: NONE, PullStartedAt: 2021-09-30 08:58:55.809460935 +0000 UTC m=+5 2315.765706001, PullStoppedAt: 2021-09-30 08:58:55.919351717 +0000 UTC m=+52315.875596782, ExecutionStoppedAt: 2021-09-30 08:58:56.159356552 +0000 UTC m=+52316.115601617, container change: arn:aws:ecs:eu-west-1:116589935960:task/GravitonID-ecs-ECSGraviton2DA545608-tzdG3bupgLcn/ef8d9ea15a434c298a9623 551d39c6ab web -> STOPPED, Reason CannotStartContainerError: Error response from daemon: driver failed programming external connectivity on endpoint e cs-GravitonIDecsTaskDefA2CA7A76-4-web-9eb9aba094eccadb1300 (db13dc1931d5be70284cac4de6899246035db8e5f9e0cf9ee3773000801a70b0): (iptables failed: ipta bles --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:3000 ! -i docker0: iptables v1.8.7 (legacy): unknown optio n \"--to-destination\"\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2)), Known Sent: NONE] sent: false" module=task_ha ndler_types.go > > > > > > Possibly linked to this issue and nftables support? https://github.com/moby/moby/issues/38099 > > > > > > Any ideas for workarounds would be very much appreciated! > > > > It's your kernel configuration, coupled with the iptables modules > > available .. but most often, it is a missing kernel module. > > > > So without knowing exactly what kernel and hardware you are running, > > it is hard to say more. > > > > Bruce > > > > > > > > > > > > > > > > > > > > -- > > - Thou shalt not follow the NULL pointer, for chaos and madness await > > thee at its end > > - "Use the force Harry" - Gandalf, Star Trek II > > > > > > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 > > > > Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, UK Establishment No. BR019315 > > > > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 > > Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, UK Establishment No. BR019315 > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, UK Establishment No. BR019315 ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-10-01 13:42 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-30 14:41 Docker 20.10.3 breaks due to iptables v1.8.7 (legacy) incompatibility #meta-virtualization Stephen 2021-09-30 15:29 ` [meta-virtualization] " Bruce Ashfield 2021-09-30 15:39 ` Stephen 2021-09-30 16:43 ` Bruce Ashfield 2021-10-01 8:34 ` Stephen 2021-10-01 12:49 ` Bruce Ashfield 2021-10-01 13:42 ` Stephen
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.