Re: ❌ FAIL: Test report for kernel 5.10.0-rc6 (mainline.kernel.org)

[Eric Dumazet <edumazet@google.com>]

Hmm... maybe the ECN stuff has always been buggy then, and nobody cared...


On Wed, Dec 9, 2020 at 6:20 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> Eric, could this possibly be commit 4179b00c04d1 ("geneve: pull IP
> header before ECN decapsulation")?
>
> On Wed, 9 Dec 2020 10:05:14 +0800 Jianlin Shi wrote:
> > Hi ,
> >
> > I reported a bug in bugzilla.kernel.org for geneve issue:
> > https://bugzilla.kernel.org/show_bug.cgi?id=210569
> >
> > Thanks & Best Regards,
> > Jianlin Shi
> >
> >
> > On Tue, Dec 8, 2020 at 9:38 AM Jianlin Shi <jishi@redhat.com> wrote:
> > >
> > > Hi ,
> > >
> > >
> > > On Tue, Dec 8, 2020 at 8:25 AM CKI Project <cki-project@redhat.com> wrote:
> > >>
> > >>
> > >> Hello,
> > >>
> > >> We ran automated tests on a recent commit from this kernel tree:
> > >>
> > >>        Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > >>             Commit: 7059c2c00a21 - Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
> > >>
> > >> The results of these automated tests are provided below.
> > >>
> > >>     Overall result: FAILED (see details below)
> > >>              Merge: OK
> > >>            Compile: OK
> > >>  Selftests compile: FAILED
> > >>              Tests: FAILED
> > >>
> > >>     Pipeline: https://xci32.lab.eng.rdu2.redhat.com/cki-project/cki-pipeline/-/pipelines/619303
> > >>
> > >>     Check out our dashboard including known failures tagged by our bot at
> > >>       https://datawarehouse.internal.cki-project.org/kcidb/revisions/7748
> > >>
> > >> One or more kernel tests failed:
> > >>
> > >>     s390x:
> > >>      ❌ LTP
> > >>      ❌ Networking tunnel: geneve basic test
> > >>
> > >>     ppc64le:
> > >>      ❌ Boot test
> > >>      ❌ LTP
> > >>      ❌ Networking tunnel: geneve basic test
> > >>
> > >>     aarch64:
> > >>      ❌ storage: software RAID testing
> > >>      ❌ LTP
> > >>      ❌ Networking tunnel: geneve basic test
> > >>
> > >>     x86_64:
> > >>      ❌ LTP
> > >>      ❌ Networking tunnel: geneve basic test
> > >
> > >
> > > the basic traffic over geneve would fail, could you help to check. Should we report a bug or not?
> > >
> > >
> > > ip netns add client
> > >
> > > ip netns add server
> > >
> > >
> > >
> > > ip link add veth0_c type veth peer name veth0_s
> > >
> > > ip link set veth0_c netns client
> > >
> > > ip link set veth0_s netns server
> > >
> > >
> > >
> > > ip netns exec client ip link set lo up
> > >
> > > ip netns exec client ip link set veth0_c up
> > >
> > >
> > >
> > > ip netns exec server ip link set lo up
> > >
> > > ip netns exec server ip link set veth0_s up
> > >
> > >
> > >
> > >
> > >
> > > ip netns exec client ip addr add 2000::1/64 dev veth0_c
> > >
> > > ip netns exec client ip addr add 10.10.0.1/24 dev veth0_c
> > >
> > >
> > >
> > > ip netns exec server ip addr add 2000::2/64 dev veth0_s
> > >
> > > ip netns exec server ip addr add 10.10.0.2/24 dev veth0_s
> > >
> > >
> > >
> > > ip netns exec client ping 10.10.0.2 -c 2
> > >
> > > ip netns exec client ping6 2000::2 -c 2
> > >
> > >
> > >
> > > ip netns exec client ip link add geneve1 type geneve vni 1234 remote 10.10.0.2 ttl 64
> > >
> > > ip netns exec server ip link add geneve1 type geneve vni 1234 remote 10.10.0.1 ttl 64
> > >
> > >
> > >
> > > ip netns exec client ip link set geneve1 up
> > >
> > > ip netns exec client ip addr add 1.1.1.1/24 dev geneve1
> > >
> > > ip netns exec server ip link set geneve1 up
> > >
> > > ip netns exec server ip addr add 1.1.1.2/24 dev geneve1
> > > ip netns exec client ping 1.1.1.2 -c 3
> > >
> > >
> > >>
> > >>      ❌ storage: software RAID testing
> > >>
> > >> We hope that these logs can help you find the problem quickly. For the full
> > >> detail on our testing procedures, please scroll to the bottom of this message.
> > >>
> > >> Please reply to this email if you have any questions about the tests that we
> > >> ran or if you have any suggestions on how to make future tests more effective.
> > >>
> > >>         ,-.   ,-.
> > >>        ( C ) ( K )  Continuous
> > >>         `-',-.`-'   Kernel
> > >>           ( I )     Integration
> > >>            `-'
> > >> ______________________________________________________________________________
> > >>
> > >> Compile testing
> > >> ---------------
> > >>
> > >> We compiled the kernel for 4 architectures:
> > >>
> > >>     aarch64:
> > >>       make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg
> > >>
> > >>     ppc64le:
> > >>       make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg
> > >>
> > >>     s390x:
> > >>       make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg
> > >>
> > >>     x86_64:
> > >>       make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg
> > >>
> > >>
> > >> We built the following selftests:
> > >>
> > >>   x86_64:
> > >>       net: OK
> > >>       bpf: fail
> > >>       install and packaging: OK
> > >>
> > >> You can find the full log (build-selftests.log) in the artifact storage above.
> > >>
> > >>
> > >> Hardware testing
> > >> ----------------
> > >> All the testing jobs are listed here:
> > >>
> > >>   https://beaker.engineering.redhat.com/jobs/?jobsearch-0.table=Whiteboard&jobsearch-0.operation=contains&jobsearch-0.value=cki%40gitlab%3A619303
> > >>
> > >> We booted each kernel and ran the following tests:
> > >>
> > >>   aarch64:
> > >>     Host 1: https://beaker.engineering.redhat.com/recipes/9156933
> > >>        ✅ Boot test
> > >>        ✅ selinux-policy: serge-testsuite
> > >>        ❌ storage: software RAID testing
> > >>        ✅ stress: stress-ng
> > >>         ❌ xfstests - ext4
> > >>         ✅ xfstests - xfs
> > >>         ✅ xfstests - btrfs
> > >>         ❌ IPMI driver test
> > >>         ✅ IPMItool loop stress test
> > >>         ✅ Storage blktests
> > >>         ✅ Storage block - filesystem fio test
> > >>         ✅ Storage block - queue scheduler test
> > >>         ✅ Storage nvme - tcp
> > >>         ✅ Storage: swraid mdadm raid_module test
> > >>
> > >>     Host 2: https://beaker.engineering.redhat.com/recipes/9156932
> > >>        ✅ Boot test
> > >>        ✅ ACPI table test
> > >>        ✅ ACPI enabled test
> > >>        ❌ LTP
> > >>        ✅ Loopdev Sanity
> > >>        ✅ Memory: fork_mem
> > >>        ✅ Memory function: memfd_create
> > >>        ✅ AMTU (Abstract Machine Test Utility)
> > >>        ✅ Networking bridge: sanity
> > >>        ✅ Networking socket: fuzz
> > >>        ✅ Networking: igmp conformance test
> > >>        ✅ Networking route: pmtu
> > >>        ✅ Networking route_func - local
> > >>        ✅ Networking route_func - forward
> > >>        ✅ Networking TCP: keepalive test
> > >>        ✅ Networking UDP: socket
> > >>        ❌ Networking tunnel: geneve basic test
> > >>        ✅ Networking tunnel: gre basic
> > >>        ✅ L2TP basic test
> > >>        ✅ Networking tunnel: vxlan basic
> > >>        ✅ Networking ipsec: basic netns - transport
> > >>        ✅ Networking ipsec: basic netns - tunnel
> > >>        ✅ Libkcapi AF_ALG test
> > >>        ✅ pciutils: update pci ids test
> > >>        ✅ ALSA PCM loopback test
> > >>        ✅ ALSA Control (mixer) Userspace Element test
> > >>        ✅ storage: SCSI VPD
> > >>         ✅ CIFS Connectathon
> > >>         ✅ POSIX pjd-fstest suites
> > >>         ✅ Firmware test suite
> > >>         ✅ jvm - jcstress tests
> > >>         ✅ Memory function: kaslr
> > >>         ✅ Ethernet drivers sanity
> > >>         ✅ Networking firewall: basic netfilter test
> > >>         ✅ audit: audit testsuite test
> > >>         ✅ trace: ftrace/tracer
> > >>         ✅ kdump - kexec_boot
> > >>
> > >>   ppc64le:
> > >>     Host 1: https://beaker.engineering.redhat.com/recipes/9156935
> > >>        ❌ Boot test
> > >>        ⚡⚡⚡ selinux-policy: serge-testsuite
> > >>        ⚡⚡⚡ storage: software RAID testing
> > >>         ⚡⚡⚡ xfstests - ext4
> > >>         ⚡⚡⚡ xfstests - xfs
> > >>         ⚡⚡⚡ xfstests - btrfs
> > >>         ⚡⚡⚡ IPMI driver test
> > >>         ⚡⚡⚡ IPMItool loop stress test
> > >>         ⚡⚡⚡ Storage blktests
> > >>         ⚡⚡⚡ Storage block - filesystem fio test
> > >>         ⚡⚡⚡ Storage block - queue scheduler test
> > >>         ⚡⚡⚡ Storage nvme - tcp
> > >>         ⚡⚡⚡ Storage: swraid mdadm raid_module test
> > >>
> > >>     Host 2: https://beaker.engineering.redhat.com/recipes/9156934
> > >>        ✅ Boot test
> > >>        ❌ LTP
> > >>        ✅ Loopdev Sanity
> > >>        ✅ Memory: fork_mem
> > >>        ✅ Memory function: memfd_create
> > >>        ✅ AMTU (Abstract Machine Test Utility)
> > >>        ✅ Networking bridge: sanity
> > >>        ✅ Networking socket: fuzz
> > >>        ✅ Networking route: pmtu
> > >>        ✅ Networking route_func - local
> > >>        ✅ Networking route_func - forward
> > >>        ✅ Networking TCP: keepalive test
> > >>        ✅ Networking UDP: socket
> > >>        ❌ Networking tunnel: geneve basic test
> > >>        ✅ Networking tunnel: gre basic
> > >>        ✅ L2TP basic test
> > >>        ✅ Networking tunnel: vxlan basic
> > >>        ✅ Networking ipsec: basic netns - tunnel
> > >>        ✅ Libkcapi AF_ALG test
> > >>        ✅ pciutils: update pci ids test
> > >>        ✅ ALSA PCM loopback test
> > >>        ✅ ALSA Control (mixer) Userspace Element test
> > >>         ✅ CIFS Connectathon
> > >>         ✅ POSIX pjd-fstest suites
> > >>         ✅ jvm - jcstress tests
> > >>         ✅ Memory function: kaslr
> > >>         ✅ Ethernet drivers sanity
> > >>         ✅ Networking firewall: basic netfilter test
> > >>         ✅ audit: audit testsuite test
> > >>         ✅ trace: ftrace/tracer
> > >>
> > >>   s390x:
> > >>     Host 1: https://beaker.engineering.redhat.com/recipes/9156940
> > >>        ✅ Boot test
> > >>        ✅ selinux-policy: serge-testsuite
> > >>        ✅ stress: stress-ng
> > >>         ✅ Storage blktests
> > >>         ❌ Storage nvme - tcp
> > >>         ✅ Storage: swraid mdadm raid_module test
> > >>
> > >>     Host 2: https://beaker.engineering.redhat.com/recipes/9156939
> > >>        ✅ Boot test
> > >>        ❌ LTP
> > >>        ✅ Loopdev Sanity
> > >>        ✅ Memory: fork_mem
> > >>        ✅ Memory function: memfd_create
> > >>        ✅ AMTU (Abstract Machine Test Utility)
> > >>        ✅ Networking bridge: sanity
> > >>        ✅ Networking route: pmtu
> > >>        ✅ Networking route_func - local
> > >>        ✅ Networking route_func - forward
> > >>        ✅ Networking TCP: keepalive test
> > >>        ✅ Networking UDP: socket
> > >>        ❌ Networking tunnel: geneve basic test
> > >>        ✅ Networking tunnel: gre basic
> > >>        ✅ L2TP basic test
> > >>        ✅ Networking tunnel: vxlan basic
> > >>        ✅ Networking ipsec: basic netns - transport
> > >>        ✅ Networking ipsec: basic netns - tunnel
> > >>        ✅ Libkcapi AF_ALG test
> > >>         ✅ CIFS Connectathon
> > >>         ✅ POSIX pjd-fstest suites
> > >>         ✅ jvm - jcstress tests
> > >>         ✅ Memory function: kaslr
> > >>         ✅ Ethernet drivers sanity
> > >>         ✅ Networking firewall: basic netfilter test
> > >>         ❌ audit: audit testsuite test
> > >>         ✅ trace: ftrace/tracer
> > >>
> > >>   x86_64:
> > >>     Host 1: https://beaker.engineering.redhat.com/recipes/9156936
> > >>        ✅ Boot test
> > >>        ✅ ACPI table test
> > >>        ❌ LTP
> > >>        ✅ Loopdev Sanity
> > >>        ✅ Memory: fork_mem
> > >>        ✅ Memory function: memfd_create
> > >>        ✅ AMTU (Abstract Machine Test Utility)
> > >>        ✅ Networking bridge: sanity
> > >>        ✅ Networking socket: fuzz
> > >>        ✅ Networking: igmp conformance test
> > >>        ✅ Networking route: pmtu
> > >>        ✅ Networking route_func - local
> > >>        ✅ Networking route_func - forward
> > >>        ✅ Networking TCP: keepalive test
> > >>        ✅ Networking UDP: socket
> > >>        ❌ Networking tunnel: geneve basic test
> > >>        ✅ Networking tunnel: gre basic
> > >>        ✅ L2TP basic test
> > >>        ✅ Networking tunnel: vxlan basic
> > >>        ✅ Networking ipsec: basic netns - transport
> > >>        ✅ Networking ipsec: basic netns - tunnel
> > >>        ✅ Libkcapi AF_ALG test
> > >>        ✅ pciutils: sanity smoke test
> > >>        ✅ pciutils: update pci ids test
> > >>        ✅ ALSA PCM loopback test
> > >>        ✅ ALSA Control (mixer) Userspace Element test
> > >>        ✅ storage: SCSI VPD
> > >>         ✅ CIFS Connectathon
> > >>         ✅ POSIX pjd-fstest suites
> > >>         ✅ Firmware test suite
> > >>         ✅ jvm - jcstress tests
> > >>         ✅ Memory function: kaslr
> > >>         ✅ Ethernet drivers sanity
> > >>         ✅ Networking firewall: basic netfilter test
> > >>         ✅ audit: audit testsuite test
> > >>         ✅ trace: ftrace/tracer
> > >>         ✅ kdump - kexec_boot
> > >>
> > >>     Host 2: https://beaker.engineering.redhat.com/recipes/9156937
> > >>        ✅ Boot test
> > >>         ✅ kdump - sysrq-c
> > >>         ✅ kdump - file-load
> > >>
> > >>     Host 3: https://beaker.engineering.redhat.com/recipes/9156938
> > >>
> > >>        ⚡ Internal infrastructure issues prevented one or more tests (marked
> > >>        with ⚡⚡⚡) from running on this architecture.
> > >>        This is not the fault of the kernel that was tested.
> > >>
> > >>        ✅ Boot test
> > >>        ✅ selinux-policy: serge-testsuite
> > >>        ❌ storage: software RAID testing
> > >>        ✅ stress: stress-ng
> > >>         ❌ CPU: Frequency Driver Test
> > >>         ✅ CPU: Idle Test
> > >>         ❌ xfstests - ext4
> > >>         ✅ xfstests - xfs
> > >>         ✅ xfstests - btrfs
> > >>         ⚡⚡⚡ IPMI driver test
> > >>         ⚡⚡⚡ IPMItool loop stress test
> > >>         ⚡⚡⚡ power-management: cpupower/sanity test
> > >>         ⚡⚡⚡ Storage blktests
> > >>         ⚡⚡⚡ Storage block - filesystem fio test
> > >>         ⚡⚡⚡ Storage block - queue scheduler test
> > >>         ⚡⚡⚡ Storage nvme - tcp
> > >>         ⚡⚡⚡ Storage: swraid mdadm raid_module test
> > >>
> > >>   Test sources: https://gitlab.com/cki-project/kernel-tests
> > >>     Pull requests are welcome for new tests or improvements to existing tests!
> > >>
> > >> Aborted tests
> > >> -------------
> > >> Tests that didn't complete running successfully are marked with ⚡⚡⚡.
> > >> If this was caused by an infrastructure issue, we try to mark that
> > >> explicitly in the report.
> > >>
> > >> Waived tests
> > >> ------------
> > >> If the test run included waived tests, they are marked with . Such tests are
> > >> executed but their results are not taken into account. Tests are waived when
> > >> their results are not reliable enough, e.g. when they're just introduced or are
> > >> being fixed.
> > >>
> > >> Testing timeout
> > >> ---------------
> > >> We aim to provide a report within reasonable timeframe. Tests that haven't
> > >> finished running yet are marked with ⏱.
> > >>
> > >> Reproducing results
> > >> -------------------
> > >> Click on a link below to access a web page that allows you to adjust the
> > >> Beaker job and re-run any failed tests. These links are generated for
> > >> failed or aborted tests that are not waived. Please adjust the Beaker
> > >> job whiteboard string in the web page so that it is easy for you to find
> > >> and so that it is not confused with a regular CKI job.
> > >>
> > >> After clicking the "Submit the job!" button, a dialog will open that should
> > >> contain a link to the newly submitted Beaker job.
> > >>
> > >>   https://beaker-respin.internal.cki-project.org/respin?whiteboard=respin_job_4794082&recipe_id=9156933&recipe_id=9156932&job_id=4794082
> > >>   https://beaker-respin.internal.cki-project.org/respin?whiteboard=respin_job_4794083&recipe_id=9156935&recipe_id=9156934&job_id=4794083
> > >>   https://beaker-respin.internal.cki-project.org/respin?whiteboard=respin_job_4794085&recipe_id=9156939&job_id=4794085
> > >>   https://beaker-respin.internal.cki-project.org/respin?whiteboard=respin_job_4794084&recipe_id=9156936&recipe_id=9156938&job_id=4794084
> > >>
> >
>