❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

[CKI Project]

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
            Commit: 4176e42aa565 - Merge branch 'for-next/cpufeature' into for-kernelci

The results of these automated tests are provided below.

    Overall result: FAILED (see details below)
             Merge: OK
           Compile: OK
             Tests: FAILED

All kernel binaries, config files, and logs are available for download here:

  https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/09/623431

One or more kernel tests failed:

    aarch64:
     ❌ Boot test

We hope that these logs can help you find the problem quickly. For the full
detail on our testing procedures, please scroll to the bottom of this message.

Please reply to this email if you have any questions about the tests that we
ran or if you have any suggestions on how to make future tests more effective.

        ,-.   ,-.
       ( C ) ( K )  Continuous
        `-',-.`-'   Kernel
          ( I )     Integration
           `-'
______________________________________________________________________________

Compile testing
---------------

We compiled the kernel for 1 architecture:

    aarch64:
      make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg



Hardware testing
----------------
We booted each kernel and ran the following tests:

  aarch64:
    Host 1:
       ✅ Boot test
       ✅ ACPI table test
       ✅ ACPI enabled test
       ✅ LTP
       ✅ Loopdev Sanity
       ✅ Memory: fork_mem
       ✅ Memory function: memfd_create
       ✅ AMTU (Abstract Machine Test Utility)
       ✅ Networking bridge: sanity
       ✅ Networking socket: fuzz
       ✅ Networking: igmp conformance test
       ✅ Networking route: pmtu
       ✅ Networking route_func - local
       ✅ Networking route_func - forward
       ✅ Networking TCP: keepalive test
       ✅ Networking UDP: socket
       ✅ Networking tunnel: geneve basic test
       ✅ Networking tunnel: gre basic
       ✅ L2TP basic test
       ✅ Networking tunnel: vxlan basic
       ✅ Networking ipsec: basic netns - transport
       ✅ Networking ipsec: basic netns - tunnel
       ✅ Libkcapi AF_ALG test
       ✅ pciutils: update pci ids test
       ✅ ALSA PCM loopback test
       ✅ ALSA Control (mixer) Userspace Element test
       ✅ storage: SCSI VPD
       🚧 ✅ CIFS Connectathon
       🚧 ✅ POSIX pjd-fstest suites
       🚧 ✅ Firmware test suite
       🚧 ✅ jvm - jcstress tests
       🚧 ✅ Memory function: kaslr
       🚧 ✅ Ethernet drivers sanity
       🚧 ✅ Networking firewall: basic netfilter test
       🚧 ✅ audit: audit testsuite test
       🚧 ✅ trace: ftrace/tracer

    Host 2:
       ❌ Boot test
       ⚡⚡⚡ selinux-policy: serge-testsuite
       ⚡⚡⚡ storage: software RAID testing
       🚧 ⚡⚡⚡ xfstests - ext4
       🚧 ⚡⚡⚡ xfstests - xfs
       🚧 ⚡⚡⚡ xfstests - btrfs
       🚧 ⚡⚡⚡ IPMI driver test
       🚧 ⚡⚡⚡ IPMItool loop stress test
       🚧 ⚡⚡⚡ Storage blktests
       🚧 ⚡⚡⚡ Storage block - filesystem fio test
       🚧 ⚡⚡⚡ Storage block - queue scheduler test
       🚧 ⚡⚡⚡ Storage nvme - tcp
       🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
       🚧 ⚡⚡⚡ stress: stress-ng

  Test sources: https://gitlab.com/cki-project/kernel-tests
    💚 Pull requests are welcome for new tests or improvements to existing tests!

Aborted tests
-------------
Tests that didn't complete running successfully are marked with ⚡⚡⚡.
If this was caused by an infrastructure issue, we try to mark that
explicitly in the report.

Waived tests
------------
If the test run included waived tests, they are marked with 🚧. Such tests are
executed but their results are not taken into account. Tests are waived when
their results are not reliable enough, e.g. when they're just introduced or are
being fixed.

Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't
finished running yet are marked with ⏱.


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

[Will Deacon]

Hi CKI folks,

Please can you help me with the error below?

On Tue, Feb 09, 2021 at 09:07:50PM -0000, CKI Project wrote:
> We ran automated tests on a recent commit from this kernel tree:
> 
>        Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
>             Commit: 4176e42aa565 - Merge branch 'for-next/cpufeature' into for-kernelci
> 
> The results of these automated tests are provided below.
> 
>     Overall result: FAILED (see details below)
>              Merge: OK
>            Compile: OK
>              Tests: FAILED
> 
> All kernel binaries, config files, and logs are available for download here:
> 
>   https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/09/623431
> 
> One or more kernel tests failed:
> 
>     aarch64:
>      ❌ Boot test

[...]

>     Host 2:
>        ❌ Boot test
>        ⚡⚡⚡ selinux-policy: serge-testsuite
>        ⚡⚡⚡ storage: software RAID testing
>        🚧 ⚡⚡⚡ xfstests - ext4
>        🚧 ⚡⚡⚡ xfstests - xfs
>        🚧 ⚡⚡⚡ xfstests - btrfs
>        🚧 ⚡⚡⚡ IPMI driver test
>        🚧 ⚡⚡⚡ IPMItool loop stress test
>        🚧 ⚡⚡⚡ Storage blktests
>        🚧 ⚡⚡⚡ Storage block - filesystem fio test
>        🚧 ⚡⚡⚡ Storage block - queue scheduler test
>        🚧 ⚡⚡⚡ Storage nvme - tcp
>        🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
>        🚧 ⚡⚡⚡ stress: stress-ng

Which system (e.g. soc) is host 2 and are there are known infra issues at
the moment? I did push some changes which affect the early boot path, so we
may well be running into a kernel bug, but I'd just like to make sure before
we dive in trying to debug that, especially as we haven't seen failures on
other systems (and host 1 seems ok).

In the meantime, I'll push a new branch without those early changes to see
if that does any better.

Thanks,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

[Veronika Kabatova]

----- Original Message -----
> From: "Will Deacon" <will@kernel.org>
> To: "CKI Project" <cki-project@redhat.com>
> Cc: "catalin marinas" <catalin.marinas@arm.com>, linux-arm-kernel@lists.infradead.org
> Sent: Wednesday, February 10, 2021 10:29:46 AM
> Subject: Re: ❌ FAIL: Test report for kernel	5.11.0-rc7 (arm-next)
> 
> Hi CKI folks,
> 
> Please can you help me with the error below?
> 
> On Tue, Feb 09, 2021 at 09:07:50PM -0000, CKI Project wrote:
> > We ran automated tests on a recent commit from this kernel tree:
> > 
> >        Kernel repo:
> >        https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
> >             Commit: 4176e42aa565 - Merge branch 'for-next/cpufeature' into
> >             for-kernelci
> > 
> > The results of these automated tests are provided below.
> > 
> >     Overall result: FAILED (see details below)
> >              Merge: OK
> >            Compile: OK
> >              Tests: FAILED
> > 
> > All kernel binaries, config files, and logs are available for download
> > here:
> > 
> >   https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/09/623431
> > 
> > One or more kernel tests failed:
> > 
> >     aarch64:
> >      ❌ Boot test
> 
> [...]
> 
> >     Host 2:
> >        ❌ Boot test
> >        ⚡⚡⚡ selinux-policy: serge-testsuite
> >        ⚡⚡⚡ storage: software RAID testing
> >        🚧 ⚡⚡⚡ xfstests - ext4
> >        🚧 ⚡⚡⚡ xfstests - xfs
> >        🚧 ⚡⚡⚡ xfstests - btrfs
> >        🚧 ⚡⚡⚡ IPMI driver test
> >        🚧 ⚡⚡⚡ IPMItool loop stress test
> >        🚧 ⚡⚡⚡ Storage blktests
> >        🚧 ⚡⚡⚡ Storage block - filesystem fio test
> >        🚧 ⚡⚡⚡ Storage block - queue scheduler test
> >        🚧 ⚡⚡⚡ Storage nvme - tcp
> >        🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
> >        🚧 ⚡⚡⚡ stress: stress-ng
> 
> Which system (e.g. soc) is host 2 and are there are known infra issues at
> the moment? I did push some changes which affect the early boot path, so we
> may well be running into a kernel bug, but I'd just like to make sure before
> we dive in trying to debug that, especially as we haven't seen failures on
> other systems (and host 1 seems ok).
> 

Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable
kernel just a few days back okay. The last messages I can see in the raw
console log from this run are:

EFI stub: Booting Linux Kernel... 
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled 
EFI stub: Using DTB from configuration table 
EFI stub: Exiting boot services and installing virtual address map... 

and then it times out after hour and half. I'm not aware of any ongoing
issues, however sometimes the link between the lab controller and the
machines can sometimes go wrong after reboot and lead to a similarly
looking problem.

I'll resubmit the test job on that same machine to check if that was
the case and let you know right after it boots.


Veronika

> In the meantime, I'll push a new branch without those early changes to see
> if that does any better.
> 
> Thanks,
> 
> Will
> 
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

[Veronika Kabatova]

----- Original Message -----
> From: "Veronika Kabatova" <vkabatov@redhat.com>
> To: "Will Deacon" <will@kernel.org>
> Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <cki-project@redhat.com>,
> linux-arm-kernel@lists.infradead.org
> Sent: Wednesday, February 10, 2021 1:01:36 PM
> Subject: Re: ❌ FAIL: Test report for kernel	5.11.0-rc7 (arm-next)
> 
> 
> 
> ----- Original Message -----
> > From: "Will Deacon" <will@kernel.org>
> > To: "CKI Project" <cki-project@redhat.com>
> > Cc: "catalin marinas" <catalin.marinas@arm.com>,
> > linux-arm-kernel@lists.infradead.org
> > Sent: Wednesday, February 10, 2021 10:29:46 AM
> > Subject: Re: ❌ FAIL: Test report for kernel	5.11.0-rc7 (arm-next)
> > 
> > Hi CKI folks,
> > 
> > Please can you help me with the error below?
> > 
> > On Tue, Feb 09, 2021 at 09:07:50PM -0000, CKI Project wrote:
> > > We ran automated tests on a recent commit from this kernel tree:
> > > 
> > >        Kernel repo:
> > >        https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
> > >             Commit: 4176e42aa565 - Merge branch 'for-next/cpufeature'
> > >             into
> > >             for-kernelci
> > > 
> > > The results of these automated tests are provided below.
> > > 
> > >     Overall result: FAILED (see details below)
> > >              Merge: OK
> > >            Compile: OK
> > >              Tests: FAILED
> > > 
> > > All kernel binaries, config files, and logs are available for download
> > > here:
> > > 
> > >   https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/09/623431
> > > 
> > > One or more kernel tests failed:
> > > 
> > >     aarch64:
> > >      ❌ Boot test
> > 
> > [...]
> > 
> > >     Host 2:
> > >        ❌ Boot test
> > >        ⚡⚡⚡ selinux-policy: serge-testsuite
> > >        ⚡⚡⚡ storage: software RAID testing
> > >        🚧 ⚡⚡⚡ xfstests - ext4
> > >        🚧 ⚡⚡⚡ xfstests - xfs
> > >        🚧 ⚡⚡⚡ xfstests - btrfs
> > >        🚧 ⚡⚡⚡ IPMI driver test
> > >        🚧 ⚡⚡⚡ IPMItool loop stress test
> > >        🚧 ⚡⚡⚡ Storage blktests
> > >        🚧 ⚡⚡⚡ Storage block - filesystem fio test
> > >        🚧 ⚡⚡⚡ Storage block - queue scheduler test
> > >        🚧 ⚡⚡⚡ Storage nvme - tcp
> > >        🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
> > >        🚧 ⚡⚡⚡ stress: stress-ng
> > 
> > Which system (e.g. soc) is host 2 and are there are known infra issues at
> > the moment? I did push some changes which affect the early boot path, so we
> > may well be running into a kernel bug, but I'd just like to make sure
> > before
> > we dive in trying to debug that, especially as we haven't seen failures on
> > other systems (and host 1 seems ok).
> > 
> 
> Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable
> kernel just a few days back okay. The last messages I can see in the raw
> console log from this run are:
> 
> EFI stub: Booting Linux Kernel...
> EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
> EFI stub: Using DTB from configuration table
> EFI stub: Exiting boot services and installing virtual address map...
> 
> and then it times out after hour and half. I'm not aware of any ongoing
> issues, however sometimes the link between the lab controller and the
> machines can sometimes go wrong after reboot and lead to a similarly
> looking problem.
> 
> I'll resubmit the test job on that same machine to check if that was
> the case and let you know right after it boots.
> 

Hi, I have a few results back:

- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot
- tried the version from last week: boots ok

There is an extra message from the run that managed to boot, which is not
present with any of the runs that failed:

EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value

But this message is not present with the stable run that I mentioned
previously.

We didn't change the builder container for a while now and since I tested
things on the same machine one run after other, there should also be no
distro/package changes involved and the only difference should be the
kernel.



Veronika

> 
> Veronika
> 
> > In the meantime, I'll push a new branch without those early changes to see
> > if that does any better.
> > 
> > Thanks,
> > 
> > Will
> > 
> > 
> > 
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test report for kernel?5.11.0-rc7 (arm-next)

[Will Deacon]

Hi Veronika,

Thanks for the help with this.

On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
> > > On Tue, Feb 09, 2021 at 09:07:50PM -0000, CKI Project wrote:
> > > >     Host 2:
> > > >        ❌ Boot test
> > > >        ⚡⚡⚡ selinux-policy: serge-testsuite
> > > >        ⚡⚡⚡ storage: software RAID testing
> > > >        🚧 ⚡⚡⚡ xfstests - ext4
> > > >        🚧 ⚡⚡⚡ xfstests - xfs
> > > >        🚧 ⚡⚡⚡ xfstests - btrfs
> > > >        🚧 ⚡⚡⚡ IPMI driver test
> > > >        🚧 ⚡⚡⚡ IPMItool loop stress test
> > > >        🚧 ⚡⚡⚡ Storage blktests
> > > >        🚧 ⚡⚡⚡ Storage block - filesystem fio test
> > > >        🚧 ⚡⚡⚡ Storage block - queue scheduler test
> > > >        🚧 ⚡⚡⚡ Storage nvme - tcp
> > > >        🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
> > > >        🚧 ⚡⚡⚡ stress: stress-ng
> > > 
> > > Which system (e.g. soc) is host 2 and are there are known infra issues at
> > > the moment? I did push some changes which affect the early boot path, so we
> > > may well be running into a kernel bug, but I'd just like to make sure
> > > before
> > > we dive in trying to debug that, especially as we haven't seen failures on
> > > other systems (and host 1 seems ok).
> > > 
> > 
> > Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable
> > kernel just a few days back okay. The last messages I can see in the raw
> > console log from this run are:
> > 
> > EFI stub: Booting Linux Kernel...
> > EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
> > EFI stub: Using DTB from configuration table
> > EFI stub: Exiting boot services and installing virtual address map...
> > 
> > and then it times out after hour and half. I'm not aware of any ongoing
> > issues, however sometimes the link between the lab controller and the
> > machines can sometimes go wrong after reboot and lead to a similarly
> > looking problem.
> > 
> > I'll resubmit the test job on that same machine to check if that was
> > the case and let you know right after it boots.
> > 
> 
> Hi, I have a few results back:
> 
> - resubmitted the same kernel: gets stuck in the same spot
> - tried the new version pushed today: gets stuck in the same spot

That's odd, as I just received a pass report for that branch!

https://lore.kernel.org/r/cki.598435E2D5.M3C5MKJ1NV@redhat.com

Is it just flakey, perhaps? Obviously, that's not great either, but it will
make bisection more challenging.

> - tried the version from last week: boots ok
>
> There is an extra message from the run that managed to boot, which is not
> present with any of the runs that failed:
> 
> EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
> 
> But this message is not present with the stable run that I mentioned
> previously.

Interesting. Are those messages in the logs anywhere? It would be handy to
include them, if possible.

Cheers,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test report for kernel?5.11.0-rc7 (arm-next)

[Veronika Kabatova]

----- Original Message -----
> From: "Will Deacon" <will@kernel.org>
> To: "Veronika Kabatova" <vkabatov@redhat.com>
> Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <cki-project@redhat.com>,
> linux-arm-kernel@lists.infradead.org
> Sent: Wednesday, February 10, 2021 5:09:37 PM
> Subject: Re: ❌ FAIL: Test report for	kernel?5.11.0-rc7 (arm-next)
> 
> Hi Veronika,
> 
> Thanks for the help with this.
> 
> On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
> > > > On Tue, Feb 09, 2021 at 09:07:50PM -0000, CKI Project wrote:
> > > > >     Host 2:
> > > > >        ❌ Boot test
> > > > >        ⚡⚡⚡ selinux-policy: serge-testsuite
> > > > >        ⚡⚡⚡ storage: software RAID testing
> > > > >        🚧 ⚡⚡⚡ xfstests - ext4
> > > > >        🚧 ⚡⚡⚡ xfstests - xfs
> > > > >        🚧 ⚡⚡⚡ xfstests - btrfs
> > > > >        🚧 ⚡⚡⚡ IPMI driver test
> > > > >        🚧 ⚡⚡⚡ IPMItool loop stress test
> > > > >        🚧 ⚡⚡⚡ Storage blktests
> > > > >        🚧 ⚡⚡⚡ Storage block - filesystem fio test
> > > > >        🚧 ⚡⚡⚡ Storage block - queue scheduler test
> > > > >        🚧 ⚡⚡⚡ Storage nvme - tcp
> > > > >        🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
> > > > >        🚧 ⚡⚡⚡ stress: stress-ng
> > > > 
> > > > Which system (e.g. soc) is host 2 and are there are known infra issues
> > > > at
> > > > the moment? I did push some changes which affect the early boot path,
> > > > so we
> > > > may well be running into a kernel bug, but I'd just like to make sure
> > > > before
> > > > we dive in trying to debug that, especially as we haven't seen failures
> > > > on
> > > > other systems (and host 1 seems ok).
> > > > 
> > > 
> > > Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a
> > > stable
> > > kernel just a few days back okay. The last messages I can see in the raw
> > > console log from this run are:
> > > 
> > > EFI stub: Booting Linux Kernel...
> > > EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
> > > EFI stub: Using DTB from configuration table
> > > EFI stub: Exiting boot services and installing virtual address map...
> > > 
> > > and then it times out after hour and half. I'm not aware of any ongoing
> > > issues, however sometimes the link between the lab controller and the
> > > machines can sometimes go wrong after reboot and lead to a similarly
> > > looking problem.
> > > 
> > > I'll resubmit the test job on that same machine to check if that was
> > > the case and let you know right after it boots.
> > > 
> > 
> > Hi, I have a few results back:
> > 
> > - resubmitted the same kernel: gets stuck in the same spot
> > - tried the new version pushed today: gets stuck in the same spot
> 
> That's odd, as I just received a pass report for that branch!
> 
> https://lore.kernel.org/r/cki.598435E2D5.M3C5MKJ1NV@redhat.com
> 
> Is it just flakey, perhaps? Obviously, that's not great either, but it will
> make bisection more challenging.
> 

We have a large number of machines (both physical and virtual) and it's
impossible to run all tests on all of them, so they are randomly picked as
long as they fit the distro and test requirements. The distribution for
ARM tree is 1 physical and 1 "any" machine (which usually ends up being
virtual). The jobs from the report you linked ran on different machines
and didn't pick the one that failed to boot previously, so I manually
forced my testing to pick that machine to eliminate some variables.

The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new kernels.

> > - tried the version from last week: boots ok
> >
> > There is an extra message from the run that managed to boot, which is not
> > present with any of the runs that failed:
> > 
> > EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus
> > value
> > 
> > But this message is not present with the stable run that I mentioned
> > previously.
> 
> Interesting. Are those messages in the logs anywhere? It would be handy to
> include them, if possible.
> 

The messages are from before the kernel boot banner which is the marker
we use for log inclusion (to reduce console log spam from distro
installation which uses a different kernel and thus makes debugging less
straightforward). The same EFI messages are present before the kernel
banner in the new report you linked, and with the passing job from the
previous runs as well:

EFI stub: Booting Linux Kernel... 
EFI stub: Using DTB from configuration table 
EFI stub: Exiting boot services and installing virtual address map... 
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1] 
[    0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021 
[    0.000000] efi: EFI v2.70 by EDK II 
[    0.000000] efi: SMBIOS 3.0=0x1bf760000 MEMATTR=0x1be656018 ACPI 2.0=0x1bc030000 RNG=0x1bf86cf98 MEMRESERVE=0x1bc3d3e18  
[    0.000000] efi: seeding entropy pool 
....

and

EFI stub: Booting Linux Kernel... 
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled 
EFI stub: Using DTB from configuration table 
EFI stub: Exiting boot services and installing virtual address map... 
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1] 
[    0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021 
[    0.000000] efi: EFI v2.70 by American Megatrends 
....

The failing machine/kernel combos get stuck right after that last EFI
line before the kernel messages come in.


Let me know if I should test some other versions or if you need some
other information!

Veronika

> Cheers,
> 
> Will
> 
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test report for?kernel?5.11.0-rc7 (arm-next)

[Will Deacon]

On Wed, Feb 10, 2021 at 12:07:23PM -0500, Veronika Kabatova wrote:
> > On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
> > > Hi, I have a few results back:
> > > 
> > > - resubmitted the same kernel: gets stuck in the same spot
> > > - tried the new version pushed today: gets stuck in the same spot
> > 
> > That's odd, as I just received a pass report for that branch!
> > 
> > https://lore.kernel.org/r/cki.598435E2D5.M3C5MKJ1NV@redhat.com
> > 
> > Is it just flakey, perhaps? Obviously, that's not great either, but it will
> > make bisection more challenging.
> > 
> 
> We have a large number of machines (both physical and virtual) and it's
> impossible to run all tests on all of them, so they are randomly picked as
> long as they fit the distro and test requirements. The distribution for
> ARM tree is 1 physical and 1 "any" machine (which usually ends up being
> virtual). The jobs from the report you linked ran on different machines
> and didn't pick the one that failed to boot previously, so I manually
> forced my testing to pick that machine to eliminate some variables.

Ah thanks, I hadn't twigged that it was a different set of hosts each time.
Makes sense.

> The machine in question can on course be somewhat flaky (hard to eliminate
> that possibility completely), but I checked our historical data and it
> didn't fail to boot a single time other than with these two new kernels.

So the first thing we should probably try is whether vanilla -rc7 fails on
the machine causing us problems. If it does, then the arm64 queue for 5.12
is out of the equation, if not then we can try a targetted bisection.

Would you be able to try v5.11-rc7 please?

> > > - tried the version from last week: boots ok
> > >
> > > There is an extra message from the run that managed to boot, which is not
> > > present with any of the runs that failed:
> > > 
> > > EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus
> > > value
> > > 
> > > But this message is not present with the stable run that I mentioned
> > > previously.
> > 
> > Interesting. Are those messages in the logs anywhere? It would be handy to
> > include them, if possible.
> > 
> 
> The messages are from before the kernel boot banner which is the marker
> we use for log inclusion (to reduce console log spam from distro
> installation which uses a different kernel and thus makes debugging less
> straightforward). The same EFI messages are present before the kernel
> banner in the new report you linked, and with the passing job from the
> previous runs as well:
> 
> EFI stub: Booting Linux Kernel... 
> EFI stub: Using DTB from configuration table 
> EFI stub: Exiting boot services and installing virtual address map... 
> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1] 
> [    0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021 
> [    0.000000] efi: EFI v2.70 by EDK II 
> [    0.000000] efi: SMBIOS 3.0=0x1bf760000 MEMATTR=0x1be656018 ACPI 2.0=0x1bc030000 RNG=0x1bf86cf98 MEMRESERVE=0x1bc3d3e18  
> [    0.000000] efi: seeding entropy pool 
> ....
> 
> and
> 
> EFI stub: Booting Linux Kernel... 
> EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled 
> EFI stub: Using DTB from configuration table 
> EFI stub: Exiting boot services and installing virtual address map... 
> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1] 
> [    0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021 
> [    0.000000] efi: EFI v2.70 by American Megatrends 
> ....
> 
> The failing machine/kernel combos get stuck right after that last EFI
> line before the kernel messages come in.

Sorry, just to be clear here: do we always fail when we have the "KASLR
will be disabled" message, or do some machines pass with that?

Thanks again,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test report for?kernel?5.11.0-rc7 (arm-next)

[Veronika Kabatova]

----- Original Message -----
> From: "Will Deacon" <will@kernel.org>
> To: "Veronika Kabatova" <vkabatov@redhat.com>
> Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <cki-project@redhat.com>,
> linux-arm-kernel@lists.infradead.org
> Sent: Wednesday, February 10, 2021 6:31:34 PM
> Subject: Re: ❌ FAIL: Test report	for?kernel?5.11.0-rc7 (arm-next)
> 
> On Wed, Feb 10, 2021 at 12:07:23PM -0500, Veronika Kabatova wrote:
> > > On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
> > > > Hi, I have a few results back:
> > > > 
> > > > - resubmitted the same kernel: gets stuck in the same spot
> > > > - tried the new version pushed today: gets stuck in the same spot
> > > 
> > > That's odd, as I just received a pass report for that branch!
> > > 
> > > https://lore.kernel.org/r/cki.598435E2D5.M3C5MKJ1NV@redhat.com
> > > 
> > > Is it just flakey, perhaps? Obviously, that's not great either, but it
> > > will
> > > make bisection more challenging.
> > > 
> > 
> > We have a large number of machines (both physical and virtual) and it's
> > impossible to run all tests on all of them, so they are randomly picked as
> > long as they fit the distro and test requirements. The distribution for
> > ARM tree is 1 physical and 1 "any" machine (which usually ends up being
> > virtual). The jobs from the report you linked ran on different machines
> > and didn't pick the one that failed to boot previously, so I manually
> > forced my testing to pick that machine to eliminate some variables.
> 
> Ah thanks, I hadn't twigged that it was a different set of hosts each time.
> Makes sense.
> 
> > The machine in question can on course be somewhat flaky (hard to eliminate
> > that possibility completely), but I checked our historical data and it
> > didn't fail to boot a single time other than with these two new kernels.
> 
> So the first thing we should probably try is whether vanilla -rc7 fails on
> the machine causing us problems. If it does, then the arm64 queue for 5.12
> is out of the equation, if not then we can try a targetted bisection.
> 
> Would you be able to try v5.11-rc7 please?

Can do.

Someone snatched up the machine in the meanwhile and appears to have it
reserved till next Monday :/ I sincerely hope that they'll release it sooner
and have queued up the test job with high priority. Will let you know right
as I get the results.

> 
> > > > - tried the version from last week: boots ok
> > > >
> > > > There is an extra message from the run that managed to boot, which is
> > > > not
> > > > present with any of the runs that failed:
> > > > 
> > > > EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus
> > > > value
> > > > 
> > > > But this message is not present with the stable run that I mentioned
> > > > previously.
> > > 
> > > Interesting. Are those messages in the logs anywhere? It would be handy
> > > to
> > > include them, if possible.
> > > 
> > 
> > The messages are from before the kernel boot banner which is the marker
> > we use for log inclusion (to reduce console log spam from distro
> > installation which uses a different kernel and thus makes debugging less
> > straightforward). The same EFI messages are present before the kernel
> > banner in the new report you linked, and with the passing job from the
> > previous runs as well:
> > 
> > EFI stub: Booting Linux Kernel...
> > EFI stub: Using DTB from configuration table
> > EFI stub: Exiting boot services and installing virtual address map...
> > [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
> > [    0.000000] Linux version 5.11.0-rc7
> > (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc
> > (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version
> > 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
> > [    0.000000] efi: EFI v2.70 by EDK II
> > [    0.000000] efi: SMBIOS 3.0=0x1bf760000 MEMATTR=0x1be656018 ACPI
> > 2.0=0x1bc030000 RNG=0x1bf86cf98 MEMRESERVE=0x1bc3d3e18
> > [    0.000000] efi: seeding entropy pool
> > ....
> > 
> > and
> > 
> > EFI stub: Booting Linux Kernel...
> > EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
> > EFI stub: Using DTB from configuration table
> > EFI stub: Exiting boot services and installing virtual address map...
> > [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
> > [    0.000000] Linux version 5.11.0-rc7
> > (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc
> > (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version
> > 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
> > [    0.000000] efi: EFI v2.70 by American Megatrends
> > ....
> > 
> > The failing machine/kernel combos get stuck right after that last EFI
> > line before the kernel messages come in.
> 
> Sorry, just to be clear here: do we always fail when we have the "KASLR
> will be disabled" message, or do some machines pass with that?
> 

No. The message is sometimes present, sometimes not (likely machine
dependent), and doesn't seem to influence the boot results. Both of the
log excerpts above are from jobs that booted.


Veronika

> Thanks again,
> 
> Will
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test report?for?kernel?5.11.0-rc7 (arm-next)

[Will Deacon]

On Wed, Feb 10, 2021 at 01:06:08PM -0500, Veronika Kabatova wrote:
> 
> 
> ----- Original Message -----
> > From: "Will Deacon" <will@kernel.org>
> > To: "Veronika Kabatova" <vkabatov@redhat.com>
> > Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <cki-project@redhat.com>,
> > linux-arm-kernel@lists.infradead.org
> > Sent: Wednesday, February 10, 2021 6:31:34 PM
> > Subject: Re: ❌ FAIL: Test report	for?kernel?5.11.0-rc7 (arm-next)
> > 
> > On Wed, Feb 10, 2021 at 12:07:23PM -0500, Veronika Kabatova wrote:
> > > > On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
> > > > > Hi, I have a few results back:
> > > > > 
> > > > > - resubmitted the same kernel: gets stuck in the same spot
> > > > > - tried the new version pushed today: gets stuck in the same spot
> > > > 
> > > > That's odd, as I just received a pass report for that branch!
> > > > 
> > > > https://lore.kernel.org/r/cki.598435E2D5.M3C5MKJ1NV@redhat.com
> > > > 
> > > > Is it just flakey, perhaps? Obviously, that's not great either, but it
> > > > will
> > > > make bisection more challenging.
> > > > 
> > > 
> > > We have a large number of machines (both physical and virtual) and it's
> > > impossible to run all tests on all of them, so they are randomly picked as
> > > long as they fit the distro and test requirements. The distribution for
> > > ARM tree is 1 physical and 1 "any" machine (which usually ends up being
> > > virtual). The jobs from the report you linked ran on different machines
> > > and didn't pick the one that failed to boot previously, so I manually
> > > forced my testing to pick that machine to eliminate some variables.
> > 
> > Ah thanks, I hadn't twigged that it was a different set of hosts each time.
> > Makes sense.
> > 
> > > The machine in question can on course be somewhat flaky (hard to eliminate
> > > that possibility completely), but I checked our historical data and it
> > > didn't fail to boot a single time other than with these two new kernels.
> > 
> > So the first thing we should probably try is whether vanilla -rc7 fails on
> > the machine causing us problems. If it does, then the arm64 queue for 5.12
> > is out of the equation, if not then we can try a targetted bisection.
> > 
> > Would you be able to try v5.11-rc7 please?
> 
> Can do.

Brill, thanks.

> Someone snatched up the machine in the meanwhile and appears to have it
> reserved till next Monday :/ I sincerely hope that they'll release it sooner
> and have queued up the test job with high priority. Will let you know right
> as I get the results.

Just tell them the machine is broken and they really don't want to use it
for anything important ;)

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test report?for?kernel?5.11.0-rc7 (arm-next)

[Veronika Kabatova]

----- Original Message -----
> From: "Will Deacon" <will@kernel.org>
> To: "Veronika Kabatova" <vkabatov@redhat.com>
> Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <cki-project@redhat.com>,
> linux-arm-kernel@lists.infradead.org
> Sent: Wednesday, February 10, 2021 7:56:03 PM
> Subject: Re: ❌ FAIL: Test	report?for?kernel?5.11.0-rc7 (arm-next)
> 
> On Wed, Feb 10, 2021 at 01:06:08PM -0500, Veronika Kabatova wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Will Deacon" <will@kernel.org>
> > > To: "Veronika Kabatova" <vkabatov@redhat.com>
> > > Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project"
> > > <cki-project@redhat.com>,
> > > linux-arm-kernel@lists.infradead.org
> > > Sent: Wednesday, February 10, 2021 6:31:34 PM
> > > Subject: Re: ❌ FAIL: Test report	for?kernel?5.11.0-rc7 (arm-next)
> > > 
> > > On Wed, Feb 10, 2021 at 12:07:23PM -0500, Veronika Kabatova wrote:
> > > > > On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
> > > > > > Hi, I have a few results back:
> > > > > > 
> > > > > > - resubmitted the same kernel: gets stuck in the same spot
> > > > > > - tried the new version pushed today: gets stuck in the same spot
> > > > > 
> > > > > That's odd, as I just received a pass report for that branch!
> > > > > 
> > > > > https://lore.kernel.org/r/cki.598435E2D5.M3C5MKJ1NV@redhat.com
> > > > > 
> > > > > Is it just flakey, perhaps? Obviously, that's not great either, but
> > > > > it
> > > > > will
> > > > > make bisection more challenging.
> > > > > 
> > > > 
> > > > We have a large number of machines (both physical and virtual) and it's
> > > > impossible to run all tests on all of them, so they are randomly picked
> > > > as
> > > > long as they fit the distro and test requirements. The distribution for
> > > > ARM tree is 1 physical and 1 "any" machine (which usually ends up being
> > > > virtual). The jobs from the report you linked ran on different machines
> > > > and didn't pick the one that failed to boot previously, so I manually
> > > > forced my testing to pick that machine to eliminate some variables.
> > > 
> > > Ah thanks, I hadn't twigged that it was a different set of hosts each
> > > time.
> > > Makes sense.
> > > 
> > > > The machine in question can on course be somewhat flaky (hard to
> > > > eliminate
> > > > that possibility completely), but I checked our historical data and it
> > > > didn't fail to boot a single time other than with these two new
> > > > kernels.
> > > 
> > > So the first thing we should probably try is whether vanilla -rc7 fails
> > > on
> > > the machine causing us problems. If it does, then the arm64 queue for
> > > 5.12
> > > is out of the equation, if not then we can try a targetted bisection.
> > > 
> > > Would you be able to try v5.11-rc7 please?
> > 
> > Can do.
> 
> Brill, thanks.
> 
> > Someone snatched up the machine in the meanwhile and appears to have it
> > reserved till next Monday :/ I sincerely hope that they'll release it
> > sooner
> > and have queued up the test job with high priority. Will let you know right
> > as I get the results.
> 
> Just tell them the machine is broken and they really don't want to use it
> for anything important ;)
> 

Our wishes were granted by the lab fairy and the machine was returned
rather quickly :)

The 5.11-rc7 kernel boots.


Veronika

> Will
> 
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test?report?for?kernel?5.11.0-rc7 (arm-next)

[Will Deacon]

On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
> > > > > The machine in question can on course be somewhat flaky (hard to
> > > > > eliminate
> > > > > that possibility completely), but I checked our historical data and it
> > > > > didn't fail to boot a single time other than with these two new
> > > > > kernels.
> > > > 
> > > > So the first thing we should probably try is whether vanilla -rc7 fails
> > > > on
> > > > the machine causing us problems. If it does, then the arm64 queue for
> > > > 5.12
> > > > is out of the equation, if not then we can try a targetted bisection.
> > > > 
> > > > Would you be able to try v5.11-rc7 please?
> > > 
> > > Can do.
> > 
> > Brill, thanks.
> > 
> > > Someone snatched up the machine in the meanwhile and appears to have it
> > > reserved till next Monday :/ I sincerely hope that they'll release it
> > > sooner
> > > and have queued up the test job with high priority. Will let you know right
> > > as I get the results.
> > 
> > Just tell them the machine is broken and they really don't want to use it
> > for anything important ;)
> > 
> 
> Our wishes were granted by the lab fairy and the machine was returned
> rather quickly :)
> 
> The 5.11-rc7 kernel boots.

Fantastic! Then it's something in the arm64 for-next/core tree that was
added since then. The diff isn't huge and one change stands out for me, so
let me try reverting that and I'll update the branch...

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test?report?for?kernel?5.11.0-rc7 (arm-next)

[Will Deacon]

On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
> On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
> > > > > > The machine in question can on course be somewhat flaky (hard to
> > > > > > eliminate
> > > > > > that possibility completely), but I checked our historical data and it
> > > > > > didn't fail to boot a single time other than with these two new
> > > > > > kernels.
> > > > > 
> > > > > So the first thing we should probably try is whether vanilla -rc7 fails
> > > > > on
> > > > > the machine causing us problems. If it does, then the arm64 queue for
> > > > > 5.12
> > > > > is out of the equation, if not then we can try a targetted bisection.
> > > > > 
> > > > > Would you be able to try v5.11-rc7 please?
> > > > 
> > > > Can do.
> > > 
> > > Brill, thanks.
> > > 
> > > > Someone snatched up the machine in the meanwhile and appears to have it
> > > > reserved till next Monday :/ I sincerely hope that they'll release it
> > > > sooner
> > > > and have queued up the test job with high priority. Will let you know right
> > > > as I get the results.
> > > 
> > > Just tell them the machine is broken and they really don't want to use it
> > > for anything important ;)
> > > 
> > 
> > Our wishes were granted by the lab fairy and the machine was returned
> > rather quickly :)
> > 
> > The 5.11-rc7 kernel boots.
> 
> Fantastic! Then it's something in the arm64 for-next/core tree that was
> added since then. The diff isn't huge and one change stands out for me, so
> let me try reverting that and I'll update the branch...

Ok, I updated for-kernelci so that it contains the arm64 for-next/core
branch merged into -rc7, but with a couple of patches reverted on top.

HEAD is	e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
based memory""). Please can you try that?

Cheers,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL: Test?report?for?kernel?5.11.0-rc7 (arm-next)

[Veronika Kabatova]

----- Original Message -----
> From: "Will Deacon" <will@kernel.org>
> To: "Veronika Kabatova" <vkabatov@redhat.com>
> Cc: "catalin marinas" <catalin.marinas@arm.com>, linux-arm-kernel@lists.infradead.org, "CKI Project"
> <cki-project@redhat.com>
> Sent: Wednesday, February 10, 2021 9:32:06 PM
> Subject: Re: ❌ FAIL:	Test?report?for?kernel?5.11.0-rc7 (arm-next)
> 
> On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
> > On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
> > > > > > > The machine in question can on course be somewhat flaky (hard to
> > > > > > > eliminate
> > > > > > > that possibility completely), but I checked our historical data
> > > > > > > and it
> > > > > > > didn't fail to boot a single time other than with these two new
> > > > > > > kernels.
> > > > > > 
> > > > > > So the first thing we should probably try is whether vanilla -rc7
> > > > > > fails
> > > > > > on
> > > > > > the machine causing us problems. If it does, then the arm64 queue
> > > > > > for
> > > > > > 5.12
> > > > > > is out of the equation, if not then we can try a targetted
> > > > > > bisection.
> > > > > > 
> > > > > > Would you be able to try v5.11-rc7 please?
> > > > > 
> > > > > Can do.
> > > > 
> > > > Brill, thanks.
> > > > 
> > > > > Someone snatched up the machine in the meanwhile and appears to have
> > > > > it
> > > > > reserved till next Monday :/ I sincerely hope that they'll release it
> > > > > sooner
> > > > > and have queued up the test job with high priority. Will let you know
> > > > > right
> > > > > as I get the results.
> > > > 
> > > > Just tell them the machine is broken and they really don't want to use
> > > > it
> > > > for anything important ;)
> > > > 
> > > 
> > > Our wishes were granted by the lab fairy and the machine was returned
> > > rather quickly :)
> > > 
> > > The 5.11-rc7 kernel boots.
> > 
> > Fantastic! Then it's something in the arm64 for-next/core tree that was
> > added since then. The diff isn't huge and one change stands out for me, so
> > let me try reverting that and I'll update the branch...
> 
> Ok, I updated for-kernelci so that it contains the arm64 for-next/core
> branch merged into -rc7, but with a couple of patches reverted on top.
> 
> HEAD is	e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
> based memory""). Please can you try that?
> 

Hi,

I didn't actually have to, the autopick picked the machine for the testing
and it happily booted :) \o/

Apparently IT cut off our email sending access because we're sending too
many emails so that's why you don't have the report yet. When we work
around it I'll make sure to retrigger the sending so you have it.


Veronika

> Cheers,
> 
> Will
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)

[Will Deacon]

On Thu, Feb 11, 2021 at 05:46:02AM -0500, Veronika Kabatova wrote:
> > On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
> > > On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
> > > > > > > > The machine in question can on course be somewhat flaky (hard to
> > > > > > > > eliminate
> > > > > > > > that possibility completely), but I checked our historical data
> > > > > > > > and it
> > > > > > > > didn't fail to boot a single time other than with these two new
> > > > > > > > kernels.
> > > > > > > 
> > > > > > > So the first thing we should probably try is whether vanilla -rc7
> > > > > > > fails
> > > > > > > on
> > > > > > > the machine causing us problems. If it does, then the arm64 queue
> > > > > > > for
> > > > > > > 5.12
> > > > > > > is out of the equation, if not then we can try a targetted
> > > > > > > bisection.
> > > > > > > 
> > > > > > > Would you be able to try v5.11-rc7 please?
> > > > > > 
> > > > > > Can do.
> > > > > 
> > > > > Brill, thanks.
> > > > > 
> > > > > > Someone snatched up the machine in the meanwhile and appears to have
> > > > > > it
> > > > > > reserved till next Monday :/ I sincerely hope that they'll release it
> > > > > > sooner
> > > > > > and have queued up the test job with high priority. Will let you know
> > > > > > right
> > > > > > as I get the results.
> > > > > 
> > > > > Just tell them the machine is broken and they really don't want to use
> > > > > it
> > > > > for anything important ;)
> > > > > 
> > > > 
> > > > Our wishes were granted by the lab fairy and the machine was returned
> > > > rather quickly :)
> > > > 
> > > > The 5.11-rc7 kernel boots.
> > > 
> > > Fantastic! Then it's something in the arm64 for-next/core tree that was
> > > added since then. The diff isn't huge and one change stands out for me, so
> > > let me try reverting that and I'll update the branch...
> > 
> > Ok, I updated for-kernelci so that it contains the arm64 for-next/core
> > branch merged into -rc7, but with a couple of patches reverted on top.
> > 
> > HEAD is	e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
> > based memory""). Please can you try that?
> > 
> 
> I didn't actually have to, the autopick picked the machine for the testing
> and it happily booted :) \o/

Phew, so I'll drop those from linux-next as well. Do you know if "earlycon"
works on the problematic machine? If possible, it would be helpful to try
booting the bad kernel with earlycon on the cmdline to see if it manages to
say anything in its dying breath.

> Apparently IT cut off our email sending access because we're sending too
> many emails so that's why you don't have the report yet. When we work
> around it I'll make sure to retrigger the sending so you have it.

That's nice of them...

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)

[Veronika Kabatova]

----- Original Message -----
> From: "Will Deacon" <will@kernel.org>
> To: "Veronika Kabatova" <vkabatov@redhat.com>
> Cc: "catalin marinas" <catalin.marinas@arm.com>, linux-arm-kernel@lists.infradead.org, "CKI Project"
> <cki-project@redhat.com>
> Sent: Thursday, February 11, 2021 12:50:50 PM
> Subject: Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)
> 
> On Thu, Feb 11, 2021 at 05:46:02AM -0500, Veronika Kabatova wrote:
> > > On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
> > > > On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
> > > > > > > > > The machine in question can on course be somewhat flaky (hard
> > > > > > > > > to
> > > > > > > > > eliminate
> > > > > > > > > that possibility completely), but I checked our historical
> > > > > > > > > data
> > > > > > > > > and it
> > > > > > > > > didn't fail to boot a single time other than with these two
> > > > > > > > > new
> > > > > > > > > kernels.
> > > > > > > > 
> > > > > > > > So the first thing we should probably try is whether vanilla
> > > > > > > > -rc7
> > > > > > > > fails
> > > > > > > > on
> > > > > > > > the machine causing us problems. If it does, then the arm64
> > > > > > > > queue
> > > > > > > > for
> > > > > > > > 5.12
> > > > > > > > is out of the equation, if not then we can try a targetted
> > > > > > > > bisection.
> > > > > > > > 
> > > > > > > > Would you be able to try v5.11-rc7 please?
> > > > > > > 
> > > > > > > Can do.
> > > > > > 
> > > > > > Brill, thanks.
> > > > > > 
> > > > > > > Someone snatched up the machine in the meanwhile and appears to
> > > > > > > have
> > > > > > > it
> > > > > > > reserved till next Monday :/ I sincerely hope that they'll
> > > > > > > release it
> > > > > > > sooner
> > > > > > > and have queued up the test job with high priority. Will let you
> > > > > > > know
> > > > > > > right
> > > > > > > as I get the results.
> > > > > > 
> > > > > > Just tell them the machine is broken and they really don't want to
> > > > > > use
> > > > > > it
> > > > > > for anything important ;)
> > > > > > 
> > > > > 
> > > > > Our wishes were granted by the lab fairy and the machine was returned
> > > > > rather quickly :)
> > > > > 
> > > > > The 5.11-rc7 kernel boots.
> > > > 
> > > > Fantastic! Then it's something in the arm64 for-next/core tree that was
> > > > added since then. The diff isn't huge and one change stands out for me,
> > > > so
> > > > let me try reverting that and I'll update the branch...
> > > 
> > > Ok, I updated for-kernelci so that it contains the arm64 for-next/core
> > > branch merged into -rc7, but with a couple of patches reverted on top.
> > > 
> > > HEAD is	e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
> > > based memory""). Please can you try that?
> > > 
> > 
> > I didn't actually have to, the autopick picked the machine for the testing
> > and it happily booted :) \o/
> 
> Phew, so I'll drop those from linux-next as well. Do you know if "earlycon"
> works on the problematic machine? If possible, it would be helpful to try
> booting the bad kernel with earlycon on the cmdline to see if it manages to
> say anything in its dying breath.
> 

No idea, we don't have the option enabled and I can't find any information
about it. I submitted a new job with the option and let's see whether it
works or not. The machine is reserved again so there may be some delays.

> > Apparently IT cut off our email sending access because we're sending too
> > many emails so that's why you don't have the report yet. When we work
> > around it I'll make sure to retrigger the sending so you have it.
> 
> That's nice of them...
> 

Update for the drama loving folks: IT is actually innocent in this.
Someone deployed a misconfigured application in the same cluster as our
reporting system is in, and this application is sending out emails
every 3 seconds. We're collateral damage. Working on resolution, but
right now it seems that the situation is under control.


Veronika

> Will
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)

[Veronika Kabatova]

----- Original Message -----
> From: "Veronika Kabatova" <vkabatov@redhat.com>
> To: "Will Deacon" <will@kernel.org>
> Cc: "catalin marinas" <catalin.marinas@arm.com>, linux-arm-kernel@lists.infradead.org, "CKI Project"
> <cki-project@redhat.com>
> Sent: Thursday, February 11, 2021 1:25:34 PM
> Subject: Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)
> 
> 
> 
> ----- Original Message -----
> > From: "Will Deacon" <will@kernel.org>
> > To: "Veronika Kabatova" <vkabatov@redhat.com>
> > Cc: "catalin marinas" <catalin.marinas@arm.com>,
> > linux-arm-kernel@lists.infradead.org, "CKI Project"
> > <cki-project@redhat.com>
> > Sent: Thursday, February 11, 2021 12:50:50 PM
> > Subject: Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)
> > 
> > On Thu, Feb 11, 2021 at 05:46:02AM -0500, Veronika Kabatova wrote:
> > > > On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
> > > > > On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
> > > > > > > > > > The machine in question can on course be somewhat flaky
> > > > > > > > > > (hard
> > > > > > > > > > to
> > > > > > > > > > eliminate
> > > > > > > > > > that possibility completely), but I checked our historical
> > > > > > > > > > data
> > > > > > > > > > and it
> > > > > > > > > > didn't fail to boot a single time other than with these two
> > > > > > > > > > new
> > > > > > > > > > kernels.
> > > > > > > > > 
> > > > > > > > > So the first thing we should probably try is whether vanilla
> > > > > > > > > -rc7
> > > > > > > > > fails
> > > > > > > > > on
> > > > > > > > > the machine causing us problems. If it does, then the arm64
> > > > > > > > > queue
> > > > > > > > > for
> > > > > > > > > 5.12
> > > > > > > > > is out of the equation, if not then we can try a targetted
> > > > > > > > > bisection.
> > > > > > > > > 
> > > > > > > > > Would you be able to try v5.11-rc7 please?
> > > > > > > > 
> > > > > > > > Can do.
> > > > > > > 
> > > > > > > Brill, thanks.
> > > > > > > 
> > > > > > > > Someone snatched up the machine in the meanwhile and appears to
> > > > > > > > have
> > > > > > > > it
> > > > > > > > reserved till next Monday :/ I sincerely hope that they'll
> > > > > > > > release it
> > > > > > > > sooner
> > > > > > > > and have queued up the test job with high priority. Will let
> > > > > > > > you
> > > > > > > > know
> > > > > > > > right
> > > > > > > > as I get the results.
> > > > > > > 
> > > > > > > Just tell them the machine is broken and they really don't want
> > > > > > > to
> > > > > > > use
> > > > > > > it
> > > > > > > for anything important ;)
> > > > > > > 
> > > > > > 
> > > > > > Our wishes were granted by the lab fairy and the machine was
> > > > > > returned
> > > > > > rather quickly :)
> > > > > > 
> > > > > > The 5.11-rc7 kernel boots.
> > > > > 
> > > > > Fantastic! Then it's something in the arm64 for-next/core tree that
> > > > > was
> > > > > added since then. The diff isn't huge and one change stands out for
> > > > > me,
> > > > > so
> > > > > let me try reverting that and I'll update the branch...
> > > > 
> > > > Ok, I updated for-kernelci so that it contains the arm64 for-next/core
> > > > branch merged into -rc7, but with a couple of patches reverted on top.
> > > > 
> > > > HEAD is	e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for
> > > > ZONE_DEVICE
> > > > based memory""). Please can you try that?
> > > > 
> > > 
> > > I didn't actually have to, the autopick picked the machine for the
> > > testing
> > > and it happily booted :) \o/
> > 
> > Phew, so I'll drop those from linux-next as well. Do you know if "earlycon"
> > works on the problematic machine? If possible, it would be helpful to try
> > booting the bad kernel with earlycon on the cmdline to see if it manages to
> > say anything in its dying breath.
> > 
> 
> No idea, we don't have the option enabled and I can't find any information
> about it. I submitted a new job with the option and let's see whether it
> works or not. The machine is reserved again so there may be some delays.
> 

I finally got access to the machine and don't have any good news. The hang
is from before early printk takes effect. The output is identical to what
I sent before, with nothing resembling kernel lines in there.


Veronika

> > > Apparently IT cut off our email sending access because we're sending too
> > > many emails so that's why you don't have the report yet. When we work
> > > around it I'll make sure to retrigger the sending so you have it.
> > 
> > That's nice of them...
> > 
> 
> Update for the drama loving folks: IT is actually innocent in this.
> Someone deployed a misconfigured application in the same cluster as our
> reporting system is in, and this application is sending out emails
> every 3 seconds. We're collateral damage. Working on resolution, but
> right now it seems that the situation is under control.
> 
> 
> Veronika
> 
> > Will
> > 
> > 
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)

[Will Deacon]

On Mon, Feb 15, 2021 at 08:13:18AM -0500, Veronika Kabatova wrote:
> > > On Thu, Feb 11, 2021 at 05:46:02AM -0500, Veronika Kabatova wrote:
> > > > > On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
> > > > > > On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
> > > > > Ok, I updated for-kernelci so that it contains the arm64 for-next/core
> > > > > branch merged into -rc7, but with a couple of patches reverted on top.
> > > > > 
> > > > > HEAD is	e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for
> > > > > ZONE_DEVICE
> > > > > based memory""). Please can you try that?
> > > > > 
> > > > 
> > > > I didn't actually have to, the autopick picked the machine for the
> > > > testing
> > > > and it happily booted :) \o/
> > > 
> > > Phew, so I'll drop those from linux-next as well. Do you know if "earlycon"
> > > works on the problematic machine? If possible, it would be helpful to try
> > > booting the bad kernel with earlycon on the cmdline to see if it manages to
> > > say anything in its dying breath.
> > > 
> > 
> > No idea, we don't have the option enabled and I can't find any information
> > about it. I submitted a new job with the option and let's see whether it
> > works or not. The machine is reserved again so there may be some delays.
> > 
> 
> I finally got access to the machine and don't have any good news. The hang
> is from before early printk takes effect. The output is identical to what
> I sent before, with nothing resembling kernel lines in there.

Oh well, thanks for giving it a shot. I've already dropped the for-next/mm
branch from my pull request for 5.12, but please keep a note of the
problematic machine if possible so that we can try further debugging in
future.

Cheers,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel