cip/linux-4.19.y-cip baseline: 121 runs, 1 regressions (v4.19.217-cip62) #kernelci

cip/linux-4.19.y-cip baseline: 121 runs, 1 regressions (v4.19.217-cip62) #kernelci

[kernelci.org bot]

cip/linux-4.19.y-cip baseline: 121 runs, 1 regressions (v4.19.217-cip62)

Regressions Summary
-------------------

platform | arch | lab           | compiler | defconfig           | regressions
---------+------+---------------+----------+---------------------+------------
panda    | arm  | lab-collabora | gcc-10   | omap2plus_defconfig | 1          

  Details:  https://kernelci.org/test/job/cip/branch/linux-4.19.y-cip/kernel/v4.19.217-cip62/plan/baseline/

  Test:     baseline
  Tree:     cip
  Branch:   linux-4.19.y-cip
  Describe: v4.19.217-cip62
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
  SHA:      dc62e26e3be875a7324b85b8274c13a335e610dd 


Test Regressions
---------------- 


platform | arch | lab           | compiler | defconfig           | regressions
---------+------+---------------+----------+---------------------+------------
panda    | arm  | lab-collabora | gcc-10   | omap2plus_defconfig | 1          

  Details:     https://kernelci.org/test/plan/id/61a587a0ab3b0079bd18f6d7

  Results:     5 PASS, 1 FAIL, 0 SKIP
  Full config: omap2plus_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.txt
  HTML log:    https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05-6-g8983f3b738df/armel/baseline/rootfs.cpio.gz 


  * baseline.dmesg.emerg: https://kernelci.org/test/case/id/61a587a0ab3b0079bd18f6dd
        new failure (last pass: v4.19.216-cip61)
        2 lines

    2021-11-30T02:08:20.120053  kern  :emerg : BUG: spinlock bad magic on CPU#0, udevd/110
    2021-11-30T02:08:20.129545  kern  :emerg :  lock: emif_lock+0x0/0xffffecfc [emif], .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1
    2021-11-30T02:08:20.143995  <8>[   21.198120] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=emerg RESULT=fail UNITS=lines MEASUREMENT=2>   
  

Re: [cip-dev] cip/linux-4.19.y-cip baseline: 121 runs, 1 regressions (v4.19.217-cip62) #kernelci

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2309 bytes --]

Hi!

> platform | arch | lab           | compiler | defconfig           | regressions
> ---------+------+---------------+----------+---------------------+------------
> panda    | arm  | lab-collabora | gcc-10   | omap2plus_defconfig | 1          
> 
>   Details:     https://kernelci.org/test/plan/id/61a587a0ab3b0079bd18f6d7
> 
>   Results:     5 PASS, 1 FAIL, 0 SKIP
>   Full config: omap2plus_defconfig
>   Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
>   Plain log:   https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.txt
>   HTML log:
> https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.html

So this has enough information to tell me that we have a kernel
problem there, good:

02:08:17.792502  <6>[   18.847320] omap-mailbox 4a0f4000.mailbox: omap
mailbox rev 0x400
02:08:17.800596  <0>[   18.857543] BUG: spinlock bad magic on CPU#0,
udevd/110
02:08:17.808843  <6>[   18.857788] emif 4c000000.emif: emif_probe:
device configured with addr = (ptrval) and IRQ26
02:08:17.818753  <0>[   18.863311]  lock: emif_lock+0x0/0xffffecfc
[emif], .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1
02:08:17.825277  <4>[   18.877502] CPU: 0 PID: 110 Comm: udevd Not
tainted 4.19.217-cip62 #1
02:08:17.831655  <4>[   18.888275] Hardware name: Generic OMAP4
(Flattened Device Tree)
02:08:17.839660  <4>[   18.894195] [<c01123bc>] (unwind_backtrace)
from [<c010cc28>] (show_stack+0x10/0x14)
02:08:17.847570  <4>[   18.900482] [<c010cc28>] (show_stack) from
[<c0949dd4>] (dump_stack+0xe0/0x114)
02:08:17.855211  <4>[   18.905761] [<c0949dd4>] (dump_stack) from
[<c01a9168>] (do_raw_spin_lock+0xbc/0x124)

Unfortunately, not enough information to debug the problem, I'm
afraid. Questions to start debugging this would be:

a) is the problem deterministic?

b) is the problem present in v4.19.217 ?

c) what were the last -stable and -cip kernels that worked?

d) what device/module is causing the BUG?

Best regards,
								Pavel
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [cip-dev] cip/linux-4.19.y-cip baseline: 121 runs, 1 regressions (v4.19.217-cip62) #kernelci

[Alice Ferrazzi]

On Tue, Nov 30, 2021 at 5:41 PM Pavel Machek <pavel@denx.de> wrote:
>
> Hi!
>
> > platform | arch | lab           | compiler | defconfig           | regressions
> > ---------+------+---------------+----------+---------------------+------------
> > panda    | arm  | lab-collabora | gcc-10   | omap2plus_defconfig | 1
> >
> >   Details:     https://kernelci.org/test/plan/id/61a587a0ab3b0079bd18f6d7
> >
> >   Results:     5 PASS, 1 FAIL, 0 SKIP
> >   Full config: omap2plus_defconfig
> >   Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
> >   Plain log:   https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.txt
> >   HTML log:
> > https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.html
>
> So this has enough information to tell me that we have a kernel
> problem there, good:
>
> 02:08:17.792502  <6>[   18.847320] omap-mailbox 4a0f4000.mailbox: omap
> mailbox rev 0x400
> 02:08:17.800596  <0>[   18.857543] BUG: spinlock bad magic on CPU#0,
> udevd/110
> 02:08:17.808843  <6>[   18.857788] emif 4c000000.emif: emif_probe:
> device configured with addr = (ptrval) and IRQ26
> 02:08:17.818753  <0>[   18.863311]  lock: emif_lock+0x0/0xffffecfc
> [emif], .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1
> 02:08:17.825277  <4>[   18.877502] CPU: 0 PID: 110 Comm: udevd Not
> tainted 4.19.217-cip62 #1
> 02:08:17.831655  <4>[   18.888275] Hardware name: Generic OMAP4
> (Flattened Device Tree)
> 02:08:17.839660  <4>[   18.894195] [<c01123bc>] (unwind_backtrace)
> from [<c010cc28>] (show_stack+0x10/0x14)
> 02:08:17.847570  <4>[   18.900482] [<c010cc28>] (show_stack) from
> [<c0949dd4>] (dump_stack+0xe0/0x114)
> 02:08:17.855211  <4>[   18.905761] [<c0949dd4>] (dump_stack) from
> [<c01a9168>] (do_raw_spin_lock+0xbc/0x124)
>
> Unfortunately, not enough information to debug the problem, I'm
> afraid. Questions to start debugging this would be:
>
> a) is the problem deterministic?
>

as the testing board are actually different is unclear.
The board and results can be seen here:
https://linux.kernelci.org/test/plan/id/61a587a0ab3b0079bd18f6d7/

the output log is here:
https://storage.kernelci.org/cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.html
working board example log is here (works for 4.19.216-cip61 but using
a board on baylibre that could be different from the board on
collabora):
https://storage.kernelci.org/cip/linux-4.19.y-cip/v4.19.216-cip61/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-panda.html
looks like there is also a passed test on lab-baylibre for the panda
board with the same kernel:
https://storage.kernelci.org/cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-panda.html

The job json file for replicate the job on a lava lab with panda board
can be found here:
https://lava.collabora.co.uk/scheduler/job/5113437/definition


> b) is the problem present in v4.19.217 ?
>

here is the diff with 4.19.217 stable
https://www.diffchecker.com/JjmXiP7b

> c) what were the last -stable and -cip kernels that worked?
>

if you click on the link on the details or the link here under:
https://linux.kernelci.org/test/case/id/61a587a0ab3b0079bd18f6dd/
is wrote:
New regression, last pass: v4.19.216-cip61

This is the one that passed:
https://linux.kernelci.org/test/plan/id/618f6407f33d0dd1e633595d/


- About bisecting the kernel.
We are currently in the process of activating CIP to the list of
automated bisected kernel
for getting more information on this and future problems.
The following issue is the request to KernelCI for adding CIP to the
automated bisected kernel list.
https://github.com/kernelci/kernelci-core/issues/922



Thanks,
Alice


--
======================================
Cybertrust Japan Co.,Ltd.
Alice Ferrazzi
alice.ferrazzi@miraclelinux.com
======================================