All of lore.kernel.org
 help / color / mirror / Atom feed
* KernelCI reports
@ 2022-06-23 18:13 Pavel Machek
  2022-07-14  9:20 ` Alice Ferrazzi
  0 siblings, 1 reply; 6+ messages in thread
From: Pavel Machek @ 2022-06-23 18:13 UTC (permalink / raw)
  To: alice.ferrazzi, cip-dev

[-- Attachment #1: Type: text/plain, Size: 5860 bytes --]

Hi!

First... what I am currently using for testing: gitlab; I simply watch
for green ticks there. It takes ~15 minutes to get results. Red cross
in compile stage is pretty sure sign of build error. Red cross in
testing means I need to look at the results, but it is pretty easy to
see "network timeout" or "board unavailable" problems, and mostly I
can just hit retry and get the green tick.

What wories me about gitlab is that we have single board for 4.4-st,
and just two for 4.4-cip.

Now, kernelci.

https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html

I got this link in the irc meeting:

https://linux.kernelci.org/job/cip/

Lets start with build results.

linux-5.10.y-cip      v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3

I click on "3" and get build error summary:

Errors Summary
2      arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
2	arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

...but those are not kernel problems, that's buildsystem incompatible
with config or something like that. Nothing for me to solve.

Let me go through more:

linux-4.19.y-cip	v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
linux-4.19.y-cip	v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
linux-4.19.y-cip-rt	v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
linux-4.4.y-cip	v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
linux-5.10.y-cip-rt	v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
linux-5.10.y-cip	v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
linux-4.19.y-cip	v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120

That's 19 errors total + outlier of v4.19.239-cip72 with 120
errors. None of the 19 errors is real. KernelCI needs to fix their
configs/build environment.

120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as

cm_x2xx_defconfig ‐ arm ‐ gcc-10  0 warnings 0 errors. When I click for details I get "No test results found".

https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/

Nothing for me to debug, either, agreed? So lets look at test results.

Branch	Kernel Commit Build Status	Test Results	Date
linux-5.10.y-cip      v5.10.121-cip9	bd24696dd7ad23517733728da675a00cd3073916 169 7 3  2457 256 21 2022-06-12

21 failures.

If I click that, I can see that most regressions (11) are at baseline
test plan. The machine that looks relevant to us is
qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --

https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/

New regression, last pass: v5.10.112-cip6

I get to
https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
which looks quite useful, it lists successful run and failed run,
v5.10.121-cip9 fails while v5.10.112-cip6 works.

Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .

And sorry, no I don't see anything I could debug there. Did the docker
fail to start? Hmm. Lets compare it with the older results.

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible. 

Lets try one more.

rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10

https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/

It gets me to:

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

I see some errors... Like missing firmware (not a kernel problem). At
the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?

 1368 12:21:39.212881  /usr/bin/tpm2_getcap
  1369 12:21:39.251829  ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
   1370 12:21:39.261387  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error
    1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
     1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
      1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap

Userland seems to say that it does not support this device. I guess I could compare it with older boot?

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

Aha, no, could not, 404.

Now, let's go through Jan's
example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
. Again, errors look like missconfigured build system, with

arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.

I spent quite some time trying to click around to find an error I
could debug. I could not find one. All of above looks like kernelCI
configuration problems. Can they be fixed and/or filtered out?

Thanks and best regards,
								Pavel

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: KernelCI reports
  2022-06-23 18:13 KernelCI reports Pavel Machek
@ 2022-07-14  9:20 ` Alice Ferrazzi
  2022-07-14  9:45   ` [cip-dev] " Bezdeka, Florian
  0 siblings, 1 reply; 6+ messages in thread
From: Alice Ferrazzi @ 2022-07-14  9:20 UTC (permalink / raw)
  To: Pavel Machek; +Cc: cip-dev, chris.paterson2

Hi Pavel,

On Fri, Jun 24, 2022 at 3:13 AM Pavel Machek <pavel@denx.de> wrote:
>
> Hi!
>
> First... what I am currently using for testing: gitlab; I simply watch
> for green ticks there. It takes ~15 minutes to get results. Red cross
> in compile stage is pretty sure sign of build error. Red cross in
> testing means I need to look at the results, but it is pretty easy to
> see "network timeout" or "board unavailable" problems, and mostly I
> can just hit retry and get the green tick.
>
> What wories me about gitlab is that we have single board for 4.4-st,
> and just two for 4.4-cip.
>
> Now, kernelci.
>
> https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html
>
> I got this link in the irc meeting:
>
> https://linux.kernelci.org/job/cip/
>
> Lets start with build results.
>
> linux-5.10.y-cip      v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3
>
> I click on "3" and get build error summary:
>
> Errors Summary
> 2      arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
> 2       arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
>
> ...but those are not kernel problems, that's buildsystem incompatible
> with config or something like that. Nothing for me to solve.
>
> Let me go through more:
>
> linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
> linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
> linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
> linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
> linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
> linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
> linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120
>
> That's 19 errors total + outlier of v4.19.239-cip72 with 120
> errors. None of the 19 errors is real. KernelCI needs to fix their
> configs/build environment.
>
> 120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as
>
> cm_x2xx_defconfig ‐ arm ‐ gcc-10  0 warnings 0 errors. When I click for details I get "No test results found".
>
> https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/
>
> Nothing for me to debug, either, agreed? So lets look at test results.
>
> Branch  Kernel Commit Build Status      Test Results    Date
> linux-5.10.y-cip      v5.10.121-cip9    bd24696dd7ad23517733728da675a00cd3073916 169 7 3  2457 256 21 2022-06-12
>
> 21 failures.
>
> If I click that, I can see that most regressions (11) are at baseline
> test plan. The machine that looks relevant to us is
> qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --
>
> https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/
>
> New regression, last pass: v5.10.112-cip6
>
> I get to
> https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
> which looks quite useful, it lists successful run and failed run,
> v5.10.121-cip9 fails while v5.10.112-cip6 works.

Yes, checking for regressions is quite useful.

>
> Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .
>
> And sorry, no I don't see anything I could debug there. Did the docker
> fail to start? Hmm. Lets compare it with the older results.
>
> https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.

The garbage collector removes everything that's older than 28 days.

>
> Lets try one more.
>
> rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10
>
> https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/
>
> It gets me to:
>
> https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
>
> I see some errors... Like missing firmware (not a kernel problem). At
> the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?
>
>  1368 12:21:39.212881  /usr/bin/tpm2_getcap
>   1369 12:21:39.251829  ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
>    1370 12:21:39.261387  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error
>     1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
>      1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
>       1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap
>
> Userland seems to say that it does not support this device. I guess I could compare it with older boot?
>
> https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
>
> Aha, no, could not, 404.

same here, the garbage collector removes everything that's older than 28 days.

>
> Now, let's go through Jan's
> example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
> . Again, errors look like missconfigured build system, with
>
> arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
> arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
>
> Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.

same here, the garbage collector removes everything that's older than 28 days.

>
> I spent quite some time trying to click around to find an error I
> could debug. I could not find one. All of above looks like kernelCI
> configuration problems. Can they be fixed and/or filtered out?
>

If you can write which configurations you want to filter out (because
they are not needed for CIP)
to KernelCI, KernelCI could blocklist such configurations.

for reference: https://github.com/kernelci/kernelci-core/issues/1053#issuecomment-1050679725

-- 
======================================
Cybertrust Japan Co.,Ltd.
Alice Ferrazzi
alice.ferrazzi@miraclelinux.com
======================================


^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [cip-dev] KernelCI reports
  2022-07-14  9:20 ` Alice Ferrazzi
@ 2022-07-14  9:45   ` Bezdeka, Florian
  2022-07-14 10:17     ` Alice Ferrazzi
  0 siblings, 1 reply; 6+ messages in thread
From: Bezdeka, Florian @ 2022-07-14  9:45 UTC (permalink / raw)
  To: cip-dev, pavel; +Cc: chris.paterson2

On Thu, 2022-07-14 at 18:20 +0900, Alice Ferrazzi via lists.cip-
project.org wrote:
> Hi Pavel,
> 
> On Fri, Jun 24, 2022 at 3:13 AM Pavel Machek <pavel@denx.de> wrote:
> > 
> > Hi!
> > 
> > First... what I am currently using for testing: gitlab; I simply watch
> > for green ticks there. It takes ~15 minutes to get results. Red cross
> > in compile stage is pretty sure sign of build error. Red cross in
> > testing means I need to look at the results, but it is pretty easy to
> > see "network timeout" or "board unavailable" problems, and mostly I
> > can just hit retry and get the green tick.
> > 
> > What wories me about gitlab is that we have single board for 4.4-st,
> > and just two for 4.4-cip.
> > 
> > Now, kernelci.
> > 
> > https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html
> > 
> > I got this link in the irc meeting:
> > 
> > https://linux.kernelci.org/job/cip/
> > 
> > Lets start with build results.
> > 
> > linux-5.10.y-cip      v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3
> > 
> > I click on "3" and get build error summary:
> > 
> > Errors Summary
> > 2      arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
> > 2       arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
> > 
> > ...but those are not kernel problems, that's buildsystem incompatible
> > with config or something like that. Nothing for me to solve.
> > 
> > Let me go through more:
> > 
> > linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
> > linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
> > linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
> > linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
> > linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
> > linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
> > linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120
> > 
> > That's 19 errors total + outlier of v4.19.239-cip72 with 120
> > errors. None of the 19 errors is real. KernelCI needs to fix their
> > configs/build environment.
> > 
> > 120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as
> > 
> > cm_x2xx_defconfig ‐ arm ‐ gcc-10  0 warnings 0 errors. When I click for details I get "No test results found".
> > 
> > https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/
> > 
> > Nothing for me to debug, either, agreed? So lets look at test results.
> > 
> > Branch  Kernel Commit Build Status      Test Results    Date
> > linux-5.10.y-cip      v5.10.121-cip9    bd24696dd7ad23517733728da675a00cd3073916 169 7 3  2457 256 21 2022-06-12
> > 
> > 21 failures.
> > 
> > If I click that, I can see that most regressions (11) are at baseline
> > test plan. The machine that looks relevant to us is
> > qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --
> > 
> > https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/
> > 
> > New regression, last pass: v5.10.112-cip6
> > 
> > I get to
> > https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
> > which looks quite useful, it lists successful run and failed run,
> > v5.10.121-cip9 fails while v5.10.112-cip6 works.
> 
> Yes, checking for regressions is quite useful.
> 
> > 
> > Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .
> > 
> > And sorry, no I don't see anything I could debug there. Did the docker
> > fail to start? Hmm. Lets compare it with the older results.
> > 
> > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.
> 
> The garbage collector removes everything that's older than 28 days.
> 
> > 
> > Lets try one more.
> > 
> > rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10
> > 
> > https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/
> > 
> > It gets me to:
> > 
> > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
> > 
> > I see some errors... Like missing firmware (not a kernel problem). At
> > the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?
> > 
> >  1368 12:21:39.212881  /usr/bin/tpm2_getcap
> >   1369 12:21:39.251829  ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
> >    1370 12:21:39.261387  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error
> >     1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
> >      1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
> >       1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap
> > 
> > Userland seems to say that it does not support this device. I guess I could compare it with older boot?
> > 
> > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
> > 
> > Aha, no, could not, 404.
> 
> same here, the garbage collector removes everything that's older than 28 days.
> 
> > 
> > Now, let's go through Jan's
> > example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
> > . Again, errors look like missconfigured build system, with
> > 
> > arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
> > arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
> > 
> > Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.
> 
> same here, the garbage collector removes everything that's older than 28 days.
> 
> > 
> > I spent quite some time trying to click around to find an error I
> > could debug. I could not find one. All of above looks like kernelCI
> > configuration problems. Can they be fixed and/or filtered out?
> > 
> 
> If you can write which configurations you want to filter out (because
> they are not needed for CIP)
> to KernelCI, KernelCI could blocklist such configurations.
> 
> for reference: https://github.com/kernelci/kernelci-core/issues/1053#issuecomment-1050679725

Jan asked me to check the kernelci related issues you summarized here
related to the 4.4-cip branch. I tried to find the latest 4.4 test
report that failed. Can someone point me there?

With some guidance from Alice it should be possible for me to come up
with a exclude/blocklist to bring 4.4 back to a "green" state.
Afterwards we would have to sort out all the problems (decoupled from
Pavel's testing)

Best regards,
Florian

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#8754): https://lists.cip-project.org/g/cip-dev/message/8754
> Mute This Topic: https://lists.cip-project.org/mt/91949361/5792637
> Group Owner: cip-dev+owner@lists.cip-project.org
> Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/9882880/5792637/2090341516/xyzzy [florian.bezdeka@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [cip-dev] KernelCI reports
  2022-07-14  9:45   ` [cip-dev] " Bezdeka, Florian
@ 2022-07-14 10:17     ` Alice Ferrazzi
  2022-07-14 10:27       ` Bezdeka, Florian
  0 siblings, 1 reply; 6+ messages in thread
From: Alice Ferrazzi @ 2022-07-14 10:17 UTC (permalink / raw)
  To: cip-dev; +Cc: pavel, chris.paterson2, florian.bezdeka

On Thu, Jul 14, 2022 at 6:45 PM florian.bezdeka@siemens.com
<florian.bezdeka@siemens.com> wrote:
>
> On Thu, 2022-07-14 at 18:20 +0900, Alice Ferrazzi via lists.cip-
> project.org wrote:
> > Hi Pavel,
> >
> > On Fri, Jun 24, 2022 at 3:13 AM Pavel Machek <pavel@denx.de> wrote:
> > >
> > > Hi!
> > >
> > > First... what I am currently using for testing: gitlab; I simply watch
> > > for green ticks there. It takes ~15 minutes to get results. Red cross
> > > in compile stage is pretty sure sign of build error. Red cross in
> > > testing means I need to look at the results, but it is pretty easy to
> > > see "network timeout" or "board unavailable" problems, and mostly I
> > > can just hit retry and get the green tick.
> > >
> > > What wories me about gitlab is that we have single board for 4.4-st,
> > > and just two for 4.4-cip.
> > >
> > > Now, kernelci.
> > >
> > > https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html
> > >
> > > I got this link in the irc meeting:
> > >
> > > https://linux.kernelci.org/job/cip/
> > >
> > > Lets start with build results.
> > >
> > > linux-5.10.y-cip      v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3
> > >
> > > I click on "3" and get build error summary:
> > >
> > > Errors Summary
> > > 2      arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
> > > 2       arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
> > >
> > > ...but those are not kernel problems, that's buildsystem incompatible
> > > with config or something like that. Nothing for me to solve.
> > >
> > > Let me go through more:
> > >
> > > linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
> > > linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
> > > linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
> > > linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
> > > linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
> > > linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
> > > linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120
> > >
> > > That's 19 errors total + outlier of v4.19.239-cip72 with 120
> > > errors. None of the 19 errors is real. KernelCI needs to fix their
> > > configs/build environment.
> > >
> > > 120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as
> > >
> > > cm_x2xx_defconfig ‐ arm ‐ gcc-10  0 warnings 0 errors. When I click for details I get "No test results found".
> > >
> > > https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/
> > >
> > > Nothing for me to debug, either, agreed? So lets look at test results.
> > >
> > > Branch  Kernel Commit Build Status      Test Results    Date
> > > linux-5.10.y-cip      v5.10.121-cip9    bd24696dd7ad23517733728da675a00cd3073916 169 7 3  2457 256 21 2022-06-12
> > >
> > > 21 failures.
> > >
> > > If I click that, I can see that most regressions (11) are at baseline
> > > test plan. The machine that looks relevant to us is
> > > qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --
> > >
> > > https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/
> > >
> > > New regression, last pass: v5.10.112-cip6
> > >
> > > I get to
> > > https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
> > > which looks quite useful, it lists successful run and failed run,
> > > v5.10.121-cip9 fails while v5.10.112-cip6 works.
> >
> > Yes, checking for regressions is quite useful.
> >
> > >
> > > Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .
> > >
> > > And sorry, no I don't see anything I could debug there. Did the docker
> > > fail to start? Hmm. Lets compare it with the older results.
> > >
> > > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.
> >
> > The garbage collector removes everything that's older than 28 days.
> >
> > >
> > > Lets try one more.
> > >
> > > rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10
> > >
> > > https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/
> > >
> > > It gets me to:
> > >
> > > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
> > >
> > > I see some errors... Like missing firmware (not a kernel problem). At
> > > the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?
> > >
> > >  1368 12:21:39.212881  /usr/bin/tpm2_getcap
> > >   1369 12:21:39.251829  ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
> > >    1370 12:21:39.261387  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error
> > >     1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
> > >      1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
> > >       1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap
> > >
> > > Userland seems to say that it does not support this device. I guess I could compare it with older boot?
> > >
> > > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
> > >
> > > Aha, no, could not, 404.
> >
> > same here, the garbage collector removes everything that's older than 28 days.
> >
> > >
> > > Now, let's go through Jan's
> > > example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
> > > . Again, errors look like missconfigured build system, with
> > >
> > > arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
> > > arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
> > >
> > > Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.
> >
> > same here, the garbage collector removes everything that's older than 28 days.
> >
> > >
> > > I spent quite some time trying to click around to find an error I
> > > could debug. I could not find one. All of above looks like kernelCI
> > > configuration problems. Can they be fixed and/or filtered out?
> > >
> >
> > If you can write which configurations you want to filter out (because
> > they are not needed for CIP)
> > to KernelCI, KernelCI could blocklist such configurations.
> >
> > for reference: https://github.com/kernelci/kernelci-core/issues/1053#issuecomment-1050679725
>
> Jan asked me to check the kernelci related issues you summarized here
> related to the 4.4-cip branch. I tried to find the latest 4.4 test
> report that failed. Can someone point me there?
>
> With some guidance from Alice it should be possible for me to come up
> with a exclude/blocklist to bring 4.4 back to a "green" state.
> Afterwards we would have to sort out all the problems (decoupled from
> Pavel's testing)
>

Hello Florian

Latest build 4.4 cip is here
https://linux.kernelci.org/test/job/cip/branch/linux-4.4.y-cip/kernel/v4.4.302-cip69-296-g52de9431d314/
is a really old build as the latest 4.4 cip tag is v4.4.302-cip69
tagged on date 2022-04-04 13:18:30 +0900
Unfortunately as I previously said KernelCI garbage collector removes
everything that's older than 28 days.
You can find CIP builds here: https://linux.kernelci.org/job/cip/

Thanks,
Alicef


-- 
======================================
Cybertrust Japan Co.,Ltd.
Alice Ferrazzi
alice.ferrazzi@miraclelinux.com
======================================


^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [cip-dev] KernelCI reports
  2022-07-14 10:17     ` Alice Ferrazzi
@ 2022-07-14 10:27       ` Bezdeka, Florian
  2022-07-14 10:45         ` Alice Ferrazzi
  0 siblings, 1 reply; 6+ messages in thread
From: Bezdeka, Florian @ 2022-07-14 10:27 UTC (permalink / raw)
  To: cip-dev, alice.ferrazzi; +Cc: pavel, chris.paterson2

On Thu, 2022-07-14 at 19:17 +0900, Alice Ferrazzi wrote:
> On Thu, Jul 14, 2022 at 6:45 PM florian.bezdeka@siemens.com
> <florian.bezdeka@siemens.com> wrote:
> > 
> > On Thu, 2022-07-14 at 18:20 +0900, Alice Ferrazzi via lists.cip-
> > project.org wrote:
> > > Hi Pavel,
> > > 
> > > On Fri, Jun 24, 2022 at 3:13 AM Pavel Machek <pavel@denx.de> wrote:
> > > > 
> > > > Hi!
> > > > 
> > > > First... what I am currently using for testing: gitlab; I simply watch
> > > > for green ticks there. It takes ~15 minutes to get results. Red cross
> > > > in compile stage is pretty sure sign of build error. Red cross in
> > > > testing means I need to look at the results, but it is pretty easy to
> > > > see "network timeout" or "board unavailable" problems, and mostly I
> > > > can just hit retry and get the green tick.
> > > > 
> > > > What wories me about gitlab is that we have single board for 4.4-st,
> > > > and just two for 4.4-cip.
> > > > 
> > > > Now, kernelci.
> > > > 
> > > > https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html
> > > > 
> > > > I got this link in the irc meeting:
> > > > 
> > > > https://linux.kernelci.org/job/cip/
> > > > 
> > > > Lets start with build results.
> > > > 
> > > > linux-5.10.y-cip      v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3
> > > > 
> > > > I click on "3" and get build error summary:
> > > > 
> > > > Errors Summary
> > > > 2      arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
> > > > 2       arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
> > > > 
> > > > ...but those are not kernel problems, that's buildsystem incompatible
> > > > with config or something like that. Nothing for me to solve.
> > > > 
> > > > Let me go through more:
> > > > 
> > > > linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
> > > > linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
> > > > linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
> > > > linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
> > > > linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
> > > > linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
> > > > linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120
> > > > 
> > > > That's 19 errors total + outlier of v4.19.239-cip72 with 120
> > > > errors. None of the 19 errors is real. KernelCI needs to fix their
> > > > configs/build environment.
> > > > 
> > > > 120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as
> > > > 
> > > > cm_x2xx_defconfig ‐ arm ‐ gcc-10  0 warnings 0 errors. When I click for details I get "No test results found".
> > > > 
> > > > https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/
> > > > 
> > > > Nothing for me to debug, either, agreed? So lets look at test results.
> > > > 
> > > > Branch  Kernel Commit Build Status      Test Results    Date
> > > > linux-5.10.y-cip      v5.10.121-cip9    bd24696dd7ad23517733728da675a00cd3073916 169 7 3  2457 256 21 2022-06-12
> > > > 
> > > > 21 failures.
> > > > 
> > > > If I click that, I can see that most regressions (11) are at baseline
> > > > test plan. The machine that looks relevant to us is
> > > > qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --
> > > > 
> > > > https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/
> > > > 
> > > > New regression, last pass: v5.10.112-cip6
> > > > 
> > > > I get to
> > > > https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
> > > > which looks quite useful, it lists successful run and failed run,
> > > > v5.10.121-cip9 fails while v5.10.112-cip6 works.
> > > 
> > > Yes, checking for regressions is quite useful.
> > > 
> > > > 
> > > > Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .
> > > > 
> > > > And sorry, no I don't see anything I could debug there. Did the docker
> > > > fail to start? Hmm. Lets compare it with the older results.
> > > > 
> > > > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.
> > > 
> > > The garbage collector removes everything that's older than 28 days.
> > > 
> > > > 
> > > > Lets try one more.
> > > > 
> > > > rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10
> > > > 
> > > > https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/
> > > > 
> > > > It gets me to:
> > > > 
> > > > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
> > > > 
> > > > I see some errors... Like missing firmware (not a kernel problem). At
> > > > the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?
> > > > 
> > > >  1368 12:21:39.212881  /usr/bin/tpm2_getcap
> > > >   1369 12:21:39.251829  ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
> > > >    1370 12:21:39.261387  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error
> > > >     1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
> > > >      1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
> > > >       1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap
> > > > 
> > > > Userland seems to say that it does not support this device. I guess I could compare it with older boot?
> > > > 
> > > > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
> > > > 
> > > > Aha, no, could not, 404.
> > > 
> > > same here, the garbage collector removes everything that's older than 28 days.
> > > 
> > > > 
> > > > Now, let's go through Jan's
> > > > example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
> > > > . Again, errors look like missconfigured build system, with
> > > > 
> > > > arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
> > > > arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
> > > > 
> > > > Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.
> > > 
> > > same here, the garbage collector removes everything that's older than 28 days.
> > > 
> > > > 
> > > > I spent quite some time trying to click around to find an error I
> > > > could debug. I could not find one. All of above looks like kernelCI
> > > > configuration problems. Can they be fixed and/or filtered out?
> > > > 
> > > 
> > > If you can write which configurations you want to filter out (because
> > > they are not needed for CIP)
> > > to KernelCI, KernelCI could blocklist such configurations.
> > > 
> > > for reference: https://github.com/kernelci/kernelci-core/issues/1053#issuecomment-1050679725
> > 
> > Jan asked me to check the kernelci related issues you summarized here
> > related to the 4.4-cip branch. I tried to find the latest 4.4 test
> > report that failed. Can someone point me there?
> > 
> > With some guidance from Alice it should be possible for me to come up
> > with a exclude/blocklist to bring 4.4 back to a "green" state.
> > Afterwards we would have to sort out all the problems (decoupled from
> > Pavel's testing)
> > 
> 
> Hello Florian
> 
> Latest build 4.4 cip is here
> https://linux.kernelci.org/test/job/cip/branch/linux-4.4.y-cip/kernel/v4.4.302-cip69-296-g52de9431d314/
> is a really old build as the latest 4.4 cip tag is v4.4.302-cip69
> tagged on date 2022-04-04 13:18:30 +0900
> Unfortunately as I previously said KernelCI garbage collector removes
> everything that's older than 28 days.
> You can find CIP builds here: https://linux.kernelci.org/job/cip/

Thanks a lot Alice!

Any chance for me to trigger a re-build? 

> 
> Thanks,
> Alicef
> 
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [cip-dev] KernelCI reports
  2022-07-14 10:27       ` Bezdeka, Florian
@ 2022-07-14 10:45         ` Alice Ferrazzi
  0 siblings, 0 replies; 6+ messages in thread
From: Alice Ferrazzi @ 2022-07-14 10:45 UTC (permalink / raw)
  To: Bezdeka, Florian; +Cc: cip-dev, pavel, chris.paterson2

hello Florian,

On Thu, Jul 14, 2022 at 7:27 PM Bezdeka, Florian
<florian.bezdeka@siemens.com> wrote:
>
> On Thu, 2022-07-14 at 19:17 +0900, Alice Ferrazzi wrote:
> > On Thu, Jul 14, 2022 at 6:45 PM florian.bezdeka@siemens.com
> > <florian.bezdeka@siemens.com> wrote:
> > >
> > > On Thu, 2022-07-14 at 18:20 +0900, Alice Ferrazzi via lists.cip-
> > > project.org wrote:
> > > > Hi Pavel,
> > > >
> > > > On Fri, Jun 24, 2022 at 3:13 AM Pavel Machek <pavel@denx.de> wrote:
> > > > >
> > > > > Hi!
> > > > >
> > > > > First... what I am currently using for testing: gitlab; I simply watch
> > > > > for green ticks there. It takes ~15 minutes to get results. Red cross
> > > > > in compile stage is pretty sure sign of build error. Red cross in
> > > > > testing means I need to look at the results, but it is pretty easy to
> > > > > see "network timeout" or "board unavailable" problems, and mostly I
> > > > > can just hit retry and get the green tick.
> > > > >
> > > > > What wories me about gitlab is that we have single board for 4.4-st,
> > > > > and just two for 4.4-cip.
> > > > >
> > > > > Now, kernelci.
> > > > >
> > > > > https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html
> > > > >
> > > > > I got this link in the irc meeting:
> > > > >
> > > > > https://linux.kernelci.org/job/cip/
> > > > >
> > > > > Lets start with build results.
> > > > >
> > > > > linux-5.10.y-cip      v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3
> > > > >
> > > > > I click on "3" and get build error summary:
> > > > >
> > > > > Errors Summary
> > > > > 2      arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
> > > > > 2       arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
> > > > >
> > > > > ...but those are not kernel problems, that's buildsystem incompatible
> > > > > with config or something like that. Nothing for me to solve.
> > > > >
> > > > > Let me go through more:
> > > > >
> > > > > linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
> > > > > linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
> > > > > linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
> > > > > linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
> > > > > linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
> > > > > linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
> > > > > linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120
> > > > >
> > > > > That's 19 errors total + outlier of v4.19.239-cip72 with 120
> > > > > errors. None of the 19 errors is real. KernelCI needs to fix their
> > > > > configs/build environment.
> > > > >
> > > > > 120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as
> > > > >
> > > > > cm_x2xx_defconfig ‐ arm ‐ gcc-10  0 warnings 0 errors. When I click for details I get "No test results found".
> > > > >
> > > > > https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/
> > > > >
> > > > > Nothing for me to debug, either, agreed? So lets look at test results.
> > > > >
> > > > > Branch  Kernel Commit Build Status      Test Results    Date
> > > > > linux-5.10.y-cip      v5.10.121-cip9    bd24696dd7ad23517733728da675a00cd3073916 169 7 3  2457 256 21 2022-06-12
> > > > >
> > > > > 21 failures.
> > > > >
> > > > > If I click that, I can see that most regressions (11) are at baseline
> > > > > test plan. The machine that looks relevant to us is
> > > > > qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --
> > > > >
> > > > > https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/
> > > > >
> > > > > New regression, last pass: v5.10.112-cip6
> > > > >
> > > > > I get to
> > > > > https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
> > > > > which looks quite useful, it lists successful run and failed run,
> > > > > v5.10.121-cip9 fails while v5.10.112-cip6 works.
> > > >
> > > > Yes, checking for regressions is quite useful.
> > > >
> > > > >
> > > > > Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .
> > > > >
> > > > > And sorry, no I don't see anything I could debug there. Did the docker
> > > > > fail to start? Hmm. Lets compare it with the older results.
> > > > >
> > > > > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.
> > > >
> > > > The garbage collector removes everything that's older than 28 days.
> > > >
> > > > >
> > > > > Lets try one more.
> > > > >
> > > > > rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10
> > > > >
> > > > > https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/
> > > > >
> > > > > It gets me to:
> > > > >
> > > > > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
> > > > >
> > > > > I see some errors... Like missing firmware (not a kernel problem). At
> > > > > the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?
> > > > >
> > > > >  1368 12:21:39.212881  /usr/bin/tpm2_getcap
> > > > >   1369 12:21:39.251829  ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
> > > > >    1370 12:21:39.261387  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error
> > > > >     1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
> > > > >      1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
> > > > >       1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap
> > > > >
> > > > > Userland seems to say that it does not support this device. I guess I could compare it with older boot?
> > > > >
> > > > > https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html
> > > > >
> > > > > Aha, no, could not, 404.
> > > >
> > > > same here, the garbage collector removes everything that's older than 28 days.
> > > >
> > > > >
> > > > > Now, let's go through Jan's
> > > > > example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
> > > > > . Again, errors look like missconfigured build system, with
> > > > >
> > > > > arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
> > > > > arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’
> > > > >
> > > > > Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.
> > > >
> > > > same here, the garbage collector removes everything that's older than 28 days.
> > > >
> > > > >
> > > > > I spent quite some time trying to click around to find an error I
> > > > > could debug. I could not find one. All of above looks like kernelCI
> > > > > configuration problems. Can they be fixed and/or filtered out?
> > > > >
> > > >
> > > > If you can write which configurations you want to filter out (because
> > > > they are not needed for CIP)
> > > > to KernelCI, KernelCI could blocklist such configurations.
> > > >
> > > > for reference: https://github.com/kernelci/kernelci-core/issues/1053#issuecomment-1050679725
> > >
> > > Jan asked me to check the kernelci related issues you summarized here
> > > related to the 4.4-cip branch. I tried to find the latest 4.4 test
> > > report that failed. Can someone point me there?
> > >
> > > With some guidance from Alice it should be possible for me to come up
> > > with a exclude/blocklist to bring 4.4 back to a "green" state.
> > > Afterwards we would have to sort out all the problems (decoupled from
> > > Pavel's testing)
> > >
> >
> > Hello Florian
> >
> > Latest build 4.4 cip is here
> > https://linux.kernelci.org/test/job/cip/branch/linux-4.4.y-cip/kernel/v4.4.302-cip69-296-g52de9431d314/
> > is a really old build as the latest 4.4 cip tag is v4.4.302-cip69
> > tagged on date 2022-04-04 13:18:30 +0900
> > Unfortunately as I previously said KernelCI garbage collector removes
> > everything that's older than 28 days.
> > You can find CIP builds here: https://linux.kernelci.org/job/cip/
>
> Thanks a lot Alice!
>
> Any chance for me to trigger a re-build?
>

I'm trying to triggering CIP_4.4 again it should appear in
https://staging.kernelci.org/job/
but Pavel have is own repository for triggering re-builds
"""
tree: cip-gitlab
branch: 'ci/pavel/linux-test'
variants: *cip_variants
cip_pavel-test:
"""

Thanks,
Alicef

-- 
======================================
Cybertrust Japan Co.,Ltd.
Alice Ferrazzi
alice.ferrazzi@miraclelinux.com
======================================


^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-07-14 10:46 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-23 18:13 KernelCI reports Pavel Machek
2022-07-14  9:20 ` Alice Ferrazzi
2022-07-14  9:45   ` [cip-dev] " Bezdeka, Florian
2022-07-14 10:17     ` Alice Ferrazzi
2022-07-14 10:27       ` Bezdeka, Florian
2022-07-14 10:45         ` Alice Ferrazzi

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.