❌ FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

❌ FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[CKI Project]

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/sashal/linux-stable.git
            Commit: 4b17a56708d9 - kcov: remote coverage support

The results of these automated tests are provided below.

    Overall result: FAILED (see details below)
             Merge: OK
           Compile: OK
             Tests: FAILED

All kernel binaries, config files, and logs are available for download here:

  https://artifacts.cki-project.org/pipelines/296781

One or more kernel tests failed:

    ppc64le:
     ❌ LTP lite
     ❌ xfstests: ext4

We hope that these logs can help you find the problem quickly. For the full
detail on our testing procedures, please scroll to the bottom of this message.

Please reply to this email if you have any questions about the tests that we
ran or if you have any suggestions on how to make future tests more effective.

        ,-.   ,-.
       ( C ) ( K )  Continuous
        `-',-.`-'   Kernel
          ( I )     Integration
           `-'
______________________________________________________________________________

Compile testing
---------------

We compiled the kernel for 3 architectures:

    aarch64:
      make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

    ppc64le:
      make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

    x86_64:
      make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg


Hardware testing
----------------
We booted each kernel and ran the following tests:

  aarch64:
    Host 1:
       ✅ Boot test
       ✅ Podman system integration test (as root)
       ✅ Podman system integration test (as user)
       ✅ LTP lite
       ✅ Loopdev Sanity
       ✅ jvm test suite
       ✅ Memory function: memfd_create
       ✅ Memory function: kaslr
       ✅ AMTU (Abstract Machine Test Utility)
       ✅ LTP: openposix test suite
       ✅ Networking bridge: sanity
       ✅ Ethernet drivers sanity
       ✅ Networking MACsec: sanity
       ✅ Networking socket: fuzz
       ✅ Networking sctp-auth: sockopts test
       ✅ Networking: igmp conformance test
       ✅ Networking route: pmtu
       ✅ Networking route_func: local
       ✅ Networking route_func: forward
       ✅ Networking TCP: keepalive test
       ✅ Networking UDP: socket
       ✅ Networking tunnel: geneve basic test
       ✅ Networking tunnel: gre basic
       ✅ L2TP basic test
       ✅ Networking tunnel: vxlan basic
       ✅ Networking ipsec: basic netns transport
       ✅ Networking ipsec: basic netns tunnel
       ✅ audit: audit testsuite test
       ✅ httpd: mod_ssl smoke sanity
       ✅ iotop: sanity
       ✅ tuned: tune-processes-through-perf
       ✅ ALSA PCM loopback test
       ✅ ALSA Control (mixer) Userspace Element test
       ✅ Usex - version 1.9-29
       ✅ storage: SCSI VPD
       ✅ stress: stress-ng
       ✅ trace: ftrace/tracer
       🚧 ✅ CIFS Connectathon
       🚧 ✅ POSIX pjd-fstest suites
       🚧 ✅ Networking vnic: ipvlan/basic
       🚧 ✅ storage: dm/common

    Host 2:
       ✅ Boot test
       ✅ xfstests: ext4
       ✅ xfstests: xfs
       ✅ lvm thinp sanity
       ✅ storage: software RAID testing
       🚧 ✅ selinux-policy: serge-testsuite
       🚧 ✅ Storage blktests

  ppc64le:
    Host 1:
       ✅ Boot test
       ✅ Podman system integration test (as root)
       ✅ Podman system integration test (as user)
       ❌ LTP lite
       ✅ Loopdev Sanity
       ✅ jvm test suite
       ✅ Memory function: memfd_create
       ✅ Memory function: kaslr
       ✅ AMTU (Abstract Machine Test Utility)
       ✅ LTP: openposix test suite
       ✅ Networking bridge: sanity
       ✅ Ethernet drivers sanity
       ✅ Networking MACsec: sanity
       ✅ Networking socket: fuzz
       ✅ Networking sctp-auth: sockopts test
       ✅ Networking route: pmtu
       ✅ Networking route_func: local
       ✅ Networking route_func: forward
       ✅ Networking TCP: keepalive test
       ✅ Networking UDP: socket
       ✅ Networking tunnel: geneve basic test
       ✅ Networking tunnel: gre basic
       ✅ L2TP basic test
       ✅ Networking tunnel: vxlan basic
       ✅ Networking ipsec: basic netns tunnel
       ✅ audit: audit testsuite test
       ✅ httpd: mod_ssl smoke sanity
       ✅ iotop: sanity
       ✅ tuned: tune-processes-through-perf
       ✅ ALSA PCM loopback test
       ✅ ALSA Control (mixer) Userspace Element test
       ✅ Usex - version 1.9-29
       ✅ trace: ftrace/tracer
       🚧 ✅ CIFS Connectathon
       🚧 ✅ POSIX pjd-fstest suites
       🚧 ✅ Networking vnic: ipvlan/basic
       🚧 ✅ storage: dm/common

    Host 2:
       ✅ Boot test
       ❌ xfstests: ext4
       ✅ xfstests: xfs
       ✅ lvm thinp sanity
       ✅ storage: software RAID testing
       🚧 ✅ selinux-policy: serge-testsuite
       🚧 ✅ Storage blktests

  x86_64:
    Host 1:
       ✅ Boot test
       ✅ Podman system integration test (as root)
       ✅ Podman system integration test (as user)
       ✅ LTP lite
       ✅ Loopdev Sanity
       ✅ jvm test suite
       ✅ Memory function: memfd_create
       ✅ Memory function: kaslr
       ✅ AMTU (Abstract Machine Test Utility)
       ✅ LTP: openposix test suite
       ✅ Networking bridge: sanity
       ✅ Ethernet drivers sanity
       ✅ Networking MACsec: sanity
       ✅ Networking socket: fuzz
       ✅ Networking sctp-auth: sockopts test
       ✅ Networking: igmp conformance test
       ✅ Networking route: pmtu
       ✅ Networking route_func: local
       ✅ Networking route_func: forward
       ✅ Networking TCP: keepalive test
       ✅ Networking UDP: socket
       ✅ Networking tunnel: geneve basic test
       ✅ Networking tunnel: gre basic
       ✅ L2TP basic test
       ✅ Networking tunnel: vxlan basic
       ✅ Networking ipsec: basic netns transport
       ✅ Networking ipsec: basic netns tunnel
       ✅ audit: audit testsuite test
       ✅ httpd: mod_ssl smoke sanity
       ✅ iotop: sanity
       ✅ tuned: tune-processes-through-perf
       ✅ pciutils: sanity smoke test
       ✅ ALSA PCM loopback test
       ✅ ALSA Control (mixer) Userspace Element test
       ✅ Usex - version 1.9-29
       ✅ storage: SCSI VPD
       ✅ stress: stress-ng
       ✅ trace: ftrace/tracer
       🚧 ✅ CIFS Connectathon
       🚧 ✅ POSIX pjd-fstest suites
       🚧 ✅ Networking vnic: ipvlan/basic
       🚧 ✅ storage: dm/common

    Host 2:
       ✅ Boot test
       ✅ Storage SAN device stress - mpt3sas driver

    Host 3:
       ✅ Boot test
       🚧 ✅ IPMI driver test
       🚧 ✅ IPMItool loop stress test

    Host 4:
       ✅ Boot test
       ✅ xfstests: ext4
       ✅ xfstests: xfs
       ✅ lvm thinp sanity
       ✅ storage: software RAID testing
       🚧 ✅ IOMMU boot test
       🚧 ✅ selinux-policy: serge-testsuite
       🚧 ✅ Storage blktests

    Host 5:
       ✅ Boot test
       ✅ Storage SAN device stress - megaraid_sas

  Test sources: https://github.com/CKI-project/tests-beaker
    💚 Pull requests are welcome for new tests or improvements to existing tests!

Waived tests
------------
If the test run included waived tests, they are marked with 🚧. Such tests are
executed but their results are not taken into account. Tests are waived when
their results are not reliable enough, e.g. when they're just introduced or are
being fixed.

Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't
finished running are marked with ⏱. Reports for non-upstream kernels have
a Beaker recipe linked to next to each host.

[LTP] ❌ FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[CKI Project]

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/sashal/linux-stable.git
            Commit: 4b17a56708d9 - kcov: remote coverage support

The results of these automated tests are provided below.

    Overall result: FAILED (see details below)
             Merge: OK
           Compile: OK
             Tests: FAILED

All kernel binaries, config files, and logs are available for download here:

  https://artifacts.cki-project.org/pipelines/296781

One or more kernel tests failed:

    ppc64le:
     ? LTP lite
     ? xfstests: ext4

We hope that these logs can help you find the problem quickly. For the full
detail on our testing procedures, please scroll to the bottom of this message.

Please reply to this email if you have any questions about the tests that we
ran or if you have any suggestions on how to make future tests more effective.

        ,-.   ,-.
       ( C ) ( K )  Continuous
        `-',-.`-'   Kernel
          ( I )     Integration
           `-'
______________________________________________________________________________

Compile testing
---------------

We compiled the kernel for 3 architectures:

    aarch64:
      make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

    ppc64le:
      make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

    x86_64:
      make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg


Hardware testing
----------------
We booted each kernel and ran the following tests:

  aarch64:
    Host 1:
       ? Boot test
       ? Podman system integration test (as root)
       ? Podman system integration test (as user)
       ? LTP lite
       ? Loopdev Sanity
       ? jvm test suite
       ? Memory function: memfd_create
       ? Memory function: kaslr
       ? AMTU (Abstract Machine Test Utility)
       ? LTP: openposix test suite
       ? Networking bridge: sanity
       ? Ethernet drivers sanity
       ? Networking MACsec: sanity
       ? Networking socket: fuzz
       ? Networking sctp-auth: sockopts test
       ? Networking: igmp conformance test
       ? Networking route: pmtu
       ? Networking route_func: local
       ? Networking route_func: forward
       ? Networking TCP: keepalive test
       ? Networking UDP: socket
       ? Networking tunnel: geneve basic test
       ? Networking tunnel: gre basic
       ? L2TP basic test
       ? Networking tunnel: vxlan basic
       ? Networking ipsec: basic netns transport
       ? Networking ipsec: basic netns tunnel
       ? audit: audit testsuite test
       ? httpd: mod_ssl smoke sanity
       ? iotop: sanity
       ? tuned: tune-processes-through-perf
       ? ALSA PCM loopback test
       ? ALSA Control (mixer) Userspace Element test
       ? Usex - version 1.9-29
       ? storage: SCSI VPD
       ? stress: stress-ng
       ? trace: ftrace/tracer
       ? ? CIFS Connectathon
       ? ? POSIX pjd-fstest suites
       ? ? Networking vnic: ipvlan/basic
       ? ? storage: dm/common

    Host 2:
       ? Boot test
       ? xfstests: ext4
       ? xfstests: xfs
       ? lvm thinp sanity
       ? storage: software RAID testing
       ? ? selinux-policy: serge-testsuite
       ? ? Storage blktests

  ppc64le:
    Host 1:
       ? Boot test
       ? Podman system integration test (as root)
       ? Podman system integration test (as user)
       ? LTP lite
       ? Loopdev Sanity
       ? jvm test suite
       ? Memory function: memfd_create
       ? Memory function: kaslr
       ? AMTU (Abstract Machine Test Utility)
       ? LTP: openposix test suite
       ? Networking bridge: sanity
       ? Ethernet drivers sanity
       ? Networking MACsec: sanity
       ? Networking socket: fuzz
       ? Networking sctp-auth: sockopts test
       ? Networking route: pmtu
       ? Networking route_func: local
       ? Networking route_func: forward
       ? Networking TCP: keepalive test
       ? Networking UDP: socket
       ? Networking tunnel: geneve basic test
       ? Networking tunnel: gre basic
       ? L2TP basic test
       ? Networking tunnel: vxlan basic
       ? Networking ipsec: basic netns tunnel
       ? audit: audit testsuite test
       ? httpd: mod_ssl smoke sanity
       ? iotop: sanity
       ? tuned: tune-processes-through-perf
       ? ALSA PCM loopback test
       ? ALSA Control (mixer) Userspace Element test
       ? Usex - version 1.9-29
       ? trace: ftrace/tracer
       ? ? CIFS Connectathon
       ? ? POSIX pjd-fstest suites
       ? ? Networking vnic: ipvlan/basic
       ? ? storage: dm/common

    Host 2:
       ? Boot test
       ? xfstests: ext4
       ? xfstests: xfs
       ? lvm thinp sanity
       ? storage: software RAID testing
       ? ? selinux-policy: serge-testsuite
       ? ? Storage blktests

  x86_64:
    Host 1:
       ? Boot test
       ? Podman system integration test (as root)
       ? Podman system integration test (as user)
       ? LTP lite
       ? Loopdev Sanity
       ? jvm test suite
       ? Memory function: memfd_create
       ? Memory function: kaslr
       ? AMTU (Abstract Machine Test Utility)
       ? LTP: openposix test suite
       ? Networking bridge: sanity
       ? Ethernet drivers sanity
       ? Networking MACsec: sanity
       ? Networking socket: fuzz
       ? Networking sctp-auth: sockopts test
       ? Networking: igmp conformance test
       ? Networking route: pmtu
       ? Networking route_func: local
       ? Networking route_func: forward
       ? Networking TCP: keepalive test
       ? Networking UDP: socket
       ? Networking tunnel: geneve basic test
       ? Networking tunnel: gre basic
       ? L2TP basic test
       ? Networking tunnel: vxlan basic
       ? Networking ipsec: basic netns transport
       ? Networking ipsec: basic netns tunnel
       ? audit: audit testsuite test
       ? httpd: mod_ssl smoke sanity
       ? iotop: sanity
       ? tuned: tune-processes-through-perf
       ? pciutils: sanity smoke test
       ? ALSA PCM loopback test
       ? ALSA Control (mixer) Userspace Element test
       ? Usex - version 1.9-29
       ? storage: SCSI VPD
       ? stress: stress-ng
       ? trace: ftrace/tracer
       ? ? CIFS Connectathon
       ? ? POSIX pjd-fstest suites
       ? ? Networking vnic: ipvlan/basic
       ? ? storage: dm/common

    Host 2:
       ? Boot test
       ? Storage SAN device stress - mpt3sas driver

    Host 3:
       ? Boot test
       ? ? IPMI driver test
       ? ? IPMItool loop stress test

    Host 4:
       ? Boot test
       ? xfstests: ext4
       ? xfstests: xfs
       ? lvm thinp sanity
       ? storage: software RAID testing
       ? ? IOMMU boot test
       ? ? selinux-policy: serge-testsuite
       ? ? Storage blktests

    Host 5:
       ? Boot test
       ? Storage SAN device stress - megaraid_sas

  Test sources: https://github.com/CKI-project/tests-beaker
    ? Pull requests are welcome for new tests or improvements to existing tests!

Waived tests
------------
If the test run included waived tests, they are marked with ?. Such tests are
executed but their results are not taken into account. Tests are waived when
their results are not reliable enough, e.g. when they're just introduced or are
being fixed.

Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't
finished running are marked with ?. Reports for non-upstream kernels have
a Beaker recipe linked to next to each host.

Re: [LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Cyril Hrubis]

Hi!
> One or more kernel tests failed:
> 
>     ppc64le:
>      ??? LTP lite
>      ??? xfstests: ext4

Both logs shows missing files, that may be an infrastructure problem as
well.

Also can we include links to the logfiles here? Bonus points for showing
the snippet with the actual failure in the email as well. I takes a fair
amount of time locating them manually in the pipeline repository, it
would be much much easier just with the links to the right logfile...

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Cyril Hrubis]

Hi!
> One or more kernel tests failed:
> 
>     ppc64le:
>      ??? LTP lite
>      ??? xfstests: ext4

Both logs shows missing files, that may be an infrastructure problem as
well.

Also can we include links to the logfiles here? Bonus points for showing
the snippet with the actual failure in the email as well. I takes a fair
amount of time locating them manually in the pipeline repository, it
would be much much easier just with the links to the right logfile...

-- 
Cyril Hrubis
chrubis@suse.cz

Re: [LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Rachel Sibley]

On 11/20/19 6:35 AM, Cyril Hrubis wrote:
> Hi!
>> One or more kernel tests failed:
>>
>>      ppc64le:
>>       ??? LTP lite
>>       ??? xfstests: ext4
> 
> Both logs shows missing files, that may be an infrastructure problem as
> well.
> 
> Also can we include links to the logfiles here? Bonus points for showing
> the snippet with the actual failure in the email as well. I takes a fair
> amount of time locating them manually in the pipeline repository, it
> would be much much easier just with the links to the right logfile...
> 

Thanks for the feedback Cyril, we did have links to each failure listed
before but we were told it made the email look cluttered especially
if there are multiple failures.

The test logs are sorted by arch|host|TC, is there something we can
do to make it easier to find related logs ?
https://artifacts.cki-project.org/pipelines/296781/logs/

Maybe we can look into adding the linked logs to the bottom of the
email with a reference id next to the failures in the summary, so
for example:

     ppc64le:
      ❌ LTP lite [1]
      ❌ xfstests: ext4 [2]

We could also look into merging the ltp run logs into a single file
as well.

-Rachel

[LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Rachel Sibley]

On 11/20/19 6:35 AM, Cyril Hrubis wrote:
> Hi!
>> One or more kernel tests failed:
>>
>>      ppc64le:
>>       ??? LTP lite
>>       ??? xfstests: ext4
> 
> Both logs shows missing files, that may be an infrastructure problem as
> well.
> 
> Also can we include links to the logfiles here? Bonus points for showing
> the snippet with the actual failure in the email as well. I takes a fair
> amount of time locating them manually in the pipeline repository, it
> would be much much easier just with the links to the right logfile...
> 

Thanks for the feedback Cyril, we did have links to each failure listed
before but we were told it made the email look cluttered especially
if there are multiple failures.

The test logs are sorted by arch|host|TC, is there something we can
do to make it easier to find related logs ?
https://artifacts.cki-project.org/pipelines/296781/logs/

Maybe we can look into adding the linked logs to the bottom of the
email with a reference id next to the failures in the summary, so
for example:

     ppc64le:
      ? LTP lite [1]
      ? xfstests: ext4 [2]

We could also look into merging the ltp run logs into a single file
as well.

-Rachel

Re: [LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Cyril Hrubis]

Hi!
> >> One or more kernel tests failed:
> >>
> >>      ppc64le:
> >>       ??? LTP lite
> >>       ??? xfstests: ext4
> > 
> > Both logs shows missing files, that may be an infrastructure problem as
> > well.
> > 
> > Also can we include links to the logfiles here? Bonus points for showing
> > the snippet with the actual failure in the email as well. I takes a fair
> > amount of time locating them manually in the pipeline repository, it
> > would be much much easier just with the links to the right logfile...
> > 
> 
> Thanks for the feedback Cyril, we did have links to each failure listed
> before but we were told it made the email look cluttered especially
> if there are multiple failures.

So it's exactly how Dmitry described it, you can't please everyone..,

> The test logs are sorted by arch|host|TC, is there something we can
> do to make it easier to find related logs ?
> https://artifacts.cki-project.org/pipelines/296781/logs/
> 
> Maybe we can look into adding the linked logs to the bottom of the
> email with a reference id next to the failures in the summary, so
> for example:
> 
>      ppc64le:
>       ??? LTP lite [1]
>       ??? xfstests: ext4 [2]

That would work for me.

> We could also look into merging the ltp run logs into a single file
> as well.

That would make it too big I guess. Actually the only part I'm
interested in most of the time is the part of the log with the failing
test. I would be quite happy if we had logs/failures file on the
pipelines sever that would contain only failures extracted from
different logfiles. The question is if that's feasible with your
framework.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Cyril Hrubis]

Hi!
> >> One or more kernel tests failed:
> >>
> >>      ppc64le:
> >>       ??? LTP lite
> >>       ??? xfstests: ext4
> > 
> > Both logs shows missing files, that may be an infrastructure problem as
> > well.
> > 
> > Also can we include links to the logfiles here? Bonus points for showing
> > the snippet with the actual failure in the email as well. I takes a fair
> > amount of time locating them manually in the pipeline repository, it
> > would be much much easier just with the links to the right logfile...
> > 
> 
> Thanks for the feedback Cyril, we did have links to each failure listed
> before but we were told it made the email look cluttered especially
> if there are multiple failures.

So it's exactly how Dmitry described it, you can't please everyone..,

> The test logs are sorted by arch|host|TC, is there something we can
> do to make it easier to find related logs ?
> https://artifacts.cki-project.org/pipelines/296781/logs/
> 
> Maybe we can look into adding the linked logs to the bottom of the
> email with a reference id next to the failures in the summary, so
> for example:
> 
>      ppc64le:
>       ??? LTP lite [1]
>       ??? xfstests: ext4 [2]

That would work for me.

> We could also look into merging the ltp run logs into a single file
> as well.

That would make it too big I guess. Actually the only part I'm
interested in most of the time is the part of the log with the failing
test. I would be quite happy if we had logs/failures file on the
pipelines sever that would contain only failures extracted from
different logfiles. The question is if that's feasible with your
framework.

-- 
Cyril Hrubis
chrubis@suse.cz

RE: [LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Tim.Bird]

> -----Original Message-----
> From:  Cyril Hrubis
> 
> Hi!
> > >> One or more kernel tests failed:
> > >>
> > >>      ppc64le:
> > >>       ??? LTP lite
> > >>       ??? xfstests: ext4
> > >
> > > Both logs shows missing files, that may be an infrastructure problem as
> > > well.
> > >
> > > Also can we include links to the logfiles here? Bonus points for showing
> > > the snippet with the actual failure in the email as well. I takes a fair
> > > amount of time locating them manually in the pipeline repository, it
> > > would be much much easier just with the links to the right logfile...

My preference would be to include the failure snippet somewhere in
the e-mail as well (as opposed to just a link).

> > >
> >
> > Thanks for the feedback Cyril, we did have links to each failure listed
> > before but we were told it made the email look cluttered especially
> > if there are multiple failures.
> 
> So it's exactly how Dmitry described it, you can't please everyone..,
> 
> > The test logs are sorted by arch|host|TC, is there something we can
> > do to make it easier to find related logs ?
> > https://artifacts.cki-project.org/pipelines/296781/logs/
> >
> > Maybe we can look into adding the linked logs to the bottom of the
> > email with a reference id next to the failures in the summary, so
> > for example:
> >
> >      ppc64le:
> >       ??? LTP lite [1]
> >       ??? xfstests: ext4 [2]
> 
> That would work for me.
> 

Maybe combine the 'footnote' idea with the 'inline' idea, and have
the footnote include a link to the full log and a snippet with just the
output from the failing testcase, from the full log?

> > We could also look into merging the ltp run logs into a single file
> > as well.
> 
> That would make it too big I guess. Actually the only part I'm
> interested in most of the time is the part of the log with the failing
> test. I would be quite happy if we had logs/failures file on the
> pipelines sever that would contain only failures extracted from
> different logfiles. The question is if that's feasible with your
> framework.

Fuego has an LTP log-splitter and link generator.
It's Fuego-specific and generates
files referred to by links in the result  tables that Fuego
shows to users.

I don't know how CKI is generating it's data or storing it,
but I can take a look and see if it could be applied
to their use case.  It's a python program that is fairly small.

See here:
https://bitbucket.org/fuegotest/fuego-core/src/master/tests/Functional.LTP/parser.py

It might not be applicable, depending on whether CKI stores their LTP output similarly
to how Fuego does, but IMHO it's worth taking a look.  If there is sufficient interest,
maybe this could be generalized and submitted to upstream LTP.  The Fuego log-splitter
produces individual files.

Another idea would be to write a program that takes an LTP log,
and the name of a failing testcase, and outputs (on stdout)  the snippet from the log
for that testcase.  I think this would be very easy to do, and might be suitable to
use in multiple contexts: on the command line, in a report
generator, or as a CGI script for a results server.
 -- Tim

[LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Tim.Bird]

> -----Original Message-----
> From:  Cyril Hrubis
> 
> Hi!
> > >> One or more kernel tests failed:
> > >>
> > >>      ppc64le:
> > >>       ??? LTP lite
> > >>       ??? xfstests: ext4
> > >
> > > Both logs shows missing files, that may be an infrastructure problem as
> > > well.
> > >
> > > Also can we include links to the logfiles here? Bonus points for showing
> > > the snippet with the actual failure in the email as well. I takes a fair
> > > amount of time locating them manually in the pipeline repository, it
> > > would be much much easier just with the links to the right logfile...

My preference would be to include the failure snippet somewhere in
the e-mail as well (as opposed to just a link).

> > >
> >
> > Thanks for the feedback Cyril, we did have links to each failure listed
> > before but we were told it made the email look cluttered especially
> > if there are multiple failures.
> 
> So it's exactly how Dmitry described it, you can't please everyone..,
> 
> > The test logs are sorted by arch|host|TC, is there something we can
> > do to make it easier to find related logs ?
> > https://artifacts.cki-project.org/pipelines/296781/logs/
> >
> > Maybe we can look into adding the linked logs to the bottom of the
> > email with a reference id next to the failures in the summary, so
> > for example:
> >
> >      ppc64le:
> >       ??? LTP lite [1]
> >       ??? xfstests: ext4 [2]
> 
> That would work for me.
> 

Maybe combine the 'footnote' idea with the 'inline' idea, and have
the footnote include a link to the full log and a snippet with just the
output from the failing testcase, from the full log?

> > We could also look into merging the ltp run logs into a single file
> > as well.
> 
> That would make it too big I guess. Actually the only part I'm
> interested in most of the time is the part of the log with the failing
> test. I would be quite happy if we had logs/failures file on the
> pipelines sever that would contain only failures extracted from
> different logfiles. The question is if that's feasible with your
> framework.

Fuego has an LTP log-splitter and link generator.
It's Fuego-specific and generates
files referred to by links in the result  tables that Fuego
shows to users.

I don't know how CKI is generating it's data or storing it,
but I can take a look and see if it could be applied
to their use case.  It's a python program that is fairly small.

See here:
https://bitbucket.org/fuegotest/fuego-core/src/master/tests/Functional.LTP/parser.py

It might not be applicable, depending on whether CKI stores their LTP output similarly
to how Fuego does, but IMHO it's worth taking a look.  If there is sufficient interest,
maybe this could be generalized and submitted to upstream LTP.  The Fuego log-splitter
produces individual files.

Another idea would be to write a program that takes an LTP log,
and the name of a failing testcase, and outputs (on stdout)  the snippet from the log
for that testcase.  I think this would be very easy to do, and might be suitable to
use in multiple contexts: on the command line, in a report
generator, or as a CGI script for a results server.
 -- Tim

Re: [LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Rachel Sibley]

On 11/21/19 5:58 AM, Tim.Bird@sony.com wrote:
> 
> 
>> -----Original Message-----
>> From:  Cyril Hrubis
>>
>> Hi!
>>>>> One or more kernel tests failed:
>>>>>
>>>>>       ppc64le:
>>>>>        ??? LTP lite
>>>>>        ??? xfstests: ext4
>>>>
>>>> Both logs shows missing files, that may be an infrastructure problem as
>>>> well.
>>>>
>>>> Also can we include links to the logfiles here? Bonus points for showing
>>>> the snippet with the actual failure in the email as well. I takes a fair
>>>> amount of time locating them manually in the pipeline repository, it
>>>> would be much much easier just with the links to the right logfile...
> 
> My preference would be to include the failure snippet somewhere in
> the e-mail as well (as opposed to just a link).
> 
>>>>
>>>
>>> Thanks for the feedback Cyril, we did have links to each failure listed
>>> before but we were told it made the email look cluttered especially
>>> if there are multiple failures.
>>
>> So it's exactly how Dmitry described it, you can't please everyone..,
>>
>>> The test logs are sorted by arch|host|TC, is there something we can
>>> do to make it easier to find related logs ?
>>> https://artifacts.cki-project.org/pipelines/296781/logs/
>>>
>>> Maybe we can look into adding the linked logs to the bottom of the
>>> email with a reference id next to the failures in the summary, so
>>> for example:
>>>
>>>       ppc64le:
>>>        ??? LTP lite [1]
>>>        ??? xfstests: ext4 [2]
>>
>> That would work for me.
>>
> 
> Maybe combine the 'footnote' idea with the 'inline' idea, and have
> the footnote include a link to the full log and a snippet with just the
> output from the failing testcase, from the full log?
> 
>>> We could also look into merging the ltp run logs into a single file
>>> as well.
>>
>> That would make it too big I guess. Actually the only part I'm
>> interested in most of the time is the part of the log with the failing
>> test. I would be quite happy if we had logs/failures file on the
>> pipelines sever that would contain only failures extracted from
>> different logfiles. The question is if that's feasible with your
>> framework.
> 
> Fuego has an LTP log-splitter and link generator.
> It's Fuego-specific and generates
> files referred to by links in the result  tables that Fuego
> shows to users.
> 
> I don't know how CKI is generating it's data or storing it,
> but I can take a look and see if it could be applied
> to their use case.  It's a python program that is fairly small.

There is a summary log which captures overall results:
https://artifacts.cki-project.org/pipelines/296781/logs/aarch64_host_1_LTP_lite_resultoutputfile.log

Then an individual log file for each LTP testsuite, e.g:
https://artifacts.cki-project.org/pipelines/296781/logs/aarch64_host_1_LTP_lite_fs.run.log

> 
> See here:
> https://bitbucket.org/fuegotest/fuego-core/src/master/tests/Functional.LTP/parser.py

Thanks!
> 
> It might not be applicable, depending on whether CKI stores their LTP output similarly
> to how Fuego does, but IMHO it's worth taking a look.  If there is sufficient interest,
> maybe this could be generalized and submitted to upstream LTP.  The Fuego log-splitter
> produces individual files.

I think it's a good idea, as long as it can be generic enough where
someone could modify a config file for example to indicate the log
path and naming convention.

> 
> Another idea would be to write a program that takes an LTP log,
> and the name of a failing testcase, and outputs (on stdout)  the snippet from the log
> for that testcase.  I think this would be very easy to do, and might be suitable to
> use in multiple contexts: on the command line, in a report
> generator, or as a CGI script for a results server.

I logged a few tickets so our team can take a closer look and discuss 
for both failure snippets and linking LTP logs directly. I'm also
checking to see if we have anything internally.

Thanks for all the feedback.

>   -- Tim
> 

[LTP] ??? FAIL: Test report for kernel 5.4.0-rc8-4b17a56.cki (stable-next)

[Rachel Sibley]

On 11/21/19 5:58 AM, Tim.Bird@sony.com wrote:
> 
> 
>> -----Original Message-----
>> From:  Cyril Hrubis
>>
>> Hi!
>>>>> One or more kernel tests failed:
>>>>>
>>>>>       ppc64le:
>>>>>        ??? LTP lite
>>>>>        ??? xfstests: ext4
>>>>
>>>> Both logs shows missing files, that may be an infrastructure problem as
>>>> well.
>>>>
>>>> Also can we include links to the logfiles here? Bonus points for showing
>>>> the snippet with the actual failure in the email as well. I takes a fair
>>>> amount of time locating them manually in the pipeline repository, it
>>>> would be much much easier just with the links to the right logfile...
> 
> My preference would be to include the failure snippet somewhere in
> the e-mail as well (as opposed to just a link).
> 
>>>>
>>>
>>> Thanks for the feedback Cyril, we did have links to each failure listed
>>> before but we were told it made the email look cluttered especially
>>> if there are multiple failures.
>>
>> So it's exactly how Dmitry described it, you can't please everyone..,
>>
>>> The test logs are sorted by arch|host|TC, is there something we can
>>> do to make it easier to find related logs ?
>>> https://artifacts.cki-project.org/pipelines/296781/logs/
>>>
>>> Maybe we can look into adding the linked logs to the bottom of the
>>> email with a reference id next to the failures in the summary, so
>>> for example:
>>>
>>>       ppc64le:
>>>        ??? LTP lite [1]
>>>        ??? xfstests: ext4 [2]
>>
>> That would work for me.
>>
> 
> Maybe combine the 'footnote' idea with the 'inline' idea, and have
> the footnote include a link to the full log and a snippet with just the
> output from the failing testcase, from the full log?
> 
>>> We could also look into merging the ltp run logs into a single file
>>> as well.
>>
>> That would make it too big I guess. Actually the only part I'm
>> interested in most of the time is the part of the log with the failing
>> test. I would be quite happy if we had logs/failures file on the
>> pipelines sever that would contain only failures extracted from
>> different logfiles. The question is if that's feasible with your
>> framework.
> 
> Fuego has an LTP log-splitter and link generator.
> It's Fuego-specific and generates
> files referred to by links in the result  tables that Fuego
> shows to users.
> 
> I don't know how CKI is generating it's data or storing it,
> but I can take a look and see if it could be applied
> to their use case.  It's a python program that is fairly small.

There is a summary log which captures overall results:
https://artifacts.cki-project.org/pipelines/296781/logs/aarch64_host_1_LTP_lite_resultoutputfile.log

Then an individual log file for each LTP testsuite, e.g:
https://artifacts.cki-project.org/pipelines/296781/logs/aarch64_host_1_LTP_lite_fs.run.log

> 
> See here:
> https://bitbucket.org/fuegotest/fuego-core/src/master/tests/Functional.LTP/parser.py

Thanks!
> 
> It might not be applicable, depending on whether CKI stores their LTP output similarly
> to how Fuego does, but IMHO it's worth taking a look.  If there is sufficient interest,
> maybe this could be generalized and submitted to upstream LTP.  The Fuego log-splitter
> produces individual files.

I think it's a good idea, as long as it can be generic enough where
someone could modify a config file for example to indicate the log
path and naming convention.

> 
> Another idea would be to write a program that takes an LTP log,
> and the name of a failing testcase, and outputs (on stdout)  the snippet from the log
> for that testcase.  I think this would be very easy to do, and might be suitable to
> use in multiple contexts: on the command line, in a report
> generator, or as a CGI script for a results server.

I logged a few tickets so our team can take a closer look and discuss 
for both failure snippets and linking LTP logs directly. I'm also
checking to see if we have anything internally.

Thanks for all the feedback.

>   -- Tim
>