next/master boot: 265 boots: 17 failed, 184 passed with 64 offline (next-20190730)

next/master boot: 265 boots: 17 failed, 184 passed with 64 offline (next-20190730)

[kernelci.org bot]

next/master boot: 265 boots: 17 failed, 184 passed with 64 offline (next-20190730)

Full Boot Summary: https://kernelci.org/boot/all/job/next/branch/master/kernel/next-20190730/
Full Build Summary: https://kernelci.org/build/next/branch/master/kernel/next-20190730/

Tree: next
Branch: master
Git Describe: next-20190730
Git Commit: 70f4b4ac1655dffd33cb38f369b31bdecc588daa
Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
Tested: 84 unique boards, 28 SoC families, 22 builds out of 230

Boot Failures Detected:

arm64:
    defconfig:
        gcc-8:
            qcom-qdf2400: 1 failed lab

    defconfig+CONFIG_CPU_BIG_ENDIAN=y:
        gcc-8:
            meson-gxm-khadas-vim2: 1 failed lab

    defconfig+CONFIG_RANDOMIZE_BASE=y:
        gcc-8:
            meson-gxm-khadas-vim2: 1 failed lab
            qcom-qdf2400: 1 failed lab

arm:
    oxnas_v6_defconfig:
        gcc-8:
            ox820-cloudengines-pogoplug-series-3: 1 failed lab

    qcom_defconfig:
        gcc-8:
            qcom-apq8064-cm-qs600: 1 failed lab

    multi_v7_defconfig+CONFIG_CPU_BIG_ENDIAN=y:
        gcc-8:
            armada-xp-openblocks-ax3-4: 1 failed lab

Offline Platforms:

riscv:

    defconfig:
        gcc-8
            sifive_fu540: 1 offline lab

arm64:

    defconfig:
        gcc-8
            meson-axg-s400: 1 offline lab
            meson-g12a-u200: 1 offline lab
            meson-g12a-x96-max: 1 offline lab
            meson-gxbb-odroidc2: 1 offline lab
            meson-gxl-s905d-p230: 1 offline lab
            meson-gxl-s905x-libretech-cc: 1 offline lab
            meson-gxl-s905x-nexbox-a95x: 1 offline lab
            meson-gxl-s905x-p212: 1 offline lab
            meson-gxm-nexbox-a1: 1 offline lab
            rk3399-firefly: 1 offline lab
            sun50i-a64-pine64-plus: 1 offline lab

    defconfig+CONFIG_CPU_BIG_ENDIAN=y:
        gcc-8
            meson-axg-s400: 1 offline lab
            meson-g12a-u200: 1 offline lab
            meson-g12a-x96-max: 1 offline lab
            meson-gxbb-odroidc2: 1 offline lab
            meson-gxl-s905d-p230: 1 offline lab
            meson-gxl-s905x-libretech-cc: 1 offline lab
            meson-gxl-s905x-nexbox-a95x: 1 offline lab
            meson-gxl-s905x-p212: 1 offline lab

    defconfig+CONFIG_RANDOMIZE_BASE=y:
        gcc-8
            meson-axg-s400: 1 offline lab
            meson-g12a-u200: 1 offline lab
            meson-g12a-x96-max: 1 offline lab
            meson-gxbb-odroidc2: 1 offline lab
            meson-gxl-s905d-p230: 1 offline lab
            meson-gxl-s905x-libretech-cc: 1 offline lab
            meson-gxl-s905x-nexbox-a95x: 1 offline lab
            meson-gxl-s905x-p212: 1 offline lab
            meson-gxm-nexbox-a1: 1 offline lab
            rk3399-firefly: 1 offline lab
            sun50i-a64-pine64-plus: 1 offline lab

mips:

    pistachio_defconfig:
        gcc-8
            pistachio_marduk: 1 offline lab

arm:

    exynos_defconfig:
        gcc-8
            exynos5250-arndale: 1 offline lab
            exynos5420-arndale-octa: 1 offline lab
            exynos5800-peach-pi: 1 offline lab

    multi_v7_defconfig:
        gcc-8
            bcm72521-bcm97252sffe: 1 offline lab
            bcm7445-bcm97445c: 1 offline lab
            exynos5250-arndale: 1 offline lab
            exynos5420-arndale-octa: 1 offline lab
            exynos5800-peach-pi: 1 offline lab
            imx6dl-wandboard_dual: 1 offline lab
            imx6dl-wandboard_solo: 1 offline lab
            imx6q-wandboard: 1 offline lab
            imx7s-warp: 1 offline lab
            meson8b-odroidc1: 1 offline lab
            omap3-beagle: 1 offline lab
            omap4-panda: 1 offline lab
            qcom-apq8064-ifc6410: 1 offline lab
            stih410-b2120: 1 offline lab
            sun4i-a10-cubieboard: 1 offline lab
            sun7i-a20-bananapi: 1 offline lab
            vf610-colibri-eval-v3: 1 offline lab

    omap2plus_defconfig:
        gcc-8
            omap3-beagle: 1 offline lab
            omap4-panda: 1 offline lab

    qcom_defconfig:
        gcc-8
            qcom-apq8064-ifc6410: 1 offline lab

    davinci_all_defconfig:
        gcc-8
            da850-evm: 1 offline lab
            dm365evm,legacy: 1 offline lab

    imx_v6_v7_defconfig:
        gcc-8
            imx6dl-wandboard_dual: 1 offline lab
            imx6dl-wandboard_solo: 1 offline lab
            imx6q-wandboard: 1 offline lab
            imx7s-warp: 1 offline lab
            vf610-colibri-eval-v3: 1 offline lab

    sunxi_defconfig:
        gcc-8
            sun4i-a10-cubieboard: 1 offline lab
            sun7i-a20-bananapi: 1 offline lab

---
For more info write to <info@kernelci.org>

Re: next/master boot: 265 boots: 17 failed, 184 passed with 64 offline (next-20190730)

[Neil Armstrong]

[-- Attachment #1.1: Type: text/plain, Size: 1731 bytes --]

Hi Mark,

On 30/07/2019 14:34, Mark Brown wrote:
> On Tue, Jul 30, 2019 at 05:17:56AM -0700, kernelci.org bot wrote:
> 
>> Boot Failures Detected:
>>
>> arm:
>>     oxnas_v6_defconfig:
>>         gcc-8:
>>             ox820-cloudengines-pogoplug-series-3: 1 failed lab
> 
> For some time now -next and mainline have been failing to boot on
> Pogoplug 3 with the oxnas_v6_defconfig, the kernel seems to start fine
> but fails to parse the ramdisk it's passed:
> 
> 08:50:02.086589  <6>[    7.719854] IP-Config: Complete:
> 08:50:02.087213  <6>[    7.723330]      device=eth0, hwaddr=0a:a2:89:27:10:1b, ipaddr=10.201.4.144, mask=255.255.0.0, gw=10.201.0.1
> 08:50:02.087413  <6>[    7.733409]      host=10.201.4.144, domain=, nis-domain=(none)
> 08:50:02.088056  <6>[    7.739499]      bootserver=10.201.1.1, rootserver=10.201.1.1, rootpath=
> 08:50:02.088248  <6>[    7.739504]      nameserver0=10.201.1.1
> 08:50:02.129966  <5>[    7.752025] RAMDISK: Couldn't find valid RAM disk image starting at 0.
> 08:50:02.130381  <4>[    7.759616] List of all partitions:
> 08:50:02.131333  <4>[    7.763363] 0100           65536 ram0 
> 
> Possibly an issue with the ramdisk getting overwritten or something?

Thanks for reporting, it's my suspicion since my multiple bisect runs all point to
this merge commit :
a318423b61e8 Merge tag 'upstream-5.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rw/ubifs

This merge doesn't introduce notable changes for the oxnas_v6_defconfig, but disabling UBI entirely makes
it work again.

Continuing my investigations...

Neil

> 
> Full details for today's -next can be seen here:
> 
> 	https://kernelci.org/boot/id/5d4004bb59b51489d631b28d/
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: next/master boot: 265 boots: 17 failed, 184 passed with 64 offline (next-20190730)

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2313 bytes --]

On Tue, Jul 30, 2019 at 05:17:56AM -0700, kernelci.org bot wrote:

> next/master boot: 265 boots: 17 failed, 184 passed with 64 offline (next-20190730)
> Full Boot Summary: https://kernelci.org/boot/all/job/next/branch/master/kernel/next-20190730/
> Full Build Summary: https://kernelci.org/build/next/branch/master/kernel/next-20190730/

For a while now all arm64 big endian clang built kernels have been
failing, the kernel mounts the root filesystem but is unable to execute
init due to an inability to understand the executable format:

08:55:25.999629  <6>[  226.077194] Run /init as init process
08:55:31.066490  <4>[  226.086518] request_module: kmod_concurrent_max (0) close to 0 (max_modprobes: 50), for module binfmt-464c, throttling...
08:55:31.085167  <4>[  231.135458] request_module: modprobe binfmt-464c cannot be processed, kmod busy with 50 threads for more than 5 seconds now
08:55:35.745340  ShellCommand command timed out.: Sending # in case of corruption. Connection timeout 00:01:54, retry in 00:00:11
08:55:35.846536  #
08:55:35.849523  #
08:55:36.185339  <4>[  231.154208] request_module: kmod_concurrent_max (0) close to 0 (max_modprobes: 50), for module binfmt-464c, throttling...
08:55:36.208673  <4>[  236.255449] request_module: modprobe binfmt-464c cannot be processed, kmod busy with 50 threads for more than 5 seconds now
08:55:36.209013  <3>[  236.269366] Failed to execute /init (error -8)
08:55:36.210161  <6>[  236.285459] Run /sbin/init as init process
08:55:41.306737  <4>[  236.294490] request_module: kmod_concurrent_max (0) close to 0 (max_modprobes: 50), for module binfmt-464c, throttling...
08:55:41.331547  <4>[  241.375455] request_module: modprobe binfmt-464c cannot be processed, kmod busy with 50 threads for more than 5 seconds now
08:55:41.331837  <3>[  241.389316] Starting init: /sbin/init exists but couldn't execute it (error -8)

(binfmt-464c is binfmt-misc, the fallback for unknown executable
formats).  The same kernel version built with GCC boots fine.

You can see a bunch of reports here (all the big endian failures):

	https://kernelci.org/boot/all/job/next/branch/master/kernel/next-20190730/

It's possible that there's some infrastructure error that's causing the
wrong ramdisk to be sent to the boards only for clang but I'd be a bit
surprised.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: next/master boot: 265 boots: 17 failed, 184 passed with 64 offline (next-20190730)

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

On Tue, Jul 30, 2019 at 05:17:56AM -0700, kernelci.org bot wrote:

The previously reported issues with booting -next on
meson-gxm-khadas-vim2 are still present today, though seemingly only
manifesting with CONFIG_RANDOMIZE_BASE and not defconfig (there are
failures with big endian too but they don't look device specific):

> arm64:

>     defconfig+CONFIG_RANDOMIZE_BASE=y:
>         gcc-8:
>             meson-gxm-khadas-vim2: 1 failed lab

It looks like it gets to userspace and then hangs (end of the log
below).  More details at:

	https://kernelci.org/boot/id/5d40069859b5148b3931b2bf/

The last message in the log indicates it was initializing the Panfrost
driver:

08:53:47.332143  <6>[   15.172833] panfrost d00c0000.gpu: clock rate = 666666666
08:55:40.299880  ShellCommand command timed out.: Sending # in case of corruption. Connection timeout 00:04:14, retry in 00:02:07

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: next/master boot: 265 boots: 17 failed, 184 passed with 64 offline (next-20190730)

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1246 bytes --]

On Tue, Jul 30, 2019 at 05:17:56AM -0700, kernelci.org bot wrote:

> Boot Failures Detected:
> 
> arm:
>     oxnas_v6_defconfig:
>         gcc-8:
>             ox820-cloudengines-pogoplug-series-3: 1 failed lab

For some time now -next and mainline have been failing to boot on
Pogoplug 3 with the oxnas_v6_defconfig, the kernel seems to start fine
but fails to parse the ramdisk it's passed:

08:50:02.086589  <6>[    7.719854] IP-Config: Complete:
08:50:02.087213  <6>[    7.723330]      device=eth0, hwaddr=0a:a2:89:27:10:1b, ipaddr=10.201.4.144, mask=255.255.0.0, gw=10.201.0.1
08:50:02.087413  <6>[    7.733409]      host=10.201.4.144, domain=, nis-domain=(none)
08:50:02.088056  <6>[    7.739499]      bootserver=10.201.1.1, rootserver=10.201.1.1, rootpath=
08:50:02.088248  <6>[    7.739504]      nameserver0=10.201.1.1
08:50:02.129966  <5>[    7.752025] RAMDISK: Couldn't find valid RAM disk image starting at 0.
08:50:02.130381  <4>[    7.759616] List of all partitions:
08:50:02.131333  <4>[    7.763363] 0100           65536 ram0 

Possibly an issue with the ramdisk getting overwritten or something?

Full details for today's -next can be seen here:

	https://kernelci.org/boot/id/5d4004bb59b51489d631b28d/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]