Widespread crashes in next-20180906

Widespread crashes in next-20180906

[Guenter Roeck]

Build results:
	total: 134 pass: 133 fail: 1
Failed builds:
	sparc32:allmodconfig
Qemu test results:
	total: 311 pass: 76 fail: 235
Failed builds:
	<pretty much everything trying to boot from disk>

Error message is always something like

Filesystem requires source device
VFS: Cannot open root device "hda" or unknown-block(3,0): error -2

The only variance is the boot device. Logs in full glory are available
at https://kerneltests.org/builders/, in the "next" column.

I did not run bisect, but the recent filesystem changes are a definite suspect.

Guenter

Re: Widespread crashes in next-20180906

[Theodore Y. Ts'o]

On Thu, Sep 06, 2018 at 06:45:15AM -0700, Guenter Roeck wrote:
> Build results:
> 	total: 134 pass: 133 fail: 1
> Failed builds:
> 	sparc32:allmodconfig
> Qemu test results:
> 	total: 311 pass: 76 fail: 235
> Failed builds:
> 	<pretty much everything trying to boot from disk>
> 
> Error message is always something like
> 
> Filesystem requires source device
> VFS: Cannot open root device "hda" or unknown-block(3,0): error -2
> 
> The only variance is the boot device. Logs in full glory are available
> at https://kerneltests.org/builders/, in the "next" column.
> 
> I did not run bisect, but the recent filesystem changes are a definite suspect.

Yes, this is the vm_fault_t changes.  See the other thread on LKML.
The guilty commit was: 83c0adddcc6e: fs: convert return type int to
vm_fault_t

This is the *second* time vm_fault_t patches have broken things.  The
first time it went through the ext4 tree, and I NACK'ed it after
running a 60 second smoke test showed it was broken.  The seocnd time
the problem was supposedly fixed, but it went through the mm tree, and
so I didn't have a chance regression test or stop it...

      		       		  	      	     - Ted

Re: Widespread crashes in next-20180906

[Matt Hart]

On 6 September 2018 at 15:04, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Thu, Sep 06, 2018 at 06:45:15AM -0700, Guenter Roeck wrote:
>> Build results:
>>       total: 134 pass: 133 fail: 1

Do you build arm64? Because KernelCI is seeing build failures in arm64
defconfig for next-20180906
Clearly it's a module build problem for sunxi but I'm not sure who to
CC about this.

https://kernelci.org/build/next/branch/master/kernel/next-20180906/
https://storage.kernelci.org/next/master/next-20180906/arm64/defconfig/build.log

ERROR: "sun8i_tcon_top_de_config"
[drivers/gpu/drm/sun4i/sun4i-tcon.ko] undefined!
ERROR: "sun8i_tcon_top_set_hdmi_src"
[drivers/gpu/drm/sun4i/sun4i-tcon.ko] undefined!
ERROR: "sun8i_tcon_top_of_table" [drivers/gpu/drm/sun4i/sun4i-tcon.ko]
undefined!
../scripts/Makefile.modpost:92: recipe for target '__modpost' failed
make[2]: *** [__modpost] Error 1
make[2]: Target '_modpost' not remade because of errors.
/home/buildslave/workspace/kernel-build/linux/Makefile:1223: recipe
for target 'modules' failed
make[1]: *** [modules] Error 2
make[1]: Target '_all' not remade because of errors.
Makefile:146: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2
make: Target '_all' not remade because of errors.

>> Failed builds:
>>       sparc32:allmodconfig
>> Qemu test results:
>>       total: 311 pass: 76 fail: 235
>> Failed builds:
>>       <pretty much everything trying to boot from disk>
>>
>> Error message is always something like
>>
>> Filesystem requires source device
>> VFS: Cannot open root device "hda" or unknown-block(3,0): error -2
>>
>> The only variance is the boot device. Logs in full glory are available
>> at https://kerneltests.org/builders/, in the "next" column.
>>
>> I did not run bisect, but the recent filesystem changes are a definite suspect.
>
> Yes, this is the vm_fault_t changes.  See the other thread on LKML.
> The guilty commit was: 83c0adddcc6e: fs: convert return type int to
> vm_fault_t
>
> This is the *second* time vm_fault_t patches have broken things.  The
> first time it went through the ext4 tree, and I NACK'ed it after
> running a 60 second smoke test showed it was broken.  The seocnd time
> the problem was supposedly fixed, but it went through the mm tree, and
> so I didn't have a chance regression test or stop it...
>
>                                                      - Ted

Re: Widespread crashes in next-20180906

[Guenter Roeck]

On Thu, Sep 06, 2018 at 03:13:23PM +0100, Matt Hart wrote:
> On 6 September 2018 at 15:04, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> > On Thu, Sep 06, 2018 at 06:45:15AM -0700, Guenter Roeck wrote:
> >> Build results:
> >>       total: 134 pass: 133 fail: 1
> 
> Do you build arm64? Because KernelCI is seeing build failures in arm64
> defconfig for next-20180906
> Clearly it's a module build problem for sunxi but I'm not sure who to
> CC about this.
> 
I do, but as part of the boot tests, not as explicit build test,
to save a bit of time. Maybe I should run a build test as well.

> https://kernelci.org/build/next/branch/master/kernel/next-20180906/
> https://storage.kernelci.org/next/master/next-20180906/arm64/defconfig/build.log
> 
> ERROR: "sun8i_tcon_top_de_config"
> [drivers/gpu/drm/sun4i/sun4i-tcon.ko] undefined!
> ERROR: "sun8i_tcon_top_set_hdmi_src"
> [drivers/gpu/drm/sun4i/sun4i-tcon.ko] undefined!
> ERROR: "sun8i_tcon_top_of_table" [drivers/gpu/drm/sun4i/sun4i-tcon.ko]
> undefined!

Same error here, in the boot tests. Sorry, there are just too many
failures there, and I didn't go through all the logs.

Do you have bisect results ? Otherwise I can run bisect later today;
that should hopefully identify the culprit.

Thanks,
Guenter

Re: Widespread crashes in next-20180906

[Guenter Roeck]

On Thu, Sep 06, 2018 at 10:04:13AM -0400, Theodore Y. Ts'o wrote:
> On Thu, Sep 06, 2018 at 06:45:15AM -0700, Guenter Roeck wrote:
> > Build results:
> > 	total: 134 pass: 133 fail: 1
> > Failed builds:
> > 	sparc32:allmodconfig
> > Qemu test results:
> > 	total: 311 pass: 76 fail: 235
> > Failed builds:
> > 	<pretty much everything trying to boot from disk>
> > 
> > Error message is always something like
> > 
> > Filesystem requires source device
> > VFS: Cannot open root device "hda" or unknown-block(3,0): error -2
> > 
> > The only variance is the boot device. Logs in full glory are available
> > at https://kerneltests.org/builders/, in the "next" column.
> > 
> > I did not run bisect, but the recent filesystem changes are a definite suspect.
> 
> Yes, this is the vm_fault_t changes.  See the other thread on LKML.
> The guilty commit was: 83c0adddcc6e: fs: convert return type int to
> vm_fault_t
> 
That thing is just asking for trouble. Why not leave return type
and value alone and add vm_fault_t * (assuming it really adds value)
as another parameter ? Is it really a good idea to deviate from "return
well defined error as integer" as used everywhere else in the kernel ?
Do we really need "my_favored_error_return_t" in every subsystem going
forward ? Oh well, I guess (hope) that is all discussed in the other
thread.

> This is the *second* time vm_fault_t patches have broken things.  The
> first time it went through the ext4 tree, and I NACK'ed it after
> running a 60 second smoke test showed it was broken.  The seocnd time
> the problem was supposedly fixed, but it went through the mm tree, and
> so I didn't have a chance regression test or stop it...
> 
Looking at the patch, NACK seems like the proper response to me, maybe
augmented with "please refrain from shooting yourself (and everyone else)
in the foot".

Guenter

Re: Widespread crashes in next-20180906

[Matt Hart]

On 6 September 2018 at 16:23, Guenter Roeck <linux@roeck-us.net> wrote:
> On Thu, Sep 06, 2018 at 03:13:23PM +0100, Matt Hart wrote:
>> On 6 September 2018 at 15:04, Theodore Y. Ts'o <tytso@mit.edu> wrote:
>> > On Thu, Sep 06, 2018 at 06:45:15AM -0700, Guenter Roeck wrote:
>> >> Build results:
>> >>       total: 134 pass: 133 fail: 1
>>
>> Do you build arm64? Because KernelCI is seeing build failures in arm64
>> defconfig for next-20180906
>> Clearly it's a module build problem for sunxi but I'm not sure who to
>> CC about this.
>>
> I do, but as part of the boot tests, not as explicit build test,
> to save a bit of time. Maybe I should run a build test as well.
>
>> https://kernelci.org/build/next/branch/master/kernel/next-20180906/
>> https://storage.kernelci.org/next/master/next-20180906/arm64/defconfig/build.log
>>
>> ERROR: "sun8i_tcon_top_de_config"
>> [drivers/gpu/drm/sun4i/sun4i-tcon.ko] undefined!
>> ERROR: "sun8i_tcon_top_set_hdmi_src"
>> [drivers/gpu/drm/sun4i/sun4i-tcon.ko] undefined!
>> ERROR: "sun8i_tcon_top_of_table" [drivers/gpu/drm/sun4i/sun4i-tcon.ko]
>> undefined!
>
> Same error here, in the boot tests. Sorry, there are just too many
> failures there, and I didn't go through all the logs.
>
> Do you have bisect results ? Otherwise I can run bisect later today;
> that should hopefully identify the culprit.

Bisecting it now, though it might finish after I'm done for the night.

>
> Thanks,
> Guenter

Re: Widespread crashes in next-20180906

[Theodore Y. Ts'o]

On Thu, Sep 06, 2018 at 03:13:23PM +0100, Matt Hart wrote:
> On 6 September 2018 at 15:04, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> > On Thu, Sep 06, 2018 at 06:45:15AM -0700, Guenter Roeck wrote:
> >> Build results:
> >>       total: 134 pass: 133 fail: 1
> 
> Do you build arm64? Because KernelCI is seeing build failures in arm64
> defconfig for next-20180906
> Clearly it's a module build problem for sunxi but I'm not sure who to
> CC about this.
> 
> https://kernelci.org/build/next/branch/master/kernel/next-20180906/
> https://storage.kernelci.org/next/master/next-20180906/arm64/defconfig/build.log

At a guess, from the MAINTAINERS file

ARM/Allwinner sunXi SoC support
M:	Maxime Ripard <maxime.ripard@bootlin.com>
M:	Chen-Yu Tsai <wens@csie.org>

						- Ted

Re: Widespread crashes in next-20180906

[Matt Hart]

On 6 September 2018 at 20:43, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Thu, Sep 06, 2018 at 03:13:23PM +0100, Matt Hart wrote:
>> On 6 September 2018 at 15:04, Theodore Y. Ts'o <tytso@mit.edu> wrote:
>> > On Thu, Sep 06, 2018 at 06:45:15AM -0700, Guenter Roeck wrote:
>> >> Build results:
>> >>       total: 134 pass: 133 fail: 1
>>
>> Do you build arm64? Because KernelCI is seeing build failures in arm64
>> defconfig for next-20180906
>> Clearly it's a module build problem for sunxi but I'm not sure who to
>> CC about this.
>>
>> https://kernelci.org/build/next/branch/master/kernel/next-20180906/
>> https://storage.kernelci.org/next/master/next-20180906/arm64/defconfig/build.log
>
> At a guess, from the MAINTAINERS file
>
> ARM/Allwinner sunXi SoC support
> M:      Maxime Ripard <maxime.ripard@bootlin.com>
> M:      Chen-Yu Tsai <wens@csie.org>

Seems they are already on the case
https://www.spinics.net/lists/arm-kernel/msg675227.html

>
>                                                 - Ted