next: /dev/root: Can't open blockdev

next: /dev/root: Can't open blockdev

[Naresh Kamboju]

All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
kernel crash.

Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>

Crash log:
---------
<3>[    3.257960] /dev/root: Can't open blockdev
<4>[    3.258940] VFS: Cannot open root device "/dev/sda" or
unknown-block(8,0): error -16
<4>[    3.259704] Please append a correct "root=" boot option; here
are the available partitions:
<4>[    3.261088] 0800         2500336 sda
<4>[    3.261186]  driver: sd
<4>[    3.262260] 0b00         1048575 sr0
<4>[    3.262409]  driver: sr
<3>[    3.263022] List of all bdev filesystems:
<3>[    3.263553]  ext3
<3>[    3.263708]  ext2
<3>[    3.263994]  ext4
<3>[    3.264160]  vfat
<3>[    3.264419]  msdos
<3>[    3.264589]  iso9660
<3>[    3.264773]
<0>[    3.265665] Kernel panic - not syncing: VFS: Unable to mount
root fs on unknown-block(8,0)
<4>[    3.266991] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
6.8.0-rc3-next-20240206 #1
<4>[    3.267593] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
BIOS 1.16.3-debian-1.16.3-2 04/01/2014
<4>[    3.268937] Call Trace:
<4>[    3.269316]  <TASK>
<4>[    3.270113]  dump_stack_lvl+0x71/0xb0
<4>[    3.271837]  dump_stack+0x14/0x20
<4>[    3.272128]  panic+0x12f/0x2f0
<4>[    3.272812]  ? _printk+0x5d/0x80
<4>[    3.273097]  mount_root_generic+0x26e/0x2b0
<4>[    3.273941]  mount_block_root+0x3f/0x50
<4>[    3.274212]  mount_root+0x60/0x80
<4>[    3.274610]  prepare_namespace+0x7a/0xb0
<4>[    3.276008]  kernel_init_freeable+0x137/0x180
<4>[    3.276285]  ? __pfx_kernel_init+0x10/0x10
<4>[    3.276563]  kernel_init+0x1e/0x1a0
<4>[    3.276837]  ret_from_fork+0x45/0x50
<4>[    3.277319]  ? __pfx_kernel_init+0x10/0x10
<4>[    3.278176]  ret_from_fork_asm+0x1a/0x30
<4>[    3.278560]  </TASK>
<0>[    3.280750] Kernel Offset: 0x1a800000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
<0>[    3.281985] ---[ end Kernel panic - not syncing: VFS: Unable to
mount root fs on unknown-block(8,0) ]---


Links:
 - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/testrun/22547673/suite/log-parser-test/test/check-kernel-panic/log
 - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/testrun/22547673/suite/log-parser-test/tests/

--
Linaro LKFT
https://lkft.linaro.org

Re: next: /dev/root: Can't open blockdev

[Jan Kara]

On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
> All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
> kernel crash.
> 
> Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
> 
> Crash log:
> ---------
> <3>[    3.257960] /dev/root: Can't open blockdev
> <4>[    3.258940] VFS: Cannot open root device "/dev/sda" or
> unknown-block(8,0): error -16

Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are
suspect? Do you have some sample kconfig available somewhere?

								Honza

> <4>[    3.259704] Please append a correct "root=" boot option; here
> are the available partitions:
> <4>[    3.261088] 0800         2500336 sda
> <4>[    3.261186]  driver: sd
> <4>[    3.262260] 0b00         1048575 sr0
> <4>[    3.262409]  driver: sr
> <3>[    3.263022] List of all bdev filesystems:
> <3>[    3.263553]  ext3
> <3>[    3.263708]  ext2
> <3>[    3.263994]  ext4
> <3>[    3.264160]  vfat
> <3>[    3.264419]  msdos
> <3>[    3.264589]  iso9660
> <3>[    3.264773]
> <0>[    3.265665] Kernel panic - not syncing: VFS: Unable to mount
> root fs on unknown-block(8,0)
> <4>[    3.266991] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> 6.8.0-rc3-next-20240206 #1
> <4>[    3.267593] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
> BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> <4>[    3.268937] Call Trace:
> <4>[    3.269316]  <TASK>
> <4>[    3.270113]  dump_stack_lvl+0x71/0xb0
> <4>[    3.271837]  dump_stack+0x14/0x20
> <4>[    3.272128]  panic+0x12f/0x2f0
> <4>[    3.272812]  ? _printk+0x5d/0x80
> <4>[    3.273097]  mount_root_generic+0x26e/0x2b0
> <4>[    3.273941]  mount_block_root+0x3f/0x50
> <4>[    3.274212]  mount_root+0x60/0x80
> <4>[    3.274610]  prepare_namespace+0x7a/0xb0
> <4>[    3.276008]  kernel_init_freeable+0x137/0x180
> <4>[    3.276285]  ? __pfx_kernel_init+0x10/0x10
> <4>[    3.276563]  kernel_init+0x1e/0x1a0
> <4>[    3.276837]  ret_from_fork+0x45/0x50
> <4>[    3.277319]  ? __pfx_kernel_init+0x10/0x10
> <4>[    3.278176]  ret_from_fork_asm+0x1a/0x30
> <4>[    3.278560]  </TASK>
> <0>[    3.280750] Kernel Offset: 0x1a800000 from 0xffffffff81000000
> (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> <0>[    3.281985] ---[ end Kernel panic - not syncing: VFS: Unable to
> mount root fs on unknown-block(8,0) ]---
> 
> 
> Links:
>  - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/testrun/22547673/suite/log-parser-test/test/check-kernel-panic/log
>  - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/testrun/22547673/suite/log-parser-test/tests/
> 
> --
> Linaro LKFT
> https://lkft.linaro.org
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: next: /dev/root: Can't open blockdev

[Naresh Kamboju]

On Tue, 6 Feb 2024 at 15:45, Jan Kara <jack@suse.cz> wrote:
>
> On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
> > All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
> > kernel crash.
> >
> > Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
> >
> > Crash log:
> > ---------
> > <3>[    3.257960] /dev/root: Can't open blockdev
> > <4>[    3.258940] VFS: Cannot open root device "/dev/sda" or
> > unknown-block(8,0): error -16
>
> Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are
> suspect? Do you have some sample kconfig available somewhere?

All build information is in this url,
https://storage.tuxsuite.com/public/linaro/lkft/builds/2byqguFVp7MYAEjKo6nJGba2FcP/

- Naresh

Re: next: /dev/root: Can't open blockdev

[Jan Kara]

On Tue 06-02-24 15:53:34, Naresh Kamboju wrote:
> On Tue, 6 Feb 2024 at 15:45, Jan Kara <jack@suse.cz> wrote:
> >
> > On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
> > > All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
> > > kernel crash.
> > >
> > > Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
> > >
> > > Crash log:
> > > ---------
> > > <3>[    3.257960] /dev/root: Can't open blockdev
> > > <4>[    3.258940] VFS: Cannot open root device "/dev/sda" or
> > > unknown-block(8,0): error -16
> >
> > Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are
> > suspect? Do you have some sample kconfig available somewhere?
> 
> All build information is in this url,
> https://storage.tuxsuite.com/public/linaro/lkft/builds/2byqguFVp7MYAEjKo6nJGba2FcP/

Thanks! So for record the config has:

CONFIG_BLK_DEV_WRITE_MOUNTED=y

So we are not hitting any weird corner case with blocking writes to mounted
filesystems. It must be something else.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: next: /dev/root: Can't open blockdev

[Naresh Kamboju]

On Tue, 6 Feb 2024 at 17:58, Jan Kara <jack@suse.cz> wrote:
>
> On Tue 06-02-24 15:53:34, Naresh Kamboju wrote:
> > On Tue, 6 Feb 2024 at 15:45, Jan Kara <jack@suse.cz> wrote:
> > >
> > > On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
> > > > All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
> > > > kernel crash.
> > > >
> > > > Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
> > > >
> > > > Crash log:
> > > > ---------
> > > > <3>[    3.257960] /dev/root: Can't open blockdev
> > > > <4>[    3.258940] VFS: Cannot open root device "/dev/sda" or
> > > > unknown-block(8,0): error -16
> > >
> > > Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are
> > > suspect? Do you have some sample kconfig available somewhere?
> >
> > All build information is in this url,
> > https://storage.tuxsuite.com/public/linaro/lkft/builds/2byqguFVp7MYAEjKo6nJGba2FcP/
>
> Thanks! So for record the config has:
>
> CONFIG_BLK_DEV_WRITE_MOUNTED=y
>
> So we are not hitting any weird corner case with blocking writes to mounted
> filesystems. It must be something else.

As per Anders bisection results the first bad commit pointing to
   ba858e55b205 ("bdev: open block device as files")

- Naresh

Re: next: /dev/root: Can't open blockdev

[Christian Brauner]

> As per Anders bisection results the first bad commit pointing to
>    ba858e55b205 ("bdev: open block device as files")

On it.

Re: next: /dev/root: Can't open blockdev

[Christian Brauner]

> On it.

Ok, can you try:
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.super.debug
please?

Re: next: /dev/root: Can't open blockdev

[Marek Szyprowski]

Hi Christian,

On 06.02.2024 16:53, Christian Brauner wrote:
>> On it.
> Ok, can you try:
> git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.super.debug
> please?


I've also encountered this issue during my linux-next daily tests and I 
confirm that the above branch works fine.


I've applied the diff between e3bfad989976^2 and the above branch 
(bc7cb6c829e2), which looks following:

diff --git a/init/do_mounts.c b/init/do_mounts.c
index 279ad28bf4fb..d8ea839463a5 100644
--- a/init/do_mounts.c
+++ b/init/do_mounts.c
@@ -19,6 +19,7 @@
  #include <linux/ramfs.h>
  #include <linux/shmem_fs.h>
  #include <linux/ktime.h>
+#include <linux/task_work.h>

  #include <linux/nfs_fs.h>
  #include <linux/nfs_fs_sb.h>
@@ -208,6 +209,10 @@ void __init mount_root_generic(char *name, char 
*pretty_name, int flags)
                                 goto out;
                         case -EACCES:
                         case -EINVAL:
+#ifdef CONFIG_BLOCK
+                               flush_delayed_fput();
+                               task_work_run();
+#endif
                                 continue;
                 }
                 /*


onto next-20240206 and it fixed all boot problems I've observed on my 
test farm. :)

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: next: /dev/root: Can't open blockdev

[Aithal, Srikanth]

On 2/7/2024 4:35 AM, Marek Szyprowski wrote:
> Hi Christian,
> 
> On 06.02.2024 16:53, Christian Brauner wrote:
>>> On it.
>> Ok, can you try:
>> git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.super.debug
>> please?
> 
> 
> I've also encountered this issue during my linux-next daily tests and I
> confirm that the above branch works fine.
> 
> 
> I've applied the diff between e3bfad989976^2 and the above branch
> (bc7cb6c829e2), which looks following:
> 
> diff --git a/init/do_mounts.c b/init/do_mounts.c
> index 279ad28bf4fb..d8ea839463a5 100644
> --- a/init/do_mounts.c
> +++ b/init/do_mounts.c
> @@ -19,6 +19,7 @@
>    #include <linux/ramfs.h>
>    #include <linux/shmem_fs.h>
>    #include <linux/ktime.h>
> +#include <linux/task_work.h>
> 
>    #include <linux/nfs_fs.h>
>    #include <linux/nfs_fs_sb.h>
> @@ -208,6 +209,10 @@ void __init mount_root_generic(char *name, char
> *pretty_name, int flags)
>                                   goto out;
>                           case -EACCES:
>                           case -EINVAL:
> +#ifdef CONFIG_BLOCK
> +                               flush_delayed_fput();
> +                               task_work_run();
> +#endif
>                                   continue;
>                   }
>                   /*
> 
> 
> onto next-20240206 and it fixed all boot problems I've observed on my
> test farm. :)
> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> 
> Best regards

We as well encountered this issue during our CI run. The given branch 
fixes the issues seen in our run.

Tested-by: Srikanth Aithal <sraithal@amd.com>