Re: [linux-next] PPC Lpar fail to boot with error hid: module verification failed: signature and/or required key missing - tainting kernel

Re: [linux-next] PPC Lpar fail to boot with error hid: module verification failed: signature and/or required key missing - tainting kernel

[Jiri Kosina]

On Thu, 25 May 2017, Abdul Haleem wrote:

> Hi,
> 
> Test : boot
> Machine : Power 8 PowerVM LPAR
> Kernel version : 4.12.0-rc2-next-20170524
> Toolchain : gcc version 5.2.1
> Config : attached
> 
> next-20170517 was last good version where we did not see the failure.
> from next-20170519 to today's next-20170524, we see the problem
> everyday.
> 
> Kernels fails to boot on PowerVM LPAR with these errors;
> 
> Freeing unused kernel memory: 4416K
> This architecture does not have kernel memory protection.
> Loading, please wait...
> mount: mounting udev on /dev failed: Device or resource busy
> W: devtmpfs not available, falling back to tmpfs for /dev
> starting version 222
> Begin: Loading essential drivers ... done.
> Begin: Running /scripts/init-premount ... done.
> Begin: Mounting root file system ... Begin:
> Running /scripts/local-top ... done.
> chvt: can't open console
> Gave up waiting for root device.  Common problems:
>  - Boot args (cat /proc/cmdline)
>    - Check rootdelay= (did the system wait long enough?)
>    - Check root= (did the system wait for the right device?)
>  - Missing modules (cat /proc/modules; ls /dev)
> chvt: can't open console
> ALERT!  /dev/disk/by-uuid/3aee8de3-5b8f-4a89-9e89-108efe213983 does not
> exist.  Dropping to a shell!

Seems like the by-uuid node for your harddrive containing rootfs didn't 
show up.

The EBUSY from mount(/dev) seems like the biggest suspect.

These:

> hid: module verification failed: signature and/or required key missing -
> tainting kernel
> hidraw: raw HID events driver (C) Jiri Kosina
> usbcore: registered new interface driver usbhid
> usbhid: USB HID core driver

are for sure unrelated.

Please focus on figuring out why mounting /dev doesn't succeed.

-- 
Jiri Kosina
SUSE Labs

Re: [linux-next] PPC Lpar fail to boot with error hid: module verification failed: signature and/or required key missing - tainting kernel

[Michael Ellerman]

Abdul Haleem <abdhalee@linux.vnet.ibm.com> writes:

> Hi,
>
> Test : boot
> Machine : Power 8 PowerVM LPAR
> Kernel version : 4.12.0-rc2-next-20170524
> Toolchain : gcc version 5.2.1
> Config : attached
>
> next-20170517 was last good version where we did not see the failure.
> from next-20170519 to today's next-20170524, we see the problem
> everyday.
>
> Kernels fails to boot on PowerVM LPAR with these errors;
>
> Freeing unused kernel memory: 4416K
> This architecture does not have kernel memory protection.
> Loading, please wait...
> mount: mounting udev on /dev failed: Device or resource busy
> W: devtmpfs not available, falling back to tmpfs for /dev

It'll be:

ee35011fd032 ("initramfs: make initramfs honor CONFIG_DEVTMPFS_MOUNT")

cheers

Re: [linux-next] PPC Lpar fail to boot with error hid: module verification failed: signature and/or required key missing - tainting kernel

[Stephen Rothwell]

Hi Michael,

On Thu, 25 May 2017 23:02:06 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote:
>
> It'll be:
> 
> ee35011fd032 ("initramfs: make initramfs honor CONFIG_DEVTMPFS_MOUNT")

And Andrew has asked me to drop that patch from linux-next which will
happen today.

-- 
Cheers,
Stephen Rothwell

Re: [linux-next] PPC Lpar fail to boot with error hid: module verification failed: signature and/or required key missing - tainting kernel

[Rob Landley]

On 05/25/2017 04:24 PM, Stephen Rothwell wrote:
> Hi Michael,
> 
> On Thu, 25 May 2017 23:02:06 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote:
>>
>> It'll be:
>>
>> ee35011fd032 ("initramfs: make initramfs honor CONFIG_DEVTMPFS_MOUNT")
>
> And Andrew has asked me to drop that patch from linux-next which will
> happen today.

What approach do the kernel developers suggest I take here?

I would have thought letting it soak in linux-next for a release so
people could fix userspace bugs would be the next step, but this sounds
like that's not an option?

Is the behavior the patch implements wrong?

Rob

Re: [linux-next] PPC Lpar fail to boot with error hid: module verification failed: signature and/or required key missing - tainting kernel

[Michael Ellerman]

Rob Landley <rob@landley.net> writes:

> On 05/25/2017 04:24 PM, Stephen Rothwell wrote:
>> Hi Michael,
>> 
>> On Thu, 25 May 2017 23:02:06 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote:
>>>
>>> It'll be:
>>>
>>> ee35011fd032 ("initramfs: make initramfs honor CONFIG_DEVTMPFS_MOUNT")
>>
>> And Andrew has asked me to drop that patch from linux-next which will
>> happen today.
>
> What approach do the kernel developers suggest I take here?

Well I'm just *a* kernel developer, but rule #1 is don't break userspace.

> I would have thought letting it soak in linux-next for a release so
> people could fix userspace bugs would be the next step, but this sounds
> like that's not an option?

You say they're userspace bugs, userspace will say it's a bug that the
kernel has changed its behaviour.

> Is the behavior the patch implements wrong?

Yes, because it breaks existing setups for no particularly good reason.

If CONFIG_DEVTMPFS_MOUNT had always meant devtmpfs was mounted in the
initramfs then that would have been fine.

But because it didn't, there are now systems out there that depend on
the existing behaviour, and changing it is therefore wrong IMHO.

As I said in another mail you can avoid breaking existing setups by
adding a new config option to control mounting devtmpfs in the
initramfs. It's a pity to need yet another config option, but such is
life.

cheers

[PATCH v3] Make initramfs honor CONFIG_DEVTMPFS_MOUNT

[Rob Landley]

From: Rob Landley <rob@landley.net>

Make initramfs honor CONFIG_DEVTMPFS_MOUNT, and move
/dev/console open after devtmpfs mount.

Add workaround for Debian bug that was copied by Ubuntu.

Signed-off-by: Rob Landley <rob@landley.net>
---

v2 discussion: http://lkml.iu.edu/hypermail/linux/kernel/1705.2/05611.html

 drivers/base/Kconfig |   14 ++++----------
 fs/namespace.c       |   14 ++++++++++++++
 init/main.c          |   15 +++++++++------
 3 files changed, 27 insertions(+), 16 deletions(-)

diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index f046d21..97352d4 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -48,16 +48,10 @@ config DEVTMPFS_MOUNT
 	bool "Automount devtmpfs at /dev, after the kernel mounted the rootfs"
 	depends on DEVTMPFS
 	help
-	  This will instruct the kernel to automatically mount the
-	  devtmpfs filesystem at /dev, directly after the kernel has
-	  mounted the root filesystem. The behavior can be overridden
-	  with the commandline parameter: devtmpfs.mount=0|1.
-	  This option does not affect initramfs based booting, here
-	  the devtmpfs filesystem always needs to be mounted manually
-	  after the rootfs is mounted.
-	  With this option enabled, it allows to bring up a system in
-	  rescue mode with init=/bin/sh, even when the /dev directory
-	  on the rootfs is completely empty.
+	  Automatically mount devtmpfs at /dev on the root filesystem, which
+	  lets the system to come up in rescue mode with [rd]init=/bin/sh.
+	  Override with devtmpfs.mount=0 on the commandline. Initramfs can
+	  create a /dev dir as needed, other rootfs needs the mount point.
 
 config STANDALONE
 	bool "Select only drivers that don't need compile-time external firmware"
diff --git a/fs/namespace.c b/fs/namespace.c
index f8893dc..06057d7 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -2417,7 +2417,21 @@ static int do_add_mount(struct mount *newmnt, struct path *path, int mnt_flags)
 	err = -EBUSY;
 	if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb &&
 	    path->mnt->mnt_root == path->dentry)
+	{
+		if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT) &&
+		    !strcmp(path->mnt->mnt_sb->s_type->name, "devtmpfs"))
+		{
+			/* Debian's kernel config enables DEVTMPFS_MOUNT, then
+			   its initramfs setup script tries to mount devtmpfs
+			   again, and if the second mount-over-itself fails
+			   the script overmounts a tmpfs on /dev to hide the
+			   existing contents, then boot fails with empty /dev. */
+			printk(KERN_WARNING "Debian bug workaround for devtmpfs overmount.");
+
+			err = 0;
+		}
 		goto unlock;
+	}
 
 	err = -EINVAL;
 	if (d_is_symlink(newmnt->mnt.mnt_root))
diff --git a/init/main.c b/init/main.c
index 0ee9c686..0d8e5ec 100644
--- a/init/main.c
+++ b/init/main.c
@@ -1065,12 +1065,6 @@ static noinline void __init kernel_init_freeable(void)
 
 	do_basic_setup();
 
-	/* Open the /dev/console on the rootfs, this should never fail */
-	if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
-		pr_err("Warning: unable to open an initial console.\n");
-
-	(void) sys_dup(0);
-	(void) sys_dup(0);
 	/*
 	 * check if there is an early userspace init.  If yes, let it do all
 	 * the work
@@ -1082,8 +1076,17 @@ static noinline void __init kernel_init_freeable(void)
 	if (sys_access((const char __user *) ramdisk_execute_command, 0) != 0) {
 		ramdisk_execute_command = NULL;
 		prepare_namespace();
+	} else if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT)) {
+		sys_mkdir("/dev", 0755);
+		devtmpfs_mount("/dev");
 	}
 
+	/* Open the /dev/console on the rootfs, this should never fail */
+	if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
+		pr_err("Warning: unable to open an initial console.\n");
+	(void) sys_dup(0);
+	(void) sys_dup(0);
+
 	/*
 	 * Ok, we have completed the initial bootup, and
 	 * we're essentially up and running. Get rid of the

Re: [PATCH v3] Make initramfs honor CONFIG_DEVTMPFS_MOUNT

[Christophe LEROY]

Le 14/09/2017 à 01:51, Rob Landley a écrit :
> From: Rob Landley <rob@landley.net>
> 
> Make initramfs honor CONFIG_DEVTMPFS_MOUNT, and move
> /dev/console open after devtmpfs mount.
> 
> Add workaround for Debian bug that was copied by Ubuntu.

Is that a bug only for Debian ? Why ?
Why should a Debian bug be fixed by a workaround in the mainline kernel ?

> 
> Signed-off-by: Rob Landley <rob@landley.net>
> ---
> 
> v2 discussion: http://lkml.iu.edu/hypermail/linux/kernel/1705.2/05611.html
> 
>   drivers/base/Kconfig |   14 ++++----------
>   fs/namespace.c       |   14 ++++++++++++++
>   init/main.c          |   15 +++++++++------
>   3 files changed, 27 insertions(+), 16 deletions(-)
> 
> diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
> index f046d21..97352d4 100644
> --- a/drivers/base/Kconfig
> +++ b/drivers/base/Kconfig
> @@ -48,16 +48,10 @@ config DEVTMPFS_MOUNT
>   	bool "Automount devtmpfs at /dev, after the kernel mounted the rootfs"
>   	depends on DEVTMPFS
>   	help
> -	  This will instruct the kernel to automatically mount the
> -	  devtmpfs filesystem at /dev, directly after the kernel has
> -	  mounted the root filesystem. The behavior can be overridden
> -	  with the commandline parameter: devtmpfs.mount=0|1.
> -	  This option does not affect initramfs based booting, here
> -	  the devtmpfs filesystem always needs to be mounted manually
> -	  after the rootfs is mounted.
> -	  With this option enabled, it allows to bring up a system in
> -	  rescue mode with init=/bin/sh, even when the /dev directory
> -	  on the rootfs is completely empty.
> +	  Automatically mount devtmpfs at /dev on the root filesystem, which
> +	  lets the system to come up in rescue mode with [rd]init=/bin/sh.
> +	  Override with devtmpfs.mount=0 on the commandline. Initramfs can
> +	  create a /dev dir as needed, other rootfs needs the mount point.

Why modifying the initial text ?
Why talking about rescue mode only, whereas this feature mainly concerns 
the standard mode.

>   
>   config STANDALONE
>   	bool "Select only drivers that don't need compile-time external firmware"
> diff --git a/fs/namespace.c b/fs/namespace.c
> index f8893dc..06057d7 100644
> --- a/fs/namespace.c
> +++ b/fs/namespace.c
> @@ -2417,7 +2417,21 @@ static int do_add_mount(struct mount *newmnt, struct path *path, int mnt_flags)
>   	err = -EBUSY;
>   	if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb &&
>   	    path->mnt->mnt_root == path->dentry)
> +	{
> +		if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT) &&
> +		    !strcmp(path->mnt->mnt_sb->s_type->name, "devtmpfs"))
> +		{
> +			/* Debian's kernel config enables DEVTMPFS_MOUNT, then
> +			   its initramfs setup script tries to mount devtmpfs
> +			   again, and if the second mount-over-itself fails
> +			   the script overmounts a tmpfs on /dev to hide the
> +			   existing contents, then boot fails with empty /dev. */

Does it matter for the kernel code what Debian's kernel config does ?

> +			printk(KERN_WARNING "Debian bug workaround for devtmpfs overmount.");

Is this log message worth it when this modification goes in mainline 
kernel ?

If so, pr_err() should be used instead.

> +
> +			err = 0;
> +		}
>   		goto unlock;
> +	}
>   
>   	err = -EINVAL;
>   	if (d_is_symlink(newmnt->mnt.mnt_root))
> diff --git a/init/main.c b/init/main.c
> index 0ee9c686..0d8e5ec 100644
> --- a/init/main.c
> +++ b/init/main.c
> @@ -1065,12 +1065,6 @@ static noinline void __init kernel_init_freeable(void)
>   
>   	do_basic_setup();
>   
> -	/* Open the /dev/console on the rootfs, this should never fail */
> -	if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
> -		pr_err("Warning: unable to open an initial console.\n");
> -
> -	(void) sys_dup(0);
> -	(void) sys_dup(0);
>   	/*
>   	 * check if there is an early userspace init.  If yes, let it do all
>   	 * the work
> @@ -1082,8 +1076,17 @@ static noinline void __init kernel_init_freeable(void)
>   	if (sys_access((const char __user *) ramdisk_execute_command, 0) != 0) {
>   		ramdisk_execute_command = NULL;
>   		prepare_namespace();
> +	} else if (IS_ENABLED(CONFIG_DEVTMPFS_MOUNT)) {
> +		sys_mkdir("/dev", 0755);

Why not, but couldn't we also expect the initramfs to already contains 
that mountpoint ?

> +		devtmpfs_mount("/dev");
>   	}
>   
> +	/* Open the /dev/console on the rootfs, this should never fail */
> +	if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
> +		pr_err("Warning: unable to open an initial console.\n");
> +	(void) sys_dup(0);
> +	(void) sys_dup(0);
> +
>   	/*
>   	 * Ok, we have completed the initial bootup, and
>   	 * we're essentially up and running. Get rid of the
> 


Christophe

Re: [PATCH v3] Make initramfs honor CONFIG_DEVTMPFS_MOUNT

[Greg KH]

On Wed, Sep 13, 2017 at 06:51:25PM -0500, Rob Landley wrote:
> From: Rob Landley <rob@landley.net>
> 
> Make initramfs honor CONFIG_DEVTMPFS_MOUNT, and move
> /dev/console open after devtmpfs mount.
> 
> Add workaround for Debian bug that was copied by Ubuntu.
> 
> Signed-off-by: Rob Landley <rob@landley.net>
> ---
> 
> v2 discussion: http://lkml.iu.edu/hypermail/linux/kernel/1705.2/05611.html
> 
>  drivers/base/Kconfig |   14 ++++----------
>  fs/namespace.c       |   14 ++++++++++++++
>  init/main.c          |   15 +++++++++------
>  3 files changed, 27 insertions(+), 16 deletions(-)

Always run scripts/checkpatch.pl so you don't get grumpy emails from
reviewers telling you to run scripts/checkpatch.pl... telling you to run
scripts/checkpatch.pl... telling you to run scripts/checkpatch.pl...
telling you to run scripts/checkpatch.pl...

Re: [PATCH v3] Make initramfs honor CONFIG_DEVTMPFS_MOUNT

[Rob Landley]

On 09/14/2017 04:17 AM, Christophe LEROY wrote:
> Le 14/09/2017 à 01:51, Rob Landley a écrit :
>> From: Rob Landley <rob@landley.net>
>>
>> Make initramfs honor CONFIG_DEVTMPFS_MOUNT, and move
>> /dev/console open after devtmpfs mount.
>>
>> Add workaround for Debian bug that was copied by Ubuntu.
> 
> Is that a bug only for Debian ? Why ?

Look down, specifically this bit:

>> v2 discussion:
>> http://lkml.iu.edu/hypermail/linux/kernel/1705.2/05611.html

That's some discussion of version 2 of this patch, which was merged for
a while last dev cycle, then backed out again because it triggered the
same bug in a number of system init scripts:

  http://lkml.iu.edu/hypermail/linux/kernel/1705.2/07072.html
  http://lkml.iu.edu/hypermail/linux/kernel/1705.3/01182.html
  http://lkml.iu.edu/hypermail/linux/kernel/1705.3/01505.html
  http://lkml.iu.edu/hypermail/linux/kernel/1705.3/01320.html

All of whom copied the broken error "recovery" path from debian. If they
checked whether it was already mounted, or didn't _blank_ the /dev
directory in response to mounting the exact same filesystem over itself
giving -EBUSY, the system would work fine. Heck, if you built a kernel
with a static /dev in initramfs and no devtmpfs configured in, the
script would break things exactly the same way. The breakage is that
script takes a hammer to a perfectly functional /dev directory and then
continues the boot with an empty /dev. That's bonkers.

> Why should a Debian bug be fixed by a workaround in the mainline kernel ?

That was my argument last time, and the answer was "Breaking userspace
is bad, mmmkay." Even when userspace is doing something REALLY OBVIOUSLY
STUPID and it is _clearly_ their fault, as long as they got there first
they've established the status quo and it doesn't matter how silly it is.

This was explicitly stated to me here:

  http://lkml.iu.edu/hypermail/linux/kernel/1705.3/03292.html

I.E. don't argue with me, argue with him. :)

So, I added a workaround with a printk in hopes of embarassing them into
someday fixing it.

Rob

Re: [PATCH v3] Make initramfs honor CONFIG_DEVTMPFS_MOUNT

[Henrique de Moraes Holschuh]

On Sat, 16 Sep 2017, Rob Landley wrote:
> So, I added a workaround with a printk in hopes of embarassing them into
> someday fixing it.

Oh, it will be fixed in Debian alright.  I am just waiting the issue to
settle a bit to file the bug reports, or maybe even send in the Debian
patches myself (note that I am not responsible for the code in question,
so I am not wearing a brown paperbag at this time).  Even if I didn't do
it, there are several other Debian Developers reading LKML that could do
it (provided they noticed this specific thread and are aware of the
situation) :p

I can even push for the fixes to be accepted into the stable and
oldstable branches of Debian, but that can take anything from a few
weeks to several months, due to the way our stable releases work.  But
it would eventually happen.

Whether such fixes will ever make it to LTS branches, especially
Ubuntu's, *that* I don't know.

-- 
  Henrique Holschuh

Re: [PATCH v3] Make initramfs honor CONFIG_DEVTMPFS_MOUNT

[Rob Landley]

On 09/17/2017 08:51 AM, Henrique de Moraes Holschuh wrote:
> On Sat, 16 Sep 2017, Rob Landley wrote:
>> So, I added a workaround with a printk in hopes of embarassing them into
>> someday fixing it.
> 
> Oh, it will be fixed in Debian alright.

Cool!

But part of the problem is people upgrade the kernel on existing
deployed root filesystems, some of which are a fork off of a fork off of
debian, so we won't exhaust the broken userspace for probably a couple
years.

I'd put it in feature-removal-schedule.txt but Linus zapped that, so...

> I am just waiting the issue to
> settle a bit to file the bug reports, or maybe even send in the Debian
> patches myself (note that I am not responsible for the code in question,
> so I am not wearing a brown paperbag at this time).  Even if I didn't do
> it, there are several other Debian Developers reading LKML that could do
> it (provided they noticed this specific thread and are aware of the
> situation) :p

There was a previous thread last merge window they didn't notice. I was
hoping the warning would be obvious enough. :)

> I can even push for the fixes to be accepted into the stable and
> oldstable branches of Debian, but that can take anything from a few
> weeks to several months, due to the way our stable releases work.  But
> it would eventually happen.
> 
> Whether such fixes will ever make it to LTS branches, especially
> Ubuntu's, *that* I don't know.

I have no idea what that powerpc system was, the guy didn't say...

Rob

Re: [PATCH v3] Make initramfs honor CONFIG_DEVTMPFS_MOUNT

[Michael Ellerman]

Rob Landley <rob@landley.net> writes:

> On 09/14/2017 04:17 AM, Christophe LEROY wrote:
>> Le 14/09/2017 à 01:51, Rob Landley a écrit :
>>> From: Rob Landley <rob@landley.net>
>>>
>>> Make initramfs honor CONFIG_DEVTMPFS_MOUNT, and move
>>> /dev/console open after devtmpfs mount.
>>>
>>> Add workaround for Debian bug that was copied by Ubuntu.
>> 
>> Is that a bug only for Debian ? Why ?
>
> Look down, specifically this bit:
>
>>> v2 discussion:
>>> http://lkml.iu.edu/hypermail/linux/kernel/1705.2/05611.html
>
> That's some discussion of version 2 of this patch, which was merged for
> a while last dev cycle, then backed out again because it triggered the
> same bug in a number of system init scripts:
>
>   http://lkml.iu.edu/hypermail/linux/kernel/1705.2/07072.html
>   http://lkml.iu.edu/hypermail/linux/kernel/1705.3/01182.html
>   http://lkml.iu.edu/hypermail/linux/kernel/1705.3/01505.html
>   http://lkml.iu.edu/hypermail/linux/kernel/1705.3/01320.html
>
> All of whom copied the broken error "recovery" path from debian. If they
> checked whether it was already mounted, or didn't _blank_ the /dev
> directory in response to mounting the exact same filesystem over itself
> giving -EBUSY, the system would work fine. Heck, if you built a kernel
> with a static /dev in initramfs and no devtmpfs configured in, the
> script would break things exactly the same way. The breakage is that
> script takes a hammer to a perfectly functional /dev directory and then
> continues the boot with an empty /dev. That's bonkers.
>
>> Why should a Debian bug be fixed by a workaround in the mainline kernel ?
>
> That was my argument last time, and the answer was "Breaking userspace
> is bad, mmmkay." Even when userspace is doing something REALLY OBVIOUSLY
> STUPID and it is _clearly_ their fault, as long as they got there first
> they've established the status quo and it doesn't matter how silly it is.
>
> This was explicitly stated to me here:
>
>   http://lkml.iu.edu/hypermail/linux/kernel/1705.3/03292.html
>
> I.E. don't argue with me, argue with him. :)

I'm still here. And I'm still right :)

No one wants their system to stop booting because of this obscure
functionality.

Just put it behind a new config option which defaults off. No
workarounds required, no broken systems, no long email threads required.

cheers