[PATCH RESEND 0/3] Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

[PATCH RESEND 0/3] Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

[Baoquan He]

This is a RESEND post.

The original v1 post can be found here:
http://lkml.kernel.org/r/20180829141624.13985-1-bhe@redhat.com

It's trying to fix several corner case issues for kexec/kdump when
dynamic switching of paging mode is enabled in x86_64. Please click 
above link to check the details.

In v1, hpa raised concern that why the paging mode checking is not done
before kexec jumping, the discussion can be found here:

http://lkml.kernel.org/r/alpine.DEB.2.21.1809051002020.1416@nanos.tec.linutronix.de

As tglx said, it might be not doable for kdump since kdump kernel's
reserved crashkernel region only owns a portion of memory, may
be above 4G; and might be not safer to do paging mode checking and
switching thing after crash.

So resend this patchset.

Baoquan He (3):
  x86/boot: Add bit fields into xloadflags for 5-level kernel checking
  x86/kexec/64: Error out if try to jump to old 4-level kernel from
    5-level kernel
  x86/kdump/64: Change the upper limit of crashkernel reservation

 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 arch/x86/kernel/kexec-bzimage64.c     |  5 +++++
 arch/x86/kernel/setup.c               | 18 ++++++++++++++----
 4 files changed, 32 insertions(+), 5 deletions(-)

-- 
2.13.6

[PATCH RESEND 1/3] x86/boot: Add bit fields into xloadflags for 5-level kernel checking

[Baoquan He]

Add two bit fields XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel.
Bit XLF_5LEVEL indicates if 5-level related code is contained
in this kernel.
Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

They are being used in later patch to check if kexec/kdump kernel
is loaded in right place.

Signed-off-by: Baoquan He <bhe@redhat.com>
---
 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/arch/x86/boot/header.S b/arch/x86/boot/header.S
index 850b8762e889..be19f4199727 100644
--- a/arch/x86/boot/header.S
+++ b/arch/x86/boot/header.S
@@ -419,7 +419,17 @@ xloadflags:
 # define XLF4 0
 #endif
 
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
 
 cmdline_size:   .long   COMMAND_LINE_SIZE-1     #length of the command line,
                                                 #added with boot protocol
diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h
index 60733f137e9a..c895df5482c5 100644
--- a/arch/x86/include/uapi/asm/bootparam.h
+++ b/arch/x86/include/uapi/asm/bootparam.h
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 #ifndef __ASSEMBLY__
 
-- 
2.13.6

[PATCH RESEND 2/3] x86/kexec/64: Error out if try to jump to old 4-level kernel from 5-level kernel

[Baoquan He]

In relocate_kernel() CR4.LA57 flag is set before kexec jumping if
the kernel has 5-level paging enabled. Then in boot/compressed/head_64.S,
it will check if the booting kernel is in 4-level or 5-level paging
mode, and handle accordingly. However, the old kernel which doesn't
contain the 5-level codes doesn't know how to cope with it, then #GP
triggered.

Instead of triggering #GP during kexec kernel boot, error out during
kexec loading if find out we are trying to jump to old 4-level kernel
from 5-level kernel.

Signed-off-by: Baoquan He <bhe@redhat.com>
---
 arch/x86/kernel/kexec-bzimage64.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/x86/kernel/kexec-bzimage64.c b/arch/x86/kernel/kexec-bzimage64.c
index 278cd07228dd..5bfb39fc56fe 100644
--- a/arch/x86/kernel/kexec-bzimage64.c
+++ b/arch/x86/kernel/kexec-bzimage64.c
@@ -316,6 +316,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)
 		return ret;
 	}
 
+	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
+		pr_err("Can not jump to old 4-level kernel from 5-level kernel.\n");
+		return ret;
+	}
+
 	/* I've got a bzImage */
 	pr_debug("It's a relocatable bzImage64\n");
 	ret = 0;
-- 
2.13.6

[PATCH RESEND 3/3] x86/kdump/64: Change the upper limit of crashkernel reservation

[Baoquan He]

Restrict kdump to only reserve crashkernel below 64TB. Since the kdump
jumping may be from 5-level to 4-level, and the kdump kernel is put
above 64TB in 5-level kernel, then the jumping will fail. And the
crashkernel reservation is done during the 1st kernel bootup, there's
no way to detect the paging mode of kdump kernel at that time.

Hence change the upper limit of crashkernel reservation to 64TB
on x86_64.

Signed-off-by: Baoquan He <bhe@redhat.com>
---
 arch/x86/kernel/setup.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 3d872a527cd9..d4d1366738a4 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -451,16 +451,26 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 #define CRASH_ALIGN		(16 << 20)
 
 /*
- * Keep the crash kernel below this limit.  On 32 bits earlier kernels
- * would limit the kernel to the low 512 MiB due to mapping restrictions.
- * On 64bit, old kexec-tools need to under 896MiB.
+ * Keep the crash kernel below this limit.
+ *
+ * On 32 bits earlier kernels would limit the kernel to the low
+ * 512 MiB due to mapping restrictions.
+ *
+ * On 64bit, old kexec-tools need to be under 896MiB. The later
+ * supports to put kernel above 4G, up to system RAM top. Here
+ * kdump kernel need be restricted to be under 64TB, which is
+ * the upper limit of system RAM in 4-level paing mode. Since
+ * the kdump jumping could be from 5-level to 4-level, the jumping
+ * will fail if kernel is put above 64TB, and there's no way to
+ * detect the paging mode of the kernel which will be loaded for
+ * dumping during the 1st kernel bootup.
  */
 #ifdef CONFIG_X86_32
 # define CRASH_ADDR_LOW_MAX	(512 << 20)
 # define CRASH_ADDR_HIGH_MAX	(512 << 20)
 #else
 # define CRASH_ADDR_LOW_MAX	(896UL << 20)
-# define CRASH_ADDR_HIGH_MAX	MAXMEM
+# define CRASH_ADDR_HIGH_MAX	(1ULL < 46)
 #endif
 
 static int __init reserve_crashkernel_low(void)
-- 
2.13.6

Re: [PATCH RESEND 0/3] Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

[Kirill A. Shutemov]

On Fri, Jan 25, 2019 at 10:28:14AM +0800, Baoquan He wrote:
> This is a RESEND post.
> 
> The original v1 post can be found here:
> http://lkml.kernel.org/r/20180829141624.13985-1-bhe@redhat.com
> 
> It's trying to fix several corner case issues for kexec/kdump when
> dynamic switching of paging mode is enabled in x86_64. Please click 
> above link to check the details.
> 
> In v1, hpa raised concern that why the paging mode checking is not done
> before kexec jumping, the discussion can be found here:
> 
> http://lkml.kernel.org/r/alpine.DEB.2.21.1809051002020.1416@nanos.tec.linutronix.de
> 
> As tglx said, it might be not doable for kdump since kdump kernel's
> reserved crashkernel region only owns a portion of memory, may
> be above 4G; and might be not safer to do paging mode checking and
> switching thing after crash.
> 
> So resend this patchset.

Changes look good to me:

Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>

-- 
 Kirill A. Shutemov

Re: [PATCH RESEND 1/3] x86/boot: Add bit fields into xloadflags for 5-level kernel checking

[Thomas Gleixner]

On Fri, 25 Jan 2019, Baoquan He wrote:

> Add two bit fields XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel.

These are not bit fields. These are simple bits.

> Bit XLF_5LEVEL indicates if 5-level related code is contained
> in this kernel.
> Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

I'm confused. 

> -			.word XLF0 | XLF1 | XLF23 | XLF4
> +#ifdef CONFIG_X86_64
> +#ifdef CONFIG_X86_5LEVEL
> +#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
> +#else
> +#define XLF56 XLF_5LEVEL
> +#endif
> +#else
> +#define XLF56 0
> +#endif
> +
> +			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56

So this actually stores the bits, but looking at the following patch which
fixes the real issue:

> +	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
> +		pr_err("Can not jump to old 4-level kernel from 5-level kernel.\n");
> +		return ret;
> +	}

So what is XLF_5LEVEL_ENABLED used for and why does it exist at all?

Thanks,

	tglx

Re: [PATCH RESEND 1/3] x86/boot: Add bit fields into xloadflags for 5-level kernel checking

[Baoquan He]

Thanks for reviewing. I was in vacation, sorry for late reply.

On 01/29/19 at 09:05pm, Thomas Gleixner wrote:
> On Fri, 25 Jan 2019, Baoquan He wrote:
> 
> > Add two bit fields XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel.
> 
> These are not bit fields. These are simple bits.

Indeed, they are only xloadflags bits, will change. Thanks.

> 
> > Bit XLF_5LEVEL indicates if 5-level related code is contained
> > in this kernel.
> > Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.
> 
> I'm confused. 
> 
> > -			.word XLF0 | XLF1 | XLF23 | XLF4
> > +#ifdef CONFIG_X86_64
> > +#ifdef CONFIG_X86_5LEVEL
> > +#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
> > +#else
> > +#define XLF56 XLF_5LEVEL
> > +#endif
> > +#else
> > +#define XLF56 0
> > +#endif
> > +
> > +			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
> 
> So this actually stores the bits, but looking at the following patch which
> fixes the real issue:
> 
> > +	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
> > +		pr_err("Can not jump to old 4-level kernel from 5-level kernel.\n");
> > +		return ret;
> > +	}
> 
> So what is XLF_5LEVEL_ENABLED used for and why does it exist at all?

Yes, this is a little bit confusing. I explained it in the v1 cover
letter:
http://lists.infradead.org/pipermail/kexec/2018-August/021419.html

As told at above, XLF_5LEVEL marks the new kernel containing 5level
code, while XLF_5LEVEL_ENABLED marking the CONFIG_X86_5LEVEL option
enabling. Hence if XLF_5LEVEL is set, XLF_5LEVEL_ENABLED not, means it's
new kernel but can't be switched into 5-level.

For kexec_load and kexec_file_load, there's difference in loading
behaviour. kexec_load will search available area top down to put
kernel in system RAM, we need check if the kexec-ed kernel is in
leve-5 paging mode, and limit the loading postion below 64 TB if
not. But for kexec_file_load, it's searching area bottom up to put
kernel, most of time area found below 4G. We don't have worry about the
kexec_file_load interface which implements the loading functionality in
kernel. That's why the XLF_5LEVEL_ENABLED bit is not used in this kernel
patch set, I would like to post patch to kexec-tools for kexec_load
after these patches have been accepted.

I ever tried to unify the behavious of these two interfaces on loading
kernel, to make both kexec_load and kexec_file_load search and put
kernel top to down, but that involves many lines of code change, seems
people are worried about it and hesitated to offere ack, I just gave up.
Please check below link:

https://lore.kernel.org/lkml/20180718024944.577-1-bhe@redhat.com/T/#u

Sorry for the inconvenience because of my missing explanation.

Thanks
Baoquan

[PATCH v2 1/3] x86/boot: Add xloadflags bits for 5-level kernel checking

[Baoquan He]

Add two bits XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel.
Bit XLF_5LEVEL indicates if 5-level related code is contained
in this kernel.
Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

They are being used in later patch to check if kexec/kdump kernel
is loaded in right place.

Signed-off-by: Baoquan He <bhe@redhat.com>
---
v1->v2:
  Change the wrong term in subject pointed out by tglx.

 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/arch/x86/boot/header.S b/arch/x86/boot/header.S
index 850b8762e889..be19f4199727 100644
--- a/arch/x86/boot/header.S
+++ b/arch/x86/boot/header.S
@@ -419,7 +419,17 @@ xloadflags:
 # define XLF4 0
 #endif
 
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
 
 cmdline_size:   .long   COMMAND_LINE_SIZE-1     #length of the command line,
                                                 #added with boot protocol
diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h
index 60733f137e9a..c895df5482c5 100644
--- a/arch/x86/include/uapi/asm/bootparam.h
+++ b/arch/x86/include/uapi/asm/bootparam.h
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 #ifndef __ASSEMBLY__
 
-- 
2.17.2