[PATCH v5 0/3] Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

[PATCH v5 0/3] Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

[Baoquan He]

The v4 cover letter tells the background about this adding, paste the
link here for reference:
http://lkml.kernel.org/r/20190509013644.1246-1-bhe@redhat.com

Changelog:
v4->v5:
  Tune code and log per Dave's comments, no functional change.
  - In patch 2, change the printed erorr message; 
  - In patch 3, add macro SZ_64T and use it in code, and remove the
    obsolete code comment.
v3->v4:
  No functional change.
  - Rewrite log of patch 1/3 tell who the newly added bits are gonna be
    used.
  - Rewrite log of patch 2/3 per tglx's words.
  - Add Kirill's Acked-by.

v2->v3:
  Change the constant to match the notation for the rest of defines as
  Kirill suggested;
v1->v2:
  Correct the subject of patch 1 according to tglx's comment;
  Add more information to cover-letter to address reviewers' concerns;

Baoquan He (3):
  x86/boot: Add xloadflags bits for 5-level kernel checking
  x86/kexec/64: Error out if try to jump to old 4-level kernel from
    5-level kernel
  x86/kdump/64: Change the upper limit of crashkernel reservation

 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 arch/x86/kernel/kexec-bzimage64.c     |  5 +++++
 arch/x86/kernel/setup.c               | 15 ++++++++++++---
 include/linux/sizes.h                 |  1 +
 5 files changed, 31 insertions(+), 4 deletions(-)

-- 
2.17.2

[PATCH v5 1/3] x86/boot: Add xloadflags bits for 5-level kernel checking

[Baoquan He]

The current kernel supports 5-level paging mode, and supports dynamically
choosing paging mode during bootup according to kernel image, hardware and
kernel parameter setting. This flexibility brings several issues to
kexec/kdump:
1)
Dynamic switching between paging modes requires code change in target
kernel. So we can't do kexec-jumping from 5-level kernel to old 4-level
kernel which lacks the code change.

2)
Kexec jumping from 5-level kernel to 4-level kernel would fail, if kexec
loading puts kernel image above 64 TB of memory. Because kexec loading
searches area to put kernel from top to down in system RAM, the 2nd kernel
will be loaded above 64 TB if the amount of system RAM is bigger than 64
TB. Here no need to worry about kexec_file loading, because it searches
area from bottom to up, and can always find area below 4 GB.

Solution:

Add two bits XLF_5LEVEL and XLF_5LEVEL_ENABLED for 5-level kernel.
- Bit XLF_5LEVEL indicates if 5-level related code is contained in this
  kernel.
- Bit XLF_5LEVEL_ENABLED indicates if CONFIG_X86_5LEVEL=y is set.

a) For issue 1), need check if XLF_5LEVEL is set, otherwise print out error
   message.
  - This checking need be added in kernel, for the code of kexec_file
    loading;
  - And also need be added into kexec_tools utility, for kexec loading.

b) For issue 2), need check if both XLF_5LEVEL and XLF_5LEVEL_ENABLED are
   set, otherwise print out error message.
  - This only need be done in kexec_tools utility, because kexec loading
    does the searching in user space kexec_tools.

So here add XLF_5LEVEL and XLF_5LEVEL_ENABLED into xloadflags. The later
code will check XLF_5LEVEL bit in kexec_file implementation of kernel. And
the kexec_tools code will check both XLF_5LEVEL and XLF_5LEVEL_ENABLED for
kexec loading.

Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/arch/x86/boot/header.S b/arch/x86/boot/header.S
index 850b8762e889..be19f4199727 100644
--- a/arch/x86/boot/header.S
+++ b/arch/x86/boot/header.S
@@ -419,7 +419,17 @@ xloadflags:
 # define XLF4 0
 #endif
 
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
 
 cmdline_size:   .long   COMMAND_LINE_SIZE-1     #length of the command line,
                                                 #added with boot protocol
diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h
index 60733f137e9a..c895df5482c5 100644
--- a/arch/x86/include/uapi/asm/bootparam.h
+++ b/arch/x86/include/uapi/asm/bootparam.h
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 #ifndef __ASSEMBLY__
 
-- 
2.17.2

[PATCH v5 2/3] x86/kexec/64: Error out if try to jump to old 4-level kernel from 5-level kernel

[Baoquan He]

If the running kernel has 5-level paging activated, the 5-level paging
mode is preserved across kexec. If the kexec'ed kernel does not contain
support for handling active 5-level paging mode in the decompressor, the
decompressor will crash with #GP.

Prevent this situation at load time. If 5-level paging is active, check the
xloadflags whether the kexec kernel can handle 5-level paging at least in
the decompressor. If not, reject the load attempt and print out error
message.

Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/kernel/kexec-bzimage64.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/x86/kernel/kexec-bzimage64.c b/arch/x86/kernel/kexec-bzimage64.c
index 22f60dd26460..7f439739ea3d 100644
--- a/arch/x86/kernel/kexec-bzimage64.c
+++ b/arch/x86/kernel/kexec-bzimage64.c
@@ -321,6 +321,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)
 		return ret;
 	}
 
+	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
+		pr_err("bzImage cannot handle 5-level paging mode.\n");
+		return ret;
+	}
+
 	/* I've got a bzImage */
 	pr_debug("It's a relocatable bzImage64\n");
 	ret = 0;
-- 
2.17.2

[PATCH v5 3/3] x86/kdump/64: Change the upper limit of crashkernel reservation

[Baoquan He]

Restrict kdump to only reserve crashkernel below 64TB.

The reaons is that the kdump may jump from 5-level to 4-level, and if
the kdump kernel is put above 64TB, then the jumping will fail. While the
1st kernel reserves crashkernel region during bootup, we don't know yet
which kind of kernel will be loaded after system bootup, 5-level kernel
or 4-level kernel.

Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Dave Young <dyoung@redhat.com>
---
 arch/x86/kernel/setup.c | 15 ++++++++++++---
 include/linux/sizes.h   |  1 +
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 08a5f4a131f5..dcbdf54fb5c1 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -453,15 +453,24 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 #define CRASH_ALIGN		SZ_16M
 
 /*
- * Keep the crash kernel below this limit.  On 32 bits earlier kernels
- * would limit the kernel to the low 512 MiB due to mapping restrictions.
+ * Keep the crash kernel below this limit.
+ *
+ * On 32 bits earlier kernels would limit the kernel to the low 512 MiB
+ * due to mapping restrictions.
+ *
+ * On 64bit, kdump kernel need be restricted to be under 64TB, which is
+ * the upper limit of system RAM in 4-level paing mode. Since the kdump
+ * jumping could be from 5-level to 4-level, the jumping will fail if
+ * kernel is put above 64TB, and there's no way to detect the paging mode
+ * of the kernel which will be loaded for dumping during the 1st kernel
+ * bootup.
  */
 #ifdef CONFIG_X86_32
 # define CRASH_ADDR_LOW_MAX	SZ_512M
 # define CRASH_ADDR_HIGH_MAX	SZ_512M
 #else
 # define CRASH_ADDR_LOW_MAX	SZ_4G
-# define CRASH_ADDR_HIGH_MAX	MAXMEM
+# define CRASH_ADDR_HIGH_MAX	SZ_64T
 #endif
 
 static int __init reserve_crashkernel_low(void)
diff --git a/include/linux/sizes.h b/include/linux/sizes.h
index fbde0bc7e882..8651269cb46c 100644
--- a/include/linux/sizes.h
+++ b/include/linux/sizes.h
@@ -47,5 +47,6 @@
 #define SZ_2G				0x80000000
 
 #define SZ_4G				_AC(0x100000000, ULL)
+#define SZ_64T				_AC(0x400000000000, ULL)
 
 #endif /* __LINUX_SIZES_H__ */
-- 
2.17.2

Re: [PATCH v5 0/3] Add restrictions for kexec/kdump jumping between 5-level and 4-level kernel

[Baoquan He]

Hi,

On 05/24/19 at 03:38pm, Baoquan He wrote:
> 

Ping.

Can anyone help do further reviewing on this patchset? Or consider
merging since people have ack-ed?

Thanks
Baoquan

> The v4 cover letter tells the background about this adding, paste the
> link here for reference:
> http://lkml.kernel.org/r/20190509013644.1246-1-bhe@redhat.com
> 
> Changelog:
> v4->v5:
>   Tune code and log per Dave's comments, no functional change.
>   - In patch 2, change the printed erorr message; 
>   - In patch 3, add macro SZ_64T and use it in code, and remove the
>     obsolete code comment.
> v3->v4:
>   No functional change.
>   - Rewrite log of patch 1/3 tell who the newly added bits are gonna be
>     used.
>   - Rewrite log of patch 2/3 per tglx's words.
>   - Add Kirill's Acked-by.
> 
> v2->v3:
>   Change the constant to match the notation for the rest of defines as
>   Kirill suggested;
> v1->v2:
>   Correct the subject of patch 1 according to tglx's comment;
>   Add more information to cover-letter to address reviewers' concerns;
> 
> Baoquan He (3):
>   x86/boot: Add xloadflags bits for 5-level kernel checking
>   x86/kexec/64: Error out if try to jump to old 4-level kernel from
>     5-level kernel
>   x86/kdump/64: Change the upper limit of crashkernel reservation
> 
>  arch/x86/boot/header.S                | 12 +++++++++++-
>  arch/x86/include/uapi/asm/bootparam.h |  2 ++
>  arch/x86/kernel/kexec-bzimage64.c     |  5 +++++
>  arch/x86/kernel/setup.c               | 15 ++++++++++++---
>  include/linux/sizes.h                 |  1 +
>  5 files changed, 31 insertions(+), 4 deletions(-)
> 
> -- 
> 2.17.2
> 

[tip:x86/boot] x86/boot: Add xloadflags bits to check for 5-level paging support

[tip-bot for Baoquan He]

Commit-ID:  f2d08c5d3bcf3f7ef788af122b57a919efa1e9d0
Gitweb:     https://git.kernel.org/tip/f2d08c5d3bcf3f7ef788af122b57a919efa1e9d0
Author:     Baoquan He <bhe@redhat.com>
AuthorDate: Fri, 24 May 2019 15:38:08 +0800
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Fri, 28 Jun 2019 07:14:59 +0200

x86/boot: Add xloadflags bits to check for 5-level paging support

The current kernel supports 5-level paging mode, and supports dynamically
choosing the paging mode during bootup depending on the kernel image,
hardware and kernel parameter settings. This flexibility brings several
issues to kexec/kdump:

1) Dynamic switching between paging modes requires support in the target
   kernel. This means kexec from a 5-level paging kernel into a kernel
   which does not support mode switching is not possible. So the loader
   needs to be able to analyze the supported paging modes of the kexec
   target kernel.

2) If running on a 5-level paging kernel and the kexec target kernel is a
   4-level paging kernel, the target immage cannot be loaded above the 64TB
   address space limit. But the kexec loader searches for a load area from
   top to bottom which would eventually put the target kernel above 64TB
   when the machine has large enough RAM size. So the loader needs to be
   able to analyze the paging mode of the target kernel to load it at a
   suitable spot in the address space.

Solution:

Add two bits XLF_5LEVEL and XLF_5LEVEL_ENABLED:

 - Bit XLF_5LEVEL indicates whether 5-level paging mode switching support
   is available. (Issue #1)

 - Bit XLF_5LEVEL_ENABLED indicates whether the kernel was compiled with
   full 5-level paging support (CONFIG_X86_5LEVEL=y). (Issue #2)

The loader will use these bits to verify whether the target kernel is
suitable to be kexec'ed to from a 5-level paging kernel and to determine
the constraints of the target kernel load address.

The flags will be used by the kernel kexec subsystem and the userspace
kexec tools.

[ tglx: Massaged changelog ]

Signed-off-by: Baoquan He <bhe@redhat.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: bp@alien8.de
Cc: hpa@zytor.com
Cc: dyoung@redhat.com
Link: https://lkml.kernel.org/r/20190524073810.24298-2-bhe@redhat.com

---
 arch/x86/boot/header.S                | 12 +++++++++++-
 arch/x86/include/uapi/asm/bootparam.h |  2 ++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/arch/x86/boot/header.S b/arch/x86/boot/header.S
index 850b8762e889..be19f4199727 100644
--- a/arch/x86/boot/header.S
+++ b/arch/x86/boot/header.S
@@ -419,7 +419,17 @@ xloadflags:
 # define XLF4 0
 #endif
 
-			.word XLF0 | XLF1 | XLF23 | XLF4
+#ifdef CONFIG_X86_64
+#ifdef CONFIG_X86_5LEVEL
+#define XLF56 (XLF_5LEVEL|XLF_5LEVEL_ENABLED)
+#else
+#define XLF56 XLF_5LEVEL
+#endif
+#else
+#define XLF56 0
+#endif
+
+			.word XLF0 | XLF1 | XLF23 | XLF4 | XLF56
 
 cmdline_size:   .long   COMMAND_LINE_SIZE-1     #length of the command line,
                                                 #added with boot protocol
diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h
index 60733f137e9a..c895df5482c5 100644
--- a/arch/x86/include/uapi/asm/bootparam.h
+++ b/arch/x86/include/uapi/asm/bootparam.h
@@ -29,6 +29,8 @@
 #define XLF_EFI_HANDOVER_32		(1<<2)
 #define XLF_EFI_HANDOVER_64		(1<<3)
 #define XLF_EFI_KEXEC			(1<<4)
+#define XLF_5LEVEL			(1<<5)
+#define XLF_5LEVEL_ENABLED		(1<<6)
 
 #ifndef __ASSEMBLY__
 

[tip:x86/boot] x86/kexec/64: Prevent kexec from 5-level paging to a 4-level only kernel

[tip-bot for Baoquan He]

Commit-ID:  ee338b9ee2822e65a85750da6129946c14962410
Gitweb:     https://git.kernel.org/tip/ee338b9ee2822e65a85750da6129946c14962410
Author:     Baoquan He <bhe@redhat.com>
AuthorDate: Fri, 24 May 2019 15:38:09 +0800
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Fri, 28 Jun 2019 07:14:59 +0200

x86/kexec/64: Prevent kexec from 5-level paging to a 4-level only kernel

If the running kernel has 5-level paging activated, the 5-level paging mode
is preserved across kexec. If the kexec'ed kernel does not contain support
for handling active 5-level paging mode in the decompressor, the
decompressor will crash with #GP.

Prevent this situation at load time. If 5-level paging is active, check the
xloadflags whether the kexec kernel can handle 5-level paging at least in
the decompressor. If not, reject the load attempt and print out an error
message.

Signed-off-by: Baoquan He <bhe@redhat.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: bp@alien8.de
Cc: hpa@zytor.com
Cc: dyoung@redhat.com
Link: https://lkml.kernel.org/r/20190524073810.24298-3-bhe@redhat.com

---
 arch/x86/kernel/kexec-bzimage64.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/x86/kernel/kexec-bzimage64.c b/arch/x86/kernel/kexec-bzimage64.c
index 22f60dd26460..7f439739ea3d 100644
--- a/arch/x86/kernel/kexec-bzimage64.c
+++ b/arch/x86/kernel/kexec-bzimage64.c
@@ -321,6 +321,11 @@ static int bzImage64_probe(const char *buf, unsigned long len)
 		return ret;
 	}
 
+	if (!(header->xloadflags & XLF_5LEVEL) && pgtable_l5_enabled()) {
+		pr_err("bzImage cannot handle 5-level paging mode.\n");
+		return ret;
+	}
+
 	/* I've got a bzImage */
 	pr_debug("It's a relocatable bzImage64\n");
 	ret = 0;

[tip:x86/boot] x86/kdump/64: Restrict kdump kernel reservation to <64TB

[tip-bot for Baoquan He]

Commit-ID:  8ff80fbe7e9870078b1cc3c2cdd8f3f223b333a9
Gitweb:     https://git.kernel.org/tip/8ff80fbe7e9870078b1cc3c2cdd8f3f223b333a9
Author:     Baoquan He <bhe@redhat.com>
AuthorDate: Fri, 24 May 2019 15:38:10 +0800
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Fri, 28 Jun 2019 07:14:59 +0200

x86/kdump/64: Restrict kdump kernel reservation to <64TB

Restrict kdump to only reserve crashkernel below 64TB.

The reaons is that the kdump may jump from a 5-level paging mode to a
4-level paging mode kernel. If a 4-level paging mode kdump kernel is put
above 64TB, then the kdump kernel cannot start.

The 1st kernel reserves the kdump kernel region during bootup. At that
point it is not known whether the kdump kernel has 5-level or 4-level
paging support.

To support both restrict the kdump kernel reservation to the lower 64TB
address space to ensure that a 4-level paging mode kdump kernel can be
loaded and successfully started.

[ tglx: Massaged changelog ]

Signed-off-by: Baoquan He <bhe@redhat.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Dave Young <dyoung@redhat.com>
Cc: bp@alien8.de
Cc: hpa@zytor.com
Link: https://lkml.kernel.org/r/20190524073810.24298-4-bhe@redhat.com

---
 arch/x86/kernel/setup.c | 15 ++++++++++++---
 include/linux/sizes.h   |  1 +
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 08a5f4a131f5..dcbdf54fb5c1 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -453,15 +453,24 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 #define CRASH_ALIGN		SZ_16M
 
 /*
- * Keep the crash kernel below this limit.  On 32 bits earlier kernels
- * would limit the kernel to the low 512 MiB due to mapping restrictions.
+ * Keep the crash kernel below this limit.
+ *
+ * On 32 bits earlier kernels would limit the kernel to the low 512 MiB
+ * due to mapping restrictions.
+ *
+ * On 64bit, kdump kernel need be restricted to be under 64TB, which is
+ * the upper limit of system RAM in 4-level paing mode. Since the kdump
+ * jumping could be from 5-level to 4-level, the jumping will fail if
+ * kernel is put above 64TB, and there's no way to detect the paging mode
+ * of the kernel which will be loaded for dumping during the 1st kernel
+ * bootup.
  */
 #ifdef CONFIG_X86_32
 # define CRASH_ADDR_LOW_MAX	SZ_512M
 # define CRASH_ADDR_HIGH_MAX	SZ_512M
 #else
 # define CRASH_ADDR_LOW_MAX	SZ_4G
-# define CRASH_ADDR_HIGH_MAX	MAXMEM
+# define CRASH_ADDR_HIGH_MAX	SZ_64T
 #endif
 
 static int __init reserve_crashkernel_low(void)
diff --git a/include/linux/sizes.h b/include/linux/sizes.h
index fbde0bc7e882..8651269cb46c 100644
--- a/include/linux/sizes.h
+++ b/include/linux/sizes.h
@@ -47,5 +47,6 @@
 #define SZ_2G				0x80000000
 
 #define SZ_4G				_AC(0x100000000, ULL)
+#define SZ_64T				_AC(0x400000000000, ULL)
 
 #endif /* __LINUX_SIZES_H__ */