kexec.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug
@ 2022-06-13 22:42 Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 1/7] crash: move crash_prepare_elf64_headers Eric DeVolder
                   ` (7 more replies)
  0 siblings, 8 replies; 20+ messages in thread
From: Eric DeVolder @ 2022-06-13 22:42 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	eric.devolder

When the kdump service is loaded, if a CPU or memory is hot
un/plugged, the crash elfcorehdr (for x86), which describes the CPUs
and memory in the system, must also be updated, else the resulting
vmcore is inaccurate (eg. missing either CPU context or memory
regions).

The current solution utilizes udev to initiate an unload-then-reload
of the kdump image (e. kernel, initrd, boot_params, puratory and
elfcorehdr) by the userspace kexec utility. In previous posts I have
outlined the significant performance problems related to offloading
this activity to userspace.

This patchset introduces a generic crash hot un/plug handler that
registers with the CPU and memory notifiers. Upon CPU or memory
changes, this generic handler is invoked and performs important
housekeeping, for example obtaining the appropriate lock, and then
invokes an architecture specific handler to do the appropriate
updates.

In the case of x86_64, the arch specific handler generates a new
elfcorehdr, and overwrites the old one in memory. No involvement
with userspace needed.

To realize the benefits/test this patchset, one must make a couple
of minor changes to userspace:

 - Prevent udev from updating kdump crash kernel on hot un/plug changes.
   Add the following as the first lines to the udev rule file
   /usr/lib/udev/rules.d/98-kexec.rules:

   # The kernel handles updates to crash elfcorehdr for cpu and memory changes
   SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
   SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"

   These lines will cause cpu and memory hot un/plug events to be
   skipped within this rule file, if the kernel has these changes
   enabled.

 - Change to the kexec_file_load for loading the kdump kernel:
   Eg. on RHEL: in /usr/bin/kdumpctl, change to:
    standard_kexec_args="-p -d -s"
   which adds the -s to select kexec_file_load syscall.

This patchset supports kexec_load with a modified kexec userspace
utility, and a working changeset to the kexec userspace utility
is provided here (and to use, the above change to standard_kexec_args
would be, for example, to append --hotplug instead of -s).

  diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
  index 9826f6d..4ed395a 100644
  --- a/kexec/arch/i386/crashdump-x86.c
  +++ b/kexec/arch/i386/crashdump-x86.c
  @@ -48,6 +48,7 @@
   #include <x86/x86-linux.h>
   
   extern struct arch_options_t arch_options;
  +extern int do_hotplug;
   
   static int get_kernel_page_offset(struct kexec_info *UNUSED(info),
   				  struct crash_elf_info *elf_info)
  @@ -975,6 +976,14 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
   	} else {
   		memsz = bufsz;
   	}
  +
  +	/* If hotplug support enabled, use larger size to accomodate changes */
  +	if (do_hotplug) {
  +		long int nr_cpus = get_nr_cpus();
  +		memsz = (nr_cpus + CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
  +	}
  +
  +    info->elfcorehdr =
   	elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
   							max_addr, -1);
   	dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
  diff --git a/kexec/crashdump-elf.c b/kexec/crashdump-elf.c
  index b8bb686..5e29f7a 100644
  --- a/kexec/crashdump-elf.c
  +++ b/kexec/crashdump-elf.c
  @@ -43,11 +43,7 @@ int FUNC(struct kexec_info *info,
   	int (*get_note_info)(int cpu, uint64_t *addr, uint64_t *len);
   	long int count_cpu;
   
  -	if (xen_present())
  -		nr_cpus = xen_get_nr_phys_cpus();
  -	else
  -		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
  -
  +	nr_cpus = get_nr_cpus();
   	if (nr_cpus < 0) {
   		return -1;
   	}
  diff --git a/kexec/crashdump.h b/kexec/crashdump.h
  index 18bd691..28d3278 100644
  --- a/kexec/crashdump.h
  +++ b/kexec/crashdump.h
  @@ -57,7 +57,6 @@ unsigned long phys_to_virt(struct crash_elf_info *elf_info,
   			   unsigned long long paddr);
   
   unsigned long xen_architecture(struct crash_elf_info *elf_info);
  -int xen_get_nr_phys_cpus(void);
   int xen_get_note(int cpu, uint64_t *addr, uint64_t *len);
   int xen_get_crashkernel_region(uint64_t *start, uint64_t *end);
   
  diff --git a/kexec/kexec-xen.h b/kexec/kexec-xen.h
  index 70fb576..f54a2dd 100644
  --- a/kexec/kexec-xen.h
  +++ b/kexec/kexec-xen.h
  @@ -83,5 +83,6 @@ extern int __xc_interface_close(xc_interface *xch);
   #endif
   
   int xen_get_kexec_range(int range, uint64_t *start, uint64_t *end);
  +int xen_get_nr_phys_cpus(void);
   
   #endif /* KEXEC_XEN_H */
  diff --git a/kexec/kexec.c b/kexec/kexec.c
  index 829a6ea..3668b73 100644
  --- a/kexec/kexec.c
  +++ b/kexec/kexec.c
  @@ -58,6 +58,7 @@
   
   unsigned long long mem_min = 0;
   unsigned long long mem_max = ULONG_MAX;
  +int do_hotplug = 0;
   static unsigned long kexec_flags = 0;
   /* Flags for kexec file (fd) based syscall */
   static unsigned long kexec_file_flags = 0;
  @@ -489,6 +490,17 @@ static int add_backup_segments(struct kexec_info *info,
   	return 0;
   }
   
  +long int get_nr_cpus(void)
  +{
  +    long int nr_cpus;
  +
  +	if (xen_present())
  +		nr_cpus = xen_get_nr_phys_cpus();
  +	else
  +		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
  +    return nr_cpus;
  +}
  +
   static char *slurp_fd(int fd, const char *filename, off_t size, off_t *nread)
   {
   	char *buf;
  @@ -672,6 +684,14 @@ static void update_purgatory(struct kexec_info *info)
   		if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
   			continue;
   		}
  +
  +		/* Don't include elfcorehdr in the checksum, if hotplug
  +		 * support enabled.
  +		 */
  +		if (do_hotplug && (info->segment[i].mem == (void *)info->elfcorehdr)) {
  +			continue;
  +		}
  +
   		sha256_update(&ctx, info->segment[i].buf,
   			      info->segment[i].bufsz);
   		nullsz = info->segment[i].memsz - info->segment[i].bufsz;
  @@ -1565,6 +1585,9 @@ int main(int argc, char *argv[])
   		case OPT_PRINT_CKR_SIZE:
   			print_crashkernel_region_size();
   			return 0;
  +		case OPT_HOTPLUG:
  +			do_hotplug = 1;
  +			break;
   		default:
   			break;
   		}
  diff --git a/kexec/kexec.h b/kexec/kexec.h
  index 0f97a97..b0428cc 100644
  --- a/kexec/kexec.h
  +++ b/kexec/kexec.h
  @@ -169,6 +169,7 @@ struct kexec_info {
   	int command_line_len;
   
   	int skip_checks;
  +	unsigned long elfcorehdr;
   };
   
   struct arch_map_entry {
  @@ -231,7 +232,8 @@ extern int file_types;
   #define OPT_PRINT_CKR_SIZE	262
   #define OPT_LOAD_LIVE_UPDATE	263
   #define OPT_EXEC_LIVE_UPDATE	264
  -#define OPT_MAX			265
  +#define OPT_HOTPLUG		265
  +#define OPT_MAX		266
   #define KEXEC_OPTIONS \
   	{ "help",		0, 0, OPT_HELP }, \
   	{ "version",		0, 0, OPT_VERSION }, \
  @@ -258,6 +260,7 @@ extern int file_types;
   	{ "debug",		0, 0, OPT_DEBUG }, \
   	{ "status",		0, 0, OPT_STATUS }, \
   	{ "print-ckr-size",     0, 0, OPT_PRINT_CKR_SIZE }, \
  +	{ "hotplug",		0, 0, OPT_HOTPLUG }, \
   
   #define KEXEC_OPT_STR "h?vdfixyluet:pscaS"
   
  @@ -290,6 +293,8 @@ extern unsigned long add_buffer_phys_virt(struct kexec_info *info,
   	int buf_end, int phys);
   extern void arch_reuse_initrd(void);
   
  +extern long int get_nr_cpus(void);
  +
   extern int ifdown(void);
   
   extern char purgatory[];

Regards,
eric
---

v9: 13jun2022
 - Rebased to 5.18.0
 - Per Sourabh, moved crash_prepare_elf64_headers() into common
   crash_core.c to avoid compile issues with kexec_load only path.
 - Per David Hildebrand, replaced mutex_trylock() with mutex_lock().
 - Changed the __weak arch_crash_handle_hotplug_event() to utilize
   WARN_ONCE() instead of WARN(). Fix some formatting issues.
 - Per Sourabh, introduced sysfs attribute crash_hotplug for memory
   and CPUs; for use by userspace (udev) to determine if the kernel
   performs crash hot un/plug support.
 - Per Sourabh, moved the code detecting the elfcorehdr segment from
   arch/x86 into crash_core:handle_hotplug_event() so both kexec_load
   and kexec_file_load can benefit.
 - Updated userspace kexec-tools kexec utility to reflect change to
   using CRASH_MAX_MEMORY_RANGES and get_nr_cpus().

v8: 5may2022
 https://lkml.org/lkml/2022/5/5/1133
 - Per Borislav Petkov, eliminated CONFIG_CRASH_HOTPLUG in favor
   of CONFIG_HOTPLUG_CPU || CONFIG_MEMORY_HOTPLUG, ie a new define
   is not needed. Also use of IS_ENABLED() rather than #ifdef's.
   Renamed crash_hotplug_handler() to handle_hotplug_event().
   And other corrections.
 - Per Baoquan, minimized the parameters to the arch_crash_
   handle_hotplug_event() to hp_action and cpu.
 - Introduce KEXEC_CRASH_HP_INVALID_CPU definition, per Baoquan.
 - Per Sourabh Jain, renamed and repurposed CRASH_HOTPLUG_ELFCOREHDR_SZ
   to CONFIG_CRASH_MAX_MEMORY_RANGES, mirroring kexec-tools change
   by David Hildebrand. Folded this patch into the x86
   kexec_file_load support patch.

v7: 13apr2022
 https://lkml.org/lkml/2022/4/13/850
 - Resolved parameter usage to crash_hotplug_handler(), per Baoquan.

v6: 1apr2022
 https://lkml.org/lkml/2022/4/1/1203
 - Reword commit messages and some comment cleanup per Baoquan.
 - Changed elf_index to elfcorehdr_index for clarity.
 - Minor code changes per Baoquan.

v5: 3mar2022
 https://lkml.org/lkml/2022/3/3/674
 - Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per
   David Hildenbrand.
 - Refactored slightly a few patches per Baoquan recommendation.

v4: 9feb2022
 https://lkml.org/lkml/2022/2/9/1406
 - Refactored patches per Baoquan suggestsions.
 - A few corrections, per Baoquan.

v3: 10jan2022
 https://lkml.org/lkml/2022/1/10/1212
 - Rebasing per Baoquan He request.
 - Changed memory notifier per David Hildenbrand.
 - Providing example kexec userspace change in cover letter.

RFC v2: 7dec2021
 https://lkml.org/lkml/2021/12/7/1088
 - Acting upon Baoquan He suggestion of removing elfcorehdr from
   the purgatory list of segments, removed purgatory code from
   patchset, and it is signficiantly simpler now.

RFC v1: 18nov2021
 https://lkml.org/lkml/2021/11/18/845
 - working patchset demonstrating kernel handling of hotplug
   updates to x86 elfcorehdr for kexec_file_load

RFC: 14dec2020
 https://lkml.org/lkml/2020/12/14/532
 - proposed concept of allowing kernel to handle hotplug update
   of elfcorehdr
---


Eric DeVolder (7):
  crash: move crash_prepare_elf64_headers
  crash: prototype change for crash_prepare_elf64_headers
  crash: add generic infrastructure for crash hotplug support
  kexec: exclude elfcorehdr from the segment digest
  kexec: exclude hot remove cpu from elfcorehdr notes
  crash: memory and cpu hotplug sysfs attributes
  x86/crash: Add x86 crash hotplug support

 .../admin-guide/mm/memory-hotplug.rst         |   8 +
 Documentation/core-api/cpu_hotplug.rst        |  18 ++
 arch/arm64/kernel/machine_kexec_file.c        |   6 +-
 arch/powerpc/kexec/file_load_64.c             |   2 +-
 arch/x86/Kconfig                              |  11 +
 arch/x86/kernel/crash.c                       | 119 ++++++++-
 drivers/base/cpu.c                            |  13 +
 drivers/base/memory.c                         |  13 +
 include/linux/crash_core.h                    |  12 +
 include/linux/kexec.h                         |  14 +-
 kernel/crash_core.c                           | 241 ++++++++++++++++++
 kernel/kexec_file.c                           | 105 +-------
 12 files changed, 456 insertions(+), 106 deletions(-)

-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* [PATCH v9 1/7] crash: move crash_prepare_elf64_headers
  2022-06-13 22:42 [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
@ 2022-06-13 22:42 ` Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 2/7] crash: prototype change for crash_prepare_elf64_headers Eric DeVolder
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 20+ messages in thread
From: Eric DeVolder @ 2022-06-13 22:42 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	eric.devolder

At the outcome of this patch set, the crash_prepare_elf64_headers()
is utilized on both the kexec_file_load and kexec_load paths. As
such, need to move this function out of kexec_file.c and into a
common location crash_core.c.

No functionality change.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kernel/crash_core.c | 100 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec_file.c |  99 -------------------------------------------
 2 files changed, 100 insertions(+), 99 deletions(-)

diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index 4d57c03714f4..85058fdce308 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -9,6 +9,7 @@
 #include <linux/init.h>
 #include <linux/utsname.h>
 #include <linux/vmalloc.h>
+#include <linux/kexec.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>
@@ -305,6 +306,105 @@ static int __init parse_crashkernel_dummy(char *arg)
 }
 early_param("crashkernel", parse_crashkernel_dummy);
 
+int crash_prepare_elf64_headers(struct crash_mem *mem, int kernel_map,
+			  void **addr, unsigned long *sz)
+{
+	Elf64_Ehdr *ehdr;
+	Elf64_Phdr *phdr;
+	unsigned long nr_cpus = num_possible_cpus(), nr_phdr, elf_sz;
+	unsigned char *buf;
+	unsigned int cpu, i;
+	unsigned long long notes_addr;
+	unsigned long mstart, mend;
+
+	/* extra phdr for vmcoreinfo ELF note */
+	nr_phdr = nr_cpus + 1;
+	nr_phdr += mem->nr_ranges;
+
+	/*
+	 * kexec-tools creates an extra PT_LOAD phdr for kernel text mapping
+	 * area (for example, ffffffff80000000 - ffffffffa0000000 on x86_64).
+	 * I think this is required by tools like gdb. So same physical
+	 * memory will be mapped in two ELF headers. One will contain kernel
+	 * text virtual addresses and other will have __va(physical) addresses.
+	 */
+
+	nr_phdr++;
+	elf_sz = sizeof(Elf64_Ehdr) + nr_phdr * sizeof(Elf64_Phdr);
+	elf_sz = ALIGN(elf_sz, ELF_CORE_HEADER_ALIGN);
+
+	buf = vzalloc(elf_sz);
+	if (!buf)
+		return -ENOMEM;
+
+	ehdr = (Elf64_Ehdr *)buf;
+	phdr = (Elf64_Phdr *)(ehdr + 1);
+	memcpy(ehdr->e_ident, ELFMAG, SELFMAG);
+	ehdr->e_ident[EI_CLASS] = ELFCLASS64;
+	ehdr->e_ident[EI_DATA] = ELFDATA2LSB;
+	ehdr->e_ident[EI_VERSION] = EV_CURRENT;
+	ehdr->e_ident[EI_OSABI] = ELF_OSABI;
+	memset(ehdr->e_ident + EI_PAD, 0, EI_NIDENT - EI_PAD);
+	ehdr->e_type = ET_CORE;
+	ehdr->e_machine = ELF_ARCH;
+	ehdr->e_version = EV_CURRENT;
+	ehdr->e_phoff = sizeof(Elf64_Ehdr);
+	ehdr->e_ehsize = sizeof(Elf64_Ehdr);
+	ehdr->e_phentsize = sizeof(Elf64_Phdr);
+
+	/* Prepare one phdr of type PT_NOTE for each present CPU */
+	for_each_present_cpu(cpu) {
+		phdr->p_type = PT_NOTE;
+		notes_addr = per_cpu_ptr_to_phys(per_cpu_ptr(crash_notes, cpu));
+		phdr->p_offset = phdr->p_paddr = notes_addr;
+		phdr->p_filesz = phdr->p_memsz = sizeof(note_buf_t);
+		(ehdr->e_phnum)++;
+		phdr++;
+	}
+
+	/* Prepare one PT_NOTE header for vmcoreinfo */
+	phdr->p_type = PT_NOTE;
+	phdr->p_offset = phdr->p_paddr = paddr_vmcoreinfo_note();
+	phdr->p_filesz = phdr->p_memsz = VMCOREINFO_NOTE_SIZE;
+	(ehdr->e_phnum)++;
+	phdr++;
+
+	/* Prepare PT_LOAD type program header for kernel text region */
+	if (kernel_map) {
+		phdr->p_type = PT_LOAD;
+		phdr->p_flags = PF_R|PF_W|PF_X;
+		phdr->p_vaddr = (unsigned long) _text;
+		phdr->p_filesz = phdr->p_memsz = _end - _text;
+		phdr->p_offset = phdr->p_paddr = __pa_symbol(_text);
+		ehdr->e_phnum++;
+		phdr++;
+	}
+
+	/* Go through all the ranges in mem->ranges[] and prepare phdr */
+	for (i = 0; i < mem->nr_ranges; i++) {
+		mstart = mem->ranges[i].start;
+		mend = mem->ranges[i].end;
+
+		phdr->p_type = PT_LOAD;
+		phdr->p_flags = PF_R|PF_W|PF_X;
+		phdr->p_offset  = mstart;
+
+		phdr->p_paddr = mstart;
+		phdr->p_vaddr = (unsigned long) __va(mstart);
+		phdr->p_filesz = phdr->p_memsz = mend - mstart + 1;
+		phdr->p_align = 0;
+		ehdr->e_phnum++;
+		pr_debug("Crash PT_LOAD ELF header. phdr=%p vaddr=0x%llx, paddr=0x%llx, sz=0x%llx e_phnum=%d p_offset=0x%llx\n",
+			phdr, phdr->p_vaddr, phdr->p_paddr, phdr->p_filesz,
+			ehdr->e_phnum, phdr->p_offset);
+		phdr++;
+	}
+
+	*addr = buf;
+	*sz = elf_sz;
+	return 0;
+}
+
 Elf_Word *append_elf_note(Elf_Word *buf, char *name, unsigned int type,
 			  void *data, size_t data_len)
 {
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index 8347fc158d2b..7adf883c091c 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -1259,102 +1259,3 @@ int crash_exclude_mem_range(struct crash_mem *mem,
 	mem->nr_ranges++;
 	return 0;
 }
-
-int crash_prepare_elf64_headers(struct crash_mem *mem, int kernel_map,
-			  void **addr, unsigned long *sz)
-{
-	Elf64_Ehdr *ehdr;
-	Elf64_Phdr *phdr;
-	unsigned long nr_cpus = num_possible_cpus(), nr_phdr, elf_sz;
-	unsigned char *buf;
-	unsigned int cpu, i;
-	unsigned long long notes_addr;
-	unsigned long mstart, mend;
-
-	/* extra phdr for vmcoreinfo ELF note */
-	nr_phdr = nr_cpus + 1;
-	nr_phdr += mem->nr_ranges;
-
-	/*
-	 * kexec-tools creates an extra PT_LOAD phdr for kernel text mapping
-	 * area (for example, ffffffff80000000 - ffffffffa0000000 on x86_64).
-	 * I think this is required by tools like gdb. So same physical
-	 * memory will be mapped in two ELF headers. One will contain kernel
-	 * text virtual addresses and other will have __va(physical) addresses.
-	 */
-
-	nr_phdr++;
-	elf_sz = sizeof(Elf64_Ehdr) + nr_phdr * sizeof(Elf64_Phdr);
-	elf_sz = ALIGN(elf_sz, ELF_CORE_HEADER_ALIGN);
-
-	buf = vzalloc(elf_sz);
-	if (!buf)
-		return -ENOMEM;
-
-	ehdr = (Elf64_Ehdr *)buf;
-	phdr = (Elf64_Phdr *)(ehdr + 1);
-	memcpy(ehdr->e_ident, ELFMAG, SELFMAG);
-	ehdr->e_ident[EI_CLASS] = ELFCLASS64;
-	ehdr->e_ident[EI_DATA] = ELFDATA2LSB;
-	ehdr->e_ident[EI_VERSION] = EV_CURRENT;
-	ehdr->e_ident[EI_OSABI] = ELF_OSABI;
-	memset(ehdr->e_ident + EI_PAD, 0, EI_NIDENT - EI_PAD);
-	ehdr->e_type = ET_CORE;
-	ehdr->e_machine = ELF_ARCH;
-	ehdr->e_version = EV_CURRENT;
-	ehdr->e_phoff = sizeof(Elf64_Ehdr);
-	ehdr->e_ehsize = sizeof(Elf64_Ehdr);
-	ehdr->e_phentsize = sizeof(Elf64_Phdr);
-
-	/* Prepare one phdr of type PT_NOTE for each present CPU */
-	for_each_present_cpu(cpu) {
-		phdr->p_type = PT_NOTE;
-		notes_addr = per_cpu_ptr_to_phys(per_cpu_ptr(crash_notes, cpu));
-		phdr->p_offset = phdr->p_paddr = notes_addr;
-		phdr->p_filesz = phdr->p_memsz = sizeof(note_buf_t);
-		(ehdr->e_phnum)++;
-		phdr++;
-	}
-
-	/* Prepare one PT_NOTE header for vmcoreinfo */
-	phdr->p_type = PT_NOTE;
-	phdr->p_offset = phdr->p_paddr = paddr_vmcoreinfo_note();
-	phdr->p_filesz = phdr->p_memsz = VMCOREINFO_NOTE_SIZE;
-	(ehdr->e_phnum)++;
-	phdr++;
-
-	/* Prepare PT_LOAD type program header for kernel text region */
-	if (kernel_map) {
-		phdr->p_type = PT_LOAD;
-		phdr->p_flags = PF_R|PF_W|PF_X;
-		phdr->p_vaddr = (unsigned long) _text;
-		phdr->p_filesz = phdr->p_memsz = _end - _text;
-		phdr->p_offset = phdr->p_paddr = __pa_symbol(_text);
-		ehdr->e_phnum++;
-		phdr++;
-	}
-
-	/* Go through all the ranges in mem->ranges[] and prepare phdr */
-	for (i = 0; i < mem->nr_ranges; i++) {
-		mstart = mem->ranges[i].start;
-		mend = mem->ranges[i].end;
-
-		phdr->p_type = PT_LOAD;
-		phdr->p_flags = PF_R|PF_W|PF_X;
-		phdr->p_offset  = mstart;
-
-		phdr->p_paddr = mstart;
-		phdr->p_vaddr = (unsigned long) __va(mstart);
-		phdr->p_filesz = phdr->p_memsz = mend - mstart + 1;
-		phdr->p_align = 0;
-		ehdr->e_phnum++;
-		pr_debug("Crash PT_LOAD ELF header. phdr=%p vaddr=0x%llx, paddr=0x%llx, sz=0x%llx e_phnum=%d p_offset=0x%llx\n",
-			phdr, phdr->p_vaddr, phdr->p_paddr, phdr->p_filesz,
-			ehdr->e_phnum, phdr->p_offset);
-		phdr++;
-	}
-
-	*addr = buf;
-	*sz = elf_sz;
-	return 0;
-}
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply related	[flat|nested] 20+ messages in thread
* [PATCH v9 2/7] crash: prototype change for crash_prepare_elf64_headers
  2022-06-13 22:42 [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 1/7] crash: move crash_prepare_elf64_headers Eric DeVolder
@ 2022-06-13 22:42 ` Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 3/7] crash: add generic infrastructure for crash hotplug support Eric DeVolder
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 20+ messages in thread
From: Eric DeVolder @ 2022-06-13 22:42 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	eric.devolder

From within crash_prepare_elf64_headers() there is a need to
reference the struct kimage hotplug members. As such, this
change passes the struct kimage as a parameter to the
crash_prepare_elf64_headers(). The hotplug members are added
in "crash: add generic infrastructure for crash hotplug support".

This is preparation for later patch, no functionality change.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: David Hildenbrand <david@redhat.com>
---
 arch/arm64/kernel/machine_kexec_file.c | 6 +++---
 arch/powerpc/kexec/file_load_64.c      | 2 +-
 arch/x86/kernel/crash.c                | 3 ++-
 include/linux/kexec.h                  | 5 +++--
 kernel/crash_core.c                    | 4 ++--
 5 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/kernel/machine_kexec_file.c b/arch/arm64/kernel/machine_kexec_file.c
index 889951291cc0..421d42c338c1 100644
--- a/arch/arm64/kernel/machine_kexec_file.c
+++ b/arch/arm64/kernel/machine_kexec_file.c
@@ -39,7 +39,7 @@ int arch_kimage_file_post_load_cleanup(struct kimage *image)
 	return kexec_image_post_load_cleanup_default(image);
 }
 
-static int prepare_elf_headers(void **addr, unsigned long *sz)
+static int prepare_elf_headers(struct kimage *image, void **addr, unsigned long *sz)
 {
 	struct crash_mem *cmem;
 	unsigned int nr_ranges;
@@ -74,7 +74,7 @@ static int prepare_elf_headers(void **addr, unsigned long *sz)
 			goto out;
 	}
 
-	ret = crash_prepare_elf64_headers(cmem, true, addr, sz);
+	ret = crash_prepare_elf64_headers(image, cmem, true, addr, sz);
 
 out:
 	kfree(cmem);
@@ -104,7 +104,7 @@ int load_other_segments(struct kimage *image,
 
 	/* load elf core header */
 	if (image->type == KEXEC_TYPE_CRASH) {
-		ret = prepare_elf_headers(&headers, &headers_sz);
+		ret = prepare_elf_headers(image, &headers, &headers_sz);
 		if (ret) {
 			pr_err("Preparing elf core header failed\n");
 			goto out_err;
diff --git a/arch/powerpc/kexec/file_load_64.c b/arch/powerpc/kexec/file_load_64.c
index b4981b651d9a..07da6bf1cf24 100644
--- a/arch/powerpc/kexec/file_load_64.c
+++ b/arch/powerpc/kexec/file_load_64.c
@@ -797,7 +797,7 @@ static int load_elfcorehdr_segment(struct kimage *image, struct kexec_buf *kbuf)
 		goto out;
 
 	/* Setup elfcorehdr segment */
-	ret = crash_prepare_elf64_headers(cmem, false, &headers, &headers_sz);
+	ret = crash_prepare_elf64_headers(image, cmem, false, &headers, &headers_sz);
 	if (ret) {
 		pr_err("Failed to prepare elf headers for the core\n");
 		goto out;
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 9730c88530fc..9db41cce8d97 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -265,7 +265,8 @@ static int prepare_elf_headers(struct kimage *image, void **addr,
 		goto out;
 
 	/* By default prepare 64bit headers */
-	ret =  crash_prepare_elf64_headers(cmem, IS_ENABLED(CONFIG_X86_64), addr, sz);
+	ret =  crash_prepare_elf64_headers(image, cmem,
+				IS_ENABLED(CONFIG_X86_64), addr, sz);
 
 out:
 	vfree(cmem);
diff --git a/include/linux/kexec.h b/include/linux/kexec.h
index 58d1b58a971e..f93f2591fc1e 100644
--- a/include/linux/kexec.h
+++ b/include/linux/kexec.h
@@ -227,8 +227,9 @@ struct crash_mem {
 extern int crash_exclude_mem_range(struct crash_mem *mem,
 				   unsigned long long mstart,
 				   unsigned long long mend);
-extern int crash_prepare_elf64_headers(struct crash_mem *mem, int kernel_map,
-				       void **addr, unsigned long *sz);
+extern int crash_prepare_elf64_headers(struct kimage *image,
+	struct crash_mem *mem, int kernel_map,
+	void **addr, unsigned long *sz);
 #endif /* CONFIG_KEXEC_FILE */
 
 #ifdef CONFIG_KEXEC_ELF
diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index 85058fdce308..59ad87a7e40d 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -306,8 +306,8 @@ static int __init parse_crashkernel_dummy(char *arg)
 }
 early_param("crashkernel", parse_crashkernel_dummy);
 
-int crash_prepare_elf64_headers(struct crash_mem *mem, int kernel_map,
-			  void **addr, unsigned long *sz)
+int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem,
+	int kernel_map, void **addr, unsigned long *sz)
 {
 	Elf64_Ehdr *ehdr;
 	Elf64_Phdr *phdr;
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply related	[flat|nested] 20+ messages in thread
* [PATCH v9 3/7] crash: add generic infrastructure for crash hotplug support
  2022-06-13 22:42 [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 1/7] crash: move crash_prepare_elf64_headers Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 2/7] crash: prototype change for crash_prepare_elf64_headers Eric DeVolder
@ 2022-06-13 22:42 ` Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 4/7] kexec: exclude elfcorehdr from the segment digest Eric DeVolder
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 20+ messages in thread
From: Eric DeVolder @ 2022-06-13 22:42 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	eric.devolder

CPU and memory change notifications are received in order to
regenerate the elfcorehdr.

To support cpu hotplug, a callback is registered to capture the
CPUHP_AP_ONLINE_DYN online and offline events via
cpuhp_setup_state_nocalls().

To support memory hotplug, a notifier is registered to capture the
MEM_ONLINE and MEM_OFFLINE events via register_memory_notifier().

The cpu callback and memory notifiers call handle_hotplug_event()
to handle the hot plug/unplug event. Then handle_hotplug_event()
dispatches the event to the architecture specific
arch_crash_handle_hotplug_event(). During the process, the
kexec_mutex is held.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 include/linux/crash_core.h |  12 ++++
 include/linux/kexec.h      |   7 +++
 kernel/crash_core.c        | 124 +++++++++++++++++++++++++++++++++++++
 3 files changed, 143 insertions(+)

diff --git a/include/linux/crash_core.h b/include/linux/crash_core.h
index de62a722431e..564534820db9 100644
--- a/include/linux/crash_core.h
+++ b/include/linux/crash_core.h
@@ -84,4 +84,16 @@ int parse_crashkernel_high(char *cmdline, unsigned long long system_ram,
 int parse_crashkernel_low(char *cmdline, unsigned long long system_ram,
 		unsigned long long *crash_size, unsigned long long *crash_base);
 
+#define KEXEC_CRASH_HP_REMOVE_CPU		0
+#define KEXEC_CRASH_HP_ADD_CPU			1
+#define KEXEC_CRASH_HP_REMOVE_MEMORY	2
+#define KEXEC_CRASH_HP_ADD_MEMORY		3
+#define KEXEC_CRASH_HP_INVALID_CPU		-1U
+
+struct kimage;
+void *arch_map_crash_pages(unsigned long paddr, unsigned long size);
+void arch_unmap_crash_pages(void **ptr);
+void arch_crash_handle_hotplug_event(struct kimage *image, unsigned int hp_action,
+									unsigned int cpu);
+
 #endif /* LINUX_CRASH_CORE_H */
diff --git a/include/linux/kexec.h b/include/linux/kexec.h
index f93f2591fc1e..f9b13882debf 100644
--- a/include/linux/kexec.h
+++ b/include/linux/kexec.h
@@ -308,6 +308,13 @@ struct kimage {
 	struct purgatory_info purgatory_info;
 #endif
 
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
+	bool hotplug_event;
+	unsigned int offlinecpu;
+	bool elfcorehdr_index_valid;
+	int elfcorehdr_index;
+#endif
+
 #ifdef CONFIG_IMA_KEXEC
 	/* Virtual address of IMA measurement buffer for kexec syscall */
 	void *ima_buffer;
diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index 59ad87a7e40d..84eb10a4241f 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -10,12 +10,16 @@
 #include <linux/utsname.h>
 #include <linux/vmalloc.h>
 #include <linux/kexec.h>
+#include <linux/memory.h>
+#include <linux/cpuhotplug.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>
 
 #include <crypto/sha1.h>
 
+#include "kexec_internal.h"
+
 /* vmcoreinfo stuff */
 unsigned char *vmcoreinfo_data;
 size_t vmcoreinfo_size;
@@ -590,3 +594,123 @@ static int __init crash_save_vmcoreinfo_init(void)
 }
 
 subsys_initcall(crash_save_vmcoreinfo_init);
+
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
+void __weak arch_crash_handle_hotplug_event(struct kimage *image,
+							unsigned int hp_action, unsigned int cpu)
+{
+	WARN_ONCE(1, "crash hotplug handler not implemented");
+}
+
+static void handle_hotplug_event(unsigned int hp_action, unsigned int cpu)
+{
+	/* Obtain lock while changing crash information */
+	mutex_lock(&kexec_mutex);
+
+	/* Check kdump is loaded */
+	if (kexec_crash_image) {
+		struct kimage *image = kexec_crash_image;
+
+		pr_debug("crash hp: hp_action %u, cpu %u", hp_action, cpu);
+
+		/*
+		 * When the struct kimage is alloced, it is wiped to zero, so
+		 * the elfcorehdr_index_valid defaults to false. Find the
+		 * segment containing the elfcorehdr, if not already found.
+		 * This works for both the kexec_load and kexec_file_load paths.
+		 */
+		if (!image->elfcorehdr_index_valid) {
+			unsigned char *ptr;
+			unsigned long mem, memsz;
+			unsigned int n;
+
+			for (n = 0; n < image->nr_segments; n++) {
+				mem = image->segment[n].mem;
+				memsz = image->segment[n].memsz;
+				ptr = arch_map_crash_pages(mem, memsz);
+				if (ptr) {
+					/* The segment containing elfcorehdr */
+					if (memcmp(ptr, ELFMAG, SELFMAG) == 0) {
+						image->elfcorehdr_index = (int)n;
+						image->elfcorehdr_index_valid = true;
+					}
+				}
+				arch_unmap_crash_pages((void **)&ptr);
+			}
+		}
+
+		if (!image->elfcorehdr_index_valid) {
+			pr_err("crash hp: unable to locate elfcorehdr segment");
+			goto out;
+		}
+
+		/* Needed in order for the segments to be updated */
+		arch_kexec_unprotect_crashkres();
+
+		/* Flag to differentiate between normal load and hotplug */
+		image->hotplug_event = true;
+
+		/* Now invoke arch-specific update handler */
+		arch_crash_handle_hotplug_event(image, hp_action, cpu);
+
+		/* No longer handling a hotplug event */
+		image->hotplug_event = false;
+
+		/* Change back to read-only */
+		arch_kexec_protect_crashkres();
+	}
+
+out:
+	/* Release lock now that update complete */
+	mutex_unlock(&kexec_mutex);
+}
+
+static int crash_memhp_notifier(struct notifier_block *nb, unsigned long val, void *v)
+{
+	switch (val) {
+	case MEM_ONLINE:
+		handle_hotplug_event(KEXEC_CRASH_HP_ADD_MEMORY, 0);
+		break;
+
+	case MEM_OFFLINE:
+		handle_hotplug_event(KEXEC_CRASH_HP_REMOVE_MEMORY, 0);
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block crash_memhp_nb = {
+	.notifier_call = crash_memhp_notifier,
+	.priority = 0
+};
+
+static int crash_cpuhp_online(unsigned int cpu)
+{
+	handle_hotplug_event(KEXEC_CRASH_HP_ADD_CPU, cpu);
+	return 0;
+}
+
+static int crash_cpuhp_offline(unsigned int cpu)
+{
+	handle_hotplug_event(KEXEC_CRASH_HP_REMOVE_CPU, cpu);
+	return 0;
+}
+
+static int __init crash_hotplug_init(void)
+{
+	int result = 0;
+
+	if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
+		register_memory_notifier(&crash_memhp_nb);
+
+	if (IS_ENABLED(CONFIG_HOTPLUG_CPU))
+		result = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+						   "crash/cpuhp",
+						   crash_cpuhp_online,
+						   crash_cpuhp_offline);
+
+	return result;
+}
+
+subsys_initcall(crash_hotplug_init);
+#endif
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply related	[flat|nested] 20+ messages in thread
* [PATCH v9 4/7] kexec: exclude elfcorehdr from the segment digest
  2022-06-13 22:42 [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
                   ` (2 preceding siblings ...)
  2022-06-13 22:42 ` [PATCH v9 3/7] crash: add generic infrastructure for crash hotplug support Eric DeVolder
@ 2022-06-13 22:42 ` Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 5/7] kexec: exclude hot remove cpu from elfcorehdr notes Eric DeVolder
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 20+ messages in thread
From: Eric DeVolder @ 2022-06-13 22:42 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	eric.devolder

When a crash kernel is loaded via the kexec_file_load syscall, the
kernel places the various segments (ie crash kernel, crash initrd,
boot_params, elfcorehdr, purgatory, etc) in memory. For those
architectures that utilize purgatory, a hash digest of the segments
is calculated for integrity checking. This digest is embedded into
the purgatory image prior to placing purgatory in memory.

Since hotplug events cause changes to the elfcorehdr, purgatory
integrity checking fails (at crash time, and no kdump created).
As a result, this change explicitly excludes the elfcorehdr segment
from the list of segments used to create the digest. By doing so,
this permits changes to the elfcorehdr in response to hotplug events,
without having to also reload purgatory due to the change to the
digest.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
Acked-by: Baoquan He <bhe@redhat.com>
---
 kernel/kexec_file.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index 7adf883c091c..ae1d3295dfdd 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -765,6 +765,12 @@ static int kexec_calculate_store_digests(struct kimage *image)
 	for (j = i = 0; i < image->nr_segments; i++) {
 		struct kexec_segment *ksegment;
 
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
+		/* This segment excluded to allow future changes via hotplug */
+		if (image->elfcorehdr_index_valid && (j == image->elfcorehdr_index))
+			continue;
+#endif
+
 		ksegment = &image->segment[i];
 		/*
 		 * Skip purgatory as it will be modified once we put digest
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply related	[flat|nested] 20+ messages in thread
* [PATCH v9 5/7] kexec: exclude hot remove cpu from elfcorehdr notes
  2022-06-13 22:42 [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
                   ` (3 preceding siblings ...)
  2022-06-13 22:42 ` [PATCH v9 4/7] kexec: exclude elfcorehdr from the segment digest Eric DeVolder
@ 2022-06-13 22:42 ` Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 6/7] crash: memory and cpu hotplug sysfs attributes Eric DeVolder
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 20+ messages in thread
From: Eric DeVolder @ 2022-06-13 22:42 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	eric.devolder

Due to use of CPUHP_AP_ONLINE_DYN, upon CPU unplug, the CPU is
still in the for_each_present_cpu() list when within the
handle_hotplug_event(). Thus the CPU must be explicitly excluded
when building the new list of CPUs.

This change identifies in handle_hotplug_event() the CPU to be
excluded, and the check for excluding the CPU in
crash_prepare_elf64_headers().

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
Acked-by: Baoquan He <bhe@redhat.com>
---
 kernel/crash_core.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index 84eb10a4241f..47f667df1c0b 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -358,6 +358,11 @@ int crash_prepare_elf64_headers(struct kimage *image, struct crash_mem *mem,
 
 	/* Prepare one phdr of type PT_NOTE for each present CPU */
 	for_each_present_cpu(cpu) {
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
+		/* Skip the soon-to-be offlined cpu */
+		if (image->hotplug_event && (cpu == image->offlinecpu))
+			continue;
+#endif
 		phdr->p_type = PT_NOTE;
 		notes_addr = per_cpu_ptr_to_phys(per_cpu_ptr(crash_notes, cpu));
 		phdr->p_offset = phdr->p_paddr = notes_addr;
@@ -650,6 +655,16 @@ static void handle_hotplug_event(unsigned int hp_action, unsigned int cpu)
 		/* Flag to differentiate between normal load and hotplug */
 		image->hotplug_event = true;
 
+		/*
+		 * Due to use of CPUHP_AP_ONLINE_DYN, upon unplug and during
+		 * this callback, the CPU is still in the for_each_present_cpu()
+		 * list. Must explicitly look to exclude this CPU when building
+		 * new list.
+		 */
+		image->offlinecpu =
+			(hp_action == KEXEC_CRASH_HP_REMOVE_CPU) ?
+				cpu : KEXEC_CRASH_HP_INVALID_CPU;
+
 		/* Now invoke arch-specific update handler */
 		arch_crash_handle_hotplug_event(image, hp_action, cpu);
 
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply related	[flat|nested] 20+ messages in thread
* [PATCH v9 6/7] crash: memory and cpu hotplug sysfs attributes
  2022-06-13 22:42 [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
                   ` (4 preceding siblings ...)
  2022-06-13 22:42 ` [PATCH v9 5/7] kexec: exclude hot remove cpu from elfcorehdr notes Eric DeVolder
@ 2022-06-13 22:42 ` Eric DeVolder
  2022-06-13 22:42 ` [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support Eric DeVolder
  2022-07-05 15:17 ` [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
  7 siblings, 0 replies; 20+ messages in thread
From: Eric DeVolder @ 2022-06-13 22:42 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	eric.devolder

This introduces the crash_hotplug attribute for memory and CPUs
for use by userspace.  This change directly facilitates the udev
rule for managing userspace re-loading of the crash kernel upon
hot un/plug changes.

For memory, this changeset introduces the crash_hotplug attribute
to the /sys/devices/system/memory directory. For example:

 # udevadm info --attribute-walk /sys/devices/system/memory/memory81
  looking at device '/devices/system/memory/memory81':
    KERNEL=="memory81"
    SUBSYSTEM=="memory"
    DRIVER==""
    ATTR{online}=="1"
    ATTR{phys_device}=="0"
    ATTR{phys_index}=="00000051"
    ATTR{removable}=="1"
    ATTR{state}=="online"
    ATTR{valid_zones}=="Movable"

  looking at parent device '/devices/system/memory':
    KERNELS=="memory"
    SUBSYSTEMS==""
    DRIVERS==""
    ATTRS{auto_online_blocks}=="offline"
    ATTRS{block_size_bytes}=="8000000"
    ATTRS{crash_hotplug}=="1"

For CPUs, this changeset introduces the crash_hotplug attribute
to the /sys/devices/system/cpu directory. For example:

 # udevadm info --attribute-walk /sys/devices/system/cpu/cpu0
  looking at device '/devices/system/cpu/cpu0':
    KERNEL=="cpu0"
    SUBSYSTEM=="cpu"
    DRIVER=="processor"
    ATTR{crash_notes}=="277c38600"
    ATTR{crash_notes_size}=="368"
    ATTR{online}=="1"

  looking at parent device '/devices/system/cpu':
    KERNELS=="cpu"
    SUBSYSTEMS==""
    DRIVERS==""
    ATTRS{crash_hotplug}=="1"
    ATTRS{isolated}==""
    ATTRS{kernel_max}=="8191"
    ATTRS{nohz_full}=="  (null)"
    ATTRS{offline}=="4-7"
    ATTRS{online}=="0-3"
    ATTRS{possible}=="0-7"
    ATTRS{present}=="0-3"

With these sysfs attributes in place, it is possible to efficiently
instruct the udev rule to skip crash kernel reloading.

For example, the following is the proposed udev rule change for RHEL
system 98-kexec.rules (as the first lines of the rule file):

 # The kernel handles updates to crash elfcorehdr for cpu and memory changes
 SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
 SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"

When examined in the context of 98-kexec.rules, the above change
tests if crash_hotplug is set, and if so, it skips the userspace
initiated unload-then-reload of the crash kernel.

Cpu and memory checks are separated in accordance with
CONFIG_HOTPLUG_CPU and CONFIG_MEMORY_HOTPLUG kernel config options.
If an architecture supports, for example, memory hotplug but not
CPU hotplug, then the /sys/devices/system/memory/crash_hotplug
attribute file is present, but the /sys/devices/system/cpu/crash_hotplug
attribute file will NOT be present. Thus the udev rule will skip
userspace processing of memory hot un/plug events, but the udev
rule will fail for CPU events, thus allowing userspace to process
cpu hot un/plug events (ie the unload-then-reload of the kdump
capture kernel).

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 .../admin-guide/mm/memory-hotplug.rst          |  8 ++++++++
 Documentation/core-api/cpu_hotplug.rst         | 18 ++++++++++++++++++
 drivers/base/cpu.c                             | 13 +++++++++++++
 drivers/base/memory.c                          | 13 +++++++++++++
 include/linux/kexec.h                          |  2 ++
 kernel/crash_core.c                            |  2 ++
 6 files changed, 56 insertions(+)

diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
index 0f56ecd8ac05..494d7a63c543 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -293,6 +293,14 @@ The following files are currently defined:
 		       Availability depends on the CONFIG_ARCH_MEMORY_PROBE
 		       kernel configuration option.
 ``uevent``	       read-write: generic udev file for device subsystems.
+``crash_hotplug``      read-only: when changes to the system memory map
+		       occur due to hot un/plug of memory, this file contains
+		       '1' if the kernel updates the kdump capture kernel memory
+		       map itself (via elfcorehdr), or '0' if userspace must update
+		       the kdump capture kernel memory map.
+
+		       Availability depends on the CONFIG_MEMORY_HOTPLUG kernel
+		       configuration option.
 ====================== =========================================================
 
 .. note::
diff --git a/Documentation/core-api/cpu_hotplug.rst b/Documentation/core-api/cpu_hotplug.rst
index c6f4ba2fb32d..13e33d098645 100644
--- a/Documentation/core-api/cpu_hotplug.rst
+++ b/Documentation/core-api/cpu_hotplug.rst
@@ -750,6 +750,24 @@ will receive all events. A script like::
 
 can process the event further.
 
+When changes to the CPUs in the system occur, the sysfs file
+/sys/devices/system/cpu/crash_hotplug contains '1' if the kernel
+updates the kdump capture kernel list of CPUs itself (via elfcorehdr),
+or '0' if userspace must update the kdump capture kernel list of CPUs.
+
+The availability depends on the CONFIG_HOTPLUG_CPU kernel configuration
+option.
+
+To skip userspace processing of CPU hot un/plug events for kdump
+(ie the unload-then-reload to obtain a current list of CPUs), this sysfs
+file can be used in a udev rule as follows:
+
+ SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
+
+For a cpu hot un/plug event, if the architecture supports kernel updates
+of the elfcorehdr (which contains the list of CPUs), then the rule skips
+the unload-then-reload of the kdump capture kernel.
+
 Kernel Inline Documentations Reference
 ======================================
 
diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index 2ef23fce0860..2afea3d2c60c 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -293,6 +293,16 @@ static ssize_t print_cpus_nohz_full(struct device *dev,
 static DEVICE_ATTR(nohz_full, 0444, print_cpus_nohz_full, NULL);
 #endif
 
+#ifdef CONFIG_HOTPLUG_CPU
+static ssize_t crash_hotplug_show(struct device *dev,
+				     struct device_attribute *attr,
+				     char *buf)
+{
+	return sprintf(buf, "%d\n", crash_hotplug_support());
+}
+static DEVICE_ATTR_ADMIN_RO(crash_hotplug);
+#endif
+
 static void cpu_device_release(struct device *dev)
 {
 	/*
@@ -469,6 +479,9 @@ static struct attribute *cpu_root_attrs[] = {
 #ifdef CONFIG_NO_HZ_FULL
 	&dev_attr_nohz_full.attr,
 #endif
+#ifdef CONFIG_HOTPLUG_CPU
+	&dev_attr_crash_hotplug.attr,
+#endif
 #ifdef CONFIG_GENERIC_CPU_AUTOPROBE
 	&dev_attr_modalias.attr,
 #endif
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 7222ff9b5e05..12fb9106ab4f 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -483,6 +483,16 @@ static ssize_t auto_online_blocks_store(struct device *dev,
 
 static DEVICE_ATTR_RW(auto_online_blocks);
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+#include <linux/kexec.h>
+static ssize_t crash_hotplug_show(struct device *dev,
+				       struct device_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%d\n", crash_hotplug_support());
+}
+static DEVICE_ATTR_RO(crash_hotplug);
+#endif
+
 /*
  * Some architectures will have custom drivers to do this, and
  * will not need to do it from userspace.  The fake hot-add code
@@ -888,6 +898,9 @@ static struct attribute *memory_root_attrs[] = {
 
 	&dev_attr_block_size_bytes.attr,
 	&dev_attr_auto_online_blocks.attr,
+#ifdef CONFIG_MEMORY_HOTPLUG
+	&dev_attr_crash_hotplug.attr,
+#endif
 	NULL
 };
 
diff --git a/include/linux/kexec.h b/include/linux/kexec.h
index f9b13882debf..9aca8bc26c13 100644
--- a/include/linux/kexec.h
+++ b/include/linux/kexec.h
@@ -342,6 +342,7 @@ extern void __crash_kexec(struct pt_regs *);
 extern void crash_kexec(struct pt_regs *);
 int kexec_should_crash(struct task_struct *);
 int kexec_crash_loaded(void);
+int crash_hotplug_support(void);
 void crash_save_cpu(struct pt_regs *regs, int cpu);
 extern int kimage_crash_copy_vmcoreinfo(struct kimage *image);
 
@@ -427,6 +428,7 @@ static inline void __crash_kexec(struct pt_regs *regs) { }
 static inline void crash_kexec(struct pt_regs *regs) { }
 static inline int kexec_should_crash(struct task_struct *p) { return 0; }
 static inline int kexec_crash_loaded(void) { return 0; }
+static inline int crash_hotplug_support(void) { return 0; }
 #define kexec_in_progress false
 #endif /* CONFIG_KEXEC_CORE */
 
diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index 47f667df1c0b..b003857ea85d 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -601,6 +601,8 @@ static int __init crash_save_vmcoreinfo_init(void)
 subsys_initcall(crash_save_vmcoreinfo_init);
 
 #if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
+int __weak crash_hotplug_support(void) { return 0; }
+
 void __weak arch_crash_handle_hotplug_event(struct kimage *image,
 							unsigned int hp_action, unsigned int cpu)
 {
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply related	[flat|nested] 20+ messages in thread
* [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support
  2022-06-13 22:42 [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
                   ` (5 preceding siblings ...)
  2022-06-13 22:42 ` [PATCH v9 6/7] crash: memory and cpu hotplug sysfs attributes Eric DeVolder
@ 2022-06-13 22:42 ` Eric DeVolder
  2022-06-20  5:06   ` Sourabh Jain
  2022-07-08 19:38   ` Fabio M. De Francesco
  2022-07-05 15:17 ` [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
  7 siblings, 2 replies; 20+ messages in thread
From: Eric DeVolder @ 2022-06-13 22:42 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	eric.devolder

For x86_64, when CPU or memory is hot un/plugged, the crash
elfcorehdr, which describes the CPUs and memory in the system,
must also be updated.

When loading the crash kernel via kexec_load or kexec_file_load,
the elfcorehdr is identified at run time in
crash_core:handle_hotplug_event().

To update the elfcorehdr for x86_64, a new elfcorehdr must be
generated from the available CPUs and memory. The new elfcorehdr
is prepared into a buffer, and then installed over the top of
the existing elfcorehdr.

In the patch 'kexec: exclude elfcorehdr from the segment digest'
the need to update purgatory due to the change in elfcorehdr was
eliminated.  As a result, no changes to purgatory or boot_params
(as the elfcorehdr= kernel command line parameter pointer
remains unchanged and correct) are needed, just elfcorehdr.

To accommodate a growing number of resources via hotplug, the
elfcorehdr segment must be sufficiently large enough to accommodate
changes, see the CRASH_MAX_MEMORY_RANGES configure item.

With this change, crash hotplug for kexec_file_load syscall
is supported. The kexec_load is also supported, but also
requires a corresponding change to userspace kexec-tools.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 arch/x86/Kconfig        |  11 ++++
 arch/x86/kernel/crash.c | 116 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 127 insertions(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 762a0b6ab8b6..e9eecff3b97d 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -2082,6 +2082,17 @@ config CRASH_DUMP
 	  (CONFIG_RELOCATABLE=y).
 	  For more details see Documentation/admin-guide/kdump/kdump.rst
 
+config CRASH_MAX_MEMORY_RANGES
+	depends on CRASH_DUMP && KEXEC_FILE && (HOTPLUG_CPU || MEMORY_HOTPLUG)
+	int
+	default 32768
+	help
+	  For the kexec_file_load path, specify the maximum number of
+	  memory regions, eg. as represented by the 'System RAM' entries
+	  in /proc/iomem, that the elfcorehdr buffer/segment can accommodate.
+	  This value is combined with NR_CPUS and multiplied by Elf64_Phdr
+	  size to determine the final buffer size.
+
 config KEXEC_JUMP
 	bool "kexec jump"
 	depends on KEXEC && HIBERNATION
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 9db41cce8d97..b9cdf7a0d868 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -25,6 +25,7 @@
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 #include <linux/memblock.h>
+#include <linux/highmem.h>
 
 #include <asm/processor.h>
 #include <asm/hardirq.h>
@@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
 	image->elf_headers = kbuf.buffer;
 	image->elf_headers_sz = kbuf.bufsz;
 
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
+	/* Ensure elfcorehdr segment large enough for hotplug changes */
+	kbuf.memsz = (CONFIG_NR_CPUS_DEFAULT + CONFIG_CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
+	/* For marking as usable to crash kernel */
+	image->elf_headers_sz = kbuf.memsz;
+	/* Record the index of the elfcorehdr segment */
+	image->elfcorehdr_index = image->nr_segments;
+	image->elfcorehdr_index_valid = true;
+#else
 	kbuf.memsz = kbuf.bufsz;
+#endif
 	kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
 	kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
 	ret = kexec_add_buffer(&kbuf);
@@ -413,3 +424,108 @@ int crash_load_segments(struct kimage *image)
 	return ret;
 }
 #endif /* CONFIG_KEXEC_FILE */
+
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
+int crash_hotplug_support(void) { return 1; }
+void *arch_map_crash_pages(unsigned long paddr, unsigned long size)
+{
+	/*
+	 * NOTE: The addresses and sizes passed to this routine have
+	 * already been fully aligned on page boundaries. There is no
+	 * need for massaging the address or size.
+	 */
+	void *ptr = NULL;
+
+	/* NOTE: requires arch_kexec_[un]protect_crashkres() for write access */
+	if (size > 0) {
+		struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
+
+		ptr = kmap(page);
+	}
+
+	return ptr;
+}
+
+void arch_unmap_crash_pages(void **ptr)
+{
+	if (ptr) {
+		if (*ptr)
+			kunmap(*ptr);
+		*ptr = NULL;
+	}
+}
+
+/**
+ * arch_crash_handle_hotplug_event() - Handle hotplug elfcorehdr changes
+ * @image: the active struct kimage
+ * @hp_action: the hot un/plug action being handled
+ * @cpu: when KEXEC_CRASH_HP_ADD/REMOVE_CPU, the cpu affected
+ *
+ * To accurately reflect hot un/plug changes, the elfcorehdr (which
+ * is passed to the crash kernel via the elfcorehdr= parameter)
+ * must be updated with the new list of CPUs and memories. The new
+ * elfcorehdr is prepared in a kernel buffer, and then it is
+ * written on top of the existing/old elfcorehdr.
+ *
+ * For hotplug changes to elfcorehdr to work, two conditions are
+ * needed:
+ * First, the segment containing the elfcorehdr must be large enough
+ * to permit a growing number of resources. See the
+ * CONFIG_CRASH_MAX_MEMORY_RANGES description.
+ * Second, purgatory must explicitly exclude the elfcorehdr from the
+ * list of segments it checks (since the elfcorehdr changes and thus
+ * would require an update to purgatory itself to update the digest).
+ *
+ */
+void arch_crash_handle_hotplug_event(struct kimage *image,
+	unsigned int hp_action, unsigned int cpu)
+{
+	struct kexec_segment *ksegment;
+	unsigned char *ptr = NULL;
+	unsigned long elfsz = 0;
+	void *elfbuf = NULL;
+	unsigned long mem, memsz;
+
+	/*
+	 * Elfcorehdr_index_valid checked in crash_core:handle_hotplug_event()
+	 */
+	ksegment = &image->segment[image->elfcorehdr_index];
+	mem = ksegment->mem;
+	memsz = ksegment->memsz;
+
+	/*
+	 * Create the new elfcorehdr reflecting the changes to CPU and/or
+	 * memory resources.
+	 */
+	if (prepare_elf_headers(image, &elfbuf, &elfsz)) {
+		pr_err("crash hp: unable to prepare elfcore headers");
+		goto out;
+	}
+	if (elfsz > memsz) {
+		pr_err("crash hp: update elfcorehdr elfsz %lu > memsz %lu",
+			elfsz, memsz);
+		goto out;
+	}
+
+	/*
+	 * At this point, we are all but assured of success.
+	 * Copy new elfcorehdr into destination.
+	 */
+	ptr = arch_map_crash_pages(mem, memsz);
+	if (ptr) {
+		/*
+		 * Temporarily invalidate the crash image while the
+		 * elfcorehdr is updated.
+		 */
+		xchg(&kexec_crash_image, NULL);
+		memcpy_flushcache((void *)ptr, elfbuf, elfsz);
+		xchg(&kexec_crash_image, image);
+	}
+	arch_unmap_crash_pages((void **)&ptr);
+	pr_debug("crash hp: re-loaded elfcorehdr at 0x%lx\n", mem);
+
+out:
+	if (elfbuf)
+		vfree(elfbuf);
+}
+#endif
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply related	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support
  2022-06-13 22:42 ` [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support Eric DeVolder
@ 2022-06-20  5:06   ` Sourabh Jain
  2022-07-07 12:55     ` Eric DeVolder
  2022-07-08 19:38   ` Fabio M. De Francesco
  1 sibling, 1 reply; 20+ messages in thread
From: Sourabh Jain @ 2022-06-20  5:06 UTC (permalink / raw)
  To: Eric DeVolder, linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, konrad.wilk, boris.ostrovsky

Hello Eric,

On 14/06/22 04:12, Eric DeVolder wrote:
> For x86_64, when CPU or memory is hot un/plugged, the crash
> elfcorehdr, which describes the CPUs and memory in the system,
> must also be updated.
>
> When loading the crash kernel via kexec_load or kexec_file_load,
> the elfcorehdr is identified at run time in
> crash_core:handle_hotplug_event().
>
> To update the elfcorehdr for x86_64, a new elfcorehdr must be
> generated from the available CPUs and memory. The new elfcorehdr
> is prepared into a buffer, and then installed over the top of
> the existing elfcorehdr.
>
> In the patch 'kexec: exclude elfcorehdr from the segment digest'
> the need to update purgatory due to the change in elfcorehdr was
> eliminated.  As a result, no changes to purgatory or boot_params
> (as the elfcorehdr= kernel command line parameter pointer
> remains unchanged and correct) are needed, just elfcorehdr.
>
> To accommodate a growing number of resources via hotplug, the
> elfcorehdr segment must be sufficiently large enough to accommodate
> changes, see the CRASH_MAX_MEMORY_RANGES configure item.
>
> With this change, crash hotplug for kexec_file_load syscall
> is supported. The kexec_load is also supported, but also
> requires a corresponding change to userspace kexec-tools.
>
> Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
> ---
>   arch/x86/Kconfig        |  11 ++++
>   arch/x86/kernel/crash.c | 116 ++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 127 insertions(+)
>
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 762a0b6ab8b6..e9eecff3b97d 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -2082,6 +2082,17 @@ config CRASH_DUMP
>   	  (CONFIG_RELOCATABLE=y).
>   	  For more details see Documentation/admin-guide/kdump/kdump.rst
>   
> +config CRASH_MAX_MEMORY_RANGES
> +	depends on CRASH_DUMP && KEXEC_FILE && (HOTPLUG_CPU || MEMORY_HOTPLUG)
> +	int
> +	default 32768
> +	help
> +	  For the kexec_file_load path, specify the maximum number of
> +	  memory regions, eg. as represented by the 'System RAM' entries
> +	  in /proc/iomem, that the elfcorehdr buffer/segment can accommodate.
> +	  This value is combined with NR_CPUS and multiplied by Elf64_Phdr
> +	  size to determine the final buffer size.
> +
>   config KEXEC_JUMP
>   	bool "kexec jump"
>   	depends on KEXEC && HIBERNATION
> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> index 9db41cce8d97..b9cdf7a0d868 100644
> --- a/arch/x86/kernel/crash.c
> +++ b/arch/x86/kernel/crash.c
> @@ -25,6 +25,7 @@
>   #include <linux/slab.h>
>   #include <linux/vmalloc.h>
>   #include <linux/memblock.h>
> +#include <linux/highmem.h>
>   
>   #include <asm/processor.h>
>   #include <asm/hardirq.h>
> @@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
>   	image->elf_headers = kbuf.buffer;
>   	image->elf_headers_sz = kbuf.bufsz;
>   
> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
> +	/* Ensure elfcorehdr segment large enough for hotplug changes */
> +	kbuf.memsz = (CONFIG_NR_CPUS_DEFAULT + CONFIG_CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
> +	/* For marking as usable to crash kernel */
> +	image->elf_headers_sz = kbuf.memsz;
> +	/* Record the index of the elfcorehdr segment */
> +	image->elfcorehdr_index = image->nr_segments;
> +	image->elfcorehdr_index_valid = true;
> +#else
>   	kbuf.memsz = kbuf.bufsz;
> +#endif
>   	kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
>   	kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
>   	ret = kexec_add_buffer(&kbuf);
> @@ -413,3 +424,108 @@ int crash_load_segments(struct kimage *image)
>   	return ret;
>   }
>   #endif /* CONFIG_KEXEC_FILE */
> +
> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
> +int crash_hotplug_support(void) { return 1; }
> +void *arch_map_crash_pages(unsigned long paddr, unsigned long size)
> +{
> +	/*
> +	 * NOTE: The addresses and sizes passed to this routine have
> +	 * already been fully aligned on page boundaries. There is no
> +	 * need for massaging the address or size.
> +	 */
> +	void *ptr = NULL;
> +
> +	/* NOTE: requires arch_kexec_[un]protect_crashkres() for write access */
> +	if (size > 0) {
> +		struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
> +
> +		ptr = kmap(page);
> +	}
> +
> +	return ptr;
> +}
> +
> +void arch_unmap_crash_pages(void **ptr)
> +{
> +	if (ptr) {
> +		if (*ptr)
> +			kunmap(*ptr);
> +		*ptr = NULL;
> +	}
> +}

Aren't arch will have build issue if arch_[un]map_crash_pages methods 
are not defined?

Thanks,
Sourabh Jain


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug
  2022-06-13 22:42 [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
                   ` (6 preceding siblings ...)
  2022-06-13 22:42 ` [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support Eric DeVolder
@ 2022-07-05 15:17 ` Eric DeVolder
  2022-07-06  1:16   ` Baoquan He
  7 siblings, 1 reply; 20+ messages in thread
From: Eric DeVolder @ 2022-07-05 15:17 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky

Nudge...
Thanks!
eric


On 6/13/22 17:42, Eric DeVolder wrote:
> When the kdump service is loaded, if a CPU or memory is hot
> un/plugged, the crash elfcorehdr (for x86), which describes the CPUs
> and memory in the system, must also be updated, else the resulting
> vmcore is inaccurate (eg. missing either CPU context or memory
> regions).
> 
> The current solution utilizes udev to initiate an unload-then-reload
> of the kdump image (e. kernel, initrd, boot_params, puratory and
> elfcorehdr) by the userspace kexec utility. In previous posts I have
> outlined the significant performance problems related to offloading
> this activity to userspace.
> 
> This patchset introduces a generic crash hot un/plug handler that
> registers with the CPU and memory notifiers. Upon CPU or memory
> changes, this generic handler is invoked and performs important
> housekeeping, for example obtaining the appropriate lock, and then
> invokes an architecture specific handler to do the appropriate
> updates.
> 
> In the case of x86_64, the arch specific handler generates a new
> elfcorehdr, and overwrites the old one in memory. No involvement
> with userspace needed.
> 
> To realize the benefits/test this patchset, one must make a couple
> of minor changes to userspace:
> 
>   - Prevent udev from updating kdump crash kernel on hot un/plug changes.
>     Add the following as the first lines to the udev rule file
>     /usr/lib/udev/rules.d/98-kexec.rules:
> 
>     # The kernel handles updates to crash elfcorehdr for cpu and memory changes
>     SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
>     SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
> 
>     These lines will cause cpu and memory hot un/plug events to be
>     skipped within this rule file, if the kernel has these changes
>     enabled.
> 
>   - Change to the kexec_file_load for loading the kdump kernel:
>     Eg. on RHEL: in /usr/bin/kdumpctl, change to:
>      standard_kexec_args="-p -d -s"
>     which adds the -s to select kexec_file_load syscall.
> 
> This patchset supports kexec_load with a modified kexec userspace
> utility, and a working changeset to the kexec userspace utility
> is provided here (and to use, the above change to standard_kexec_args
> would be, for example, to append --hotplug instead of -s).
> 
>    diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
>    index 9826f6d..4ed395a 100644
>    --- a/kexec/arch/i386/crashdump-x86.c
>    +++ b/kexec/arch/i386/crashdump-x86.c
>    @@ -48,6 +48,7 @@
>     #include <x86/x86-linux.h>
>     
>     extern struct arch_options_t arch_options;
>    +extern int do_hotplug;
>     
>     static int get_kernel_page_offset(struct kexec_info *UNUSED(info),
>     				  struct crash_elf_info *elf_info)
>    @@ -975,6 +976,14 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
>     	} else {
>     		memsz = bufsz;
>     	}
>    +
>    +	/* If hotplug support enabled, use larger size to accomodate changes */
>    +	if (do_hotplug) {
>    +		long int nr_cpus = get_nr_cpus();
>    +		memsz = (nr_cpus + CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
>    +	}
>    +
>    +    info->elfcorehdr =
>     	elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
>     							max_addr, -1);
>     	dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
>    diff --git a/kexec/crashdump-elf.c b/kexec/crashdump-elf.c
>    index b8bb686..5e29f7a 100644
>    --- a/kexec/crashdump-elf.c
>    +++ b/kexec/crashdump-elf.c
>    @@ -43,11 +43,7 @@ int FUNC(struct kexec_info *info,
>     	int (*get_note_info)(int cpu, uint64_t *addr, uint64_t *len);
>     	long int count_cpu;
>     
>    -	if (xen_present())
>    -		nr_cpus = xen_get_nr_phys_cpus();
>    -	else
>    -		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
>    -
>    +	nr_cpus = get_nr_cpus();
>     	if (nr_cpus < 0) {
>     		return -1;
>     	}
>    diff --git a/kexec/crashdump.h b/kexec/crashdump.h
>    index 18bd691..28d3278 100644
>    --- a/kexec/crashdump.h
>    +++ b/kexec/crashdump.h
>    @@ -57,7 +57,6 @@ unsigned long phys_to_virt(struct crash_elf_info *elf_info,
>     			   unsigned long long paddr);
>     
>     unsigned long xen_architecture(struct crash_elf_info *elf_info);
>    -int xen_get_nr_phys_cpus(void);
>     int xen_get_note(int cpu, uint64_t *addr, uint64_t *len);
>     int xen_get_crashkernel_region(uint64_t *start, uint64_t *end);
>     
>    diff --git a/kexec/kexec-xen.h b/kexec/kexec-xen.h
>    index 70fb576..f54a2dd 100644
>    --- a/kexec/kexec-xen.h
>    +++ b/kexec/kexec-xen.h
>    @@ -83,5 +83,6 @@ extern int __xc_interface_close(xc_interface *xch);
>     #endif
>     
>     int xen_get_kexec_range(int range, uint64_t *start, uint64_t *end);
>    +int xen_get_nr_phys_cpus(void);
>     
>     #endif /* KEXEC_XEN_H */
>    diff --git a/kexec/kexec.c b/kexec/kexec.c
>    index 829a6ea..3668b73 100644
>    --- a/kexec/kexec.c
>    +++ b/kexec/kexec.c
>    @@ -58,6 +58,7 @@
>     
>     unsigned long long mem_min = 0;
>     unsigned long long mem_max = ULONG_MAX;
>    +int do_hotplug = 0;
>     static unsigned long kexec_flags = 0;
>     /* Flags for kexec file (fd) based syscall */
>     static unsigned long kexec_file_flags = 0;
>    @@ -489,6 +490,17 @@ static int add_backup_segments(struct kexec_info *info,
>     	return 0;
>     }
>     
>    +long int get_nr_cpus(void)
>    +{
>    +    long int nr_cpus;
>    +
>    +	if (xen_present())
>    +		nr_cpus = xen_get_nr_phys_cpus();
>    +	else
>    +		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
>    +    return nr_cpus;
>    +}
>    +
>     static char *slurp_fd(int fd, const char *filename, off_t size, off_t *nread)
>     {
>     	char *buf;
>    @@ -672,6 +684,14 @@ static void update_purgatory(struct kexec_info *info)
>     		if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
>     			continue;
>     		}
>    +
>    +		/* Don't include elfcorehdr in the checksum, if hotplug
>    +		 * support enabled.
>    +		 */
>    +		if (do_hotplug && (info->segment[i].mem == (void *)info->elfcorehdr)) {
>    +			continue;
>    +		}
>    +
>     		sha256_update(&ctx, info->segment[i].buf,
>     			      info->segment[i].bufsz);
>     		nullsz = info->segment[i].memsz - info->segment[i].bufsz;
>    @@ -1565,6 +1585,9 @@ int main(int argc, char *argv[])
>     		case OPT_PRINT_CKR_SIZE:
>     			print_crashkernel_region_size();
>     			return 0;
>    +		case OPT_HOTPLUG:
>    +			do_hotplug = 1;
>    +			break;
>     		default:
>     			break;
>     		}
>    diff --git a/kexec/kexec.h b/kexec/kexec.h
>    index 0f97a97..b0428cc 100644
>    --- a/kexec/kexec.h
>    +++ b/kexec/kexec.h
>    @@ -169,6 +169,7 @@ struct kexec_info {
>     	int command_line_len;
>     
>     	int skip_checks;
>    +	unsigned long elfcorehdr;
>     };
>     
>     struct arch_map_entry {
>    @@ -231,7 +232,8 @@ extern int file_types;
>     #define OPT_PRINT_CKR_SIZE	262
>     #define OPT_LOAD_LIVE_UPDATE	263
>     #define OPT_EXEC_LIVE_UPDATE	264
>    -#define OPT_MAX			265
>    +#define OPT_HOTPLUG		265
>    +#define OPT_MAX		266
>     #define KEXEC_OPTIONS \
>     	{ "help",		0, 0, OPT_HELP }, \
>     	{ "version",		0, 0, OPT_VERSION }, \
>    @@ -258,6 +260,7 @@ extern int file_types;
>     	{ "debug",		0, 0, OPT_DEBUG }, \
>     	{ "status",		0, 0, OPT_STATUS }, \
>     	{ "print-ckr-size",     0, 0, OPT_PRINT_CKR_SIZE }, \
>    +	{ "hotplug",		0, 0, OPT_HOTPLUG }, \
>     
>     #define KEXEC_OPT_STR "h?vdfixyluet:pscaS"
>     
>    @@ -290,6 +293,8 @@ extern unsigned long add_buffer_phys_virt(struct kexec_info *info,
>     	int buf_end, int phys);
>     extern void arch_reuse_initrd(void);
>     
>    +extern long int get_nr_cpus(void);
>    +
>     extern int ifdown(void);
>     
>     extern char purgatory[];
> 
> Regards,
> eric
> ---
> 
> v9: 13jun2022
>   - Rebased to 5.18.0
>   - Per Sourabh, moved crash_prepare_elf64_headers() into common
>     crash_core.c to avoid compile issues with kexec_load only path.
>   - Per David Hildebrand, replaced mutex_trylock() with mutex_lock().
>   - Changed the __weak arch_crash_handle_hotplug_event() to utilize
>     WARN_ONCE() instead of WARN(). Fix some formatting issues.
>   - Per Sourabh, introduced sysfs attribute crash_hotplug for memory
>     and CPUs; for use by userspace (udev) to determine if the kernel
>     performs crash hot un/plug support.
>   - Per Sourabh, moved the code detecting the elfcorehdr segment from
>     arch/x86 into crash_core:handle_hotplug_event() so both kexec_load
>     and kexec_file_load can benefit.
>   - Updated userspace kexec-tools kexec utility to reflect change to
>     using CRASH_MAX_MEMORY_RANGES and get_nr_cpus().
> 
> v8: 5may2022
>   https://lkml.org/lkml/2022/5/5/1133
>   - Per Borislav Petkov, eliminated CONFIG_CRASH_HOTPLUG in favor
>     of CONFIG_HOTPLUG_CPU || CONFIG_MEMORY_HOTPLUG, ie a new define
>     is not needed. Also use of IS_ENABLED() rather than #ifdef's.
>     Renamed crash_hotplug_handler() to handle_hotplug_event().
>     And other corrections.
>   - Per Baoquan, minimized the parameters to the arch_crash_
>     handle_hotplug_event() to hp_action and cpu.
>   - Introduce KEXEC_CRASH_HP_INVALID_CPU definition, per Baoquan.
>   - Per Sourabh Jain, renamed and repurposed CRASH_HOTPLUG_ELFCOREHDR_SZ
>     to CONFIG_CRASH_MAX_MEMORY_RANGES, mirroring kexec-tools change
>     by David Hildebrand. Folded this patch into the x86
>     kexec_file_load support patch.
> 
> v7: 13apr2022
>   https://lkml.org/lkml/2022/4/13/850
>   - Resolved parameter usage to crash_hotplug_handler(), per Baoquan.
> 
> v6: 1apr2022
>   https://lkml.org/lkml/2022/4/1/1203
>   - Reword commit messages and some comment cleanup per Baoquan.
>   - Changed elf_index to elfcorehdr_index for clarity.
>   - Minor code changes per Baoquan.
> 
> v5: 3mar2022
>   https://lkml.org/lkml/2022/3/3/674
>   - Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per
>     David Hildenbrand.
>   - Refactored slightly a few patches per Baoquan recommendation.
> 
> v4: 9feb2022
>   https://lkml.org/lkml/2022/2/9/1406
>   - Refactored patches per Baoquan suggestsions.
>   - A few corrections, per Baoquan.
> 
> v3: 10jan2022
>   https://lkml.org/lkml/2022/1/10/1212
>   - Rebasing per Baoquan He request.
>   - Changed memory notifier per David Hildenbrand.
>   - Providing example kexec userspace change in cover letter.
> 
> RFC v2: 7dec2021
>   https://lkml.org/lkml/2021/12/7/1088
>   - Acting upon Baoquan He suggestion of removing elfcorehdr from
>     the purgatory list of segments, removed purgatory code from
>     patchset, and it is signficiantly simpler now.
> 
> RFC v1: 18nov2021
>   https://lkml.org/lkml/2021/11/18/845
>   - working patchset demonstrating kernel handling of hotplug
>     updates to x86 elfcorehdr for kexec_file_load
> 
> RFC: 14dec2020
>   https://lkml.org/lkml/2020/12/14/532
>   - proposed concept of allowing kernel to handle hotplug update
>     of elfcorehdr
> ---
> 
> 
> Eric DeVolder (7):
>    crash: move crash_prepare_elf64_headers
>    crash: prototype change for crash_prepare_elf64_headers
>    crash: add generic infrastructure for crash hotplug support
>    kexec: exclude elfcorehdr from the segment digest
>    kexec: exclude hot remove cpu from elfcorehdr notes
>    crash: memory and cpu hotplug sysfs attributes
>    x86/crash: Add x86 crash hotplug support
> 
>   .../admin-guide/mm/memory-hotplug.rst         |   8 +
>   Documentation/core-api/cpu_hotplug.rst        |  18 ++
>   arch/arm64/kernel/machine_kexec_file.c        |   6 +-
>   arch/powerpc/kexec/file_load_64.c             |   2 +-
>   arch/x86/Kconfig                              |  11 +
>   arch/x86/kernel/crash.c                       | 119 ++++++++-
>   drivers/base/cpu.c                            |  13 +
>   drivers/base/memory.c                         |  13 +
>   include/linux/crash_core.h                    |  12 +
>   include/linux/kexec.h                         |  14 +-
>   kernel/crash_core.c                           | 241 ++++++++++++++++++
>   kernel/kexec_file.c                           | 105 +-------
>   12 files changed, 456 insertions(+), 106 deletions(-)
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug
  2022-07-05 15:17 ` [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
@ 2022-07-06  1:16   ` Baoquan He
  2022-07-07 13:05     ` Eric DeVolder
  0 siblings, 1 reply; 20+ messages in thread
From: Baoquan He @ 2022-07-06  1:16 UTC (permalink / raw)
  To: Eric DeVolder
  Cc: linux-kernel, x86, kexec, ebiederm, dyoung, vgoyal, tglx, mingo,
	bp, dave.hansen, hpa, nramas, thomas.lendacky, robh, efault,
	rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky

On 07/05/22 at 10:17am, Eric DeVolder wrote:
> Nudge...
> Thanks!

Seems DavidH has concern and rasied it after you posted v9. Is that
solved?

> 
> On 6/13/22 17:42, Eric DeVolder wrote:
> > When the kdump service is loaded, if a CPU or memory is hot
> > un/plugged, the crash elfcorehdr (for x86), which describes the CPUs
> > and memory in the system, must also be updated, else the resulting
> > vmcore is inaccurate (eg. missing either CPU context or memory
> > regions).
> > 
> > The current solution utilizes udev to initiate an unload-then-reload
> > of the kdump image (e. kernel, initrd, boot_params, puratory and
> > elfcorehdr) by the userspace kexec utility. In previous posts I have
> > outlined the significant performance problems related to offloading
> > this activity to userspace.
> > 
> > This patchset introduces a generic crash hot un/plug handler that
> > registers with the CPU and memory notifiers. Upon CPU or memory
> > changes, this generic handler is invoked and performs important
> > housekeeping, for example obtaining the appropriate lock, and then
> > invokes an architecture specific handler to do the appropriate
> > updates.
> > 
> > In the case of x86_64, the arch specific handler generates a new
> > elfcorehdr, and overwrites the old one in memory. No involvement
> > with userspace needed.
> > 
> > To realize the benefits/test this patchset, one must make a couple
> > of minor changes to userspace:
> > 
> >   - Prevent udev from updating kdump crash kernel on hot un/plug changes.
> >     Add the following as the first lines to the udev rule file
> >     /usr/lib/udev/rules.d/98-kexec.rules:
> > 
> >     # The kernel handles updates to crash elfcorehdr for cpu and memory changes
> >     SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
> >     SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
> > 
> >     These lines will cause cpu and memory hot un/plug events to be
> >     skipped within this rule file, if the kernel has these changes
> >     enabled.
> > 
> >   - Change to the kexec_file_load for loading the kdump kernel:
> >     Eg. on RHEL: in /usr/bin/kdumpctl, change to:
> >      standard_kexec_args="-p -d -s"
> >     which adds the -s to select kexec_file_load syscall.
> > 
> > This patchset supports kexec_load with a modified kexec userspace
> > utility, and a working changeset to the kexec userspace utility
> > is provided here (and to use, the above change to standard_kexec_args
> > would be, for example, to append --hotplug instead of -s).
> > 
> >    diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
> >    index 9826f6d..4ed395a 100644
> >    --- a/kexec/arch/i386/crashdump-x86.c
> >    +++ b/kexec/arch/i386/crashdump-x86.c
> >    @@ -48,6 +48,7 @@
> >     #include <x86/x86-linux.h>
> >     extern struct arch_options_t arch_options;
> >    +extern int do_hotplug;
> >     static int get_kernel_page_offset(struct kexec_info *UNUSED(info),
> >     				  struct crash_elf_info *elf_info)
> >    @@ -975,6 +976,14 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
> >     	} else {
> >     		memsz = bufsz;
> >     	}
> >    +
> >    +	/* If hotplug support enabled, use larger size to accomodate changes */
> >    +	if (do_hotplug) {
> >    +		long int nr_cpus = get_nr_cpus();
> >    +		memsz = (nr_cpus + CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
> >    +	}
> >    +
> >    +    info->elfcorehdr =
> >     	elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
> >     							max_addr, -1);
> >     	dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
> >    diff --git a/kexec/crashdump-elf.c b/kexec/crashdump-elf.c
> >    index b8bb686..5e29f7a 100644
> >    --- a/kexec/crashdump-elf.c
> >    +++ b/kexec/crashdump-elf.c
> >    @@ -43,11 +43,7 @@ int FUNC(struct kexec_info *info,
> >     	int (*get_note_info)(int cpu, uint64_t *addr, uint64_t *len);
> >     	long int count_cpu;
> >    -	if (xen_present())
> >    -		nr_cpus = xen_get_nr_phys_cpus();
> >    -	else
> >    -		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
> >    -
> >    +	nr_cpus = get_nr_cpus();
> >     	if (nr_cpus < 0) {
> >     		return -1;
> >     	}
> >    diff --git a/kexec/crashdump.h b/kexec/crashdump.h
> >    index 18bd691..28d3278 100644
> >    --- a/kexec/crashdump.h
> >    +++ b/kexec/crashdump.h
> >    @@ -57,7 +57,6 @@ unsigned long phys_to_virt(struct crash_elf_info *elf_info,
> >     			   unsigned long long paddr);
> >     unsigned long xen_architecture(struct crash_elf_info *elf_info);
> >    -int xen_get_nr_phys_cpus(void);
> >     int xen_get_note(int cpu, uint64_t *addr, uint64_t *len);
> >     int xen_get_crashkernel_region(uint64_t *start, uint64_t *end);
> >    diff --git a/kexec/kexec-xen.h b/kexec/kexec-xen.h
> >    index 70fb576..f54a2dd 100644
> >    --- a/kexec/kexec-xen.h
> >    +++ b/kexec/kexec-xen.h
> >    @@ -83,5 +83,6 @@ extern int __xc_interface_close(xc_interface *xch);
> >     #endif
> >     int xen_get_kexec_range(int range, uint64_t *start, uint64_t *end);
> >    +int xen_get_nr_phys_cpus(void);
> >     #endif /* KEXEC_XEN_H */
> >    diff --git a/kexec/kexec.c b/kexec/kexec.c
> >    index 829a6ea..3668b73 100644
> >    --- a/kexec/kexec.c
> >    +++ b/kexec/kexec.c
> >    @@ -58,6 +58,7 @@
> >     unsigned long long mem_min = 0;
> >     unsigned long long mem_max = ULONG_MAX;
> >    +int do_hotplug = 0;
> >     static unsigned long kexec_flags = 0;
> >     /* Flags for kexec file (fd) based syscall */
> >     static unsigned long kexec_file_flags = 0;
> >    @@ -489,6 +490,17 @@ static int add_backup_segments(struct kexec_info *info,
> >     	return 0;
> >     }
> >    +long int get_nr_cpus(void)
> >    +{
> >    +    long int nr_cpus;
> >    +
> >    +	if (xen_present())
> >    +		nr_cpus = xen_get_nr_phys_cpus();
> >    +	else
> >    +		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
> >    +    return nr_cpus;
> >    +}
> >    +
> >     static char *slurp_fd(int fd, const char *filename, off_t size, off_t *nread)
> >     {
> >     	char *buf;
> >    @@ -672,6 +684,14 @@ static void update_purgatory(struct kexec_info *info)
> >     		if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
> >     			continue;
> >     		}
> >    +
> >    +		/* Don't include elfcorehdr in the checksum, if hotplug
> >    +		 * support enabled.
> >    +		 */
> >    +		if (do_hotplug && (info->segment[i].mem == (void *)info->elfcorehdr)) {
> >    +			continue;
> >    +		}
> >    +
> >     		sha256_update(&ctx, info->segment[i].buf,
> >     			      info->segment[i].bufsz);
> >     		nullsz = info->segment[i].memsz - info->segment[i].bufsz;
> >    @@ -1565,6 +1585,9 @@ int main(int argc, char *argv[])
> >     		case OPT_PRINT_CKR_SIZE:
> >     			print_crashkernel_region_size();
> >     			return 0;
> >    +		case OPT_HOTPLUG:
> >    +			do_hotplug = 1;
> >    +			break;
> >     		default:
> >     			break;
> >     		}
> >    diff --git a/kexec/kexec.h b/kexec/kexec.h
> >    index 0f97a97..b0428cc 100644
> >    --- a/kexec/kexec.h
> >    +++ b/kexec/kexec.h
> >    @@ -169,6 +169,7 @@ struct kexec_info {
> >     	int command_line_len;
> >     	int skip_checks;
> >    +	unsigned long elfcorehdr;
> >     };
> >     struct arch_map_entry {
> >    @@ -231,7 +232,8 @@ extern int file_types;
> >     #define OPT_PRINT_CKR_SIZE	262
> >     #define OPT_LOAD_LIVE_UPDATE	263
> >     #define OPT_EXEC_LIVE_UPDATE	264
> >    -#define OPT_MAX			265
> >    +#define OPT_HOTPLUG		265
> >    +#define OPT_MAX		266
> >     #define KEXEC_OPTIONS \
> >     	{ "help",		0, 0, OPT_HELP }, \
> >     	{ "version",		0, 0, OPT_VERSION }, \
> >    @@ -258,6 +260,7 @@ extern int file_types;
> >     	{ "debug",		0, 0, OPT_DEBUG }, \
> >     	{ "status",		0, 0, OPT_STATUS }, \
> >     	{ "print-ckr-size",     0, 0, OPT_PRINT_CKR_SIZE }, \
> >    +	{ "hotplug",		0, 0, OPT_HOTPLUG }, \
> >     #define KEXEC_OPT_STR "h?vdfixyluet:pscaS"
> >    @@ -290,6 +293,8 @@ extern unsigned long add_buffer_phys_virt(struct kexec_info *info,
> >     	int buf_end, int phys);
> >     extern void arch_reuse_initrd(void);
> >    +extern long int get_nr_cpus(void);
> >    +
> >     extern int ifdown(void);
> >     extern char purgatory[];
> > 
> > Regards,
> > eric
> > ---
> > 
> > v9: 13jun2022
> >   - Rebased to 5.18.0
> >   - Per Sourabh, moved crash_prepare_elf64_headers() into common
> >     crash_core.c to avoid compile issues with kexec_load only path.
> >   - Per David Hildebrand, replaced mutex_trylock() with mutex_lock().
> >   - Changed the __weak arch_crash_handle_hotplug_event() to utilize
> >     WARN_ONCE() instead of WARN(). Fix some formatting issues.
> >   - Per Sourabh, introduced sysfs attribute crash_hotplug for memory
> >     and CPUs; for use by userspace (udev) to determine if the kernel
> >     performs crash hot un/plug support.
> >   - Per Sourabh, moved the code detecting the elfcorehdr segment from
> >     arch/x86 into crash_core:handle_hotplug_event() so both kexec_load
> >     and kexec_file_load can benefit.
> >   - Updated userspace kexec-tools kexec utility to reflect change to
> >     using CRASH_MAX_MEMORY_RANGES and get_nr_cpus().
> > 
> > v8: 5may2022
> >   https://lkml.org/lkml/2022/5/5/1133
> >   - Per Borislav Petkov, eliminated CONFIG_CRASH_HOTPLUG in favor
> >     of CONFIG_HOTPLUG_CPU || CONFIG_MEMORY_HOTPLUG, ie a new define
> >     is not needed. Also use of IS_ENABLED() rather than #ifdef's.
> >     Renamed crash_hotplug_handler() to handle_hotplug_event().
> >     And other corrections.
> >   - Per Baoquan, minimized the parameters to the arch_crash_
> >     handle_hotplug_event() to hp_action and cpu.
> >   - Introduce KEXEC_CRASH_HP_INVALID_CPU definition, per Baoquan.
> >   - Per Sourabh Jain, renamed and repurposed CRASH_HOTPLUG_ELFCOREHDR_SZ
> >     to CONFIG_CRASH_MAX_MEMORY_RANGES, mirroring kexec-tools change
> >     by David Hildebrand. Folded this patch into the x86
> >     kexec_file_load support patch.
> > 
> > v7: 13apr2022
> >   https://lkml.org/lkml/2022/4/13/850
> >   - Resolved parameter usage to crash_hotplug_handler(), per Baoquan.
> > 
> > v6: 1apr2022
> >   https://lkml.org/lkml/2022/4/1/1203
> >   - Reword commit messages and some comment cleanup per Baoquan.
> >   - Changed elf_index to elfcorehdr_index for clarity.
> >   - Minor code changes per Baoquan.
> > 
> > v5: 3mar2022
> >   https://lkml.org/lkml/2022/3/3/674
> >   - Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per
> >     David Hildenbrand.
> >   - Refactored slightly a few patches per Baoquan recommendation.
> > 
> > v4: 9feb2022
> >   https://lkml.org/lkml/2022/2/9/1406
> >   - Refactored patches per Baoquan suggestsions.
> >   - A few corrections, per Baoquan.
> > 
> > v3: 10jan2022
> >   https://lkml.org/lkml/2022/1/10/1212
> >   - Rebasing per Baoquan He request.
> >   - Changed memory notifier per David Hildenbrand.
> >   - Providing example kexec userspace change in cover letter.
> > 
> > RFC v2: 7dec2021
> >   https://lkml.org/lkml/2021/12/7/1088
> >   - Acting upon Baoquan He suggestion of removing elfcorehdr from
> >     the purgatory list of segments, removed purgatory code from
> >     patchset, and it is signficiantly simpler now.
> > 
> > RFC v1: 18nov2021
> >   https://lkml.org/lkml/2021/11/18/845
> >   - working patchset demonstrating kernel handling of hotplug
> >     updates to x86 elfcorehdr for kexec_file_load
> > 
> > RFC: 14dec2020
> >   https://lkml.org/lkml/2020/12/14/532
> >   - proposed concept of allowing kernel to handle hotplug update
> >     of elfcorehdr
> > ---
> > 
> > 
> > Eric DeVolder (7):
> >    crash: move crash_prepare_elf64_headers
> >    crash: prototype change for crash_prepare_elf64_headers
> >    crash: add generic infrastructure for crash hotplug support
> >    kexec: exclude elfcorehdr from the segment digest
> >    kexec: exclude hot remove cpu from elfcorehdr notes
> >    crash: memory and cpu hotplug sysfs attributes
> >    x86/crash: Add x86 crash hotplug support
> > 
> >   .../admin-guide/mm/memory-hotplug.rst         |   8 +
> >   Documentation/core-api/cpu_hotplug.rst        |  18 ++
> >   arch/arm64/kernel/machine_kexec_file.c        |   6 +-
> >   arch/powerpc/kexec/file_load_64.c             |   2 +-
> >   arch/x86/Kconfig                              |  11 +
> >   arch/x86/kernel/crash.c                       | 119 ++++++++-
> >   drivers/base/cpu.c                            |  13 +
> >   drivers/base/memory.c                         |  13 +
> >   include/linux/crash_core.h                    |  12 +
> >   include/linux/kexec.h                         |  14 +-
> >   kernel/crash_core.c                           | 241 ++++++++++++++++++
> >   kernel/kexec_file.c                           | 105 +-------
> >   12 files changed, 456 insertions(+), 106 deletions(-)
> > 
> 


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support
  2022-06-20  5:06   ` Sourabh Jain
@ 2022-07-07 12:55     ` Eric DeVolder
  2022-07-11  9:57       ` Baoquan He
  0 siblings, 1 reply; 20+ messages in thread
From: Eric DeVolder @ 2022-07-07 12:55 UTC (permalink / raw)
  To: Sourabh Jain, linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, konrad.wilk, boris.ostrovsky



On 6/20/22 00:06, Sourabh Jain wrote:
> Hello Eric,
> 
> On 14/06/22 04:12, Eric DeVolder wrote:
>> For x86_64, when CPU or memory is hot un/plugged, the crash
>> elfcorehdr, which describes the CPUs and memory in the system,
>> must also be updated.
>>
>> When loading the crash kernel via kexec_load or kexec_file_load,
>> the elfcorehdr is identified at run time in
>> crash_core:handle_hotplug_event().
>>
>> To update the elfcorehdr for x86_64, a new elfcorehdr must be
>> generated from the available CPUs and memory. The new elfcorehdr
>> is prepared into a buffer, and then installed over the top of
>> the existing elfcorehdr.
>>
>> In the patch 'kexec: exclude elfcorehdr from the segment digest'
>> the need to update purgatory due to the change in elfcorehdr was
>> eliminated.  As a result, no changes to purgatory or boot_params
>> (as the elfcorehdr= kernel command line parameter pointer
>> remains unchanged and correct) are needed, just elfcorehdr.
>>
>> To accommodate a growing number of resources via hotplug, the
>> elfcorehdr segment must be sufficiently large enough to accommodate
>> changes, see the CRASH_MAX_MEMORY_RANGES configure item.
>>
>> With this change, crash hotplug for kexec_file_load syscall
>> is supported. The kexec_load is also supported, but also
>> requires a corresponding change to userspace kexec-tools.
>>
>> Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
>> ---
>>   arch/x86/Kconfig        |  11 ++++
>>   arch/x86/kernel/crash.c | 116 ++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 127 insertions(+)
>>
>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>> index 762a0b6ab8b6..e9eecff3b97d 100644
>> --- a/arch/x86/Kconfig
>> +++ b/arch/x86/Kconfig
>> @@ -2082,6 +2082,17 @@ config CRASH_DUMP
>>         (CONFIG_RELOCATABLE=y).
>>         For more details see Documentation/admin-guide/kdump/kdump.rst
>> +config CRASH_MAX_MEMORY_RANGES
>> +    depends on CRASH_DUMP && KEXEC_FILE && (HOTPLUG_CPU || MEMORY_HOTPLUG)
>> +    int
>> +    default 32768
>> +    help
>> +      For the kexec_file_load path, specify the maximum number of
>> +      memory regions, eg. as represented by the 'System RAM' entries
>> +      in /proc/iomem, that the elfcorehdr buffer/segment can accommodate.
>> +      This value is combined with NR_CPUS and multiplied by Elf64_Phdr
>> +      size to determine the final buffer size.
>> +
>>   config KEXEC_JUMP
>>       bool "kexec jump"
>>       depends on KEXEC && HIBERNATION
>> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
>> index 9db41cce8d97..b9cdf7a0d868 100644
>> --- a/arch/x86/kernel/crash.c
>> +++ b/arch/x86/kernel/crash.c
>> @@ -25,6 +25,7 @@
>>   #include <linux/slab.h>
>>   #include <linux/vmalloc.h>
>>   #include <linux/memblock.h>
>> +#include <linux/highmem.h>
>>   #include <asm/processor.h>
>>   #include <asm/hardirq.h>
>> @@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
>>       image->elf_headers = kbuf.buffer;
>>       image->elf_headers_sz = kbuf.bufsz;
>> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
>> +    /* Ensure elfcorehdr segment large enough for hotplug changes */
>> +    kbuf.memsz = (CONFIG_NR_CPUS_DEFAULT + CONFIG_CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
>> +    /* For marking as usable to crash kernel */
>> +    image->elf_headers_sz = kbuf.memsz;
>> +    /* Record the index of the elfcorehdr segment */
>> +    image->elfcorehdr_index = image->nr_segments;
>> +    image->elfcorehdr_index_valid = true;
>> +#else
>>       kbuf.memsz = kbuf.bufsz;
>> +#endif
>>       kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
>>       kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
>>       ret = kexec_add_buffer(&kbuf);
>> @@ -413,3 +424,108 @@ int crash_load_segments(struct kimage *image)
>>       return ret;
>>   }
>>   #endif /* CONFIG_KEXEC_FILE */
>> +
>> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
>> +int crash_hotplug_support(void) { return 1; }
>> +void *arch_map_crash_pages(unsigned long paddr, unsigned long size)
>> +{
>> +    /*
>> +     * NOTE: The addresses and sizes passed to this routine have
>> +     * already been fully aligned on page boundaries. There is no
>> +     * need for massaging the address or size.
>> +     */
>> +    void *ptr = NULL;
>> +
>> +    /* NOTE: requires arch_kexec_[un]protect_crashkres() for write access */
>> +    if (size > 0) {
>> +        struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
>> +
>> +        ptr = kmap(page);
>> +    }
>> +
>> +    return ptr;
>> +}
>> +
>> +void arch_unmap_crash_pages(void **ptr)
>> +{
>> +    if (ptr) {
>> +        if (*ptr)
>> +            kunmap(*ptr);
>> +        *ptr = NULL;
>> +    }
>> +}
> 
> Aren't arch will have build issue if arch_[un]map_crash_pages methods are not defined?
Sourabh,
Yes, you are correct. I'll add __weak versions of each in crash_core.c in the next patch.
Thanks!
Eric

> 
> Thanks,
> Sourabh Jain
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug
  2022-07-06  1:16   ` Baoquan He
@ 2022-07-07 13:05     ` Eric DeVolder
  2022-07-20 19:08       ` Eric DeVolder
  0 siblings, 1 reply; 20+ messages in thread
From: Eric DeVolder @ 2022-07-07 13:05 UTC (permalink / raw)
  To: Baoquan He
  Cc: linux-kernel, x86, kexec, ebiederm, dyoung, vgoyal, tglx, mingo,
	bp, dave.hansen, hpa, nramas, thomas.lendacky, robh, efault,
	rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky



On 7/5/22 20:16, Baoquan He wrote:
> On 07/05/22 at 10:17am, Eric DeVolder wrote:
>> Nudge...
>> Thanks!
> 
> Seems DavidH has concern and rasied it after you posted v9. Is that
> solved?

In looking at David's comments on v8:
- [PATCH v8 2/7] crash: prototype change for crash_prepare_elf64_headers
   I adjusted per feedback and responded as such. He provided an Acked-by.

- [PATCH v8 3/7] crash: add generic infrastructure for crash hotplug support
   He says do not use WARN* on expected error paths. I instead used WARN_ONCE(),
   but I didn't consider this an error path. Nonetheless, I'll remove the
   message altogether in v11.

- [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
   David muses about a better name, but that was in response to Sourabh's
   and I exchange on the sysfs attributes to use to report the presence
   of this feature. In the end, I used something altogether different, as
   presented in v10 (and the corresponding udev solution/change that goes
   hand in hand with it).

Let me know if I've missed anything else? Sourabh points out a build problem
for architectures without this feature support, so a v11 is warranted. But I'll
wait to post that until I hear that I've addressed the previous questions.

Thanks,
eric


> 
>>
>> On 6/13/22 17:42, Eric DeVolder wrote:
>>> When the kdump service is loaded, if a CPU or memory is hot
>>> un/plugged, the crash elfcorehdr (for x86), which describes the CPUs
>>> and memory in the system, must also be updated, else the resulting
>>> vmcore is inaccurate (eg. missing either CPU context or memory
>>> regions).
>>>
>>> The current solution utilizes udev to initiate an unload-then-reload
>>> of the kdump image (e. kernel, initrd, boot_params, puratory and
>>> elfcorehdr) by the userspace kexec utility. In previous posts I have
>>> outlined the significant performance problems related to offloading
>>> this activity to userspace.
>>>
>>> This patchset introduces a generic crash hot un/plug handler that
>>> registers with the CPU and memory notifiers. Upon CPU or memory
>>> changes, this generic handler is invoked and performs important
>>> housekeeping, for example obtaining the appropriate lock, and then
>>> invokes an architecture specific handler to do the appropriate
>>> updates.
>>>
>>> In the case of x86_64, the arch specific handler generates a new
>>> elfcorehdr, and overwrites the old one in memory. No involvement
>>> with userspace needed.
>>>
>>> To realize the benefits/test this patchset, one must make a couple
>>> of minor changes to userspace:
>>>
>>>    - Prevent udev from updating kdump crash kernel on hot un/plug changes.
>>>      Add the following as the first lines to the udev rule file
>>>      /usr/lib/udev/rules.d/98-kexec.rules:
>>>
>>>      # The kernel handles updates to crash elfcorehdr for cpu and memory changes
>>>      SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
>>>      SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
>>>
>>>      These lines will cause cpu and memory hot un/plug events to be
>>>      skipped within this rule file, if the kernel has these changes
>>>      enabled.
>>>
>>>    - Change to the kexec_file_load for loading the kdump kernel:
>>>      Eg. on RHEL: in /usr/bin/kdumpctl, change to:
>>>       standard_kexec_args="-p -d -s"
>>>      which adds the -s to select kexec_file_load syscall.
>>>
>>> This patchset supports kexec_load with a modified kexec userspace
>>> utility, and a working changeset to the kexec userspace utility
>>> is provided here (and to use, the above change to standard_kexec_args
>>> would be, for example, to append --hotplug instead of -s).
>>>
>>>     diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
>>>     index 9826f6d..4ed395a 100644
>>>     --- a/kexec/arch/i386/crashdump-x86.c
>>>     +++ b/kexec/arch/i386/crashdump-x86.c
>>>     @@ -48,6 +48,7 @@
>>>      #include <x86/x86-linux.h>
>>>      extern struct arch_options_t arch_options;
>>>     +extern int do_hotplug;
>>>      static int get_kernel_page_offset(struct kexec_info *UNUSED(info),
>>>      				  struct crash_elf_info *elf_info)
>>>     @@ -975,6 +976,14 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
>>>      	} else {
>>>      		memsz = bufsz;
>>>      	}
>>>     +
>>>     +	/* If hotplug support enabled, use larger size to accomodate changes */
>>>     +	if (do_hotplug) {
>>>     +		long int nr_cpus = get_nr_cpus();
>>>     +		memsz = (nr_cpus + CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
>>>     +	}
>>>     +
>>>     +    info->elfcorehdr =
>>>      	elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
>>>      							max_addr, -1);
>>>      	dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
>>>     diff --git a/kexec/crashdump-elf.c b/kexec/crashdump-elf.c
>>>     index b8bb686..5e29f7a 100644
>>>     --- a/kexec/crashdump-elf.c
>>>     +++ b/kexec/crashdump-elf.c
>>>     @@ -43,11 +43,7 @@ int FUNC(struct kexec_info *info,
>>>      	int (*get_note_info)(int cpu, uint64_t *addr, uint64_t *len);
>>>      	long int count_cpu;
>>>     -	if (xen_present())
>>>     -		nr_cpus = xen_get_nr_phys_cpus();
>>>     -	else
>>>     -		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
>>>     -
>>>     +	nr_cpus = get_nr_cpus();
>>>      	if (nr_cpus < 0) {
>>>      		return -1;
>>>      	}
>>>     diff --git a/kexec/crashdump.h b/kexec/crashdump.h
>>>     index 18bd691..28d3278 100644
>>>     --- a/kexec/crashdump.h
>>>     +++ b/kexec/crashdump.h
>>>     @@ -57,7 +57,6 @@ unsigned long phys_to_virt(struct crash_elf_info *elf_info,
>>>      			   unsigned long long paddr);
>>>      unsigned long xen_architecture(struct crash_elf_info *elf_info);
>>>     -int xen_get_nr_phys_cpus(void);
>>>      int xen_get_note(int cpu, uint64_t *addr, uint64_t *len);
>>>      int xen_get_crashkernel_region(uint64_t *start, uint64_t *end);
>>>     diff --git a/kexec/kexec-xen.h b/kexec/kexec-xen.h
>>>     index 70fb576..f54a2dd 100644
>>>     --- a/kexec/kexec-xen.h
>>>     +++ b/kexec/kexec-xen.h
>>>     @@ -83,5 +83,6 @@ extern int __xc_interface_close(xc_interface *xch);
>>>      #endif
>>>      int xen_get_kexec_range(int range, uint64_t *start, uint64_t *end);
>>>     +int xen_get_nr_phys_cpus(void);
>>>      #endif /* KEXEC_XEN_H */
>>>     diff --git a/kexec/kexec.c b/kexec/kexec.c
>>>     index 829a6ea..3668b73 100644
>>>     --- a/kexec/kexec.c
>>>     +++ b/kexec/kexec.c
>>>     @@ -58,6 +58,7 @@
>>>      unsigned long long mem_min = 0;
>>>      unsigned long long mem_max = ULONG_MAX;
>>>     +int do_hotplug = 0;
>>>      static unsigned long kexec_flags = 0;
>>>      /* Flags for kexec file (fd) based syscall */
>>>      static unsigned long kexec_file_flags = 0;
>>>     @@ -489,6 +490,17 @@ static int add_backup_segments(struct kexec_info *info,
>>>      	return 0;
>>>      }
>>>     +long int get_nr_cpus(void)
>>>     +{
>>>     +    long int nr_cpus;
>>>     +
>>>     +	if (xen_present())
>>>     +		nr_cpus = xen_get_nr_phys_cpus();
>>>     +	else
>>>     +		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
>>>     +    return nr_cpus;
>>>     +}
>>>     +
>>>      static char *slurp_fd(int fd, const char *filename, off_t size, off_t *nread)
>>>      {
>>>      	char *buf;
>>>     @@ -672,6 +684,14 @@ static void update_purgatory(struct kexec_info *info)
>>>      		if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
>>>      			continue;
>>>      		}
>>>     +
>>>     +		/* Don't include elfcorehdr in the checksum, if hotplug
>>>     +		 * support enabled.
>>>     +		 */
>>>     +		if (do_hotplug && (info->segment[i].mem == (void *)info->elfcorehdr)) {
>>>     +			continue;
>>>     +		}
>>>     +
>>>      		sha256_update(&ctx, info->segment[i].buf,
>>>      			      info->segment[i].bufsz);
>>>      		nullsz = info->segment[i].memsz - info->segment[i].bufsz;
>>>     @@ -1565,6 +1585,9 @@ int main(int argc, char *argv[])
>>>      		case OPT_PRINT_CKR_SIZE:
>>>      			print_crashkernel_region_size();
>>>      			return 0;
>>>     +		case OPT_HOTPLUG:
>>>     +			do_hotplug = 1;
>>>     +			break;
>>>      		default:
>>>      			break;
>>>      		}
>>>     diff --git a/kexec/kexec.h b/kexec/kexec.h
>>>     index 0f97a97..b0428cc 100644
>>>     --- a/kexec/kexec.h
>>>     +++ b/kexec/kexec.h
>>>     @@ -169,6 +169,7 @@ struct kexec_info {
>>>      	int command_line_len;
>>>      	int skip_checks;
>>>     +	unsigned long elfcorehdr;
>>>      };
>>>      struct arch_map_entry {
>>>     @@ -231,7 +232,8 @@ extern int file_types;
>>>      #define OPT_PRINT_CKR_SIZE	262
>>>      #define OPT_LOAD_LIVE_UPDATE	263
>>>      #define OPT_EXEC_LIVE_UPDATE	264
>>>     -#define OPT_MAX			265
>>>     +#define OPT_HOTPLUG		265
>>>     +#define OPT_MAX		266
>>>      #define KEXEC_OPTIONS \
>>>      	{ "help",		0, 0, OPT_HELP }, \
>>>      	{ "version",		0, 0, OPT_VERSION }, \
>>>     @@ -258,6 +260,7 @@ extern int file_types;
>>>      	{ "debug",		0, 0, OPT_DEBUG }, \
>>>      	{ "status",		0, 0, OPT_STATUS }, \
>>>      	{ "print-ckr-size",     0, 0, OPT_PRINT_CKR_SIZE }, \
>>>     +	{ "hotplug",		0, 0, OPT_HOTPLUG }, \
>>>      #define KEXEC_OPT_STR "h?vdfixyluet:pscaS"
>>>     @@ -290,6 +293,8 @@ extern unsigned long add_buffer_phys_virt(struct kexec_info *info,
>>>      	int buf_end, int phys);
>>>      extern void arch_reuse_initrd(void);
>>>     +extern long int get_nr_cpus(void);
>>>     +
>>>      extern int ifdown(void);
>>>      extern char purgatory[];
>>>
>>> Regards,
>>> eric
>>> ---
>>>
>>> v9: 13jun2022
>>>    - Rebased to 5.18.0
>>>    - Per Sourabh, moved crash_prepare_elf64_headers() into common
>>>      crash_core.c to avoid compile issues with kexec_load only path.
>>>    - Per David Hildebrand, replaced mutex_trylock() with mutex_lock().
>>>    - Changed the __weak arch_crash_handle_hotplug_event() to utilize
>>>      WARN_ONCE() instead of WARN(). Fix some formatting issues.
>>>    - Per Sourabh, introduced sysfs attribute crash_hotplug for memory
>>>      and CPUs; for use by userspace (udev) to determine if the kernel
>>>      performs crash hot un/plug support.
>>>    - Per Sourabh, moved the code detecting the elfcorehdr segment from
>>>      arch/x86 into crash_core:handle_hotplug_event() so both kexec_load
>>>      and kexec_file_load can benefit.
>>>    - Updated userspace kexec-tools kexec utility to reflect change to
>>>      using CRASH_MAX_MEMORY_RANGES and get_nr_cpus().
>>>
>>> v8: 5may2022
>>>    https://lkml.org/lkml/2022/5/5/1133
>>>    - Per Borislav Petkov, eliminated CONFIG_CRASH_HOTPLUG in favor
>>>      of CONFIG_HOTPLUG_CPU || CONFIG_MEMORY_HOTPLUG, ie a new define
>>>      is not needed. Also use of IS_ENABLED() rather than #ifdef's.
>>>      Renamed crash_hotplug_handler() to handle_hotplug_event().
>>>      And other corrections.
>>>    - Per Baoquan, minimized the parameters to the arch_crash_
>>>      handle_hotplug_event() to hp_action and cpu.
>>>    - Introduce KEXEC_CRASH_HP_INVALID_CPU definition, per Baoquan.
>>>    - Per Sourabh Jain, renamed and repurposed CRASH_HOTPLUG_ELFCOREHDR_SZ
>>>      to CONFIG_CRASH_MAX_MEMORY_RANGES, mirroring kexec-tools change
>>>      by David Hildebrand. Folded this patch into the x86
>>>      kexec_file_load support patch.
>>>
>>> v7: 13apr2022
>>>    https://lkml.org/lkml/2022/4/13/850
>>>    - Resolved parameter usage to crash_hotplug_handler(), per Baoquan.
>>>
>>> v6: 1apr2022
>>>    https://lkml.org/lkml/2022/4/1/1203
>>>    - Reword commit messages and some comment cleanup per Baoquan.
>>>    - Changed elf_index to elfcorehdr_index for clarity.
>>>    - Minor code changes per Baoquan.
>>>
>>> v5: 3mar2022
>>>    https://lkml.org/lkml/2022/3/3/674
>>>    - Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per
>>>      David Hildenbrand.
>>>    - Refactored slightly a few patches per Baoquan recommendation.
>>>
>>> v4: 9feb2022
>>>    https://lkml.org/lkml/2022/2/9/1406
>>>    - Refactored patches per Baoquan suggestsions.
>>>    - A few corrections, per Baoquan.
>>>
>>> v3: 10jan2022
>>>    https://lkml.org/lkml/2022/1/10/1212
>>>    - Rebasing per Baoquan He request.
>>>    - Changed memory notifier per David Hildenbrand.
>>>    - Providing example kexec userspace change in cover letter.
>>>
>>> RFC v2: 7dec2021
>>>    https://lkml.org/lkml/2021/12/7/1088
>>>    - Acting upon Baoquan He suggestion of removing elfcorehdr from
>>>      the purgatory list of segments, removed purgatory code from
>>>      patchset, and it is signficiantly simpler now.
>>>
>>> RFC v1: 18nov2021
>>>    https://lkml.org/lkml/2021/11/18/845
>>>    - working patchset demonstrating kernel handling of hotplug
>>>      updates to x86 elfcorehdr for kexec_file_load
>>>
>>> RFC: 14dec2020
>>>    https://lkml.org/lkml/2020/12/14/532
>>>    - proposed concept of allowing kernel to handle hotplug update
>>>      of elfcorehdr
>>> ---
>>>
>>>
>>> Eric DeVolder (7):
>>>     crash: move crash_prepare_elf64_headers
>>>     crash: prototype change for crash_prepare_elf64_headers
>>>     crash: add generic infrastructure for crash hotplug support
>>>     kexec: exclude elfcorehdr from the segment digest
>>>     kexec: exclude hot remove cpu from elfcorehdr notes
>>>     crash: memory and cpu hotplug sysfs attributes
>>>     x86/crash: Add x86 crash hotplug support
>>>
>>>    .../admin-guide/mm/memory-hotplug.rst         |   8 +
>>>    Documentation/core-api/cpu_hotplug.rst        |  18 ++
>>>    arch/arm64/kernel/machine_kexec_file.c        |   6 +-
>>>    arch/powerpc/kexec/file_load_64.c             |   2 +-
>>>    arch/x86/Kconfig                              |  11 +
>>>    arch/x86/kernel/crash.c                       | 119 ++++++++-
>>>    drivers/base/cpu.c                            |  13 +
>>>    drivers/base/memory.c                         |  13 +
>>>    include/linux/crash_core.h                    |  12 +
>>>    include/linux/kexec.h                         |  14 +-
>>>    kernel/crash_core.c                           | 241 ++++++++++++++++++
>>>    kernel/kexec_file.c                           | 105 +-------
>>>    12 files changed, 456 insertions(+), 106 deletions(-)
>>>
>>
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support
  2022-06-13 22:42 ` [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support Eric DeVolder
  2022-06-20  5:06   ` Sourabh Jain
@ 2022-07-08 19:38   ` Fabio M. De Francesco
  2022-07-08 21:07     ` Eric DeVolder
  1 sibling, 1 reply; 20+ messages in thread
From: Fabio M. De Francesco @ 2022-07-08 19:38 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal, Eric DeVolder
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	eric.devolder

On martedì 14 giugno 2022 00:42:40 CEST Eric DeVolder wrote:
> For x86_64, when CPU or memory is hot un/plugged, the crash
> elfcorehdr, which describes the CPUs and memory in the system,
> must also be updated.
> 
> When loading the crash kernel via kexec_load or kexec_file_load,
> the elfcorehdr is identified at run time in
> crash_core:handle_hotplug_event().
> 
> To update the elfcorehdr for x86_64, a new elfcorehdr must be
> generated from the available CPUs and memory. The new elfcorehdr
> is prepared into a buffer, and then installed over the top of
> the existing elfcorehdr.
> 
> In the patch 'kexec: exclude elfcorehdr from the segment digest'
> the need to update purgatory due to the change in elfcorehdr was
> eliminated.  As a result, no changes to purgatory or boot_params
> (as the elfcorehdr= kernel command line parameter pointer
> remains unchanged and correct) are needed, just elfcorehdr.
> 
> To accommodate a growing number of resources via hotplug, the
> elfcorehdr segment must be sufficiently large enough to accommodate
> changes, see the CRASH_MAX_MEMORY_RANGES configure item.
> 
> With this change, crash hotplug for kexec_file_load syscall
> is supported. The kexec_load is also supported, but also
> requires a corresponding change to userspace kexec-tools.
> 
> Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
> ---
>  arch/x86/Kconfig        |  11 ++++
>  arch/x86/kernel/crash.c | 116 ++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 127 insertions(+)
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 762a0b6ab8b6..e9eecff3b97d 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -2082,6 +2082,17 @@ config CRASH_DUMP
>  	  (CONFIG_RELOCATABLE=y).
>  	  For more details see Documentation/admin-guide/kdump/kdump.rst
>  
> +config CRASH_MAX_MEMORY_RANGES
> +	depends on CRASH_DUMP && KEXEC_FILE && (HOTPLUG_CPU || 
MEMORY_HOTPLUG)
> +	int
> +	default 32768
> +	help
> +	  For the kexec_file_load path, specify the maximum number of
> +	  memory regions, eg. as represented by the 'System RAM' entries
> +	  in /proc/iomem, that the elfcorehdr buffer/segment can 
accommodate.
> +	  This value is combined with NR_CPUS and multiplied by 
Elf64_Phdr
> +	  size to determine the final buffer size.
> +
>  config KEXEC_JUMP
>  	bool "kexec jump"
>  	depends on KEXEC && HIBERNATION
> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> index 9db41cce8d97..b9cdf7a0d868 100644
> --- a/arch/x86/kernel/crash.c
> +++ b/arch/x86/kernel/crash.c
> @@ -25,6 +25,7 @@
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
>  #include <linux/memblock.h>
> +#include <linux/highmem.h>
>  
>  #include <asm/processor.h>
>  #include <asm/hardirq.h>
> @@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
>  	image->elf_headers = kbuf.buffer;
>  	image->elf_headers_sz = kbuf.bufsz;
>  
> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
> +	/* Ensure elfcorehdr segment large enough for hotplug changes */
> +	kbuf.memsz = (CONFIG_NR_CPUS_DEFAULT + 
CONFIG_CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
> +	/* For marking as usable to crash kernel */
> +	image->elf_headers_sz = kbuf.memsz;
> +	/* Record the index of the elfcorehdr segment */
> +	image->elfcorehdr_index = image->nr_segments;
> +	image->elfcorehdr_index_valid = true;
> +#else
>  	kbuf.memsz = kbuf.bufsz;
> +#endif
>  	kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
>  	kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
>  	ret = kexec_add_buffer(&kbuf);
> @@ -413,3 +424,108 @@ int crash_load_segments(struct kimage *image)
>  	return ret;
>  }
>  #endif /* CONFIG_KEXEC_FILE */
> +
> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
> +int crash_hotplug_support(void) { return 1; }
> +void *arch_map_crash_pages(unsigned long paddr, unsigned long size)
> +{
> +	/*
> +	 * NOTE: The addresses and sizes passed to this routine have
> +	 * already been fully aligned on page boundaries. There is no
> +	 * need for massaging the address or size.
> +	 */
> +	void *ptr = NULL;
> +
> +	/* NOTE: requires arch_kexec_[un]protect_crashkres() for write 
access */
> +	if (size > 0) {
> +		struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
> +
> +		ptr = kmap(page);
> +	}

kmap_local_page() is preferred over kmap() which should be avoided in new 
code (please take a look at highmem.rst which I recently updated).

Can you please check if you can easily avoid using kmap()? Otherwise, this 
code will be added to my (long) queue and changed one day or the other :-)

Thanks,

Fabio

> +
> +	return ptr;
> +}
> +
> +void arch_unmap_crash_pages(void **ptr)
> +{
> +	if (ptr) {
> +		if (*ptr)
> +			kunmap(*ptr);
> +		*ptr = NULL;
> +	}
> +}
> +
> +/**
> + * arch_crash_handle_hotplug_event() - Handle hotplug elfcorehdr changes
> + * @image: the active struct kimage
> + * @hp_action: the hot un/plug action being handled
> + * @cpu: when KEXEC_CRASH_HP_ADD/REMOVE_CPU, the cpu affected
> + *
> + * To accurately reflect hot un/plug changes, the elfcorehdr (which
> + * is passed to the crash kernel via the elfcorehdr= parameter)
> + * must be updated with the new list of CPUs and memories. The new
> + * elfcorehdr is prepared in a kernel buffer, and then it is
> + * written on top of the existing/old elfcorehdr.
> + *
> + * For hotplug changes to elfcorehdr to work, two conditions are
> + * needed:
> + * First, the segment containing the elfcorehdr must be large enough
> + * to permit a growing number of resources. See the
> + * CONFIG_CRASH_MAX_MEMORY_RANGES description.
> + * Second, purgatory must explicitly exclude the elfcorehdr from the
> + * list of segments it checks (since the elfcorehdr changes and thus
> + * would require an update to purgatory itself to update the digest).
> + *
> + */
> +void arch_crash_handle_hotplug_event(struct kimage *image,
> +	unsigned int hp_action, unsigned int cpu)
> +{
> +	struct kexec_segment *ksegment;
> +	unsigned char *ptr = NULL;
> +	unsigned long elfsz = 0;
> +	void *elfbuf = NULL;
> +	unsigned long mem, memsz;
> +
> +	/*
> +	 * Elfcorehdr_index_valid checked in 
crash_core:handle_hotplug_event()
> +	 */
> +	ksegment = &image->segment[image->elfcorehdr_index];
> +	mem = ksegment->mem;
> +	memsz = ksegment->memsz;
> +
> +	/*
> +	 * Create the new elfcorehdr reflecting the changes to CPU and/or
> +	 * memory resources.
> +	 */
> +	if (prepare_elf_headers(image, &elfbuf, &elfsz)) {
> +		pr_err("crash hp: unable to prepare elfcore headers");
> +		goto out;
> +	}
> +	if (elfsz > memsz) {
> +		pr_err("crash hp: update elfcorehdr elfsz %lu > memsz 
%lu",
> +			elfsz, memsz);
> +		goto out;
> +	}
> +
> +	/*
> +	 * At this point, we are all but assured of success.
> +	 * Copy new elfcorehdr into destination.
> +	 */
> +	ptr = arch_map_crash_pages(mem, memsz);
> +	if (ptr) {
> +		/*
> +		 * Temporarily invalidate the crash image while the
> +		 * elfcorehdr is updated.
> +		 */
> +		xchg(&kexec_crash_image, NULL);
> +		memcpy_flushcache((void *)ptr, elfbuf, elfsz);
> +		xchg(&kexec_crash_image, image);
> +	}
> +	arch_unmap_crash_pages((void **)&ptr);
> +	pr_debug("crash hp: re-loaded elfcorehdr at 0x%lx\n", mem);
> +
> +out:
> +	if (elfbuf)
> +		vfree(elfbuf);
> +}
> +#endif
> -- 
> 2.31.1
> 
> 





_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support
  2022-07-08 19:38   ` Fabio M. De Francesco
@ 2022-07-08 21:07     ` Eric DeVolder
  2022-07-09 18:41       ` Fabio M. De Francesco
  0 siblings, 1 reply; 20+ messages in thread
From: Eric DeVolder @ 2022-07-08 21:07 UTC (permalink / raw)
  To: Fabio M. De Francesco, linux-kernel, x86, kexec, ebiederm,
	dyoung, bhe, vgoyal
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky



On 7/8/22 14:38, Fabio M. De Francesco wrote:
> On martedì 14 giugno 2022 00:42:40 CEST Eric DeVolder wrote:
>> For x86_64, when CPU or memory is hot un/plugged, the crash
>> elfcorehdr, which describes the CPUs and memory in the system,
>> must also be updated.
>>
>> When loading the crash kernel via kexec_load or kexec_file_load,
>> the elfcorehdr is identified at run time in
>> crash_core:handle_hotplug_event().
>>
>> To update the elfcorehdr for x86_64, a new elfcorehdr must be
>> generated from the available CPUs and memory. The new elfcorehdr
>> is prepared into a buffer, and then installed over the top of
>> the existing elfcorehdr.
>>
>> In the patch 'kexec: exclude elfcorehdr from the segment digest'
>> the need to update purgatory due to the change in elfcorehdr was
>> eliminated.  As a result, no changes to purgatory or boot_params
>> (as the elfcorehdr= kernel command line parameter pointer
>> remains unchanged and correct) are needed, just elfcorehdr.
>>
>> To accommodate a growing number of resources via hotplug, the
>> elfcorehdr segment must be sufficiently large enough to accommodate
>> changes, see the CRASH_MAX_MEMORY_RANGES configure item.
>>
>> With this change, crash hotplug for kexec_file_load syscall
>> is supported. The kexec_load is also supported, but also
>> requires a corresponding change to userspace kexec-tools.
>>
>> Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
>> ---
>>   arch/x86/Kconfig        |  11 ++++
>>   arch/x86/kernel/crash.c | 116 ++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 127 insertions(+)
>>
>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>> index 762a0b6ab8b6..e9eecff3b97d 100644
>> --- a/arch/x86/Kconfig
>> +++ b/arch/x86/Kconfig
>> @@ -2082,6 +2082,17 @@ config CRASH_DUMP
>>   	  (CONFIG_RELOCATABLE=y).
>>   	  For more details see Documentation/admin-guide/kdump/kdump.rst
>>   
>> +config CRASH_MAX_MEMORY_RANGES
>> +	depends on CRASH_DUMP && KEXEC_FILE && (HOTPLUG_CPU ||
> MEMORY_HOTPLUG)
>> +	int
>> +	default 32768
>> +	help
>> +	  For the kexec_file_load path, specify the maximum number of
>> +	  memory regions, eg. as represented by the 'System RAM' entries
>> +	  in /proc/iomem, that the elfcorehdr buffer/segment can
> accommodate.
>> +	  This value is combined with NR_CPUS and multiplied by
> Elf64_Phdr
>> +	  size to determine the final buffer size.
>> +
>>   config KEXEC_JUMP
>>   	bool "kexec jump"
>>   	depends on KEXEC && HIBERNATION
>> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
>> index 9db41cce8d97..b9cdf7a0d868 100644
>> --- a/arch/x86/kernel/crash.c
>> +++ b/arch/x86/kernel/crash.c
>> @@ -25,6 +25,7 @@
>>   #include <linux/slab.h>
>>   #include <linux/vmalloc.h>
>>   #include <linux/memblock.h>
>> +#include <linux/highmem.h>
>>   
>>   #include <asm/processor.h>
>>   #include <asm/hardirq.h>
>> @@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
>>   	image->elf_headers = kbuf.buffer;
>>   	image->elf_headers_sz = kbuf.bufsz;
>>   
>> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
>> +	/* Ensure elfcorehdr segment large enough for hotplug changes */
>> +	kbuf.memsz = (CONFIG_NR_CPUS_DEFAULT +
> CONFIG_CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
>> +	/* For marking as usable to crash kernel */
>> +	image->elf_headers_sz = kbuf.memsz;
>> +	/* Record the index of the elfcorehdr segment */
>> +	image->elfcorehdr_index = image->nr_segments;
>> +	image->elfcorehdr_index_valid = true;
>> +#else
>>   	kbuf.memsz = kbuf.bufsz;
>> +#endif
>>   	kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
>>   	kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
>>   	ret = kexec_add_buffer(&kbuf);
>> @@ -413,3 +424,108 @@ int crash_load_segments(struct kimage *image)
>>   	return ret;
>>   }
>>   #endif /* CONFIG_KEXEC_FILE */
>> +
>> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
>> +int crash_hotplug_support(void) { return 1; }
>> +void *arch_map_crash_pages(unsigned long paddr, unsigned long size)
>> +{
>> +	/*
>> +	 * NOTE: The addresses and sizes passed to this routine have
>> +	 * already been fully aligned on page boundaries. There is no
>> +	 * need for massaging the address or size.
>> +	 */
>> +	void *ptr = NULL;
>> +
>> +	/* NOTE: requires arch_kexec_[un]protect_crashkres() for write
> access */
>> +	if (size > 0) {
>> +		struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
>> +
>> +		ptr = kmap(page);
>> +	}
> 
> kmap_local_page() is preferred over kmap() which should be avoided in new
> code (please take a look at highmem.rst which I recently updated).
> 
> Can you please check if you can easily avoid using kmap()? Otherwise, this
> code will be added to my (long) queue and changed one day or the other :-)
> 
> Thanks,
> 
> Fabio
Fabio,
I converted this to kmap_local_page()!
Thanks!
eric

> 
>> +
>> +	return ptr;
>> +}
>> +
>> +void arch_unmap_crash_pages(void **ptr)
>> +{
>> +	if (ptr) {
>> +		if (*ptr)
>> +			kunmap(*ptr);
>> +		*ptr = NULL;
>> +	}
>> +}
>> +
>> +/**
>> + * arch_crash_handle_hotplug_event() - Handle hotplug elfcorehdr changes
>> + * @image: the active struct kimage
>> + * @hp_action: the hot un/plug action being handled
>> + * @cpu: when KEXEC_CRASH_HP_ADD/REMOVE_CPU, the cpu affected
>> + *
>> + * To accurately reflect hot un/plug changes, the elfcorehdr (which
>> + * is passed to the crash kernel via the elfcorehdr= parameter)
>> + * must be updated with the new list of CPUs and memories. The new
>> + * elfcorehdr is prepared in a kernel buffer, and then it is
>> + * written on top of the existing/old elfcorehdr.
>> + *
>> + * For hotplug changes to elfcorehdr to work, two conditions are
>> + * needed:
>> + * First, the segment containing the elfcorehdr must be large enough
>> + * to permit a growing number of resources. See the
>> + * CONFIG_CRASH_MAX_MEMORY_RANGES description.
>> + * Second, purgatory must explicitly exclude the elfcorehdr from the
>> + * list of segments it checks (since the elfcorehdr changes and thus
>> + * would require an update to purgatory itself to update the digest).
>> + *
>> + */
>> +void arch_crash_handle_hotplug_event(struct kimage *image,
>> +	unsigned int hp_action, unsigned int cpu)
>> +{
>> +	struct kexec_segment *ksegment;
>> +	unsigned char *ptr = NULL;
>> +	unsigned long elfsz = 0;
>> +	void *elfbuf = NULL;
>> +	unsigned long mem, memsz;
>> +
>> +	/*
>> +	 * Elfcorehdr_index_valid checked in
> crash_core:handle_hotplug_event()
>> +	 */
>> +	ksegment = &image->segment[image->elfcorehdr_index];
>> +	mem = ksegment->mem;
>> +	memsz = ksegment->memsz;
>> +
>> +	/*
>> +	 * Create the new elfcorehdr reflecting the changes to CPU and/or
>> +	 * memory resources.
>> +	 */
>> +	if (prepare_elf_headers(image, &elfbuf, &elfsz)) {
>> +		pr_err("crash hp: unable to prepare elfcore headers");
>> +		goto out;
>> +	}
>> +	if (elfsz > memsz) {
>> +		pr_err("crash hp: update elfcorehdr elfsz %lu > memsz
> %lu",
>> +			elfsz, memsz);
>> +		goto out;
>> +	}
>> +
>> +	/*
>> +	 * At this point, we are all but assured of success.
>> +	 * Copy new elfcorehdr into destination.
>> +	 */
>> +	ptr = arch_map_crash_pages(mem, memsz);
>> +	if (ptr) {
>> +		/*
>> +		 * Temporarily invalidate the crash image while the
>> +		 * elfcorehdr is updated.
>> +		 */
>> +		xchg(&kexec_crash_image, NULL);
>> +		memcpy_flushcache((void *)ptr, elfbuf, elfsz);
>> +		xchg(&kexec_crash_image, image);
>> +	}
>> +	arch_unmap_crash_pages((void **)&ptr);
>> +	pr_debug("crash hp: re-loaded elfcorehdr at 0x%lx\n", mem);
>> +
>> +out:
>> +	if (elfbuf)
>> +		vfree(elfbuf);
>> +}
>> +#endif
>> -- 
>> 2.31.1
>>
>>
> 
> 
> 
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support
  2022-07-08 21:07     ` Eric DeVolder
@ 2022-07-09 18:41       ` Fabio M. De Francesco
  0 siblings, 0 replies; 20+ messages in thread
From: Fabio M. De Francesco @ 2022-07-09 18:41 UTC (permalink / raw)
  To: linux-kernel, x86, kexec, ebiederm, dyoung, bhe, vgoyal, Eric DeVolder
  Cc: tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky,
	ira.weiny

On venerdì 8 luglio 2022 23:07:20 CEST Eric DeVolder wrote:
> 
> On 7/8/22 14:38, Fabio M. De Francesco wrote:
> > On martedì 14 giugno 2022 00:42:40 CEST Eric DeVolder wrote:
> >> For x86_64, when CPU or memory is hot un/plugged, the crash
> >> elfcorehdr, which describes the CPUs and memory in the system,
> >> must also be updated.
> >>
> >> When loading the crash kernel via kexec_load or kexec_file_load,
> >> the elfcorehdr is identified at run time in
> >> crash_core:handle_hotplug_event().
> >>
> >> To update the elfcorehdr for x86_64, a new elfcorehdr must be
> >> generated from the available CPUs and memory. The new elfcorehdr
> >> is prepared into a buffer, and then installed over the top of
> >> the existing elfcorehdr.
> >>
> >> In the patch 'kexec: exclude elfcorehdr from the segment digest'
> >> the need to update purgatory due to the change in elfcorehdr was
> >> eliminated.  As a result, no changes to purgatory or boot_params
> >> (as the elfcorehdr= kernel command line parameter pointer
> >> remains unchanged and correct) are needed, just elfcorehdr.
> >>
> >> To accommodate a growing number of resources via hotplug, the
> >> elfcorehdr segment must be sufficiently large enough to accommodate
> >> changes, see the CRASH_MAX_MEMORY_RANGES configure item.
> >>
> >> With this change, crash hotplug for kexec_file_load syscall
> >> is supported. The kexec_load is also supported, but also
> >> requires a corresponding change to userspace kexec-tools.
> >>
> >> Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
> >> ---
> >>   arch/x86/Kconfig        |  11 ++++
> >>   arch/x86/kernel/crash.c | 116 ++++++++++++++++++++++++++++++++++++++
++
> >>   2 files changed, 127 insertions(+)
> >>
> >> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> >> index 762a0b6ab8b6..e9eecff3b97d 100644
> >> --- a/arch/x86/Kconfig
> >> +++ b/arch/x86/Kconfig
> >> @@ -2082,6 +2082,17 @@ config CRASH_DUMP
> >>   	  (CONFIG_RELOCATABLE=y).
> >>   	  For more details see Documentation/admin-guide/kdump/kdump.rst
> >>   
> >> +config CRASH_MAX_MEMORY_RANGES
> >> +	depends on CRASH_DUMP && KEXEC_FILE && (HOTPLUG_CPU ||
> > MEMORY_HOTPLUG)
> >> +	int
> >> +	default 32768
> >> +	help
> >> +	  For the kexec_file_load path, specify the maximum number of
> >> +	  memory regions, eg. as represented by the 'System RAM' entries
> >> +	  in /proc/iomem, that the elfcorehdr buffer/segment can
> > accommodate.
> >> +	  This value is combined with NR_CPUS and multiplied by
> > Elf64_Phdr
> >> +	  size to determine the final buffer size.
> >> +
> >>   config KEXEC_JUMP
> >>   	bool "kexec jump"
> >>   	depends on KEXEC && HIBERNATION
> >> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> >> index 9db41cce8d97..b9cdf7a0d868 100644
> >> --- a/arch/x86/kernel/crash.c
> >> +++ b/arch/x86/kernel/crash.c
> >> @@ -25,6 +25,7 @@
> >>   #include <linux/slab.h>
> >>   #include <linux/vmalloc.h>
> >>   #include <linux/memblock.h>
> >> +#include <linux/highmem.h>
> >>   
> >>   #include <asm/processor.h>
> >>   #include <asm/hardirq.h>
> >> @@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
> >>   	image->elf_headers = kbuf.buffer;
> >>   	image->elf_headers_sz = kbuf.bufsz;
> >>   
> >> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
> >> +	/* Ensure elfcorehdr segment large enough for hotplug changes */
> >> +	kbuf.memsz = (CONFIG_NR_CPUS_DEFAULT +
> > CONFIG_CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
> >> +	/* For marking as usable to crash kernel */
> >> +	image->elf_headers_sz = kbuf.memsz;
> >> +	/* Record the index of the elfcorehdr segment */
> >> +	image->elfcorehdr_index = image->nr_segments;
> >> +	image->elfcorehdr_index_valid = true;
> >> +#else
> >>   	kbuf.memsz = kbuf.bufsz;
> >> +#endif
> >>   	kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
> >>   	kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
> >>   	ret = kexec_add_buffer(&kbuf);
> >> @@ -413,3 +424,108 @@ int crash_load_segments(struct kimage *image)
> >>   	return ret;
> >>   }
> >>   #endif /* CONFIG_KEXEC_FILE */
> >> +
> >> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
> >> +int crash_hotplug_support(void) { return 1; }
> >> +void *arch_map_crash_pages(unsigned long paddr, unsigned long size)
> >> +{
> >> +	/*
> >> +	 * NOTE: The addresses and sizes passed to this routine have
> >> +	 * already been fully aligned on page boundaries. There is no
> >> +	 * need for massaging the address or size.
> >> +	 */
> >> +	void *ptr = NULL;
> >> +
> >> +	/* NOTE: requires arch_kexec_[un]protect_crashkres() for write
> > access */
> >> +	if (size > 0) {
> >> +		struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
> >> +
> >> +		ptr = kmap(page);
> >> +	}
> > 
> > kmap_local_page() is preferred over kmap() which should be avoided in 
new
> > code (please take a look at highmem.rst which I recently updated).
> > 
> > Can you please check if you can easily avoid using kmap()? Otherwise, 
this
> > code will be added to my (long) queue and changed one day or the other 
:-)
> > 
> > Thanks,
> > 
> > Fabio
> Fabio,
> I converted this to kmap_local_page()!
> Thanks!
> eric
> 
> > 
Eric,

Thanks for the very quick reaction to my suggestion.

Fabio



_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support
  2022-07-07 12:55     ` Eric DeVolder
@ 2022-07-11  9:57       ` Baoquan He
  2022-07-12 20:31         ` Eric DeVolder
  0 siblings, 1 reply; 20+ messages in thread
From: Baoquan He @ 2022-07-11  9:57 UTC (permalink / raw)
  To: Eric DeVolder
  Cc: Sourabh Jain, linux-kernel, x86, kexec, ebiederm, dyoung, vgoyal,
	tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, konrad.wilk, boris.ostrovsky

On 07/07/22 at 07:55am, Eric DeVolder wrote:
> 
> 
> On 6/20/22 00:06, Sourabh Jain wrote:
> > Hello Eric,
> > 
> > On 14/06/22 04:12, Eric DeVolder wrote:
> > > For x86_64, when CPU or memory is hot un/plugged, the crash
> > > elfcorehdr, which describes the CPUs and memory in the system,
> > > must also be updated.
> > > 
> > > When loading the crash kernel via kexec_load or kexec_file_load,
> > > the elfcorehdr is identified at run time in
> > > crash_core:handle_hotplug_event().
> > > 
> > > To update the elfcorehdr for x86_64, a new elfcorehdr must be
> > > generated from the available CPUs and memory. The new elfcorehdr
> > > is prepared into a buffer, and then installed over the top of
> > > the existing elfcorehdr.
> > > 
> > > In the patch 'kexec: exclude elfcorehdr from the segment digest'
> > > the need to update purgatory due to the change in elfcorehdr was
> > > eliminated.  As a result, no changes to purgatory or boot_params
> > > (as the elfcorehdr= kernel command line parameter pointer
> > > remains unchanged and correct) are needed, just elfcorehdr.
> > > 
> > > To accommodate a growing number of resources via hotplug, the
> > > elfcorehdr segment must be sufficiently large enough to accommodate
> > > changes, see the CRASH_MAX_MEMORY_RANGES configure item.
> > > 
> > > With this change, crash hotplug for kexec_file_load syscall
> > > is supported. The kexec_load is also supported, but also
> > > requires a corresponding change to userspace kexec-tools.
> > > 
> > > Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
> > > ---
> > >   arch/x86/Kconfig        |  11 ++++
> > >   arch/x86/kernel/crash.c | 116 ++++++++++++++++++++++++++++++++++++++++
> > >   2 files changed, 127 insertions(+)
> > > 
> > > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> > > index 762a0b6ab8b6..e9eecff3b97d 100644
> > > --- a/arch/x86/Kconfig
> > > +++ b/arch/x86/Kconfig
> > > @@ -2082,6 +2082,17 @@ config CRASH_DUMP
> > >         (CONFIG_RELOCATABLE=y).
> > >         For more details see Documentation/admin-guide/kdump/kdump.rst
> > > +config CRASH_MAX_MEMORY_RANGES
> > > +    depends on CRASH_DUMP && KEXEC_FILE && (HOTPLUG_CPU || MEMORY_HOTPLUG)
> > > +    int
> > > +    default 32768
> > > +    help
> > > +      For the kexec_file_load path, specify the maximum number of
> > > +      memory regions, eg. as represented by the 'System RAM' entries
> > > +      in /proc/iomem, that the elfcorehdr buffer/segment can accommodate.
> > > +      This value is combined with NR_CPUS and multiplied by Elf64_Phdr
> > > +      size to determine the final buffer size.
> > > +
> > >   config KEXEC_JUMP
> > >       bool "kexec jump"
> > >       depends on KEXEC && HIBERNATION
> > > diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> > > index 9db41cce8d97..b9cdf7a0d868 100644
> > > --- a/arch/x86/kernel/crash.c
> > > +++ b/arch/x86/kernel/crash.c
> > > @@ -25,6 +25,7 @@
> > >   #include <linux/slab.h>
> > >   #include <linux/vmalloc.h>
> > >   #include <linux/memblock.h>
> > > +#include <linux/highmem.h>
> > >   #include <asm/processor.h>
> > >   #include <asm/hardirq.h>
> > > @@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
> > >       image->elf_headers = kbuf.buffer;
> > >       image->elf_headers_sz = kbuf.bufsz;
> > > +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
> > > +    /* Ensure elfcorehdr segment large enough for hotplug changes */
> > > +    kbuf.memsz = (CONFIG_NR_CPUS_DEFAULT + CONFIG_CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
> > > +    /* For marking as usable to crash kernel */
> > > +    image->elf_headers_sz = kbuf.memsz;
> > > +    /* Record the index of the elfcorehdr segment */
> > > +    image->elfcorehdr_index = image->nr_segments;
> > > +    image->elfcorehdr_index_valid = true;
> > > +#else
> > >       kbuf.memsz = kbuf.bufsz;
> > > +#endif
> > >       kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
> > >       kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
> > >       ret = kexec_add_buffer(&kbuf);
> > > @@ -413,3 +424,108 @@ int crash_load_segments(struct kimage *image)
> > >       return ret;
> > >   }
> > >   #endif /* CONFIG_KEXEC_FILE */
> > > +
> > > +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
> > > +int crash_hotplug_support(void) { return 1; }
> > > +void *arch_map_crash_pages(unsigned long paddr, unsigned long size)
> > > +{
> > > +    /*
> > > +     * NOTE: The addresses and sizes passed to this routine have
> > > +     * already been fully aligned on page boundaries. There is no
> > > +     * need for massaging the address or size.
> > > +     */
> > > +    void *ptr = NULL;
> > > +
> > > +    /* NOTE: requires arch_kexec_[un]protect_crashkres() for write access */
> > > +    if (size > 0) {
> > > +        struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
> > > +
> > > +        ptr = kmap(page);
> > > +    }
> > > +
> > > +    return ptr;
> > > +}
> > > +
> > > +void arch_unmap_crash_pages(void **ptr)
> > > +{
> > > +    if (ptr) {
> > > +        if (*ptr)
> > > +            kunmap(*ptr);
> > > +        *ptr = NULL;
> > > +    }
> > > +}
> > 
> > Aren't arch will have build issue if arch_[un]map_crash_pages methods are not defined?
> Sourabh,
> Yes, you are correct. I'll add __weak versions of each in crash_core.c in the next patch.

Just a reminder, __weak is deprecated and has been cleaned up in
kernel/kexec*.c in below patch.

[PATCH 0/2] kexec: Drop __weak attributes from functions


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support
  2022-07-11  9:57       ` Baoquan He
@ 2022-07-12 20:31         ` Eric DeVolder
  0 siblings, 0 replies; 20+ messages in thread
From: Eric DeVolder @ 2022-07-12 20:31 UTC (permalink / raw)
  To: Baoquan He
  Cc: Sourabh Jain, linux-kernel, x86, kexec, ebiederm, dyoung, vgoyal,
	tglx, mingo, bp, dave.hansen, hpa, nramas, thomas.lendacky, robh,
	efault, rppt, david, konrad.wilk, boris.ostrovsky



On 7/11/22 04:57, Baoquan He wrote:
> On 07/07/22 at 07:55am, Eric DeVolder wrote:
>>
>>
>> On 6/20/22 00:06, Sourabh Jain wrote:
>>> Hello Eric,
>>>
>>> On 14/06/22 04:12, Eric DeVolder wrote:
>>>> For x86_64, when CPU or memory is hot un/plugged, the crash
>>>> elfcorehdr, which describes the CPUs and memory in the system,
>>>> must also be updated.
>>>>
>>>> When loading the crash kernel via kexec_load or kexec_file_load,
>>>> the elfcorehdr is identified at run time in
>>>> crash_core:handle_hotplug_event().
>>>>
>>>> To update the elfcorehdr for x86_64, a new elfcorehdr must be
>>>> generated from the available CPUs and memory. The new elfcorehdr
>>>> is prepared into a buffer, and then installed over the top of
>>>> the existing elfcorehdr.
>>>>
>>>> In the patch 'kexec: exclude elfcorehdr from the segment digest'
>>>> the need to update purgatory due to the change in elfcorehdr was
>>>> eliminated.  As a result, no changes to purgatory or boot_params
>>>> (as the elfcorehdr= kernel command line parameter pointer
>>>> remains unchanged and correct) are needed, just elfcorehdr.
>>>>
>>>> To accommodate a growing number of resources via hotplug, the
>>>> elfcorehdr segment must be sufficiently large enough to accommodate
>>>> changes, see the CRASH_MAX_MEMORY_RANGES configure item.
>>>>
>>>> With this change, crash hotplug for kexec_file_load syscall
>>>> is supported. The kexec_load is also supported, but also
>>>> requires a corresponding change to userspace kexec-tools.
>>>>
>>>> Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
>>>> ---
>>>>    arch/x86/Kconfig        |  11 ++++
>>>>    arch/x86/kernel/crash.c | 116 ++++++++++++++++++++++++++++++++++++++++
>>>>    2 files changed, 127 insertions(+)
>>>>
>>>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>>>> index 762a0b6ab8b6..e9eecff3b97d 100644
>>>> --- a/arch/x86/Kconfig
>>>> +++ b/arch/x86/Kconfig
>>>> @@ -2082,6 +2082,17 @@ config CRASH_DUMP
>>>>          (CONFIG_RELOCATABLE=y).
>>>>          For more details see Documentation/admin-guide/kdump/kdump.rst
>>>> +config CRASH_MAX_MEMORY_RANGES
>>>> +    depends on CRASH_DUMP && KEXEC_FILE && (HOTPLUG_CPU || MEMORY_HOTPLUG)
>>>> +    int
>>>> +    default 32768
>>>> +    help
>>>> +      For the kexec_file_load path, specify the maximum number of
>>>> +      memory regions, eg. as represented by the 'System RAM' entries
>>>> +      in /proc/iomem, that the elfcorehdr buffer/segment can accommodate.
>>>> +      This value is combined with NR_CPUS and multiplied by Elf64_Phdr
>>>> +      size to determine the final buffer size.
>>>> +
>>>>    config KEXEC_JUMP
>>>>        bool "kexec jump"
>>>>        depends on KEXEC && HIBERNATION
>>>> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
>>>> index 9db41cce8d97..b9cdf7a0d868 100644
>>>> --- a/arch/x86/kernel/crash.c
>>>> +++ b/arch/x86/kernel/crash.c
>>>> @@ -25,6 +25,7 @@
>>>>    #include <linux/slab.h>
>>>>    #include <linux/vmalloc.h>
>>>>    #include <linux/memblock.h>
>>>> +#include <linux/highmem.h>
>>>>    #include <asm/processor.h>
>>>>    #include <asm/hardirq.h>
>>>> @@ -398,7 +399,17 @@ int crash_load_segments(struct kimage *image)
>>>>        image->elf_headers = kbuf.buffer;
>>>>        image->elf_headers_sz = kbuf.bufsz;
>>>> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
>>>> +    /* Ensure elfcorehdr segment large enough for hotplug changes */
>>>> +    kbuf.memsz = (CONFIG_NR_CPUS_DEFAULT + CONFIG_CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
>>>> +    /* For marking as usable to crash kernel */
>>>> +    image->elf_headers_sz = kbuf.memsz;
>>>> +    /* Record the index of the elfcorehdr segment */
>>>> +    image->elfcorehdr_index = image->nr_segments;
>>>> +    image->elfcorehdr_index_valid = true;
>>>> +#else
>>>>        kbuf.memsz = kbuf.bufsz;
>>>> +#endif
>>>>        kbuf.buf_align = ELF_CORE_HEADER_ALIGN;
>>>>        kbuf.mem = KEXEC_BUF_MEM_UNKNOWN;
>>>>        ret = kexec_add_buffer(&kbuf);
>>>> @@ -413,3 +424,108 @@ int crash_load_segments(struct kimage *image)
>>>>        return ret;
>>>>    }
>>>>    #endif /* CONFIG_KEXEC_FILE */
>>>> +
>>>> +#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_MEMORY_HOTPLUG)
>>>> +int crash_hotplug_support(void) { return 1; }
>>>> +void *arch_map_crash_pages(unsigned long paddr, unsigned long size)
>>>> +{
>>>> +    /*
>>>> +     * NOTE: The addresses and sizes passed to this routine have
>>>> +     * already been fully aligned on page boundaries. There is no
>>>> +     * need for massaging the address or size.
>>>> +     */
>>>> +    void *ptr = NULL;
>>>> +
>>>> +    /* NOTE: requires arch_kexec_[un]protect_crashkres() for write access */
>>>> +    if (size > 0) {
>>>> +        struct page *page = pfn_to_page(paddr >> PAGE_SHIFT);
>>>> +
>>>> +        ptr = kmap(page);
>>>> +    }
>>>> +
>>>> +    return ptr;
>>>> +}
>>>> +
>>>> +void arch_unmap_crash_pages(void **ptr)
>>>> +{
>>>> +    if (ptr) {
>>>> +        if (*ptr)
>>>> +            kunmap(*ptr);
>>>> +        *ptr = NULL;
>>>> +    }
>>>> +}
>>>
>>> Aren't arch will have build issue if arch_[un]map_crash_pages methods are not defined?
>> Sourabh,
>> Yes, you are correct. I'll add __weak versions of each in crash_core.c in the next patch.
> 
> Just a reminder, __weak is deprecated and has been cleaned up in
> kernel/kexec*.c in below patch.
> 
> [PATCH 0/2] kexec: Drop __weak attributes from functions
> 
Baoquan,
Thanks for the pointer! I have corrected this code to do away with __weak and use the style 
presented in the kexec patch.
Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug
  2022-07-07 13:05     ` Eric DeVolder
@ 2022-07-20 19:08       ` Eric DeVolder
  2022-07-21  0:07         ` Baoquan He
  0 siblings, 1 reply; 20+ messages in thread
From: Eric DeVolder @ 2022-07-20 19:08 UTC (permalink / raw)
  To: Baoquan He
  Cc: linux-kernel, x86, kexec, ebiederm, dyoung, vgoyal, tglx, mingo,
	bp, dave.hansen, hpa, nramas, thomas.lendacky, robh, efault,
	rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky

Baoquan,
I believe I've addressed all feedback, just checking to see if you agree.
I have the next patch set ready in the event you think it a good idea to post it.
Thanks!
eric


On 7/7/22 08:05, Eric DeVolder wrote:
> 
> 
> On 7/5/22 20:16, Baoquan He wrote:
>> On 07/05/22 at 10:17am, Eric DeVolder wrote:
>>> Nudge...
>>> Thanks!
>>
>> Seems DavidH has concern and rasied it after you posted v9. Is that
>> solved?
> 
> In looking at David's comments on v8:
> - [PATCH v8 2/7] crash: prototype change for crash_prepare_elf64_headers
>    I adjusted per feedback and responded as such. He provided an Acked-by.
> 
> - [PATCH v8 3/7] crash: add generic infrastructure for crash hotplug support
>    He says do not use WARN* on expected error paths. I instead used WARN_ONCE(),
>    but I didn't consider this an error path. Nonetheless, I'll remove the
>    message altogether in v11.
> 
> - [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
>    David muses about a better name, but that was in response to Sourabh's
>    and I exchange on the sysfs attributes to use to report the presence
>    of this feature. In the end, I used something altogether different, as
>    presented in v10 (and the corresponding udev solution/change that goes
>    hand in hand with it).
> 
> Let me know if I've missed anything else? Sourabh points out a build problem
> for architectures without this feature support, so a v11 is warranted. But I'll
> wait to post that until I hear that I've addressed the previous questions.
> 
> Thanks,
> eric
> 
> 
>>
>>>
>>> On 6/13/22 17:42, Eric DeVolder wrote:
>>>> When the kdump service is loaded, if a CPU or memory is hot
>>>> un/plugged, the crash elfcorehdr (for x86), which describes the CPUs
>>>> and memory in the system, must also be updated, else the resulting
>>>> vmcore is inaccurate (eg. missing either CPU context or memory
>>>> regions).
>>>>
>>>> The current solution utilizes udev to initiate an unload-then-reload
>>>> of the kdump image (e. kernel, initrd, boot_params, puratory and
>>>> elfcorehdr) by the userspace kexec utility. In previous posts I have
>>>> outlined the significant performance problems related to offloading
>>>> this activity to userspace.
>>>>
>>>> This patchset introduces a generic crash hot un/plug handler that
>>>> registers with the CPU and memory notifiers. Upon CPU or memory
>>>> changes, this generic handler is invoked and performs important
>>>> housekeeping, for example obtaining the appropriate lock, and then
>>>> invokes an architecture specific handler to do the appropriate
>>>> updates.
>>>>
>>>> In the case of x86_64, the arch specific handler generates a new
>>>> elfcorehdr, and overwrites the old one in memory. No involvement
>>>> with userspace needed.
>>>>
>>>> To realize the benefits/test this patchset, one must make a couple
>>>> of minor changes to userspace:
>>>>
>>>>    - Prevent udev from updating kdump crash kernel on hot un/plug changes.
>>>>      Add the following as the first lines to the udev rule file
>>>>      /usr/lib/udev/rules.d/98-kexec.rules:
>>>>
>>>>      # The kernel handles updates to crash elfcorehdr for cpu and memory changes
>>>>      SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
>>>>      SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
>>>>
>>>>      These lines will cause cpu and memory hot un/plug events to be
>>>>      skipped within this rule file, if the kernel has these changes
>>>>      enabled.
>>>>
>>>>    - Change to the kexec_file_load for loading the kdump kernel:
>>>>      Eg. on RHEL: in /usr/bin/kdumpctl, change to:
>>>>       standard_kexec_args="-p -d -s"
>>>>      which adds the -s to select kexec_file_load syscall.
>>>>
>>>> This patchset supports kexec_load with a modified kexec userspace
>>>> utility, and a working changeset to the kexec userspace utility
>>>> is provided here (and to use, the above change to standard_kexec_args
>>>> would be, for example, to append --hotplug instead of -s).
>>>>
>>>>     diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
>>>>     index 9826f6d..4ed395a 100644
>>>>     --- a/kexec/arch/i386/crashdump-x86.c
>>>>     +++ b/kexec/arch/i386/crashdump-x86.c
>>>>     @@ -48,6 +48,7 @@
>>>>      #include <x86/x86-linux.h>
>>>>      extern struct arch_options_t arch_options;
>>>>     +extern int do_hotplug;
>>>>      static int get_kernel_page_offset(struct kexec_info *UNUSED(info),
>>>>                        struct crash_elf_info *elf_info)
>>>>     @@ -975,6 +976,14 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
>>>>          } else {
>>>>              memsz = bufsz;
>>>>          }
>>>>     +
>>>>     +    /* If hotplug support enabled, use larger size to accomodate changes */
>>>>     +    if (do_hotplug) {
>>>>     +        long int nr_cpus = get_nr_cpus();
>>>>     +        memsz = (nr_cpus + CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
>>>>     +    }
>>>>     +
>>>>     +    info->elfcorehdr =
>>>>          elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
>>>>                                  max_addr, -1);
>>>>          dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
>>>>     diff --git a/kexec/crashdump-elf.c b/kexec/crashdump-elf.c
>>>>     index b8bb686..5e29f7a 100644
>>>>     --- a/kexec/crashdump-elf.c
>>>>     +++ b/kexec/crashdump-elf.c
>>>>     @@ -43,11 +43,7 @@ int FUNC(struct kexec_info *info,
>>>>          int (*get_note_info)(int cpu, uint64_t *addr, uint64_t *len);
>>>>          long int count_cpu;
>>>>     -    if (xen_present())
>>>>     -        nr_cpus = xen_get_nr_phys_cpus();
>>>>     -    else
>>>>     -        nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
>>>>     -
>>>>     +    nr_cpus = get_nr_cpus();
>>>>          if (nr_cpus < 0) {
>>>>              return -1;
>>>>          }
>>>>     diff --git a/kexec/crashdump.h b/kexec/crashdump.h
>>>>     index 18bd691..28d3278 100644
>>>>     --- a/kexec/crashdump.h
>>>>     +++ b/kexec/crashdump.h
>>>>     @@ -57,7 +57,6 @@ unsigned long phys_to_virt(struct crash_elf_info *elf_info,
>>>>                     unsigned long long paddr);
>>>>      unsigned long xen_architecture(struct crash_elf_info *elf_info);
>>>>     -int xen_get_nr_phys_cpus(void);
>>>>      int xen_get_note(int cpu, uint64_t *addr, uint64_t *len);
>>>>      int xen_get_crashkernel_region(uint64_t *start, uint64_t *end);
>>>>     diff --git a/kexec/kexec-xen.h b/kexec/kexec-xen.h
>>>>     index 70fb576..f54a2dd 100644
>>>>     --- a/kexec/kexec-xen.h
>>>>     +++ b/kexec/kexec-xen.h
>>>>     @@ -83,5 +83,6 @@ extern int __xc_interface_close(xc_interface *xch);
>>>>      #endif
>>>>      int xen_get_kexec_range(int range, uint64_t *start, uint64_t *end);
>>>>     +int xen_get_nr_phys_cpus(void);
>>>>      #endif /* KEXEC_XEN_H */
>>>>     diff --git a/kexec/kexec.c b/kexec/kexec.c
>>>>     index 829a6ea..3668b73 100644
>>>>     --- a/kexec/kexec.c
>>>>     +++ b/kexec/kexec.c
>>>>     @@ -58,6 +58,7 @@
>>>>      unsigned long long mem_min = 0;
>>>>      unsigned long long mem_max = ULONG_MAX;
>>>>     +int do_hotplug = 0;
>>>>      static unsigned long kexec_flags = 0;
>>>>      /* Flags for kexec file (fd) based syscall */
>>>>      static unsigned long kexec_file_flags = 0;
>>>>     @@ -489,6 +490,17 @@ static int add_backup_segments(struct kexec_info *info,
>>>>          return 0;
>>>>      }
>>>>     +long int get_nr_cpus(void)
>>>>     +{
>>>>     +    long int nr_cpus;
>>>>     +
>>>>     +    if (xen_present())
>>>>     +        nr_cpus = xen_get_nr_phys_cpus();
>>>>     +    else
>>>>     +        nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
>>>>     +    return nr_cpus;
>>>>     +}
>>>>     +
>>>>      static char *slurp_fd(int fd, const char *filename, off_t size, off_t *nread)
>>>>      {
>>>>          char *buf;
>>>>     @@ -672,6 +684,14 @@ static void update_purgatory(struct kexec_info *info)
>>>>              if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
>>>>                  continue;
>>>>              }
>>>>     +
>>>>     +        /* Don't include elfcorehdr in the checksum, if hotplug
>>>>     +         * support enabled.
>>>>     +         */
>>>>     +        if (do_hotplug && (info->segment[i].mem == (void *)info->elfcorehdr)) {
>>>>     +            continue;
>>>>     +        }
>>>>     +
>>>>              sha256_update(&ctx, info->segment[i].buf,
>>>>                        info->segment[i].bufsz);
>>>>              nullsz = info->segment[i].memsz - info->segment[i].bufsz;
>>>>     @@ -1565,6 +1585,9 @@ int main(int argc, char *argv[])
>>>>              case OPT_PRINT_CKR_SIZE:
>>>>                  print_crashkernel_region_size();
>>>>                  return 0;
>>>>     +        case OPT_HOTPLUG:
>>>>     +            do_hotplug = 1;
>>>>     +            break;
>>>>              default:
>>>>                  break;
>>>>              }
>>>>     diff --git a/kexec/kexec.h b/kexec/kexec.h
>>>>     index 0f97a97..b0428cc 100644
>>>>     --- a/kexec/kexec.h
>>>>     +++ b/kexec/kexec.h
>>>>     @@ -169,6 +169,7 @@ struct kexec_info {
>>>>          int command_line_len;
>>>>          int skip_checks;
>>>>     +    unsigned long elfcorehdr;
>>>>      };
>>>>      struct arch_map_entry {
>>>>     @@ -231,7 +232,8 @@ extern int file_types;
>>>>      #define OPT_PRINT_CKR_SIZE    262
>>>>      #define OPT_LOAD_LIVE_UPDATE    263
>>>>      #define OPT_EXEC_LIVE_UPDATE    264
>>>>     -#define OPT_MAX            265
>>>>     +#define OPT_HOTPLUG        265
>>>>     +#define OPT_MAX        266
>>>>      #define KEXEC_OPTIONS \
>>>>          { "help",        0, 0, OPT_HELP }, \
>>>>          { "version",        0, 0, OPT_VERSION }, \
>>>>     @@ -258,6 +260,7 @@ extern int file_types;
>>>>          { "debug",        0, 0, OPT_DEBUG }, \
>>>>          { "status",        0, 0, OPT_STATUS }, \
>>>>          { "print-ckr-size",     0, 0, OPT_PRINT_CKR_SIZE }, \
>>>>     +    { "hotplug",        0, 0, OPT_HOTPLUG }, \
>>>>      #define KEXEC_OPT_STR "h?vdfixyluet:pscaS"
>>>>     @@ -290,6 +293,8 @@ extern unsigned long add_buffer_phys_virt(struct kexec_info *info,
>>>>          int buf_end, int phys);
>>>>      extern void arch_reuse_initrd(void);
>>>>     +extern long int get_nr_cpus(void);
>>>>     +
>>>>      extern int ifdown(void);
>>>>      extern char purgatory[];
>>>>
>>>> Regards,
>>>> eric
>>>> ---
>>>>
>>>> v9: 13jun2022
>>>>    - Rebased to 5.18.0
>>>>    - Per Sourabh, moved crash_prepare_elf64_headers() into common
>>>>      crash_core.c to avoid compile issues with kexec_load only path.
>>>>    - Per David Hildebrand, replaced mutex_trylock() with mutex_lock().
>>>>    - Changed the __weak arch_crash_handle_hotplug_event() to utilize
>>>>      WARN_ONCE() instead of WARN(). Fix some formatting issues.
>>>>    - Per Sourabh, introduced sysfs attribute crash_hotplug for memory
>>>>      and CPUs; for use by userspace (udev) to determine if the kernel
>>>>      performs crash hot un/plug support.
>>>>    - Per Sourabh, moved the code detecting the elfcorehdr segment from
>>>>      arch/x86 into crash_core:handle_hotplug_event() so both kexec_load
>>>>      and kexec_file_load can benefit.
>>>>    - Updated userspace kexec-tools kexec utility to reflect change to
>>>>      using CRASH_MAX_MEMORY_RANGES and get_nr_cpus().
>>>>
>>>> v8: 5may2022
>>>>    https://lkml.org/lkml/2022/5/5/1133
>>>>    - Per Borislav Petkov, eliminated CONFIG_CRASH_HOTPLUG in favor
>>>>      of CONFIG_HOTPLUG_CPU || CONFIG_MEMORY_HOTPLUG, ie a new define
>>>>      is not needed. Also use of IS_ENABLED() rather than #ifdef's.
>>>>      Renamed crash_hotplug_handler() to handle_hotplug_event().
>>>>      And other corrections.
>>>>    - Per Baoquan, minimized the parameters to the arch_crash_
>>>>      handle_hotplug_event() to hp_action and cpu.
>>>>    - Introduce KEXEC_CRASH_HP_INVALID_CPU definition, per Baoquan.
>>>>    - Per Sourabh Jain, renamed and repurposed CRASH_HOTPLUG_ELFCOREHDR_SZ
>>>>      to CONFIG_CRASH_MAX_MEMORY_RANGES, mirroring kexec-tools change
>>>>      by David Hildebrand. Folded this patch into the x86
>>>>      kexec_file_load support patch.
>>>>
>>>> v7: 13apr2022
>>>>    https://lkml.org/lkml/2022/4/13/850
>>>>    - Resolved parameter usage to crash_hotplug_handler(), per Baoquan.
>>>>
>>>> v6: 1apr2022
>>>>    https://lkml.org/lkml/2022/4/1/1203
>>>>    - Reword commit messages and some comment cleanup per Baoquan.
>>>>    - Changed elf_index to elfcorehdr_index for clarity.
>>>>    - Minor code changes per Baoquan.
>>>>
>>>> v5: 3mar2022
>>>>    https://lkml.org/lkml/2022/3/3/674
>>>>    - Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per
>>>>      David Hildenbrand.
>>>>    - Refactored slightly a few patches per Baoquan recommendation.
>>>>
>>>> v4: 9feb2022
>>>>    https://lkml.org/lkml/2022/2/9/1406
>>>>    - Refactored patches per Baoquan suggestsions.
>>>>    - A few corrections, per Baoquan.
>>>>
>>>> v3: 10jan2022
>>>>    https://lkml.org/lkml/2022/1/10/1212
>>>>    - Rebasing per Baoquan He request.
>>>>    - Changed memory notifier per David Hildenbrand.
>>>>    - Providing example kexec userspace change in cover letter.
>>>>
>>>> RFC v2: 7dec2021
>>>>    https://lkml.org/lkml/2021/12/7/1088
>>>>    - Acting upon Baoquan He suggestion of removing elfcorehdr from
>>>>      the purgatory list of segments, removed purgatory code from
>>>>      patchset, and it is signficiantly simpler now.
>>>>
>>>> RFC v1: 18nov2021
>>>>    https://lkml.org/lkml/2021/11/18/845
>>>>    - working patchset demonstrating kernel handling of hotplug
>>>>      updates to x86 elfcorehdr for kexec_file_load
>>>>
>>>> RFC: 14dec2020
>>>>    https://lkml.org/lkml/2020/12/14/532
>>>>    - proposed concept of allowing kernel to handle hotplug update
>>>>      of elfcorehdr
>>>> ---
>>>>
>>>>
>>>> Eric DeVolder (7):
>>>>     crash: move crash_prepare_elf64_headers
>>>>     crash: prototype change for crash_prepare_elf64_headers
>>>>     crash: add generic infrastructure for crash hotplug support
>>>>     kexec: exclude elfcorehdr from the segment digest
>>>>     kexec: exclude hot remove cpu from elfcorehdr notes
>>>>     crash: memory and cpu hotplug sysfs attributes
>>>>     x86/crash: Add x86 crash hotplug support
>>>>
>>>>    .../admin-guide/mm/memory-hotplug.rst         |   8 +
>>>>    Documentation/core-api/cpu_hotplug.rst        |  18 ++
>>>>    arch/arm64/kernel/machine_kexec_file.c        |   6 +-
>>>>    arch/powerpc/kexec/file_load_64.c             |   2 +-
>>>>    arch/x86/Kconfig                              |  11 +
>>>>    arch/x86/kernel/crash.c                       | 119 ++++++++-
>>>>    drivers/base/cpu.c                            |  13 +
>>>>    drivers/base/memory.c                         |  13 +
>>>>    include/linux/crash_core.h                    |  12 +
>>>>    include/linux/kexec.h                         |  14 +-
>>>>    kernel/crash_core.c                           | 241 ++++++++++++++++++
>>>>    kernel/kexec_file.c                           | 105 +-------
>>>>    12 files changed, 456 insertions(+), 106 deletions(-)
>>>>
>>>
>>

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug
  2022-07-20 19:08       ` Eric DeVolder
@ 2022-07-21  0:07         ` Baoquan He
  0 siblings, 0 replies; 20+ messages in thread
From: Baoquan He @ 2022-07-21  0:07 UTC (permalink / raw)
  To: Eric DeVolder
  Cc: linux-kernel, x86, kexec, ebiederm, dyoung, vgoyal, tglx, mingo,
	bp, dave.hansen, hpa, nramas, thomas.lendacky, robh, efault,
	rppt, david, sourabhjain, konrad.wilk, boris.ostrovsky

On 07/20/22 at 02:08pm, Eric DeVolder wrote:
> Baoquan,
> I believe I've addressed all feedback, just checking to see if you agree.
> I have the next patch set ready in the event you think it a good idea to post it.
> Thanks!

Thanks for the effort. Please post them for reviewing. The newly made
change in this rounds looks good to me, esp the sysfs adding is very
great.

> 
> 
> On 7/7/22 08:05, Eric DeVolder wrote:
> > 
> > 
> > On 7/5/22 20:16, Baoquan He wrote:
> > > On 07/05/22 at 10:17am, Eric DeVolder wrote:
> > > > Nudge...
> > > > Thanks!
> > > 
> > > Seems DavidH has concern and rasied it after you posted v9. Is that
> > > solved?
> > 
> > In looking at David's comments on v8:
> > - [PATCH v8 2/7] crash: prototype change for crash_prepare_elf64_headers
> >    I adjusted per feedback and responded as such. He provided an Acked-by.
> > 
> > - [PATCH v8 3/7] crash: add generic infrastructure for crash hotplug support
> >    He says do not use WARN* on expected error paths. I instead used WARN_ONCE(),
> >    but I didn't consider this an error path. Nonetheless, I'll remove the
> >    message altogether in v11.
> > 
> > - [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
> >    David muses about a better name, but that was in response to Sourabh's
> >    and I exchange on the sysfs attributes to use to report the presence
> >    of this feature. In the end, I used something altogether different, as
> >    presented in v10 (and the corresponding udev solution/change that goes
> >    hand in hand with it).
> > 
> > Let me know if I've missed anything else? Sourabh points out a build problem
> > for architectures without this feature support, so a v11 is warranted. But I'll
> > wait to post that until I hear that I've addressed the previous questions.
> > 
> > Thanks,
> > eric
> > 
> > 
> > > 
> > > > 
> > > > On 6/13/22 17:42, Eric DeVolder wrote:
> > > > > When the kdump service is loaded, if a CPU or memory is hot
> > > > > un/plugged, the crash elfcorehdr (for x86), which describes the CPUs
> > > > > and memory in the system, must also be updated, else the resulting
> > > > > vmcore is inaccurate (eg. missing either CPU context or memory
> > > > > regions).
> > > > > 
> > > > > The current solution utilizes udev to initiate an unload-then-reload
> > > > > of the kdump image (e. kernel, initrd, boot_params, puratory and
> > > > > elfcorehdr) by the userspace kexec utility. In previous posts I have
> > > > > outlined the significant performance problems related to offloading
> > > > > this activity to userspace.
> > > > > 
> > > > > This patchset introduces a generic crash hot un/plug handler that
> > > > > registers with the CPU and memory notifiers. Upon CPU or memory
> > > > > changes, this generic handler is invoked and performs important
> > > > > housekeeping, for example obtaining the appropriate lock, and then
> > > > > invokes an architecture specific handler to do the appropriate
> > > > > updates.
> > > > > 
> > > > > In the case of x86_64, the arch specific handler generates a new
> > > > > elfcorehdr, and overwrites the old one in memory. No involvement
> > > > > with userspace needed.
> > > > > 
> > > > > To realize the benefits/test this patchset, one must make a couple
> > > > > of minor changes to userspace:
> > > > > 
> > > > >    - Prevent udev from updating kdump crash kernel on hot un/plug changes.
> > > > >      Add the following as the first lines to the udev rule file
> > > > >      /usr/lib/udev/rules.d/98-kexec.rules:
> > > > > 
> > > > >      # The kernel handles updates to crash elfcorehdr for cpu and memory changes
> > > > >      SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
> > > > >      SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
> > > > > 
> > > > >      These lines will cause cpu and memory hot un/plug events to be
> > > > >      skipped within this rule file, if the kernel has these changes
> > > > >      enabled.
> > > > > 
> > > > >    - Change to the kexec_file_load for loading the kdump kernel:
> > > > >      Eg. on RHEL: in /usr/bin/kdumpctl, change to:
> > > > >       standard_kexec_args="-p -d -s"
> > > > >      which adds the -s to select kexec_file_load syscall.
> > > > > 
> > > > > This patchset supports kexec_load with a modified kexec userspace
> > > > > utility, and a working changeset to the kexec userspace utility
> > > > > is provided here (and to use, the above change to standard_kexec_args
> > > > > would be, for example, to append --hotplug instead of -s).
> > > > > 
> > > > >     diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
> > > > >     index 9826f6d..4ed395a 100644
> > > > >     --- a/kexec/arch/i386/crashdump-x86.c
> > > > >     +++ b/kexec/arch/i386/crashdump-x86.c
> > > > >     @@ -48,6 +48,7 @@
> > > > >      #include <x86/x86-linux.h>
> > > > >      extern struct arch_options_t arch_options;
> > > > >     +extern int do_hotplug;
> > > > >      static int get_kernel_page_offset(struct kexec_info *UNUSED(info),
> > > > >                        struct crash_elf_info *elf_info)
> > > > >     @@ -975,6 +976,14 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
> > > > >          } else {
> > > > >              memsz = bufsz;
> > > > >          }
> > > > >     +
> > > > >     +    /* If hotplug support enabled, use larger size to accomodate changes */
> > > > >     +    if (do_hotplug) {
> > > > >     +        long int nr_cpus = get_nr_cpus();
> > > > >     +        memsz = (nr_cpus + CRASH_MAX_MEMORY_RANGES) * sizeof(Elf64_Phdr);
> > > > >     +    }
> > > > >     +
> > > > >     +    info->elfcorehdr =
> > > > >          elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
> > > > >                                  max_addr, -1);
> > > > >          dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
> > > > >     diff --git a/kexec/crashdump-elf.c b/kexec/crashdump-elf.c
> > > > >     index b8bb686..5e29f7a 100644
> > > > >     --- a/kexec/crashdump-elf.c
> > > > >     +++ b/kexec/crashdump-elf.c
> > > > >     @@ -43,11 +43,7 @@ int FUNC(struct kexec_info *info,
> > > > >          int (*get_note_info)(int cpu, uint64_t *addr, uint64_t *len);
> > > > >          long int count_cpu;
> > > > >     -    if (xen_present())
> > > > >     -        nr_cpus = xen_get_nr_phys_cpus();
> > > > >     -    else
> > > > >     -        nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
> > > > >     -
> > > > >     +    nr_cpus = get_nr_cpus();
> > > > >          if (nr_cpus < 0) {
> > > > >              return -1;
> > > > >          }
> > > > >     diff --git a/kexec/crashdump.h b/kexec/crashdump.h
> > > > >     index 18bd691..28d3278 100644
> > > > >     --- a/kexec/crashdump.h
> > > > >     +++ b/kexec/crashdump.h
> > > > >     @@ -57,7 +57,6 @@ unsigned long phys_to_virt(struct crash_elf_info *elf_info,
> > > > >                     unsigned long long paddr);
> > > > >      unsigned long xen_architecture(struct crash_elf_info *elf_info);
> > > > >     -int xen_get_nr_phys_cpus(void);
> > > > >      int xen_get_note(int cpu, uint64_t *addr, uint64_t *len);
> > > > >      int xen_get_crashkernel_region(uint64_t *start, uint64_t *end);
> > > > >     diff --git a/kexec/kexec-xen.h b/kexec/kexec-xen.h
> > > > >     index 70fb576..f54a2dd 100644
> > > > >     --- a/kexec/kexec-xen.h
> > > > >     +++ b/kexec/kexec-xen.h
> > > > >     @@ -83,5 +83,6 @@ extern int __xc_interface_close(xc_interface *xch);
> > > > >      #endif
> > > > >      int xen_get_kexec_range(int range, uint64_t *start, uint64_t *end);
> > > > >     +int xen_get_nr_phys_cpus(void);
> > > > >      #endif /* KEXEC_XEN_H */
> > > > >     diff --git a/kexec/kexec.c b/kexec/kexec.c
> > > > >     index 829a6ea..3668b73 100644
> > > > >     --- a/kexec/kexec.c
> > > > >     +++ b/kexec/kexec.c
> > > > >     @@ -58,6 +58,7 @@
> > > > >      unsigned long long mem_min = 0;
> > > > >      unsigned long long mem_max = ULONG_MAX;
> > > > >     +int do_hotplug = 0;
> > > > >      static unsigned long kexec_flags = 0;
> > > > >      /* Flags for kexec file (fd) based syscall */
> > > > >      static unsigned long kexec_file_flags = 0;
> > > > >     @@ -489,6 +490,17 @@ static int add_backup_segments(struct kexec_info *info,
> > > > >          return 0;
> > > > >      }
> > > > >     +long int get_nr_cpus(void)
> > > > >     +{
> > > > >     +    long int nr_cpus;
> > > > >     +
> > > > >     +    if (xen_present())
> > > > >     +        nr_cpus = xen_get_nr_phys_cpus();
> > > > >     +    else
> > > > >     +        nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
> > > > >     +    return nr_cpus;
> > > > >     +}
> > > > >     +
> > > > >      static char *slurp_fd(int fd, const char *filename, off_t size, off_t *nread)
> > > > >      {
> > > > >          char *buf;
> > > > >     @@ -672,6 +684,14 @@ static void update_purgatory(struct kexec_info *info)
> > > > >              if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
> > > > >                  continue;
> > > > >              }
> > > > >     +
> > > > >     +        /* Don't include elfcorehdr in the checksum, if hotplug
> > > > >     +         * support enabled.
> > > > >     +         */
> > > > >     +        if (do_hotplug && (info->segment[i].mem == (void *)info->elfcorehdr)) {
> > > > >     +            continue;
> > > > >     +        }
> > > > >     +
> > > > >              sha256_update(&ctx, info->segment[i].buf,
> > > > >                        info->segment[i].bufsz);
> > > > >              nullsz = info->segment[i].memsz - info->segment[i].bufsz;
> > > > >     @@ -1565,6 +1585,9 @@ int main(int argc, char *argv[])
> > > > >              case OPT_PRINT_CKR_SIZE:
> > > > >                  print_crashkernel_region_size();
> > > > >                  return 0;
> > > > >     +        case OPT_HOTPLUG:
> > > > >     +            do_hotplug = 1;
> > > > >     +            break;
> > > > >              default:
> > > > >                  break;
> > > > >              }
> > > > >     diff --git a/kexec/kexec.h b/kexec/kexec.h
> > > > >     index 0f97a97..b0428cc 100644
> > > > >     --- a/kexec/kexec.h
> > > > >     +++ b/kexec/kexec.h
> > > > >     @@ -169,6 +169,7 @@ struct kexec_info {
> > > > >          int command_line_len;
> > > > >          int skip_checks;
> > > > >     +    unsigned long elfcorehdr;
> > > > >      };
> > > > >      struct arch_map_entry {
> > > > >     @@ -231,7 +232,8 @@ extern int file_types;
> > > > >      #define OPT_PRINT_CKR_SIZE    262
> > > > >      #define OPT_LOAD_LIVE_UPDATE    263
> > > > >      #define OPT_EXEC_LIVE_UPDATE    264
> > > > >     -#define OPT_MAX            265
> > > > >     +#define OPT_HOTPLUG        265
> > > > >     +#define OPT_MAX        266
> > > > >      #define KEXEC_OPTIONS \
> > > > >          { "help",        0, 0, OPT_HELP }, \
> > > > >          { "version",        0, 0, OPT_VERSION }, \
> > > > >     @@ -258,6 +260,7 @@ extern int file_types;
> > > > >          { "debug",        0, 0, OPT_DEBUG }, \
> > > > >          { "status",        0, 0, OPT_STATUS }, \
> > > > >          { "print-ckr-size",     0, 0, OPT_PRINT_CKR_SIZE }, \
> > > > >     +    { "hotplug",        0, 0, OPT_HOTPLUG }, \
> > > > >      #define KEXEC_OPT_STR "h?vdfixyluet:pscaS"
> > > > >     @@ -290,6 +293,8 @@ extern unsigned long add_buffer_phys_virt(struct kexec_info *info,
> > > > >          int buf_end, int phys);
> > > > >      extern void arch_reuse_initrd(void);
> > > > >     +extern long int get_nr_cpus(void);
> > > > >     +
> > > > >      extern int ifdown(void);
> > > > >      extern char purgatory[];
> > > > > 
> > > > > Regards,
> > > > > eric
> > > > > ---
> > > > > 
> > > > > v9: 13jun2022
> > > > >    - Rebased to 5.18.0
> > > > >    - Per Sourabh, moved crash_prepare_elf64_headers() into common
> > > > >      crash_core.c to avoid compile issues with kexec_load only path.
> > > > >    - Per David Hildebrand, replaced mutex_trylock() with mutex_lock().
> > > > >    - Changed the __weak arch_crash_handle_hotplug_event() to utilize
> > > > >      WARN_ONCE() instead of WARN(). Fix some formatting issues.
> > > > >    - Per Sourabh, introduced sysfs attribute crash_hotplug for memory
> > > > >      and CPUs; for use by userspace (udev) to determine if the kernel
> > > > >      performs crash hot un/plug support.
> > > > >    - Per Sourabh, moved the code detecting the elfcorehdr segment from
> > > > >      arch/x86 into crash_core:handle_hotplug_event() so both kexec_load
> > > > >      and kexec_file_load can benefit.
> > > > >    - Updated userspace kexec-tools kexec utility to reflect change to
> > > > >      using CRASH_MAX_MEMORY_RANGES and get_nr_cpus().
> > > > > 
> > > > > v8: 5may2022
> > > > >    https://lkml.org/lkml/2022/5/5/1133
> > > > >    - Per Borislav Petkov, eliminated CONFIG_CRASH_HOTPLUG in favor
> > > > >      of CONFIG_HOTPLUG_CPU || CONFIG_MEMORY_HOTPLUG, ie a new define
> > > > >      is not needed. Also use of IS_ENABLED() rather than #ifdef's.
> > > > >      Renamed crash_hotplug_handler() to handle_hotplug_event().
> > > > >      And other corrections.
> > > > >    - Per Baoquan, minimized the parameters to the arch_crash_
> > > > >      handle_hotplug_event() to hp_action and cpu.
> > > > >    - Introduce KEXEC_CRASH_HP_INVALID_CPU definition, per Baoquan.
> > > > >    - Per Sourabh Jain, renamed and repurposed CRASH_HOTPLUG_ELFCOREHDR_SZ
> > > > >      to CONFIG_CRASH_MAX_MEMORY_RANGES, mirroring kexec-tools change
> > > > >      by David Hildebrand. Folded this patch into the x86
> > > > >      kexec_file_load support patch.
> > > > > 
> > > > > v7: 13apr2022
> > > > >    https://lkml.org/lkml/2022/4/13/850
> > > > >    - Resolved parameter usage to crash_hotplug_handler(), per Baoquan.
> > > > > 
> > > > > v6: 1apr2022
> > > > >    https://lkml.org/lkml/2022/4/1/1203
> > > > >    - Reword commit messages and some comment cleanup per Baoquan.
> > > > >    - Changed elf_index to elfcorehdr_index for clarity.
> > > > >    - Minor code changes per Baoquan.
> > > > > 
> > > > > v5: 3mar2022
> > > > >    https://lkml.org/lkml/2022/3/3/674
> > > > >    - Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per
> > > > >      David Hildenbrand.
> > > > >    - Refactored slightly a few patches per Baoquan recommendation.
> > > > > 
> > > > > v4: 9feb2022
> > > > >    https://lkml.org/lkml/2022/2/9/1406
> > > > >    - Refactored patches per Baoquan suggestsions.
> > > > >    - A few corrections, per Baoquan.
> > > > > 
> > > > > v3: 10jan2022
> > > > >    https://lkml.org/lkml/2022/1/10/1212
> > > > >    - Rebasing per Baoquan He request.
> > > > >    - Changed memory notifier per David Hildenbrand.
> > > > >    - Providing example kexec userspace change in cover letter.
> > > > > 
> > > > > RFC v2: 7dec2021
> > > > >    https://lkml.org/lkml/2021/12/7/1088
> > > > >    - Acting upon Baoquan He suggestion of removing elfcorehdr from
> > > > >      the purgatory list of segments, removed purgatory code from
> > > > >      patchset, and it is signficiantly simpler now.
> > > > > 
> > > > > RFC v1: 18nov2021
> > > > >    https://lkml.org/lkml/2021/11/18/845
> > > > >    - working patchset demonstrating kernel handling of hotplug
> > > > >      updates to x86 elfcorehdr for kexec_file_load
> > > > > 
> > > > > RFC: 14dec2020
> > > > >    https://lkml.org/lkml/2020/12/14/532
> > > > >    - proposed concept of allowing kernel to handle hotplug update
> > > > >      of elfcorehdr
> > > > > ---
> > > > > 
> > > > > 
> > > > > Eric DeVolder (7):
> > > > >     crash: move crash_prepare_elf64_headers
> > > > >     crash: prototype change for crash_prepare_elf64_headers
> > > > >     crash: add generic infrastructure for crash hotplug support
> > > > >     kexec: exclude elfcorehdr from the segment digest
> > > > >     kexec: exclude hot remove cpu from elfcorehdr notes
> > > > >     crash: memory and cpu hotplug sysfs attributes
> > > > >     x86/crash: Add x86 crash hotplug support
> > > > > 
> > > > >    .../admin-guide/mm/memory-hotplug.rst         |   8 +
> > > > >    Documentation/core-api/cpu_hotplug.rst        |  18 ++
> > > > >    arch/arm64/kernel/machine_kexec_file.c        |   6 +-
> > > > >    arch/powerpc/kexec/file_load_64.c             |   2 +-
> > > > >    arch/x86/Kconfig                              |  11 +
> > > > >    arch/x86/kernel/crash.c                       | 119 ++++++++-
> > > > >    drivers/base/cpu.c                            |  13 +
> > > > >    drivers/base/memory.c                         |  13 +
> > > > >    include/linux/crash_core.h                    |  12 +
> > > > >    include/linux/kexec.h                         |  14 +-
> > > > >    kernel/crash_core.c                           | 241 ++++++++++++++++++
> > > > >    kernel/kexec_file.c                           | 105 +-------
> > > > >    12 files changed, 456 insertions(+), 106 deletions(-)
> > > > > 
> > > > 
> > > 


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

^ permalink raw reply	[flat|nested] 20+ messages in thread
end of thread, other threads:[~2022-07-21  0:10 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-13 22:42 [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
2022-06-13 22:42 ` [PATCH v9 1/7] crash: move crash_prepare_elf64_headers Eric DeVolder
2022-06-13 22:42 ` [PATCH v9 2/7] crash: prototype change for crash_prepare_elf64_headers Eric DeVolder
2022-06-13 22:42 ` [PATCH v9 3/7] crash: add generic infrastructure for crash hotplug support Eric DeVolder
2022-06-13 22:42 ` [PATCH v9 4/7] kexec: exclude elfcorehdr from the segment digest Eric DeVolder
2022-06-13 22:42 ` [PATCH v9 5/7] kexec: exclude hot remove cpu from elfcorehdr notes Eric DeVolder
2022-06-13 22:42 ` [PATCH v9 6/7] crash: memory and cpu hotplug sysfs attributes Eric DeVolder
2022-06-13 22:42 ` [PATCH v9 7/7] x86/crash: Add x86 crash hotplug support Eric DeVolder
2022-06-20  5:06   ` Sourabh Jain
2022-07-07 12:55     ` Eric DeVolder
2022-07-11  9:57       ` Baoquan He
2022-07-12 20:31         ` Eric DeVolder
2022-07-08 19:38   ` Fabio M. De Francesco
2022-07-08 21:07     ` Eric DeVolder
2022-07-09 18:41       ` Fabio M. De Francesco
2022-07-05 15:17 ` [PATCH v9 0/7] crash: Kernel handling of CPU and memory hot un/plug Eric DeVolder
2022-07-06  1:16   ` Baoquan He
2022-07-07 13:05     ` Eric DeVolder
2022-07-20 19:08       ` Eric DeVolder
2022-07-21  0:07         ` Baoquan He

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).