From: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com> To: kexec@lists.infradead.org, linux-kernel@vger.kernel.org, ebiederm@xmission.com, vgoyal@redhat.com, kumagai-atsushi@mxc.nes.nec.co.jp Subject: [PATCH 0/2] kdump: Enter 2nd kernel with BSP for enabling multiple CPUs Date: Mon, 16 Apr 2012 11:21:28 +0900 [thread overview] Message-ID: <20120416021951.9303.58568.stgit@localhost6.localdomain6> (raw) Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP. Typical result of the operation I saw is the machine hanging during the 2nd kernel boot. To solve this issue, always enter 2nd kernel with BSP. To do this, I modify logic for shooting down CPUs. I use simple existing logic only in this mechanism, not complicating crash path to machine_kexec(). I did stress tests about 100 in total on the processors below: Intel(R) Xeon(R) CPU E7- 4820 @ 2.00GHz Socket x 4, Core x 8, Thread x 16 (160 LCPUS in total) Intel(R) Xeon(R) CPU E7- 8870 @ 2.40GHz Socket x 8, Core x 10, Thread x 20 (64 LCPUS in total) * Motivation of enabling multiple CPUs on the 2nd kernel This patch is aimed at doing parallel compression on the 2nd kernel. The machine that has more than tera bytes memory requires several hours to generate crash dump. There are several ways to reduce generation time of crash time, but they have different pros and cons: Fast I/O devices pros - Can obtain high-speed stably cons - Big financial cost for good performance I/O devices. It's difficult financially to prepare these for all environments as dump devices. Filtering pros - No financial cost. - Large reduction of crash dump size cons - Some data is definitely lost. So, we cannot use this on some situations: 1) High availability configuration where application triggers OS to crash and users want to debug the application later by retrieving the application's user process image from the system's crash dump. 2) KVM virtualization configuration where KVM host machine contains KVM guest machine images as user processes. 3) Page cache is needed for debugging filesystem related bugs. Compression pros - No financial cost. - No data lost. cons - Compression doesn't always reduce crash dump size. - take heavy CPU time. Slow if CPU is weak in speed. Machines with large memory tend to have a lot of CPUs. Parallel compression is sutable for parallel processing. My goal is to make compression as for free as possible. * TODO - Extend 512MB limit of reserved memory size for 2nd kernel for multiple CPUs. - Intel microcode patch loading on the 2nd kenrel is slow for the 2nd and later CPUs: about one or more minutes per one CPU. - There are a limited number of irq vectors for TLB flush IPI on x86_64: 32 for recent 3.x kernels and 8 for around 2.6.x kernels. So compression doesn't scale if a lot of page reclaim happens when reading kernel image larger than memory. Special handling without page cache could be applicable to parallel dump mechanism, but more investigation is needed. --- HATAYAMA Daisuke (2): Enter 2nd kernel with BSP Introduce crash ipi helpers to wait for APs to stop arch/x86/include/asm/reboot.h | 4 +++ arch/x86/kernel/crash.c | 15 +++++++++- arch/x86/kernel/reboot.c | 63 +++++++++++++++++++++++++++++------------ 3 files changed, 62 insertions(+), 20 deletions(-) -- HATAYAMA Daisuke
WARNING: multiple messages have this Message-ID (diff)
From: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com> To: kexec@lists.infradead.org, linux-kernel@vger.kernel.org, ebiederm@xmission.com, vgoyal@redhat.com, kumagai-atsushi@mxc.nes.nec.co.jp Subject: [PATCH 0/2] kdump: Enter 2nd kernel with BSP for enabling multiple CPUs Date: Mon, 16 Apr 2012 11:21:28 +0900 [thread overview] Message-ID: <20120416021951.9303.58568.stgit@localhost6.localdomain6> (raw) Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP. Typical result of the operation I saw is the machine hanging during the 2nd kernel boot. To solve this issue, always enter 2nd kernel with BSP. To do this, I modify logic for shooting down CPUs. I use simple existing logic only in this mechanism, not complicating crash path to machine_kexec(). I did stress tests about 100 in total on the processors below: Intel(R) Xeon(R) CPU E7- 4820 @ 2.00GHz Socket x 4, Core x 8, Thread x 16 (160 LCPUS in total) Intel(R) Xeon(R) CPU E7- 8870 @ 2.40GHz Socket x 8, Core x 10, Thread x 20 (64 LCPUS in total) * Motivation of enabling multiple CPUs on the 2nd kernel This patch is aimed at doing parallel compression on the 2nd kernel. The machine that has more than tera bytes memory requires several hours to generate crash dump. There are several ways to reduce generation time of crash time, but they have different pros and cons: Fast I/O devices pros - Can obtain high-speed stably cons - Big financial cost for good performance I/O devices. It's difficult financially to prepare these for all environments as dump devices. Filtering pros - No financial cost. - Large reduction of crash dump size cons - Some data is definitely lost. So, we cannot use this on some situations: 1) High availability configuration where application triggers OS to crash and users want to debug the application later by retrieving the application's user process image from the system's crash dump. 2) KVM virtualization configuration where KVM host machine contains KVM guest machine images as user processes. 3) Page cache is needed for debugging filesystem related bugs. Compression pros - No financial cost. - No data lost. cons - Compression doesn't always reduce crash dump size. - take heavy CPU time. Slow if CPU is weak in speed. Machines with large memory tend to have a lot of CPUs. Parallel compression is sutable for parallel processing. My goal is to make compression as for free as possible. * TODO - Extend 512MB limit of reserved memory size for 2nd kernel for multiple CPUs. - Intel microcode patch loading on the 2nd kenrel is slow for the 2nd and later CPUs: about one or more minutes per one CPU. - There are a limited number of irq vectors for TLB flush IPI on x86_64: 32 for recent 3.x kernels and 8 for around 2.6.x kernels. So compression doesn't scale if a lot of page reclaim happens when reading kernel image larger than memory. Special handling without page cache could be applicable to parallel dump mechanism, but more investigation is needed. --- HATAYAMA Daisuke (2): Enter 2nd kernel with BSP Introduce crash ipi helpers to wait for APs to stop arch/x86/include/asm/reboot.h | 4 +++ arch/x86/kernel/crash.c | 15 +++++++++- arch/x86/kernel/reboot.c | 63 +++++++++++++++++++++++++++++------------ 3 files changed, 62 insertions(+), 20 deletions(-) -- HATAYAMA Daisuke _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
next reply other threads:[~2012-04-16 2:21 UTC|newest] Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top 2012-04-16 2:21 HATAYAMA Daisuke [this message] 2012-04-16 2:21 ` [PATCH 0/2] kdump: Enter 2nd kernel with BSP for enabling multiple CPUs HATAYAMA Daisuke 2012-04-16 2:21 ` [PATCH 1/2] Introduce crash ipi helpers to wait for APs to stop HATAYAMA Daisuke 2012-04-16 2:21 ` HATAYAMA Daisuke 2012-04-16 2:21 ` [PATCH 2/2] Enter 2nd kernel with BSP HATAYAMA Daisuke 2012-04-16 2:21 ` HATAYAMA Daisuke 2012-04-23 10:46 ` Andi Kleen 2012-04-23 10:49 ` Andi Kleen 2012-04-24 2:05 ` HATAYAMA Daisuke 2012-04-24 3:04 ` Yu, Fenghua 2012-04-24 8:04 ` Andi Kleen 2012-04-24 10:46 ` Eric W. Biederman [not found] ` <beaa8ade-b7af-4e71-b4e0-a418ceb83f1e@email.android.com> 2012-04-16 6:40 ` [PATCH 0/2] kdump: Enter 2nd kernel with BSP for enabling multiple CPUs HATAYAMA Daisuke 2012-04-16 6:40 ` HATAYAMA Daisuke 2012-05-14 8:29 ` Cong Wang 2013-04-18 11:41 ` Petr Tesarik 2013-04-18 11:41 ` Petr Tesarik 2013-04-19 8:45 ` HATAYAMA Daisuke 2013-04-19 8:45 ` HATAYAMA Daisuke
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20120416021951.9303.58568.stgit@localhost6.localdomain6 \ --to=d.hatayama@jp.fujitsu.com \ --cc=ebiederm@xmission.com \ --cc=kexec@lists.infradead.org \ --cc=kumagai-atsushi@mxc.nes.nec.co.jp \ --cc=linux-kernel@vger.kernel.org \ --cc=vgoyal@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.