From: Tejun Heo <tj@kernel.org>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Ingo Molnar <mingo@elte.hu>, "H. Peter Anvin" <hpa@zytor.com>,
Thomas Gleixner <tglx@linutronix.de>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: [PATCH 3/3] x86: collect hot percpu variables into one cacheline
Date: Mon, 03 Aug 2009 14:12:19 +0900 [thread overview]
Message-ID: <4A7671B3.1050208@kernel.org> (raw)
In-Reply-To: <alpine.LFD.2.01.0908021913130.3352@localhost.localdomain>
On x86_64, percpu variables current_task and kernel_stack are used for
get_current() and current_thread_info() respectively and thus are
often used close to each other. Move definition of current_task to
kernel/cpu/common.c right above kernel_stack definition and align it
to cacheline so that they always fall into the same cacheline. Two
percpu variables defined there together - irq_stack_ptr and irq_count
- are also pretty hot and will benefit from sharing the cacheline.
For consistency, current_task definition for x86_32 is also moved to
kernel/cpu/common.c.
Putting current_task and kernel_stack into the same cacheline was
suggested by Linus Tolvards.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Linus Tolvards <torvalds@linux-foundation.org>
Cc: Ingo Molnar <mingo@elte.hu>
---
For now this should do but adding DEFINE_PER_CPU_HOT() definitely is a
viable option.
arch/x86/kernel/cpu/common.c | 15 +++++++++++++--
arch/x86/kernel/process_32.c | 3 ---
arch/x86/kernel/process_64.c | 3 ---
3 files changed, 13 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 12493c5..1bd88ed 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -987,13 +987,21 @@ struct desc_ptr idt_descr = { 256 * 16 - 1, (unsigned long) idt_table };
DEFINE_PER_CPU_FIRST(union irq_stack_union,
irq_stack_union) __aligned(PAGE_SIZE);
-DEFINE_PER_CPU(char *, irq_stack_ptr) =
- init_per_cpu_var(irq_stack_union.irq_stack) + IRQ_STACK_SIZE - 64;
+/*
+ * The following four percpu variables are hot. Align current_task to
+ * cacheline size such that all four fall in the same cacheline.
+ */
+DEFINE_PER_CPU(struct task_struct *, current_task) ____cacheline_aligned =
+ &init_task;
+EXPORT_PER_CPU_SYMBOL(current_task);
DEFINE_PER_CPU(unsigned long, kernel_stack) =
(unsigned long)&init_thread_union - KERNEL_STACK_OFFSET + THREAD_SIZE;
EXPORT_PER_CPU_SYMBOL(kernel_stack);
+DEFINE_PER_CPU(char *, irq_stack_ptr) =
+ init_per_cpu_var(irq_stack_union.irq_stack) + IRQ_STACK_SIZE - 64;
+
DEFINE_PER_CPU(unsigned int, irq_count) = -1;
/*
@@ -1041,6 +1049,9 @@ DEFINE_PER_CPU(struct orig_ist, orig_ist);
#else /* CONFIG_X86_64 */
+DEFINE_PER_CPU(struct task_struct *, current_task) = &init_task;
+EXPORT_PER_CPU_SYMBOL(current_task);
+
#ifdef CONFIG_CC_STACKPROTECTOR
DEFINE_PER_CPU(unsigned long, stack_canary);
#endif
diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
index 59f4524..daa4107 100644
--- a/arch/x86/kernel/process_32.c
+++ b/arch/x86/kernel/process_32.c
@@ -61,9 +61,6 @@
asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");
-DEFINE_PER_CPU(struct task_struct *, current_task) = &init_task;
-EXPORT_PER_CPU_SYMBOL(current_task);
-
/*
* Return saved PC of a blocked thread.
*/
diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index ebefb54..c4c675d 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -55,9 +55,6 @@
asmlinkage extern void ret_from_fork(void);
-DEFINE_PER_CPU(struct task_struct *, current_task) = &init_task;
-EXPORT_PER_CPU_SYMBOL(current_task);
-
DEFINE_PER_CPU(unsigned long, old_rsp);
static DEFINE_PER_CPU(unsigned char, is_idle);
--
1.6.0.2
next prev parent reply other threads:[~2009-08-03 5:12 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-07-31 18:13 [GIT PULL] Additional x86 fixes for 2.6.31-rc5 H. Peter Anvin
2009-07-31 19:45 ` Linus Torvalds
2009-07-31 19:57 ` Ingo Molnar
2009-08-01 19:28 ` Linus Torvalds
2009-08-01 19:38 ` H. Peter Anvin
2009-08-01 22:04 ` Linus Torvalds
2009-08-01 22:35 ` H. Peter Anvin
2009-08-02 1:20 ` Paul Mackerras
2009-08-02 3:52 ` H. Peter Anvin
2009-08-03 1:01 ` Tejun Heo
2009-08-03 1:14 ` Linus Torvalds
2009-08-03 1:49 ` Tejun Heo
2009-08-03 2:14 ` Linus Torvalds
2009-08-03 5:08 ` [PATCH 1/3] x86: Add 'percpu_read_stable()' interface for cacheable accesses Tejun Heo
2009-08-03 5:13 ` H. Peter Anvin
2009-08-03 5:18 ` Tejun Heo
2009-08-03 6:04 ` Ingo Molnar
2009-08-03 6:08 ` H. Peter Anvin
2009-08-03 6:16 ` Ingo Molnar
2009-08-03 7:00 ` Ingo Molnar
2009-08-03 15:13 ` [PATCH 1/3 UPDATED] x86, percpu: " Tejun Heo
2009-08-03 5:10 ` [PATCH 2/3] x86,percpu: fix DECLARE/DEFINE_PER_CPU_PAGE_ALIGNED() Tejun Heo
2009-08-03 5:12 ` Tejun Heo [this message]
2009-08-05 7:34 ` [GIT PULL] Additional x86 fixes for 2.6.31-rc5 Tan, Wei Chong
2009-08-05 8:06 ` Ingo Molnar
2009-08-10 0:42 ` Tan, Wei Chong
2009-08-10 9:05 ` Ingo Molnar
2009-08-10 15:32 ` Linus Torvalds
2009-08-10 9:06 ` [tip:x86/urgent] x86: Fix serialization in pit_expect_msb() tip-bot for Linus Torvalds
2009-08-10 18:01 ` tip-bot for Linus Torvalds
2009-08-05 23:10 ` [GIT PULL] Additional x86 fixes for 2.6.31-rc5 Tan, Wei Chong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4A7671B3.1050208@kernel.org \
--to=tj@kernel.org \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).