* [PATH v2 0/5] Code clean up for percpu_xxx serial functions @ 2012-01-13 15:53 Alex Shi 2012-01-13 15:53 ` [PATCH v2 1/5] net: use this_cpu_xxx replace percpu_xxx funcs Alex Shi 2012-05-04 21:29 ` [PATH v2 0/5] Code clean up for percpu_xxx serial functions Andrew Morton 0 siblings, 2 replies; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:53 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, konrad.wilk, hpa, jeremy I am sorry for spelling mistaken on konrad's email address, so resend for correct this. Please reply this resend email. --------------- Thanks for TJ's suggestion, I split the serial patch smaller for potential bisection convenience. Compare to v1 patch, this v2 patches has separate function replace patches and final dead code clean up patch. The net, xen and x86 part code are independent. After each part was accepted in kernel, the final(5th) clean up code do the real clean up in next merge window. I will refresh the patch at that time. Any further comments are appreciated! Regards Alex ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v2 1/5] net: use this_cpu_xxx replace percpu_xxx funcs 2012-01-13 15:53 [PATH v2 0/5] Code clean up for percpu_xxx serial functions Alex Shi @ 2012-01-13 15:53 ` Alex Shi 2012-01-13 15:53 ` [PATCH v2 2/5] xen: " Alex Shi 2012-05-04 21:29 ` [PATH v2 0/5] Code clean up for percpu_xxx serial functions Andrew Morton 1 sibling, 1 reply; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:53 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, konrad.wilk, hpa, jeremy percpu_xxx funcs are duplicated with this_cpu_xxx funcs, so replace them for further code clean up. And in preempt safe scenario, __this_cpu_xxx funcs has a bit better performance since __this_cpu_xxx has no redundant preempt_disable() Signed-off-by: Alex Shi <alex.shi@intel.com> Acked-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: David S. Miller <davem@davemloft.net> --- net/netfilter/xt_TEE.c | 12 ++++++------ net/socket.c | 4 ++-- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/net/netfilter/xt_TEE.c b/net/netfilter/xt_TEE.c index 3aae66f..f81eba4 100644 --- a/net/netfilter/xt_TEE.c +++ b/net/netfilter/xt_TEE.c @@ -87,7 +87,7 @@ tee_tg4(struct sk_buff *skb, const struct xt_action_param *par) const struct xt_tee_tginfo *info = par->targinfo; struct iphdr *iph; - if (percpu_read(tee_active)) + if (__this_cpu_read(tee_active)) return XT_CONTINUE; /* * Copy the skb, and route the copy. Will later return %XT_CONTINUE for @@ -124,9 +124,9 @@ tee_tg4(struct sk_buff *skb, const struct xt_action_param *par) ip_send_check(iph); if (tee_tg_route4(skb, info)) { - percpu_write(tee_active, true); + __this_cpu_write(tee_active, true); ip_local_out(skb); - percpu_write(tee_active, false); + __this_cpu_write(tee_active, false); } else { kfree_skb(skb); } @@ -167,7 +167,7 @@ tee_tg6(struct sk_buff *skb, const struct xt_action_param *par) { const struct xt_tee_tginfo *info = par->targinfo; - if (percpu_read(tee_active)) + if (__this_cpu_read(tee_active)) return XT_CONTINUE; skb = pskb_copy(skb, GFP_ATOMIC); if (skb == NULL) @@ -185,9 +185,9 @@ tee_tg6(struct sk_buff *skb, const struct xt_action_param *par) --iph->hop_limit; } if (tee_tg_route6(skb, info)) { - percpu_write(tee_active, true); + __this_cpu_write(tee_active, true); ip6_local_out(skb); - percpu_write(tee_active, false); + __this_cpu_write(tee_active, false); } else { kfree_skb(skb); } diff --git a/net/socket.c b/net/socket.c index 28a96af..784d93a 100644 --- a/net/socket.c +++ b/net/socket.c @@ -479,7 +479,7 @@ static struct socket *sock_alloc(void) inode->i_uid = current_fsuid(); inode->i_gid = current_fsgid(); - percpu_add(sockets_in_use, 1); + this_cpu_add(sockets_in_use, 1); return sock; } @@ -522,7 +522,7 @@ void sock_release(struct socket *sock) if (rcu_dereference_protected(sock->wq, 1)->fasync_list) printk(KERN_ERR "sock_release: fasync list not empty!\n"); - percpu_sub(sockets_in_use, 1); + this_cpu_sub(sockets_in_use, 1); if (!sock->file) { iput(SOCK_INODE(sock)); return; -- 1.6.3.3 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2 2/5] xen: use this_cpu_xxx replace percpu_xxx funcs 2012-01-13 15:53 ` [PATCH v2 1/5] net: use this_cpu_xxx replace percpu_xxx funcs Alex Shi @ 2012-01-13 15:53 ` Alex Shi 2012-01-13 15:53 ` [PATCH v2 3/5] x86: " Alex Shi 0 siblings, 1 reply; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:53 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, konrad.wilk, hpa, jeremy percpu_xxx funcs are duplicated with this_cpu_xxx funcs, so replace them for further code clean up. I don't know much of xen code. But, since the code is in x86 architecture, the percpu_xxx is exactly same as this_cpu_xxx serials functions. So, the change is safe. Signed-off-by: Alex Shi <alex.shi@intel.com> Acked-by: Christoph Lameter <cl@gentwo.org> Acked-by: Tejun Heo <tj@kernel.org> --- arch/x86/xen/enlighten.c | 6 +++--- arch/x86/xen/irq.c | 8 ++++---- arch/x86/xen/mmu.c | 20 ++++++++++---------- arch/x86/xen/multicalls.h | 2 +- arch/x86/xen/smp.c | 2 +- 5 files changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c index 12eb07b..312c9e3 100644 --- a/arch/x86/xen/enlighten.c +++ b/arch/x86/xen/enlighten.c @@ -777,11 +777,11 @@ static DEFINE_PER_CPU(unsigned long, xen_cr0_value); static unsigned long xen_read_cr0(void) { - unsigned long cr0 = percpu_read(xen_cr0_value); + unsigned long cr0 = this_cpu_read(xen_cr0_value); if (unlikely(cr0 == 0)) { cr0 = native_read_cr0(); - percpu_write(xen_cr0_value, cr0); + this_cpu_write(xen_cr0_value, cr0); } return cr0; @@ -791,7 +791,7 @@ static void xen_write_cr0(unsigned long cr0) { struct multicall_space mcs; - percpu_write(xen_cr0_value, cr0); + this_cpu_write(xen_cr0_value, cr0); /* Only pay attention to cr0.TS; everything else is ignored. */ diff --git a/arch/x86/xen/irq.c b/arch/x86/xen/irq.c index 8bbb465..1573376 100644 --- a/arch/x86/xen/irq.c +++ b/arch/x86/xen/irq.c @@ -26,7 +26,7 @@ static unsigned long xen_save_fl(void) struct vcpu_info *vcpu; unsigned long flags; - vcpu = percpu_read(xen_vcpu); + vcpu = this_cpu_read(xen_vcpu); /* flag has opposite sense of mask */ flags = !vcpu->evtchn_upcall_mask; @@ -50,7 +50,7 @@ static void xen_restore_fl(unsigned long flags) make sure we're don't switch CPUs between getting the vcpu pointer and updating the mask. */ preempt_disable(); - vcpu = percpu_read(xen_vcpu); + vcpu = this_cpu_read(xen_vcpu); vcpu->evtchn_upcall_mask = flags; preempt_enable_no_resched(); @@ -72,7 +72,7 @@ static void xen_irq_disable(void) make sure we're don't switch CPUs between getting the vcpu pointer and updating the mask. */ preempt_disable(); - percpu_read(xen_vcpu)->evtchn_upcall_mask = 1; + this_cpu_read(xen_vcpu)->evtchn_upcall_mask = 1; preempt_enable_no_resched(); } PV_CALLEE_SAVE_REGS_THUNK(xen_irq_disable); @@ -86,7 +86,7 @@ static void xen_irq_enable(void) the caller is confused and is trying to re-enable interrupts on an indeterminate processor. */ - vcpu = percpu_read(xen_vcpu); + vcpu = this_cpu_read(xen_vcpu); vcpu->evtchn_upcall_mask = 0; /* Doesn't matter if we get preempted here, because any diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c index 58a0e46..1a309ee 100644 --- a/arch/x86/xen/mmu.c +++ b/arch/x86/xen/mmu.c @@ -1071,14 +1071,14 @@ static void drop_other_mm_ref(void *info) struct mm_struct *mm = info; struct mm_struct *active_mm; - active_mm = percpu_read(cpu_tlbstate.active_mm); + active_mm = this_cpu_read(cpu_tlbstate.active_mm); - if (active_mm == mm && percpu_read(cpu_tlbstate.state) != TLBSTATE_OK) + if (active_mm == mm && this_cpu_read(cpu_tlbstate.state) != TLBSTATE_OK) leave_mm(smp_processor_id()); /* If this cpu still has a stale cr3 reference, then make sure it has been flushed. */ - if (percpu_read(xen_current_cr3) == __pa(mm->pgd)) + if (this_cpu_read(xen_current_cr3) == __pa(mm->pgd)) load_cr3(swapper_pg_dir); } @@ -1185,17 +1185,17 @@ static void __init xen_pagetable_setup_done(pgd_t *base) static void xen_write_cr2(unsigned long cr2) { - percpu_read(xen_vcpu)->arch.cr2 = cr2; + this_cpu_read(xen_vcpu)->arch.cr2 = cr2; } static unsigned long xen_read_cr2(void) { - return percpu_read(xen_vcpu)->arch.cr2; + return this_cpu_read(xen_vcpu)->arch.cr2; } unsigned long xen_read_cr2_direct(void) { - return percpu_read(xen_vcpu_info.arch.cr2); + return this_cpu_read(xen_vcpu_info.arch.cr2); } static void xen_flush_tlb(void) @@ -1278,12 +1278,12 @@ static void xen_flush_tlb_others(const struct cpumask *cpus, static unsigned long xen_read_cr3(void) { - return percpu_read(xen_cr3); + return this_cpu_read(xen_cr3); } static void set_current_cr3(void *v) { - percpu_write(xen_current_cr3, (unsigned long)v); + this_cpu_write(xen_current_cr3, (unsigned long)v); } static void __xen_write_cr3(bool kernel, unsigned long cr3) @@ -1306,7 +1306,7 @@ static void __xen_write_cr3(bool kernel, unsigned long cr3) xen_extend_mmuext_op(&op); if (kernel) { - percpu_write(xen_cr3, cr3); + this_cpu_write(xen_cr3, cr3); /* Update xen_current_cr3 once the batch has actually been submitted. */ @@ -1322,7 +1322,7 @@ static void xen_write_cr3(unsigned long cr3) /* Update while interrupts are disabled, so its atomic with respect to ipis */ - percpu_write(xen_cr3, cr3); + this_cpu_write(xen_cr3, cr3); __xen_write_cr3(true, cr3); diff --git a/arch/x86/xen/multicalls.h b/arch/x86/xen/multicalls.h index dee79b7..9c2e74f 100644 --- a/arch/x86/xen/multicalls.h +++ b/arch/x86/xen/multicalls.h @@ -47,7 +47,7 @@ static inline void xen_mc_issue(unsigned mode) xen_mc_flush(); /* restore flags saved in xen_mc_batch */ - local_irq_restore(percpu_read(xen_mc_irq_flags)); + local_irq_restore(this_cpu_read(xen_mc_irq_flags)); } /* Set up a callback to be called when the current batch is flushed */ diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c index 041d4fe..449f868 100644 --- a/arch/x86/xen/smp.c +++ b/arch/x86/xen/smp.c @@ -76,7 +76,7 @@ static void __cpuinit cpu_bringup(void) xen_setup_cpu_clockevents(); set_cpu_online(cpu, true); - percpu_write(cpu_state, CPU_ONLINE); + this_cpu_write(cpu_state, CPU_ONLINE); wmb(); /* We can take interrupts now: we're officially "up". */ -- 1.6.3.3 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2 3/5] x86: use this_cpu_xxx replace percpu_xxx funcs 2012-01-13 15:53 ` [PATCH v2 2/5] xen: " Alex Shi @ 2012-01-13 15:53 ` Alex Shi 2012-01-13 15:53 ` [PATCH v2 4/5] x86: change percpu_read_stable to this_cpu_read_stable Alex Shi 0 siblings, 1 reply; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:53 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, konrad.wilk, hpa, jeremy Since percpu_xxx() serial functions are duplicate with this_cpu_xxx(). Removing percpu_xxx() definition and replacing them by this_cpu_xxx() in code. And further more, as Christoph Lameter's requirement, I try to use __this_cpu_xx to replace this_cpu_xxx if it is in preempt safe scenario. The preempt safe scenarios include: 1, in irq/softirq/nmi handler 2, protected by preempt_disable 3, protected by spin_lock 4, if the code context imply that it is preempt safe, like the code is follows or be followed a preempt safe code. BTW, In fact, this_cpu_xxx are same as __this_cpu_xxx since all funcs implement in a single instruction for x86 machine. But it maybe different for other platforms, so, doing this distinguish is helpful for other platforms' performance. Signed-off-by: Alex Shi <alex.shi@intel.com> Acked-by: Christoph Lameter <cl@gentwo.org> Acked-by: Tejun Heo <tj@kernel.org> --- arch/x86/include/asm/hardirq.h | 9 +++++---- arch/x86/include/asm/irq_regs.h | 4 ++-- arch/x86/include/asm/mmu_context.h | 12 ++++++------ arch/x86/include/asm/percpu.h | 2 +- arch/x86/include/asm/smp.h | 4 ++-- arch/x86/include/asm/stackprotector.h | 4 ++-- arch/x86/include/asm/tlbflush.h | 4 ++-- arch/x86/kernel/cpu/common.c | 2 +- arch/x86/kernel/cpu/mcheck/mce.c | 4 ++-- arch/x86/kernel/paravirt.c | 12 ++++++------ arch/x86/kernel/process_32.c | 2 +- arch/x86/kernel/process_64.c | 12 ++++++------ arch/x86/mm/tlb.c | 10 +++++----- include/linux/topology.h | 4 ++-- 14 files changed, 43 insertions(+), 42 deletions(-) diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index da0b3ca..b6e5c83 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -36,14 +36,15 @@ DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); #define __ARCH_IRQ_STAT -#define inc_irq_stat(member) percpu_inc(irq_stat.member) +#define inc_irq_stat(member) __this_cpu_inc(irq_stat.member) -#define local_softirq_pending() percpu_read(irq_stat.__softirq_pending) +#define local_softirq_pending() __this_cpu_read(irq_stat.__softirq_pending) #define __ARCH_SET_SOFTIRQ_PENDING -#define set_softirq_pending(x) percpu_write(irq_stat.__softirq_pending, (x)) -#define or_softirq_pending(x) percpu_or(irq_stat.__softirq_pending, (x)) +#define set_softirq_pending(x) \ + __this_cpu_write(irq_stat.__softirq_pending, (x)) +#define or_softirq_pending(x) __this_cpu_or(irq_stat.__softirq_pending, (x)) extern void ack_bad_irq(unsigned int irq); diff --git a/arch/x86/include/asm/irq_regs.h b/arch/x86/include/asm/irq_regs.h index 7784322..15639ed 100644 --- a/arch/x86/include/asm/irq_regs.h +++ b/arch/x86/include/asm/irq_regs.h @@ -15,7 +15,7 @@ DECLARE_PER_CPU(struct pt_regs *, irq_regs); static inline struct pt_regs *get_irq_regs(void) { - return percpu_read(irq_regs); + return __this_cpu_read(irq_regs); } static inline struct pt_regs *set_irq_regs(struct pt_regs *new_regs) @@ -23,7 +23,7 @@ static inline struct pt_regs *set_irq_regs(struct pt_regs *new_regs) struct pt_regs *old_regs; old_regs = get_irq_regs(); - percpu_write(irq_regs, new_regs); + __this_cpu_write(irq_regs, new_regs); return old_regs; } diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 6902152..02ca533 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -25,8 +25,8 @@ void destroy_context(struct mm_struct *mm); static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) { #ifdef CONFIG_SMP - if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK) - percpu_write(cpu_tlbstate.state, TLBSTATE_LAZY); + if (__this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK) + __this_cpu_write(cpu_tlbstate.state, TLBSTATE_LAZY); #endif } @@ -37,8 +37,8 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, if (likely(prev != next)) { #ifdef CONFIG_SMP - percpu_write(cpu_tlbstate.state, TLBSTATE_OK); - percpu_write(cpu_tlbstate.active_mm, next); + __this_cpu_write(cpu_tlbstate.state, TLBSTATE_OK); + __this_cpu_write(cpu_tlbstate.active_mm, next); #endif cpumask_set_cpu(cpu, mm_cpumask(next)); @@ -56,8 +56,8 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, } #ifdef CONFIG_SMP else { - percpu_write(cpu_tlbstate.state, TLBSTATE_OK); - BUG_ON(percpu_read(cpu_tlbstate.active_mm) != next); + __this_cpu_write(cpu_tlbstate.state, TLBSTATE_OK); + BUG_ON(__this_cpu_read(cpu_tlbstate.active_mm) != next); if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next))) { /* We were in lazy tlb mode and leave_mm disabled diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 7a11910..276bbc0 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -46,7 +46,7 @@ #ifdef CONFIG_SMP #define __percpu_prefix "%%"__stringify(__percpu_seg)":" -#define __my_cpu_offset percpu_read(this_cpu_off) +#define __my_cpu_offset __this_cpu_read(this_cpu_off) /* * Compared to the generic __my_cpu_offset version, the following diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index 0434c40..e276f6b 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -188,11 +188,11 @@ extern unsigned disabled_cpus __cpuinitdata; * from the initial startup. We map APIC_BASE very early in page_setup(), * so this is correct in the x86 case. */ -#define raw_smp_processor_id() (percpu_read(cpu_number)) +#define raw_smp_processor_id() (this_cpu_read(cpu_number)) extern int safe_smp_processor_id(void); #elif defined(CONFIG_X86_64_SMP) -#define raw_smp_processor_id() (percpu_read(cpu_number)) +#define raw_smp_processor_id() (this_cpu_read(cpu_number)) #define stack_smp_processor_id() \ ({ \ diff --git a/arch/x86/include/asm/stackprotector.h b/arch/x86/include/asm/stackprotector.h index 1575177..e8a60c9 100644 --- a/arch/x86/include/asm/stackprotector.h +++ b/arch/x86/include/asm/stackprotector.h @@ -76,9 +76,9 @@ static __always_inline void boot_init_stack_canary(void) current->stack_canary = canary; #ifdef CONFIG_X86_64 - percpu_write(irq_stack_union.stack_canary, canary); + __this_cpu_write(irq_stack_union.stack_canary, canary); #else - percpu_write(stack_canary.canary, canary); + __this_cpu_write(stack_canary.canary, canary); #endif } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 169be89..e90eec0 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -156,8 +156,8 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate); static inline void reset_lazy_tlbstate(void) { - percpu_write(cpu_tlbstate.state, 0); - percpu_write(cpu_tlbstate.active_mm, &init_mm); + __this_cpu_write(cpu_tlbstate.state, 0); + __this_cpu_write(cpu_tlbstate.active_mm, &init_mm); } #endif /* SMP */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 850f296..6fbd2b4 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1167,7 +1167,7 @@ void __cpuinit cpu_init(void) oist = &per_cpu(orig_ist, cpu); #ifdef CONFIG_NUMA - if (cpu != 0 && percpu_read(numa_node) == 0 && + if (cpu != 0 && __this_cpu_read(numa_node) == 0 && early_cpu_to_node(cpu) != NUMA_NO_NODE) set_numa_node(early_cpu_to_node(cpu)); #endif diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c index f22a9f7..78f8900 100644 --- a/arch/x86/kernel/cpu/mcheck/mce.c +++ b/arch/x86/kernel/cpu/mcheck/mce.c @@ -562,7 +562,7 @@ void machine_check_poll(enum mcp_flags flags, mce_banks_t *b) struct mce m; int i; - percpu_inc(mce_poll_count); + __this_cpu_inc(mce_poll_count); mce_gather_info(&m, NULL); @@ -954,7 +954,7 @@ void do_machine_check(struct pt_regs *regs, long error_code) atomic_inc(&mce_entry); - percpu_inc(mce_exception_count); + __this_cpu_inc(mce_exception_count); if (!banks) goto out; diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index d90272e..2f0c1d1 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -239,16 +239,16 @@ static DEFINE_PER_CPU(enum paravirt_lazy_mode, paravirt_lazy_mode) = PARAVIRT_LA static inline void enter_lazy(enum paravirt_lazy_mode mode) { - BUG_ON(percpu_read(paravirt_lazy_mode) != PARAVIRT_LAZY_NONE); + BUG_ON(__this_cpu_read(paravirt_lazy_mode) != PARAVIRT_LAZY_NONE); - percpu_write(paravirt_lazy_mode, mode); + __this_cpu_write(paravirt_lazy_mode, mode); } static void leave_lazy(enum paravirt_lazy_mode mode) { - BUG_ON(percpu_read(paravirt_lazy_mode) != mode); + BUG_ON(__this_cpu_read(paravirt_lazy_mode) != mode); - percpu_write(paravirt_lazy_mode, PARAVIRT_LAZY_NONE); + __this_cpu_write(paravirt_lazy_mode, PARAVIRT_LAZY_NONE); } void paravirt_enter_lazy_mmu(void) @@ -265,7 +265,7 @@ void paravirt_start_context_switch(struct task_struct *prev) { BUG_ON(preemptible()); - if (percpu_read(paravirt_lazy_mode) == PARAVIRT_LAZY_MMU) { + if (__this_cpu_read(paravirt_lazy_mode) == PARAVIRT_LAZY_MMU) { arch_leave_lazy_mmu_mode(); set_ti_thread_flag(task_thread_info(prev), TIF_LAZY_MMU_UPDATES); } @@ -287,7 +287,7 @@ enum paravirt_lazy_mode paravirt_get_lazy_mode(void) if (in_interrupt()) return PARAVIRT_LAZY_NONE; - return percpu_read(paravirt_lazy_mode); + return __this_cpu_read(paravirt_lazy_mode); } void arch_flush_lazy_mmu_mode(void) diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 485204f..6acfb80 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -377,7 +377,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) if (prev->gs | next->gs) lazy_load_gs(next->gs); - percpu_write(current_task, next_p); + __this_cpu_write(current_task, next_p); return prev_p; } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 9b9fe4a..1b434d3 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -74,7 +74,7 @@ EXPORT_SYMBOL_GPL(idle_notifier_unregister); void enter_idle(void) { - percpu_write(is_idle, 1); + __this_cpu_write(is_idle, 1); atomic_notifier_call_chain(&idle_notifier, IDLE_START, NULL); } @@ -343,7 +343,7 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip, load_gs_index(0); regs->ip = new_ip; regs->sp = new_sp; - percpu_write(old_rsp, new_sp); + this_cpu_write(old_rsp, new_sp); regs->cs = _cs; regs->ss = _ss; regs->flags = X86_EFLAGS_IF; @@ -477,11 +477,11 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) /* * Switch the PDA and FPU contexts. */ - prev->usersp = percpu_read(old_rsp); - percpu_write(old_rsp, next->usersp); - percpu_write(current_task, next_p); + prev->usersp = __this_cpu_read(old_rsp); + __this_cpu_write(old_rsp, next->usersp); + __this_cpu_write(current_task, next_p); - percpu_write(kernel_stack, + __this_cpu_write(kernel_stack, (unsigned long)task_stack_page(next_p) + THREAD_SIZE - KERNEL_STACK_OFFSET); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index d6c0418..e931db0 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -61,10 +61,10 @@ static DEFINE_PER_CPU_READ_MOSTLY(int, tlb_vector_offset); */ void leave_mm(int cpu) { - if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK) + if (__this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK) BUG(); cpumask_clear_cpu(cpu, - mm_cpumask(percpu_read(cpu_tlbstate.active_mm))); + mm_cpumask(__this_cpu_read(cpu_tlbstate.active_mm))); load_cr3(swapper_pg_dir); } EXPORT_SYMBOL_GPL(leave_mm); @@ -152,8 +152,8 @@ void smp_invalidate_interrupt(struct pt_regs *regs) * BUG(); */ - if (f->flush_mm == percpu_read(cpu_tlbstate.active_mm)) { - if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK) { + if (f->flush_mm == __this_cpu_read(cpu_tlbstate.active_mm)) { + if (__this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK) { if (f->flush_va == TLB_FLUSH_ALL) local_flush_tlb(); else @@ -322,7 +322,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long va) static void do_flush_tlb_all(void *info) { __flush_tlb_all(); - if (percpu_read(cpu_tlbstate.state) == TLBSTATE_LAZY) + if (__this_cpu_read(cpu_tlbstate.state) == TLBSTATE_LAZY) leave_mm(smp_processor_id()); } diff --git a/include/linux/topology.h b/include/linux/topology.h index e26db03..b480403 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -239,7 +239,7 @@ static inline int cpu_to_node(int cpu) #ifndef set_numa_node static inline void set_numa_node(int node) { - percpu_write(numa_node, node); + __this_cpu_write(numa_node, node); } #endif @@ -274,7 +274,7 @@ DECLARE_PER_CPU(int, _numa_mem_); #ifndef set_numa_mem static inline void set_numa_mem(int node) { - percpu_write(_numa_mem_, node); + __this_cpu_write(_numa_mem_, node); } #endif -- 1.6.3.3 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2 4/5] x86: change percpu_read_stable to this_cpu_read_stable 2012-01-13 15:53 ` [PATCH v2 3/5] x86: " Alex Shi @ 2012-01-13 15:53 ` Alex Shi 2012-01-13 15:53 ` [PATCH v2 5/5] x86: Code clean up for percpu_xxx() functions Alex Shi 0 siblings, 1 reply; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:53 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, konrad.wilk, hpa, jeremy It has no function change. It's a preparation for percpu_xxx serial function change. Signed-off-by: Alex Shi <alex.shi@intel.com> Acked-by: Christoph Lameter <cl@gentwo.org> Acked-by: Tejun Heo <tj@kernel.org> --- arch/x86/include/asm/current.h | 2 +- arch/x86/include/asm/percpu.h | 6 +++--- arch/x86/include/asm/thread_info.h | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 4d447b7..9476c04 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -11,7 +11,7 @@ DECLARE_PER_CPU(struct task_struct *, current_task); static __always_inline struct task_struct *get_current(void) { - return percpu_read_stable(current_task); + return this_cpu_read_stable(current_task); } #define current get_current() diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 276bbc0..8d256ad 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -352,15 +352,15 @@ do { \ /* * percpu_read() makes gcc load the percpu variable every time it is - * accessed while percpu_read_stable() allows the value to be cached. - * percpu_read_stable() is more efficient and can be used if its value + * accessed while this_cpu_read_stable() allows the value to be cached. + * this_cpu_read_stable() is more efficient and can be used if its value * is guaranteed to be valid across cpus. The current users include * get_current() and get_thread_info() both of which are actually * per-thread variables implemented as per-cpu variables and thus * stable for the duration of the respective task. */ #define percpu_read(var) percpu_from_op("mov", var, "m" (var)) -#define percpu_read_stable(var) percpu_from_op("mov", var, "p" (&(var))) +#define this_cpu_read_stable(var) percpu_from_op("mov", var, "p" (&(var))) #define percpu_write(var, val) percpu_to_op("mov", var, val) #define percpu_add(var, val) percpu_add_op(var, val) #define percpu_sub(var, val) percpu_add_op(var, -(val)) diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thread_info.h index bc817cd..3544992 100644 --- a/arch/x86/include/asm/thread_info.h +++ b/arch/x86/include/asm/thread_info.h @@ -218,7 +218,7 @@ DECLARE_PER_CPU(unsigned long, kernel_stack); static inline struct thread_info *current_thread_info(void) { struct thread_info *ti; - ti = (void *)(percpu_read_stable(kernel_stack) + + ti = (void *)(this_cpu_read_stable(kernel_stack) + KERNEL_STACK_OFFSET - THREAD_SIZE); return ti; } -- 1.6.3.3 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2 5/5] x86: Code clean up for percpu_xxx() functions 2012-01-13 15:53 ` [PATCH v2 4/5] x86: change percpu_read_stable to this_cpu_read_stable Alex Shi @ 2012-01-13 15:53 ` Alex Shi 0 siblings, 0 replies; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:53 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, konrad.wilk, hpa, jeremy There is no percpu_xxx serial code in kerenl, so safly remove them. Signed-off-by: Alex Shi <alex.shi@intel.com> Acked-by: Christoph Lameter <cl@gentwo.org> Acked-by: Tejun Heo <tj@kernel.org> --- arch/x86/include/asm/percpu.h | 16 ++++------- include/linux/percpu.h | 54 ----------------------------------------- 2 files changed, 6 insertions(+), 64 deletions(-) diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 8d256ad..ed3e18a 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -351,7 +351,7 @@ do { \ }) /* - * percpu_read() makes gcc load the percpu variable every time it is + * this_cpu_read() makes gcc load the percpu variable every time it is * accessed while this_cpu_read_stable() allows the value to be cached. * this_cpu_read_stable() is more efficient and can be used if its value * is guaranteed to be valid across cpus. The current users include @@ -359,15 +359,7 @@ do { \ * per-thread variables implemented as per-cpu variables and thus * stable for the duration of the respective task. */ -#define percpu_read(var) percpu_from_op("mov", var, "m" (var)) #define this_cpu_read_stable(var) percpu_from_op("mov", var, "p" (&(var))) -#define percpu_write(var, val) percpu_to_op("mov", var, val) -#define percpu_add(var, val) percpu_add_op(var, val) -#define percpu_sub(var, val) percpu_add_op(var, -(val)) -#define percpu_and(var, val) percpu_to_op("and", var, val) -#define percpu_or(var, val) percpu_to_op("or", var, val) -#define percpu_xor(var, val) percpu_to_op("xor", var, val) -#define percpu_inc(var) percpu_unary_op("inc", var) #define __this_cpu_read_1(pcp) percpu_from_op("mov", (pcp), "m"(pcp)) #define __this_cpu_read_2(pcp) percpu_from_op("mov", (pcp), "m"(pcp)) @@ -512,7 +504,11 @@ static __always_inline int x86_this_cpu_constant_test_bit(unsigned int nr, { unsigned long __percpu *a = (unsigned long *)addr + nr / BITS_PER_LONG; - return ((1UL << (nr % BITS_PER_LONG)) & percpu_read(*a)) != 0; +#ifdef CONFIG_X86_64 + return ((1UL << (nr % BITS_PER_LONG)) & __this_cpu_read_8(*a)) != 0; +#else + return ((1UL << (nr % BITS_PER_LONG)) & __this_cpu_read_4(*a)) != 0; +#endif } static inline int x86_this_cpu_variable_test_bit(int nr, diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 32cd1f6..6e68d05 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -166,60 +166,6 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr); (typeof(type) __percpu *)__alloc_percpu(sizeof(type), __alignof__(type)) /* - * Optional methods for optimized non-lvalue per-cpu variable access. - * - * @var can be a percpu variable or a field of it and its size should - * equal char, int or long. percpu_read() evaluates to a lvalue and - * all others to void. - * - * These operations are guaranteed to be atomic. - * The generic versions disable interrupts. Archs are - * encouraged to implement single-instruction alternatives which don't - * require protection. - */ -#ifndef percpu_read -# define percpu_read(var) \ - ({ \ - typeof(var) *pr_ptr__ = &(var); \ - typeof(var) pr_ret__; \ - pr_ret__ = get_cpu_var(*pr_ptr__); \ - put_cpu_var(*pr_ptr__); \ - pr_ret__; \ - }) -#endif - -#define __percpu_generic_to_op(var, val, op) \ -do { \ - typeof(var) *pgto_ptr__ = &(var); \ - get_cpu_var(*pgto_ptr__) op val; \ - put_cpu_var(*pgto_ptr__); \ -} while (0) - -#ifndef percpu_write -# define percpu_write(var, val) __percpu_generic_to_op(var, (val), =) -#endif - -#ifndef percpu_add -# define percpu_add(var, val) __percpu_generic_to_op(var, (val), +=) -#endif - -#ifndef percpu_sub -# define percpu_sub(var, val) __percpu_generic_to_op(var, (val), -=) -#endif - -#ifndef percpu_and -# define percpu_and(var, val) __percpu_generic_to_op(var, (val), &=) -#endif - -#ifndef percpu_or -# define percpu_or(var, val) __percpu_generic_to_op(var, (val), |=) -#endif - -#ifndef percpu_xor -# define percpu_xor(var, val) __percpu_generic_to_op(var, (val), ^=) -#endif - -/* * Branching function to split up a function into a set of functions that * are called for different scalar sizes of the objects handled. */ -- 1.6.3.3 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATH v2 0/5] Code clean up for percpu_xxx serial functions 2012-01-13 15:53 [PATH v2 0/5] Code clean up for percpu_xxx serial functions Alex Shi 2012-01-13 15:53 ` [PATCH v2 1/5] net: use this_cpu_xxx replace percpu_xxx funcs Alex Shi @ 2012-05-04 21:29 ` Andrew Morton 2012-05-05 1:09 ` Alex Shi 1 sibling, 1 reply; 9+ messages in thread From: Andrew Morton @ 2012-05-04 21:29 UTC (permalink / raw) To: Alex Shi Cc: linux-kernel, cl, eric.dumazet, tglx, avi, davem, kaber, a.p.zijlstra, ying.huang, konrad.wilk, hpa, jeremy On Fri, 13 Jan 2012 23:53:33 +0800 Alex Shi <alex.shi@intel.com> wrote: > I am sorry for spelling mistaken on konrad's email address, so resend > for correct this. Please reply this resend email. > > --------------- > Thanks for TJ's suggestion, I split the serial patch smaller for > potential bisection convenience. > > Compare to v1 patch, this v2 patches has separate function replace > patches and final dead code clean up patch. > > The net, xen and x86 part code are independent. > > After each part was accepted in kernel, the final(5th) clean up code > do the real clean up in next merge window. I will refresh the patch > at that time. > > Any further comments are appreciated! > I'm still sitting on these patches. The review was a bit inconclusive and confusing and everyone will have forgotten all about everything. I think I'll drop them and ask for a resend, please. Be sure to update the changelogs so that they address everything which was discussed last time - so we don't end up covering the same ground. Please also Cc everyone who was involved in the discussion last time. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATH v2 0/5] Code clean up for percpu_xxx serial functions 2012-05-04 21:29 ` [PATH v2 0/5] Code clean up for percpu_xxx serial functions Andrew Morton @ 2012-05-05 1:09 ` Alex Shi 0 siblings, 0 replies; 9+ messages in thread From: Alex Shi @ 2012-05-05 1:09 UTC (permalink / raw) To: Andrew Morton Cc: linux-kernel, cl, eric.dumazet, tglx, avi, davem, kaber, a.p.zijlstra, ying.huang, konrad.wilk, hpa, jeremy On 05/05/2012 05:29 AM, Andrew Morton wrote: > On Fri, 13 Jan 2012 23:53:33 +0800 > Alex Shi <alex.shi@intel.com> wrote: > >> I am sorry for spelling mistaken on konrad's email address, so resend >> for correct this. Please reply this resend email. >> >> --------------- >> Thanks for TJ's suggestion, I split the serial patch smaller for >> potential bisection convenience. >> >> Compare to v1 patch, this v2 patches has separate function replace >> patches and final dead code clean up patch. >> >> The net, xen and x86 part code are independent. >> >> After each part was accepted in kernel, the final(5th) clean up code >> do the real clean up in next merge window. I will refresh the patch >> at that time. >> >> Any further comments are appreciated! >> > > I'm still sitting on these patches. The review was a bit inconclusive > and confusing and everyone will have forgotten all about everything. I > think I'll drop them and ask for a resend, please. > > Be sure to update the changelogs so that they address everything which > was discussed last time - so we don't end up covering the same ground. > Please also Cc everyone who was involved in the discussion last time. > Thanks for comments. I will try to refresh this upon mm tree next week. > ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATH v2 0/5] Code clean up for percpu_xxx serial functions @ 2012-01-13 15:45 Alex Shi 2012-01-13 15:45 ` [PATCH v2 1/5] net: use this_cpu_xxx replace percpu_xxx funcs Alex Shi 0 siblings, 1 reply; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:45 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, honrad.wilk, hpa, jeremy Compare to v1 patch, this v2 patches has separate function replace patches and final dead code clean up patch. The net, xen and x86 part code are independent. After each part was accepted in kernel, the final(5th) clean up code do the real clean up in next merge window. I will refresh the patch at that time. Any further comments are appreciated! Regards Alex ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v2 1/5] net: use this_cpu_xxx replace percpu_xxx funcs 2012-01-13 15:45 Alex Shi @ 2012-01-13 15:45 ` Alex Shi 2012-01-13 15:45 ` [PATCH v2 2/5] xen: " Alex Shi 0 siblings, 1 reply; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:45 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, honrad.wilk, hpa, jeremy percpu_xxx funcs are duplicated with this_cpu_xxx funcs, so replace them for further code clean up. And in preempt safe scenario, __this_cpu_xxx funcs has a bit better performance since __this_cpu_xxx has no redundant preempt_disable() Signed-off-by: Alex Shi <alex.shi@intel.com> Acked-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: David S. Miller <davem@davemloft.net> --- net/netfilter/xt_TEE.c | 12 ++++++------ net/socket.c | 4 ++-- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/net/netfilter/xt_TEE.c b/net/netfilter/xt_TEE.c index 3aae66f..f81eba4 100644 --- a/net/netfilter/xt_TEE.c +++ b/net/netfilter/xt_TEE.c @@ -87,7 +87,7 @@ tee_tg4(struct sk_buff *skb, const struct xt_action_param *par) const struct xt_tee_tginfo *info = par->targinfo; struct iphdr *iph; - if (percpu_read(tee_active)) + if (__this_cpu_read(tee_active)) return XT_CONTINUE; /* * Copy the skb, and route the copy. Will later return %XT_CONTINUE for @@ -124,9 +124,9 @@ tee_tg4(struct sk_buff *skb, const struct xt_action_param *par) ip_send_check(iph); if (tee_tg_route4(skb, info)) { - percpu_write(tee_active, true); + __this_cpu_write(tee_active, true); ip_local_out(skb); - percpu_write(tee_active, false); + __this_cpu_write(tee_active, false); } else { kfree_skb(skb); } @@ -167,7 +167,7 @@ tee_tg6(struct sk_buff *skb, const struct xt_action_param *par) { const struct xt_tee_tginfo *info = par->targinfo; - if (percpu_read(tee_active)) + if (__this_cpu_read(tee_active)) return XT_CONTINUE; skb = pskb_copy(skb, GFP_ATOMIC); if (skb == NULL) @@ -185,9 +185,9 @@ tee_tg6(struct sk_buff *skb, const struct xt_action_param *par) --iph->hop_limit; } if (tee_tg_route6(skb, info)) { - percpu_write(tee_active, true); + __this_cpu_write(tee_active, true); ip6_local_out(skb); - percpu_write(tee_active, false); + __this_cpu_write(tee_active, false); } else { kfree_skb(skb); } diff --git a/net/socket.c b/net/socket.c index 28a96af..784d93a 100644 --- a/net/socket.c +++ b/net/socket.c @@ -479,7 +479,7 @@ static struct socket *sock_alloc(void) inode->i_uid = current_fsuid(); inode->i_gid = current_fsgid(); - percpu_add(sockets_in_use, 1); + this_cpu_add(sockets_in_use, 1); return sock; } @@ -522,7 +522,7 @@ void sock_release(struct socket *sock) if (rcu_dereference_protected(sock->wq, 1)->fasync_list) printk(KERN_ERR "sock_release: fasync list not empty!\n"); - percpu_sub(sockets_in_use, 1); + this_cpu_sub(sockets_in_use, 1); if (!sock->file) { iput(SOCK_INODE(sock)); return; -- 1.6.3.3 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2 2/5] xen: use this_cpu_xxx replace percpu_xxx funcs 2012-01-13 15:45 ` [PATCH v2 1/5] net: use this_cpu_xxx replace percpu_xxx funcs Alex Shi @ 2012-01-13 15:45 ` Alex Shi 2012-01-13 15:45 ` [PATCH v2 3/5] x86: " Alex Shi 0 siblings, 1 reply; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:45 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, honrad.wilk, hpa, jeremy percpu_xxx funcs are duplicated with this_cpu_xxx funcs, so replace them for further code clean up. I don't know much of xen code. But, since the code is in x86 architecture, the percpu_xxx is exactly same as this_cpu_xxx serials functions. So, the change is safe. Signed-off-by: Alex Shi <alex.shi@intel.com> Acked-by: Christoph Lameter <cl@gentwo.org> Acked-by: Tejun Heo <tj@kernel.org> --- arch/x86/xen/enlighten.c | 6 +++--- arch/x86/xen/irq.c | 8 ++++---- arch/x86/xen/mmu.c | 20 ++++++++++---------- arch/x86/xen/multicalls.h | 2 +- arch/x86/xen/smp.c | 2 +- 5 files changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c index 12eb07b..312c9e3 100644 --- a/arch/x86/xen/enlighten.c +++ b/arch/x86/xen/enlighten.c @@ -777,11 +777,11 @@ static DEFINE_PER_CPU(unsigned long, xen_cr0_value); static unsigned long xen_read_cr0(void) { - unsigned long cr0 = percpu_read(xen_cr0_value); + unsigned long cr0 = this_cpu_read(xen_cr0_value); if (unlikely(cr0 == 0)) { cr0 = native_read_cr0(); - percpu_write(xen_cr0_value, cr0); + this_cpu_write(xen_cr0_value, cr0); } return cr0; @@ -791,7 +791,7 @@ static void xen_write_cr0(unsigned long cr0) { struct multicall_space mcs; - percpu_write(xen_cr0_value, cr0); + this_cpu_write(xen_cr0_value, cr0); /* Only pay attention to cr0.TS; everything else is ignored. */ diff --git a/arch/x86/xen/irq.c b/arch/x86/xen/irq.c index 8bbb465..1573376 100644 --- a/arch/x86/xen/irq.c +++ b/arch/x86/xen/irq.c @@ -26,7 +26,7 @@ static unsigned long xen_save_fl(void) struct vcpu_info *vcpu; unsigned long flags; - vcpu = percpu_read(xen_vcpu); + vcpu = this_cpu_read(xen_vcpu); /* flag has opposite sense of mask */ flags = !vcpu->evtchn_upcall_mask; @@ -50,7 +50,7 @@ static void xen_restore_fl(unsigned long flags) make sure we're don't switch CPUs between getting the vcpu pointer and updating the mask. */ preempt_disable(); - vcpu = percpu_read(xen_vcpu); + vcpu = this_cpu_read(xen_vcpu); vcpu->evtchn_upcall_mask = flags; preempt_enable_no_resched(); @@ -72,7 +72,7 @@ static void xen_irq_disable(void) make sure we're don't switch CPUs between getting the vcpu pointer and updating the mask. */ preempt_disable(); - percpu_read(xen_vcpu)->evtchn_upcall_mask = 1; + this_cpu_read(xen_vcpu)->evtchn_upcall_mask = 1; preempt_enable_no_resched(); } PV_CALLEE_SAVE_REGS_THUNK(xen_irq_disable); @@ -86,7 +86,7 @@ static void xen_irq_enable(void) the caller is confused and is trying to re-enable interrupts on an indeterminate processor. */ - vcpu = percpu_read(xen_vcpu); + vcpu = this_cpu_read(xen_vcpu); vcpu->evtchn_upcall_mask = 0; /* Doesn't matter if we get preempted here, because any diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c index 58a0e46..1a309ee 100644 --- a/arch/x86/xen/mmu.c +++ b/arch/x86/xen/mmu.c @@ -1071,14 +1071,14 @@ static void drop_other_mm_ref(void *info) struct mm_struct *mm = info; struct mm_struct *active_mm; - active_mm = percpu_read(cpu_tlbstate.active_mm); + active_mm = this_cpu_read(cpu_tlbstate.active_mm); - if (active_mm == mm && percpu_read(cpu_tlbstate.state) != TLBSTATE_OK) + if (active_mm == mm && this_cpu_read(cpu_tlbstate.state) != TLBSTATE_OK) leave_mm(smp_processor_id()); /* If this cpu still has a stale cr3 reference, then make sure it has been flushed. */ - if (percpu_read(xen_current_cr3) == __pa(mm->pgd)) + if (this_cpu_read(xen_current_cr3) == __pa(mm->pgd)) load_cr3(swapper_pg_dir); } @@ -1185,17 +1185,17 @@ static void __init xen_pagetable_setup_done(pgd_t *base) static void xen_write_cr2(unsigned long cr2) { - percpu_read(xen_vcpu)->arch.cr2 = cr2; + this_cpu_read(xen_vcpu)->arch.cr2 = cr2; } static unsigned long xen_read_cr2(void) { - return percpu_read(xen_vcpu)->arch.cr2; + return this_cpu_read(xen_vcpu)->arch.cr2; } unsigned long xen_read_cr2_direct(void) { - return percpu_read(xen_vcpu_info.arch.cr2); + return this_cpu_read(xen_vcpu_info.arch.cr2); } static void xen_flush_tlb(void) @@ -1278,12 +1278,12 @@ static void xen_flush_tlb_others(const struct cpumask *cpus, static unsigned long xen_read_cr3(void) { - return percpu_read(xen_cr3); + return this_cpu_read(xen_cr3); } static void set_current_cr3(void *v) { - percpu_write(xen_current_cr3, (unsigned long)v); + this_cpu_write(xen_current_cr3, (unsigned long)v); } static void __xen_write_cr3(bool kernel, unsigned long cr3) @@ -1306,7 +1306,7 @@ static void __xen_write_cr3(bool kernel, unsigned long cr3) xen_extend_mmuext_op(&op); if (kernel) { - percpu_write(xen_cr3, cr3); + this_cpu_write(xen_cr3, cr3); /* Update xen_current_cr3 once the batch has actually been submitted. */ @@ -1322,7 +1322,7 @@ static void xen_write_cr3(unsigned long cr3) /* Update while interrupts are disabled, so its atomic with respect to ipis */ - percpu_write(xen_cr3, cr3); + this_cpu_write(xen_cr3, cr3); __xen_write_cr3(true, cr3); diff --git a/arch/x86/xen/multicalls.h b/arch/x86/xen/multicalls.h index dee79b7..9c2e74f 100644 --- a/arch/x86/xen/multicalls.h +++ b/arch/x86/xen/multicalls.h @@ -47,7 +47,7 @@ static inline void xen_mc_issue(unsigned mode) xen_mc_flush(); /* restore flags saved in xen_mc_batch */ - local_irq_restore(percpu_read(xen_mc_irq_flags)); + local_irq_restore(this_cpu_read(xen_mc_irq_flags)); } /* Set up a callback to be called when the current batch is flushed */ diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c index 041d4fe..449f868 100644 --- a/arch/x86/xen/smp.c +++ b/arch/x86/xen/smp.c @@ -76,7 +76,7 @@ static void __cpuinit cpu_bringup(void) xen_setup_cpu_clockevents(); set_cpu_online(cpu, true); - percpu_write(cpu_state, CPU_ONLINE); + this_cpu_write(cpu_state, CPU_ONLINE); wmb(); /* We can take interrupts now: we're officially "up". */ -- 1.6.3.3 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2 3/5] x86: use this_cpu_xxx replace percpu_xxx funcs 2012-01-13 15:45 ` [PATCH v2 2/5] xen: " Alex Shi @ 2012-01-13 15:45 ` Alex Shi 0 siblings, 0 replies; 9+ messages in thread From: Alex Shi @ 2012-01-13 15:45 UTC (permalink / raw) To: linux-kernel, cl, eric.dumazet, tglx, avi, akpm, davem, kaber, a.p.zijlstra, ying.huang, honrad.wilk, hpa, jeremy Since percpu_xxx() serial functions are duplicate with this_cpu_xxx(). Removing percpu_xxx() definition and replacing them by this_cpu_xxx() in code. And further more, as Christoph Lameter's requirement, I try to use __this_cpu_xx to replace this_cpu_xxx if it is in preempt safe scenario. The preempt safe scenarios include: 1, in irq/softirq/nmi handler 2, protected by preempt_disable 3, protected by spin_lock 4, if the code context imply that it is preempt safe, like the code is follows or be followed a preempt safe code. BTW, In fact, this_cpu_xxx are same as __this_cpu_xxx since all funcs implement in a single instruction for x86 machine. But it maybe different for other platforms, so, doing this distinguish is helpful for other platforms' performance. Signed-off-by: Alex Shi <alex.shi@intel.com> Acked-by: Christoph Lameter <cl@gentwo.org> Acked-by: Tejun Heo <tj@kernel.org> --- arch/x86/include/asm/hardirq.h | 9 +++++---- arch/x86/include/asm/irq_regs.h | 4 ++-- arch/x86/include/asm/mmu_context.h | 12 ++++++------ arch/x86/include/asm/percpu.h | 2 +- arch/x86/include/asm/smp.h | 4 ++-- arch/x86/include/asm/stackprotector.h | 4 ++-- arch/x86/include/asm/tlbflush.h | 4 ++-- arch/x86/kernel/cpu/common.c | 2 +- arch/x86/kernel/cpu/mcheck/mce.c | 4 ++-- arch/x86/kernel/paravirt.c | 12 ++++++------ arch/x86/kernel/process_32.c | 2 +- arch/x86/kernel/process_64.c | 12 ++++++------ arch/x86/mm/tlb.c | 10 +++++----- include/linux/topology.h | 4 ++-- 14 files changed, 43 insertions(+), 42 deletions(-) diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index da0b3ca..b6e5c83 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -36,14 +36,15 @@ DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); #define __ARCH_IRQ_STAT -#define inc_irq_stat(member) percpu_inc(irq_stat.member) +#define inc_irq_stat(member) __this_cpu_inc(irq_stat.member) -#define local_softirq_pending() percpu_read(irq_stat.__softirq_pending) +#define local_softirq_pending() __this_cpu_read(irq_stat.__softirq_pending) #define __ARCH_SET_SOFTIRQ_PENDING -#define set_softirq_pending(x) percpu_write(irq_stat.__softirq_pending, (x)) -#define or_softirq_pending(x) percpu_or(irq_stat.__softirq_pending, (x)) +#define set_softirq_pending(x) \ + __this_cpu_write(irq_stat.__softirq_pending, (x)) +#define or_softirq_pending(x) __this_cpu_or(irq_stat.__softirq_pending, (x)) extern void ack_bad_irq(unsigned int irq); diff --git a/arch/x86/include/asm/irq_regs.h b/arch/x86/include/asm/irq_regs.h index 7784322..15639ed 100644 --- a/arch/x86/include/asm/irq_regs.h +++ b/arch/x86/include/asm/irq_regs.h @@ -15,7 +15,7 @@ DECLARE_PER_CPU(struct pt_regs *, irq_regs); static inline struct pt_regs *get_irq_regs(void) { - return percpu_read(irq_regs); + return __this_cpu_read(irq_regs); } static inline struct pt_regs *set_irq_regs(struct pt_regs *new_regs) @@ -23,7 +23,7 @@ static inline struct pt_regs *set_irq_regs(struct pt_regs *new_regs) struct pt_regs *old_regs; old_regs = get_irq_regs(); - percpu_write(irq_regs, new_regs); + __this_cpu_write(irq_regs, new_regs); return old_regs; } diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 6902152..02ca533 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -25,8 +25,8 @@ void destroy_context(struct mm_struct *mm); static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) { #ifdef CONFIG_SMP - if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK) - percpu_write(cpu_tlbstate.state, TLBSTATE_LAZY); + if (__this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK) + __this_cpu_write(cpu_tlbstate.state, TLBSTATE_LAZY); #endif } @@ -37,8 +37,8 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, if (likely(prev != next)) { #ifdef CONFIG_SMP - percpu_write(cpu_tlbstate.state, TLBSTATE_OK); - percpu_write(cpu_tlbstate.active_mm, next); + __this_cpu_write(cpu_tlbstate.state, TLBSTATE_OK); + __this_cpu_write(cpu_tlbstate.active_mm, next); #endif cpumask_set_cpu(cpu, mm_cpumask(next)); @@ -56,8 +56,8 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, } #ifdef CONFIG_SMP else { - percpu_write(cpu_tlbstate.state, TLBSTATE_OK); - BUG_ON(percpu_read(cpu_tlbstate.active_mm) != next); + __this_cpu_write(cpu_tlbstate.state, TLBSTATE_OK); + BUG_ON(__this_cpu_read(cpu_tlbstate.active_mm) != next); if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next))) { /* We were in lazy tlb mode and leave_mm disabled diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 7a11910..276bbc0 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -46,7 +46,7 @@ #ifdef CONFIG_SMP #define __percpu_prefix "%%"__stringify(__percpu_seg)":" -#define __my_cpu_offset percpu_read(this_cpu_off) +#define __my_cpu_offset __this_cpu_read(this_cpu_off) /* * Compared to the generic __my_cpu_offset version, the following diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index 0434c40..e276f6b 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -188,11 +188,11 @@ extern unsigned disabled_cpus __cpuinitdata; * from the initial startup. We map APIC_BASE very early in page_setup(), * so this is correct in the x86 case. */ -#define raw_smp_processor_id() (percpu_read(cpu_number)) +#define raw_smp_processor_id() (this_cpu_read(cpu_number)) extern int safe_smp_processor_id(void); #elif defined(CONFIG_X86_64_SMP) -#define raw_smp_processor_id() (percpu_read(cpu_number)) +#define raw_smp_processor_id() (this_cpu_read(cpu_number)) #define stack_smp_processor_id() \ ({ \ diff --git a/arch/x86/include/asm/stackprotector.h b/arch/x86/include/asm/stackprotector.h index 1575177..e8a60c9 100644 --- a/arch/x86/include/asm/stackprotector.h +++ b/arch/x86/include/asm/stackprotector.h @@ -76,9 +76,9 @@ static __always_inline void boot_init_stack_canary(void) current->stack_canary = canary; #ifdef CONFIG_X86_64 - percpu_write(irq_stack_union.stack_canary, canary); + __this_cpu_write(irq_stack_union.stack_canary, canary); #else - percpu_write(stack_canary.canary, canary); + __this_cpu_write(stack_canary.canary, canary); #endif } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 169be89..e90eec0 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -156,8 +156,8 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate); static inline void reset_lazy_tlbstate(void) { - percpu_write(cpu_tlbstate.state, 0); - percpu_write(cpu_tlbstate.active_mm, &init_mm); + __this_cpu_write(cpu_tlbstate.state, 0); + __this_cpu_write(cpu_tlbstate.active_mm, &init_mm); } #endif /* SMP */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 850f296..6fbd2b4 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1167,7 +1167,7 @@ void __cpuinit cpu_init(void) oist = &per_cpu(orig_ist, cpu); #ifdef CONFIG_NUMA - if (cpu != 0 && percpu_read(numa_node) == 0 && + if (cpu != 0 && __this_cpu_read(numa_node) == 0 && early_cpu_to_node(cpu) != NUMA_NO_NODE) set_numa_node(early_cpu_to_node(cpu)); #endif diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c index f22a9f7..78f8900 100644 --- a/arch/x86/kernel/cpu/mcheck/mce.c +++ b/arch/x86/kernel/cpu/mcheck/mce.c @@ -562,7 +562,7 @@ void machine_check_poll(enum mcp_flags flags, mce_banks_t *b) struct mce m; int i; - percpu_inc(mce_poll_count); + __this_cpu_inc(mce_poll_count); mce_gather_info(&m, NULL); @@ -954,7 +954,7 @@ void do_machine_check(struct pt_regs *regs, long error_code) atomic_inc(&mce_entry); - percpu_inc(mce_exception_count); + __this_cpu_inc(mce_exception_count); if (!banks) goto out; diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index d90272e..2f0c1d1 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -239,16 +239,16 @@ static DEFINE_PER_CPU(enum paravirt_lazy_mode, paravirt_lazy_mode) = PARAVIRT_LA static inline void enter_lazy(enum paravirt_lazy_mode mode) { - BUG_ON(percpu_read(paravirt_lazy_mode) != PARAVIRT_LAZY_NONE); + BUG_ON(__this_cpu_read(paravirt_lazy_mode) != PARAVIRT_LAZY_NONE); - percpu_write(paravirt_lazy_mode, mode); + __this_cpu_write(paravirt_lazy_mode, mode); } static void leave_lazy(enum paravirt_lazy_mode mode) { - BUG_ON(percpu_read(paravirt_lazy_mode) != mode); + BUG_ON(__this_cpu_read(paravirt_lazy_mode) != mode); - percpu_write(paravirt_lazy_mode, PARAVIRT_LAZY_NONE); + __this_cpu_write(paravirt_lazy_mode, PARAVIRT_LAZY_NONE); } void paravirt_enter_lazy_mmu(void) @@ -265,7 +265,7 @@ void paravirt_start_context_switch(struct task_struct *prev) { BUG_ON(preemptible()); - if (percpu_read(paravirt_lazy_mode) == PARAVIRT_LAZY_MMU) { + if (__this_cpu_read(paravirt_lazy_mode) == PARAVIRT_LAZY_MMU) { arch_leave_lazy_mmu_mode(); set_ti_thread_flag(task_thread_info(prev), TIF_LAZY_MMU_UPDATES); } @@ -287,7 +287,7 @@ enum paravirt_lazy_mode paravirt_get_lazy_mode(void) if (in_interrupt()) return PARAVIRT_LAZY_NONE; - return percpu_read(paravirt_lazy_mode); + return __this_cpu_read(paravirt_lazy_mode); } void arch_flush_lazy_mmu_mode(void) diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 485204f..6acfb80 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -377,7 +377,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) if (prev->gs | next->gs) lazy_load_gs(next->gs); - percpu_write(current_task, next_p); + __this_cpu_write(current_task, next_p); return prev_p; } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 9b9fe4a..1b434d3 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -74,7 +74,7 @@ EXPORT_SYMBOL_GPL(idle_notifier_unregister); void enter_idle(void) { - percpu_write(is_idle, 1); + __this_cpu_write(is_idle, 1); atomic_notifier_call_chain(&idle_notifier, IDLE_START, NULL); } @@ -343,7 +343,7 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip, load_gs_index(0); regs->ip = new_ip; regs->sp = new_sp; - percpu_write(old_rsp, new_sp); + this_cpu_write(old_rsp, new_sp); regs->cs = _cs; regs->ss = _ss; regs->flags = X86_EFLAGS_IF; @@ -477,11 +477,11 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) /* * Switch the PDA and FPU contexts. */ - prev->usersp = percpu_read(old_rsp); - percpu_write(old_rsp, next->usersp); - percpu_write(current_task, next_p); + prev->usersp = __this_cpu_read(old_rsp); + __this_cpu_write(old_rsp, next->usersp); + __this_cpu_write(current_task, next_p); - percpu_write(kernel_stack, + __this_cpu_write(kernel_stack, (unsigned long)task_stack_page(next_p) + THREAD_SIZE - KERNEL_STACK_OFFSET); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index d6c0418..e931db0 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -61,10 +61,10 @@ static DEFINE_PER_CPU_READ_MOSTLY(int, tlb_vector_offset); */ void leave_mm(int cpu) { - if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK) + if (__this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK) BUG(); cpumask_clear_cpu(cpu, - mm_cpumask(percpu_read(cpu_tlbstate.active_mm))); + mm_cpumask(__this_cpu_read(cpu_tlbstate.active_mm))); load_cr3(swapper_pg_dir); } EXPORT_SYMBOL_GPL(leave_mm); @@ -152,8 +152,8 @@ void smp_invalidate_interrupt(struct pt_regs *regs) * BUG(); */ - if (f->flush_mm == percpu_read(cpu_tlbstate.active_mm)) { - if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK) { + if (f->flush_mm == __this_cpu_read(cpu_tlbstate.active_mm)) { + if (__this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK) { if (f->flush_va == TLB_FLUSH_ALL) local_flush_tlb(); else @@ -322,7 +322,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long va) static void do_flush_tlb_all(void *info) { __flush_tlb_all(); - if (percpu_read(cpu_tlbstate.state) == TLBSTATE_LAZY) + if (__this_cpu_read(cpu_tlbstate.state) == TLBSTATE_LAZY) leave_mm(smp_processor_id()); } diff --git a/include/linux/topology.h b/include/linux/topology.h index e26db03..b480403 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -239,7 +239,7 @@ static inline int cpu_to_node(int cpu) #ifndef set_numa_node static inline void set_numa_node(int node) { - percpu_write(numa_node, node); + __this_cpu_write(numa_node, node); } #endif @@ -274,7 +274,7 @@ DECLARE_PER_CPU(int, _numa_mem_); #ifndef set_numa_mem static inline void set_numa_mem(int node) { - percpu_write(_numa_mem_, node); + __this_cpu_write(_numa_mem_, node); } #endif -- 1.6.3.3 ^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2012-05-05 1:09 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-01-13 15:53 [PATH v2 0/5] Code clean up for percpu_xxx serial functions Alex Shi 2012-01-13 15:53 ` [PATCH v2 1/5] net: use this_cpu_xxx replace percpu_xxx funcs Alex Shi 2012-01-13 15:53 ` [PATCH v2 2/5] xen: " Alex Shi 2012-01-13 15:53 ` [PATCH v2 3/5] x86: " Alex Shi 2012-01-13 15:53 ` [PATCH v2 4/5] x86: change percpu_read_stable to this_cpu_read_stable Alex Shi 2012-01-13 15:53 ` [PATCH v2 5/5] x86: Code clean up for percpu_xxx() functions Alex Shi 2012-05-04 21:29 ` [PATH v2 0/5] Code clean up for percpu_xxx serial functions Andrew Morton 2012-05-05 1:09 ` Alex Shi -- strict thread matches above, loose matches on Subject: below -- 2012-01-13 15:45 Alex Shi 2012-01-13 15:45 ` [PATCH v2 1/5] net: use this_cpu_xxx replace percpu_xxx funcs Alex Shi 2012-01-13 15:45 ` [PATCH v2 2/5] xen: " Alex Shi 2012-01-13 15:45 ` [PATCH v2 3/5] x86: " Alex Shi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).