* [guv v2 02/31] percpu: Make __verify_pcu_ptr handle per cpu pointers to arrays
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
__verify_pcpu_ptr() will cause a compilation failure if the type of the
pointer is a pointer to a fixed array of objects. Adding zero to the
pointer converts the type of pointer to that pointing to a single
object of the array.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/percpu-defs.h
===================================================================
--- linux.orig/include/linux/percpu-defs.h 2013-08-22 13:39:04.000000000 -0500
+++ linux/include/linux/percpu-defs.h 2013-08-22 13:41:15.333140537 -0500
@@ -22,9 +22,12 @@
* Macro which verifies @ptr is a percpu pointer without evaluating
* @ptr. This is to be used in percpu accessors to verify that the
* input parameter is a percpu pointer.
+ *
+ * + 0 is required in order to convert the pointer type from a
+ * potential array type to a pointer to a single item of the array.
*/
#define __verify_pcpu_ptr(ptr) do { \
- const void __percpu *__vpp_verify = (typeof(ptr))NULL; \
+ const void __percpu *__vpp_verify = (typeof((ptr) + 0))NULL; \
(void)__vpp_verify; \
} while (0)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 02/31] percpu: Make __verify_pcu_ptr handle per cpu pointers to arrays
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
__verify_pcpu_ptr() will cause a compilation failure if the type of the
pointer is a pointer to a fixed array of objects. Adding zero to the
pointer converts the type of pointer to that pointing to a single
object of the array.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/percpu-defs.h
===================================================================
--- linux.orig/include/linux/percpu-defs.h 2013-08-22 13:39:04.000000000 -0500
+++ linux/include/linux/percpu-defs.h 2013-08-22 13:41:15.333140537 -0500
@@ -22,9 +22,12 @@
* Macro which verifies @ptr is a percpu pointer without evaluating
* @ptr. This is to be used in percpu accessors to verify that the
* input parameter is a percpu pointer.
+ *
+ * + 0 is required in order to convert the pointer type from a
+ * potential array type to a pointer to a single item of the array.
*/
#define __verify_pcpu_ptr(ptr) do { \
- const void __percpu *__vpp_verify = (typeof(ptr))NULL; \
+ const void __percpu *__vpp_verify = (typeof((ptr) + 0))NULL; \
(void)__vpp_verify; \
} while (0)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 03/31] Coccinelle script for __get_cpu_var conversion
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
Much of the work was done by this coccinelle script.
Wont catch everything (since coccinelle aborts when it no longer can make
sense out of macros, sigh) but it covers a lot of ground.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/scripts/coccinelle/convert/__get_cpu_var.cocci
===================================================================
--- /dev/null 1970-01-01 00:00:00.000000000 +0000
+++ linux/scripts/coccinelle/convert/__get_cpu_var.cocci 2013-08-22 14:41:13.976518846 -0500
@@ -0,0 +1,70 @@
+// Convert all &__get_cpu_var to this_cpu_ptr
+//
+// _get_cpu_var is defined as:
+//
+// #define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
+//
+//&__get_cpu_var() is therefore
+//
+// this_cpu_ptr(&var);
+//
+// The use of this_cpu_ptr() instead of __get_cpu_var clarifies
+// that we are relocating a per cpu offset and that there are
+// no get/put semantics involved.
+//
+//
+// Convert all &__get_cpu_vars (and also related functions)
+// to this_cpu_ptr(). Multiple transformations are provided
+// to be able to beautify the source a bit.
+//
+// Christoph Lameter <cl@linux.com> August 20, 2013
+//
+
+@A@ expression E; @@
+
+...
+-(&__get_cpu_var(E))
++this_cpu_ptr(&E)
+...
+
+@B@ expression E; @@
+
+...
+-(&__raw_get_cpu_var(E))
++__this_cpu_ptr(&E)
+...
+
+@C@ expression E; @@
+
+...
+-&__get_cpu_var(E)
++this_cpu_ptr(&E)
+...
+
+@D@ expression E; @@
+
+...
+-&__raw_get_cpu_var(E)
++__this_cpu_ptr(&E)
+...
+
+@A2@ expression E; @@
+
+-(&__get_cpu_var(E))
++this_cpu_ptr(&E)
+
+@B2@ expression E; @@
+
+-(&__raw_get_cpu_var(E))
++__this_cpu_ptr(&E)
+
+@C2@ expression E; @@
+
+-&__get_cpu_var(E)
++this_cpu_ptr(&E)
+
+@D2@ expression E; @@
+
+-&__raw_get_cpu_var(E)
++__this_cpu_ptr(&E)
+
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 03/31] Coccinelle script for __get_cpu_var conversion
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
Much of the work was done by this coccinelle script.
Wont catch everything (since coccinelle aborts when it no longer can make
sense out of macros, sigh) but it covers a lot of ground.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/scripts/coccinelle/convert/__get_cpu_var.cocci
===================================================================
--- /dev/null 1970-01-01 00:00:00.000000000 +0000
+++ linux/scripts/coccinelle/convert/__get_cpu_var.cocci 2013-08-22 14:41:13.976518846 -0500
@@ -0,0 +1,70 @@
+// Convert all &__get_cpu_var to this_cpu_ptr
+//
+// _get_cpu_var is defined as:
+//
+// #define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
+//
+//&__get_cpu_var() is therefore
+//
+// this_cpu_ptr(&var);
+//
+// The use of this_cpu_ptr() instead of __get_cpu_var clarifies
+// that we are relocating a per cpu offset and that there are
+// no get/put semantics involved.
+//
+//
+// Convert all &__get_cpu_vars (and also related functions)
+// to this_cpu_ptr(). Multiple transformations are provided
+// to be able to beautify the source a bit.
+//
+// Christoph Lameter <cl@linux.com> August 20, 2013
+//
+
+@A@ expression E; @@
+
+...
+-(&__get_cpu_var(E))
++this_cpu_ptr(&E)
+...
+
+@B@ expression E; @@
+
+...
+-(&__raw_get_cpu_var(E))
++__this_cpu_ptr(&E)
+...
+
+@C@ expression E; @@
+
+...
+-&__get_cpu_var(E)
++this_cpu_ptr(&E)
+...
+
+@D@ expression E; @@
+
+...
+-&__raw_get_cpu_var(E)
++__this_cpu_ptr(&E)
+...
+
+@A2@ expression E; @@
+
+-(&__get_cpu_var(E))
++this_cpu_ptr(&E)
+
+@B2@ expression E; @@
+
+-(&__raw_get_cpu_var(E))
++__this_cpu_ptr(&E)
+
+@C2@ expression E; @@
+
+-&__get_cpu_var(E)
++this_cpu_ptr(&E)
+
+@D2@ expression E; @@
+
+-&__raw_get_cpu_var(E)
++__this_cpu_ptr(&E)
+
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 04/31] net: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, David S. Miller, netdev, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/net/core/dev.c
===================================================================
--- linux.orig/net/core/dev.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/core/dev.c 2013-08-26 14:18:37.214005168 -0500
@@ -2129,7 +2129,7 @@ static inline void __netif_reschedule(st
unsigned long flags;
local_irq_save(flags);
- sd = &__get_cpu_var(softnet_data);
+ sd = this_cpu_ptr(&softnet_data);
q->next_sched = NULL;
*sd->output_queue_tailp = q;
sd->output_queue_tailp = &q->next_sched;
@@ -2151,7 +2151,7 @@ void dev_kfree_skb_irq(struct sk_buff *s
unsigned long flags;
local_irq_save(flags);
- sd = &__get_cpu_var(softnet_data);
+ sd = this_cpu_ptr(&softnet_data);
skb->next = sd->completion_queue;
sd->completion_queue = skb;
raise_softirq_irqoff(NET_TX_SOFTIRQ);
@@ -3111,7 +3111,7 @@ static void rps_trigger_softirq(void *da
static int rps_ipi_queued(struct softnet_data *sd)
{
#ifdef CONFIG_RPS
- struct softnet_data *mysd = &__get_cpu_var(softnet_data);
+ struct softnet_data *mysd = this_cpu_ptr(&softnet_data);
if (sd != mysd) {
sd->rps_ipi_next = mysd->rps_ipi_list;
@@ -3138,7 +3138,7 @@ static bool skb_flow_limit(struct sk_buf
if (qlen < (netdev_max_backlog >> 1))
return false;
- sd = &__get_cpu_var(softnet_data);
+ sd = this_cpu_ptr(&softnet_data);
rcu_read_lock();
fl = rcu_dereference(sd->flow_limit);
@@ -3280,7 +3280,7 @@ EXPORT_SYMBOL(netif_rx_ni);
static void net_tx_action(struct softirq_action *h)
{
- struct softnet_data *sd = &__get_cpu_var(softnet_data);
+ struct softnet_data *sd = this_cpu_ptr(&softnet_data);
if (sd->completion_queue) {
struct sk_buff *clist;
@@ -3700,7 +3700,7 @@ EXPORT_SYMBOL(netif_receive_skb);
static void flush_backlog(void *arg)
{
struct net_device *dev = arg;
- struct softnet_data *sd = &__get_cpu_var(softnet_data);
+ struct softnet_data *sd = this_cpu_ptr(&softnet_data);
struct sk_buff *skb, *tmp;
rps_lock(sd);
@@ -4146,7 +4146,7 @@ void __napi_schedule(struct napi_struct
unsigned long flags;
local_irq_save(flags);
- ____napi_schedule(&__get_cpu_var(softnet_data), n);
+ ____napi_schedule(this_cpu_ptr(&softnet_data), n);
local_irq_restore(flags);
}
EXPORT_SYMBOL(__napi_schedule);
@@ -4274,7 +4274,7 @@ EXPORT_SYMBOL(netif_napi_del);
static void net_rx_action(struct softirq_action *h)
{
- struct softnet_data *sd = &__get_cpu_var(softnet_data);
+ struct softnet_data *sd = this_cpu_ptr(&softnet_data);
unsigned long time_limit = jiffies + 2;
int budget = netdev_budget;
void *have;
Index: linux/net/core/drop_monitor.c
===================================================================
--- linux.orig/net/core/drop_monitor.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/core/drop_monitor.c 2013-08-26 14:18:37.218005126 -0500
@@ -142,7 +142,7 @@ static void trace_drop_common(struct sk_
unsigned long flags;
local_irq_save(flags);
- data = &__get_cpu_var(dm_cpu_data);
+ data = this_cpu_ptr(&dm_cpu_data);
spin_lock(&data->lock);
dskb = data->skb;
Index: linux/net/core/skbuff.c
===================================================================
--- linux.orig/net/core/skbuff.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/core/skbuff.c 2013-08-26 14:18:37.218005126 -0500
@@ -371,7 +371,7 @@ static void *__netdev_alloc_frag(unsigne
unsigned long flags;
local_irq_save(flags);
- nc = &__get_cpu_var(netdev_alloc_cache);
+ nc = this_cpu_ptr(&netdev_alloc_cache);
if (unlikely(!nc->frag.page)) {
refill:
for (order = NETDEV_FRAG_PAGE_MAX_ORDER; ;) {
Index: linux/net/ipv4/syncookies.c
===================================================================
--- linux.orig/net/ipv4/syncookies.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/ipv4/syncookies.c 2013-08-26 14:18:37.210005210 -0500
@@ -44,7 +44,7 @@ static DEFINE_PER_CPU(__u32 [16 + 5 + SH
static u32 cookie_hash(__be32 saddr, __be32 daddr, __be16 sport, __be16 dport,
u32 count, int c)
{
- __u32 *tmp = __get_cpu_var(ipv4_cookie_scratch);
+ __u32 *tmp = this_cpu_ptr(ipv4_cookie_scratch);
memcpy(tmp + 4, syncookie_secret[c], sizeof(syncookie_secret[c]));
tmp[0] = (__force u32)saddr;
Index: linux/net/ipv4/tcp_output.c
===================================================================
--- linux.orig/net/ipv4/tcp_output.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/ipv4/tcp_output.c 2013-08-26 14:18:37.218005126 -0500
@@ -810,7 +810,7 @@ void tcp_wfree(struct sk_buff *skb)
/* queue this socket to tasklet queue */
local_irq_save(flags);
- tsq = &__get_cpu_var(tsq_tasklet);
+ tsq = this_cpu_ptr(&tsq_tasklet);
list_add(&tp->tsq_node, &tsq->head);
tasklet_schedule(&tsq->tasklet);
local_irq_restore(flags);
Index: linux/net/ipv6/syncookies.c
===================================================================
--- linux.orig/net/ipv6/syncookies.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/ipv6/syncookies.c 2013-08-26 14:18:37.214005168 -0500
@@ -66,7 +66,7 @@ static DEFINE_PER_CPU(__u32 [16 + 5 + SH
static u32 cookie_hash(const struct in6_addr *saddr, const struct in6_addr *daddr,
__be16 sport, __be16 dport, u32 count, int c)
{
- __u32 *tmp = __get_cpu_var(ipv6_cookie_scratch);
+ __u32 *tmp = this_cpu_ptr(ipv6_cookie_scratch);
/*
* we have 320 bits of information to hash, copy in the remaining
Index: linux/net/rds/ib_rdma.c
===================================================================
--- linux.orig/net/rds/ib_rdma.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/rds/ib_rdma.c 2013-08-26 14:18:37.218005126 -0500
@@ -267,7 +267,7 @@ static inline struct rds_ib_mr *rds_ib_r
unsigned long *flag;
preempt_disable();
- flag = &__get_cpu_var(clean_list_grace);
+ flag = this_cpu_ptr(&clean_list_grace);
set_bit(CLEAN_LIST_BUSY_BIT, flag);
ret = llist_del_first(&pool->clean_list);
if (ret)
Index: linux/include/net/netfilter/nf_conntrack.h
===================================================================
--- linux.orig/include/net/netfilter/nf_conntrack.h 2013-08-26 14:23:46.000000000 -0500
+++ linux/include/net/netfilter/nf_conntrack.h 2013-08-26 14:24:17.190395755 -0500
@@ -243,7 +243,7 @@ extern s16 (*nf_ct_nat_offset)(const str
DECLARE_PER_CPU(struct nf_conn, nf_conntrack_untracked);
static inline struct nf_conn *nf_ct_untracked_get(void)
{
- return &__raw_get_cpu_var(nf_conntrack_untracked);
+ return __this_cpu_ptr(&nf_conntrack_untracked);
}
extern void nf_ct_untracked_status_or(unsigned long bits);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 05/31] time: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Thomas Gleixner, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/kernel/hrtimer.c
===================================================================
--- linux.orig/kernel/hrtimer.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/hrtimer.c 2013-08-26 14:21:24.540227691 -0500
@@ -597,7 +597,7 @@ hrtimer_force_reprogram(struct hrtimer_c
static int hrtimer_reprogram(struct hrtimer *timer,
struct hrtimer_clock_base *base)
{
- struct hrtimer_cpu_base *cpu_base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
ktime_t expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
int res;
@@ -680,7 +680,7 @@ static inline ktime_t hrtimer_update_bas
*/
static void retrigger_next_event(void *arg)
{
- struct hrtimer_cpu_base *base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *base = this_cpu_ptr(&hrtimer_bases);
if (!hrtimer_hres_active())
return;
@@ -954,7 +954,7 @@ remove_hrtimer(struct hrtimer *timer, st
*/
debug_deactivate(timer);
timer_stats_hrtimer_clear_start_info(timer);
- reprogram = base->cpu_base == &__get_cpu_var(hrtimer_bases);
+ reprogram = base->cpu_base == this_cpu_ptr(&hrtimer_bases);
/*
* We must preserve the CALLBACK state flag here,
* otherwise we could move the timer base in
@@ -1009,7 +1009,7 @@ int __hrtimer_start_range_ns(struct hrti
*
* XXX send_remote_softirq() ?
*/
- if (leftmost && new_base->cpu_base == &__get_cpu_var(hrtimer_bases)
+ if (leftmost && new_base->cpu_base == this_cpu_ptr(&hrtimer_bases)
&& hrtimer_enqueue_reprogram(timer, new_base)) {
if (wakeup) {
/*
@@ -1142,7 +1142,7 @@ EXPORT_SYMBOL_GPL(hrtimer_get_remaining)
*/
ktime_t hrtimer_get_next_event(void)
{
- struct hrtimer_cpu_base *cpu_base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
struct hrtimer_clock_base *base = cpu_base->clock_base;
ktime_t delta, mindelta = { .tv64 = KTIME_MAX };
unsigned long flags;
@@ -1183,7 +1183,7 @@ static void __hrtimer_init(struct hrtime
memset(timer, 0, sizeof(struct hrtimer));
- cpu_base = &__raw_get_cpu_var(hrtimer_bases);
+ cpu_base = __this_cpu_ptr(&hrtimer_bases);
if (clock_id == CLOCK_REALTIME && mode != HRTIMER_MODE_ABS)
clock_id = CLOCK_MONOTONIC;
@@ -1226,7 +1226,7 @@ int hrtimer_get_res(const clockid_t whic
struct hrtimer_cpu_base *cpu_base;
int base = hrtimer_clockid_to_base(which_clock);
- cpu_base = &__raw_get_cpu_var(hrtimer_bases);
+ cpu_base = __this_cpu_ptr(&hrtimer_bases);
*tp = ktime_to_timespec(cpu_base->clock_base[base].resolution);
return 0;
@@ -1281,7 +1281,7 @@ static void __run_hrtimer(struct hrtimer
*/
void hrtimer_interrupt(struct clock_event_device *dev)
{
- struct hrtimer_cpu_base *cpu_base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
ktime_t expires_next, now, entry_time, delta;
int i, retries = 0;
@@ -1415,7 +1415,7 @@ static void __hrtimer_peek_ahead_timers(
if (!hrtimer_hres_active())
return;
- td = &__get_cpu_var(tick_cpu_device);
+ td = this_cpu_ptr(&tick_cpu_device);
if (td && td->evtdev)
hrtimer_interrupt(td->evtdev);
}
@@ -1479,7 +1479,7 @@ void hrtimer_run_pending(void)
void hrtimer_run_queues(void)
{
struct timerqueue_node *node;
- struct hrtimer_cpu_base *cpu_base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
struct hrtimer_clock_base *base;
int index, gettime = 1;
@@ -1717,7 +1717,7 @@ static void migrate_hrtimers(int scpu)
local_irq_disable();
old_base = &per_cpu(hrtimer_bases, scpu);
- new_base = &__get_cpu_var(hrtimer_bases);
+ new_base = this_cpu_ptr(&hrtimer_bases);
/*
* The caller is globally serialized and nobody else
* takes two locks at once, deadlock is not possible.
Index: linux/kernel/irq_work.c
===================================================================
--- linux.orig/kernel/irq_work.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/irq_work.c 2013-08-26 14:21:24.540227691 -0500
@@ -70,7 +70,7 @@ void irq_work_queue(struct irq_work *wor
/* Queue the entry and raise the IPI if needed. */
preempt_disable();
- llist_add(&work->llnode, &__get_cpu_var(irq_work_list));
+ llist_add(&work->llnode, this_cpu_ptr(&irq_work_list));
/*
* If the work is not "lazy" or the tick is stopped, raise the irq
@@ -90,7 +90,7 @@ bool irq_work_needs_cpu(void)
{
struct llist_head *this_list;
- this_list = &__get_cpu_var(irq_work_list);
+ this_list = this_cpu_ptr(&irq_work_list);
if (llist_empty(this_list))
return false;
@@ -115,7 +115,7 @@ static void __irq_work_run(void)
__this_cpu_write(irq_work_raised, 0);
barrier();
- this_list = &__get_cpu_var(irq_work_list);
+ this_list = this_cpu_ptr(&irq_work_list);
if (llist_empty(this_list))
return;
Index: linux/kernel/sched/clock.c
===================================================================
--- linux.orig/kernel/sched/clock.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/sched/clock.c 2013-08-26 14:21:24.540227691 -0500
@@ -94,7 +94,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(str
static inline struct sched_clock_data *this_scd(void)
{
- return &__get_cpu_var(sched_clock_data);
+ return this_cpu_ptr(&sched_clock_data);
}
static inline struct sched_clock_data *cpu_sdc(int cpu)
Index: linux/kernel/softirq.c
===================================================================
--- linux.orig/kernel/softirq.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/softirq.c 2013-08-26 14:21:24.540227691 -0500
@@ -466,7 +466,7 @@ static void tasklet_action(struct softir
local_irq_disable();
list = __this_cpu_read(tasklet_vec.head);
__this_cpu_write(tasklet_vec.head, NULL);
- __this_cpu_write(tasklet_vec.tail, &__get_cpu_var(tasklet_vec).head);
+ __this_cpu_write(tasklet_vec.tail, this_cpu_ptr(&tasklet_vec.head));
local_irq_enable();
while (list) {
@@ -501,7 +501,7 @@ static void tasklet_hi_action(struct sof
local_irq_disable();
list = __this_cpu_read(tasklet_hi_vec.head);
__this_cpu_write(tasklet_hi_vec.head, NULL);
- __this_cpu_write(tasklet_hi_vec.tail, &__get_cpu_var(tasklet_hi_vec).head);
+ __this_cpu_write(tasklet_hi_vec.tail, this_cpu_ptr(&tasklet_hi_vec.head));
local_irq_enable();
while (list) {
@@ -618,7 +618,7 @@ EXPORT_PER_CPU_SYMBOL(softirq_work_list)
static void __local_trigger(struct call_single_data *cp, int softirq)
{
- struct list_head *head = &__get_cpu_var(softirq_work_list[softirq]);
+ struct list_head *head = this_cpu_ptr(&softirq_work_list[softirq]);
list_add_tail(&cp->list, head);
@@ -718,7 +718,7 @@ static int remote_softirq_cpu_notify(str
if (list_empty(head))
continue;
- local_head = &__get_cpu_var(softirq_work_list[i]);
+ local_head = this_cpu_ptr(&softirq_work_list[i]);
list_splice_init(head, local_head);
raise_softirq_irqoff(i);
}
Index: linux/kernel/time/tick-common.c
===================================================================
--- linux.orig/kernel/time/tick-common.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/time/tick-common.c 2013-08-26 14:21:24.544227649 -0500
@@ -208,7 +208,7 @@ static void tick_setup_device(struct tic
void tick_install_replacement(struct clock_event_device *newdev)
{
- struct tick_device *td = &__get_cpu_var(tick_cpu_device);
+ struct tick_device *td = this_cpu_ptr(&tick_cpu_device);
int cpu = smp_processor_id();
clockevents_exchange_device(td->evtdev, newdev);
@@ -358,14 +358,14 @@ void tick_shutdown(unsigned int *cpup)
void tick_suspend(void)
{
- struct tick_device *td = &__get_cpu_var(tick_cpu_device);
+ struct tick_device *td = this_cpu_ptr(&tick_cpu_device);
clockevents_shutdown(td->evtdev);
}
void tick_resume(void)
{
- struct tick_device *td = &__get_cpu_var(tick_cpu_device);
+ struct tick_device *td = this_cpu_ptr(&tick_cpu_device);
int broadcast = tick_resume_broadcast();
clockevents_set_mode(td->evtdev, CLOCK_EVT_MODE_RESUME);
Index: linux/kernel/time/tick-oneshot.c
===================================================================
--- linux.orig/kernel/time/tick-oneshot.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/time/tick-oneshot.c 2013-08-26 14:21:24.544227649 -0500
@@ -59,7 +59,7 @@ void tick_setup_oneshot(struct clock_eve
*/
int tick_switch_to_oneshot(void (*handler)(struct clock_event_device *))
{
- struct tick_device *td = &__get_cpu_var(tick_cpu_device);
+ struct tick_device *td = this_cpu_ptr(&tick_cpu_device);
struct clock_event_device *dev = td->evtdev;
if (!dev || !(dev->features & CLOCK_EVT_FEAT_ONESHOT) ||
Index: linux/kernel/time/tick-sched.c
===================================================================
--- linux.orig/kernel/time/tick-sched.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/time/tick-sched.c 2013-08-26 14:21:24.540227691 -0500
@@ -199,7 +199,7 @@ static void tick_nohz_restart_sched_tick
*/
void tick_nohz_full_check(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
if (tick_nohz_full_cpu(smp_processor_id())) {
if (ts->tick_stopped && !is_idle_task(current)) {
@@ -225,7 +225,7 @@ static DEFINE_PER_CPU(struct irq_work, n
void tick_nohz_full_kick(void)
{
if (tick_nohz_full_cpu(smp_processor_id()))
- irq_work_queue(&__get_cpu_var(nohz_full_kick_work));
+ irq_work_queue(this_cpu_ptr(&nohz_full_kick_work));
}
static void nohz_full_kick_ipi(void *info)
@@ -536,7 +536,7 @@ static ktime_t tick_nohz_stop_sched_tick
unsigned long seq, last_jiffies, next_jiffies, delta_jiffies;
ktime_t last_update, expires, ret = { .tv64 = 0 };
unsigned long rcu_delta_jiffies;
- struct clock_event_device *dev = __get_cpu_var(tick_cpu_device).evtdev;
+ struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
u64 time_delta;
/* Read jiffies and the time when jiffies were updated last */
@@ -801,7 +801,7 @@ void tick_nohz_idle_enter(void)
local_irq_disable();
- ts = &__get_cpu_var(tick_cpu_sched);
+ ts = this_cpu_ptr(&tick_cpu_sched);
/*
* set ts->inidle unconditionally. even if the system did not
* switch to nohz mode the cpu frequency governers rely on the
@@ -824,7 +824,7 @@ EXPORT_SYMBOL_GPL(tick_nohz_idle_enter);
*/
void tick_nohz_irq_exit(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
if (ts->inidle)
__tick_nohz_idle_enter(ts);
@@ -839,7 +839,7 @@ void tick_nohz_irq_exit(void)
*/
ktime_t tick_nohz_get_sleep_length(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
return ts->sleep_length;
}
@@ -953,7 +953,7 @@ static int tick_nohz_reprogram(struct ti
*/
static void tick_nohz_handler(struct clock_event_device *dev)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
struct pt_regs *regs = get_irq_regs();
ktime_t now = ktime_get();
@@ -973,7 +973,7 @@ static void tick_nohz_handler(struct clo
*/
static void tick_nohz_switch_to_nohz(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
ktime_t next;
if (!tick_nohz_enabled)
@@ -1111,7 +1111,7 @@ early_param("skew_tick", skew_tick);
*/
void tick_setup_sched_timer(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
ktime_t now = ktime_get();
/*
@@ -1178,7 +1178,7 @@ void tick_clock_notify(void)
*/
void tick_oneshot_notify(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
set_bit(0, &ts->check_clocks);
}
@@ -1193,7 +1193,7 @@ void tick_oneshot_notify(void)
*/
int tick_check_oneshot_change(int allow_nohz)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
if (!test_and_clear_bit(0, &ts->check_clocks))
return 0;
Index: linux/kernel/timer.c
===================================================================
--- linux.orig/kernel/timer.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/timer.c 2013-08-26 14:21:24.544227649 -0500
@@ -621,7 +621,7 @@ static inline void debug_assert_init(str
static void do_init_timer(struct timer_list *timer, unsigned int flags,
const char *name, struct lock_class_key *key)
{
- struct tvec_base *base = __raw_get_cpu_var(tvec_bases);
+ struct tvec_base *base = __this_cpu_read(tvec_bases);
timer->entry.next = NULL;
timer->base = (void *)((unsigned long)base | flags);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 04/31] net: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, David S. Miller, netdev, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/net/core/dev.c
===================================================================
--- linux.orig/net/core/dev.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/core/dev.c 2013-08-26 14:18:37.214005168 -0500
@@ -2129,7 +2129,7 @@ static inline void __netif_reschedule(st
unsigned long flags;
local_irq_save(flags);
- sd = &__get_cpu_var(softnet_data);
+ sd = this_cpu_ptr(&softnet_data);
q->next_sched = NULL;
*sd->output_queue_tailp = q;
sd->output_queue_tailp = &q->next_sched;
@@ -2151,7 +2151,7 @@ void dev_kfree_skb_irq(struct sk_buff *s
unsigned long flags;
local_irq_save(flags);
- sd = &__get_cpu_var(softnet_data);
+ sd = this_cpu_ptr(&softnet_data);
skb->next = sd->completion_queue;
sd->completion_queue = skb;
raise_softirq_irqoff(NET_TX_SOFTIRQ);
@@ -3111,7 +3111,7 @@ static void rps_trigger_softirq(void *da
static int rps_ipi_queued(struct softnet_data *sd)
{
#ifdef CONFIG_RPS
- struct softnet_data *mysd = &__get_cpu_var(softnet_data);
+ struct softnet_data *mysd = this_cpu_ptr(&softnet_data);
if (sd != mysd) {
sd->rps_ipi_next = mysd->rps_ipi_list;
@@ -3138,7 +3138,7 @@ static bool skb_flow_limit(struct sk_buf
if (qlen < (netdev_max_backlog >> 1))
return false;
- sd = &__get_cpu_var(softnet_data);
+ sd = this_cpu_ptr(&softnet_data);
rcu_read_lock();
fl = rcu_dereference(sd->flow_limit);
@@ -3280,7 +3280,7 @@ EXPORT_SYMBOL(netif_rx_ni);
static void net_tx_action(struct softirq_action *h)
{
- struct softnet_data *sd = &__get_cpu_var(softnet_data);
+ struct softnet_data *sd = this_cpu_ptr(&softnet_data);
if (sd->completion_queue) {
struct sk_buff *clist;
@@ -3700,7 +3700,7 @@ EXPORT_SYMBOL(netif_receive_skb);
static void flush_backlog(void *arg)
{
struct net_device *dev = arg;
- struct softnet_data *sd = &__get_cpu_var(softnet_data);
+ struct softnet_data *sd = this_cpu_ptr(&softnet_data);
struct sk_buff *skb, *tmp;
rps_lock(sd);
@@ -4146,7 +4146,7 @@ void __napi_schedule(struct napi_struct
unsigned long flags;
local_irq_save(flags);
- ____napi_schedule(&__get_cpu_var(softnet_data), n);
+ ____napi_schedule(this_cpu_ptr(&softnet_data), n);
local_irq_restore(flags);
}
EXPORT_SYMBOL(__napi_schedule);
@@ -4274,7 +4274,7 @@ EXPORT_SYMBOL(netif_napi_del);
static void net_rx_action(struct softirq_action *h)
{
- struct softnet_data *sd = &__get_cpu_var(softnet_data);
+ struct softnet_data *sd = this_cpu_ptr(&softnet_data);
unsigned long time_limit = jiffies + 2;
int budget = netdev_budget;
void *have;
Index: linux/net/core/drop_monitor.c
===================================================================
--- linux.orig/net/core/drop_monitor.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/core/drop_monitor.c 2013-08-26 14:18:37.218005126 -0500
@@ -142,7 +142,7 @@ static void trace_drop_common(struct sk_
unsigned long flags;
local_irq_save(flags);
- data = &__get_cpu_var(dm_cpu_data);
+ data = this_cpu_ptr(&dm_cpu_data);
spin_lock(&data->lock);
dskb = data->skb;
Index: linux/net/core/skbuff.c
===================================================================
--- linux.orig/net/core/skbuff.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/core/skbuff.c 2013-08-26 14:18:37.218005126 -0500
@@ -371,7 +371,7 @@ static void *__netdev_alloc_frag(unsigne
unsigned long flags;
local_irq_save(flags);
- nc = &__get_cpu_var(netdev_alloc_cache);
+ nc = this_cpu_ptr(&netdev_alloc_cache);
if (unlikely(!nc->frag.page)) {
refill:
for (order = NETDEV_FRAG_PAGE_MAX_ORDER; ;) {
Index: linux/net/ipv4/syncookies.c
===================================================================
--- linux.orig/net/ipv4/syncookies.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/ipv4/syncookies.c 2013-08-26 14:18:37.210005210 -0500
@@ -44,7 +44,7 @@ static DEFINE_PER_CPU(__u32 [16 + 5 + SH
static u32 cookie_hash(__be32 saddr, __be32 daddr, __be16 sport, __be16 dport,
u32 count, int c)
{
- __u32 *tmp = __get_cpu_var(ipv4_cookie_scratch);
+ __u32 *tmp = this_cpu_ptr(ipv4_cookie_scratch);
memcpy(tmp + 4, syncookie_secret[c], sizeof(syncookie_secret[c]));
tmp[0] = (__force u32)saddr;
Index: linux/net/ipv4/tcp_output.c
===================================================================
--- linux.orig/net/ipv4/tcp_output.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/ipv4/tcp_output.c 2013-08-26 14:18:37.218005126 -0500
@@ -810,7 +810,7 @@ void tcp_wfree(struct sk_buff *skb)
/* queue this socket to tasklet queue */
local_irq_save(flags);
- tsq = &__get_cpu_var(tsq_tasklet);
+ tsq = this_cpu_ptr(&tsq_tasklet);
list_add(&tp->tsq_node, &tsq->head);
tasklet_schedule(&tsq->tasklet);
local_irq_restore(flags);
Index: linux/net/ipv6/syncookies.c
===================================================================
--- linux.orig/net/ipv6/syncookies.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/ipv6/syncookies.c 2013-08-26 14:18:37.214005168 -0500
@@ -66,7 +66,7 @@ static DEFINE_PER_CPU(__u32 [16 + 5 + SH
static u32 cookie_hash(const struct in6_addr *saddr, const struct in6_addr *daddr,
__be16 sport, __be16 dport, u32 count, int c)
{
- __u32 *tmp = __get_cpu_var(ipv6_cookie_scratch);
+ __u32 *tmp = this_cpu_ptr(ipv6_cookie_scratch);
/*
* we have 320 bits of information to hash, copy in the remaining
Index: linux/net/rds/ib_rdma.c
===================================================================
--- linux.orig/net/rds/ib_rdma.c 2013-08-26 14:16:59.000000000 -0500
+++ linux/net/rds/ib_rdma.c 2013-08-26 14:18:37.218005126 -0500
@@ -267,7 +267,7 @@ static inline struct rds_ib_mr *rds_ib_r
unsigned long *flag;
preempt_disable();
- flag = &__get_cpu_var(clean_list_grace);
+ flag = this_cpu_ptr(&clean_list_grace);
set_bit(CLEAN_LIST_BUSY_BIT, flag);
ret = llist_del_first(&pool->clean_list);
if (ret)
Index: linux/include/net/netfilter/nf_conntrack.h
===================================================================
--- linux.orig/include/net/netfilter/nf_conntrack.h 2013-08-26 14:23:46.000000000 -0500
+++ linux/include/net/netfilter/nf_conntrack.h 2013-08-26 14:24:17.190395755 -0500
@@ -243,7 +243,7 @@ extern s16 (*nf_ct_nat_offset)(const str
DECLARE_PER_CPU(struct nf_conn, nf_conntrack_untracked);
static inline struct nf_conn *nf_ct_untracked_get(void)
{
- return &__raw_get_cpu_var(nf_conntrack_untracked);
+ return __this_cpu_ptr(&nf_conntrack_untracked);
}
extern void nf_ct_untracked_status_or(unsigned long bits);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 05/31] time: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Thomas Gleixner, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/kernel/hrtimer.c
===================================================================
--- linux.orig/kernel/hrtimer.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/hrtimer.c 2013-08-26 14:21:24.540227691 -0500
@@ -597,7 +597,7 @@ hrtimer_force_reprogram(struct hrtimer_c
static int hrtimer_reprogram(struct hrtimer *timer,
struct hrtimer_clock_base *base)
{
- struct hrtimer_cpu_base *cpu_base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
ktime_t expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
int res;
@@ -680,7 +680,7 @@ static inline ktime_t hrtimer_update_bas
*/
static void retrigger_next_event(void *arg)
{
- struct hrtimer_cpu_base *base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *base = this_cpu_ptr(&hrtimer_bases);
if (!hrtimer_hres_active())
return;
@@ -954,7 +954,7 @@ remove_hrtimer(struct hrtimer *timer, st
*/
debug_deactivate(timer);
timer_stats_hrtimer_clear_start_info(timer);
- reprogram = base->cpu_base == &__get_cpu_var(hrtimer_bases);
+ reprogram = base->cpu_base == this_cpu_ptr(&hrtimer_bases);
/*
* We must preserve the CALLBACK state flag here,
* otherwise we could move the timer base in
@@ -1009,7 +1009,7 @@ int __hrtimer_start_range_ns(struct hrti
*
* XXX send_remote_softirq() ?
*/
- if (leftmost && new_base->cpu_base == &__get_cpu_var(hrtimer_bases)
+ if (leftmost && new_base->cpu_base == this_cpu_ptr(&hrtimer_bases)
&& hrtimer_enqueue_reprogram(timer, new_base)) {
if (wakeup) {
/*
@@ -1142,7 +1142,7 @@ EXPORT_SYMBOL_GPL(hrtimer_get_remaining)
*/
ktime_t hrtimer_get_next_event(void)
{
- struct hrtimer_cpu_base *cpu_base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
struct hrtimer_clock_base *base = cpu_base->clock_base;
ktime_t delta, mindelta = { .tv64 = KTIME_MAX };
unsigned long flags;
@@ -1183,7 +1183,7 @@ static void __hrtimer_init(struct hrtime
memset(timer, 0, sizeof(struct hrtimer));
- cpu_base = &__raw_get_cpu_var(hrtimer_bases);
+ cpu_base = __this_cpu_ptr(&hrtimer_bases);
if (clock_id == CLOCK_REALTIME && mode != HRTIMER_MODE_ABS)
clock_id = CLOCK_MONOTONIC;
@@ -1226,7 +1226,7 @@ int hrtimer_get_res(const clockid_t whic
struct hrtimer_cpu_base *cpu_base;
int base = hrtimer_clockid_to_base(which_clock);
- cpu_base = &__raw_get_cpu_var(hrtimer_bases);
+ cpu_base = __this_cpu_ptr(&hrtimer_bases);
*tp = ktime_to_timespec(cpu_base->clock_base[base].resolution);
return 0;
@@ -1281,7 +1281,7 @@ static void __run_hrtimer(struct hrtimer
*/
void hrtimer_interrupt(struct clock_event_device *dev)
{
- struct hrtimer_cpu_base *cpu_base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
ktime_t expires_next, now, entry_time, delta;
int i, retries = 0;
@@ -1415,7 +1415,7 @@ static void __hrtimer_peek_ahead_timers(
if (!hrtimer_hres_active())
return;
- td = &__get_cpu_var(tick_cpu_device);
+ td = this_cpu_ptr(&tick_cpu_device);
if (td && td->evtdev)
hrtimer_interrupt(td->evtdev);
}
@@ -1479,7 +1479,7 @@ void hrtimer_run_pending(void)
void hrtimer_run_queues(void)
{
struct timerqueue_node *node;
- struct hrtimer_cpu_base *cpu_base = &__get_cpu_var(hrtimer_bases);
+ struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
struct hrtimer_clock_base *base;
int index, gettime = 1;
@@ -1717,7 +1717,7 @@ static void migrate_hrtimers(int scpu)
local_irq_disable();
old_base = &per_cpu(hrtimer_bases, scpu);
- new_base = &__get_cpu_var(hrtimer_bases);
+ new_base = this_cpu_ptr(&hrtimer_bases);
/*
* The caller is globally serialized and nobody else
* takes two locks at once, deadlock is not possible.
Index: linux/kernel/irq_work.c
===================================================================
--- linux.orig/kernel/irq_work.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/irq_work.c 2013-08-26 14:21:24.540227691 -0500
@@ -70,7 +70,7 @@ void irq_work_queue(struct irq_work *wor
/* Queue the entry and raise the IPI if needed. */
preempt_disable();
- llist_add(&work->llnode, &__get_cpu_var(irq_work_list));
+ llist_add(&work->llnode, this_cpu_ptr(&irq_work_list));
/*
* If the work is not "lazy" or the tick is stopped, raise the irq
@@ -90,7 +90,7 @@ bool irq_work_needs_cpu(void)
{
struct llist_head *this_list;
- this_list = &__get_cpu_var(irq_work_list);
+ this_list = this_cpu_ptr(&irq_work_list);
if (llist_empty(this_list))
return false;
@@ -115,7 +115,7 @@ static void __irq_work_run(void)
__this_cpu_write(irq_work_raised, 0);
barrier();
- this_list = &__get_cpu_var(irq_work_list);
+ this_list = this_cpu_ptr(&irq_work_list);
if (llist_empty(this_list))
return;
Index: linux/kernel/sched/clock.c
===================================================================
--- linux.orig/kernel/sched/clock.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/sched/clock.c 2013-08-26 14:21:24.540227691 -0500
@@ -94,7 +94,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(str
static inline struct sched_clock_data *this_scd(void)
{
- return &__get_cpu_var(sched_clock_data);
+ return this_cpu_ptr(&sched_clock_data);
}
static inline struct sched_clock_data *cpu_sdc(int cpu)
Index: linux/kernel/softirq.c
===================================================================
--- linux.orig/kernel/softirq.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/softirq.c 2013-08-26 14:21:24.540227691 -0500
@@ -466,7 +466,7 @@ static void tasklet_action(struct softir
local_irq_disable();
list = __this_cpu_read(tasklet_vec.head);
__this_cpu_write(tasklet_vec.head, NULL);
- __this_cpu_write(tasklet_vec.tail, &__get_cpu_var(tasklet_vec).head);
+ __this_cpu_write(tasklet_vec.tail, this_cpu_ptr(&tasklet_vec.head));
local_irq_enable();
while (list) {
@@ -501,7 +501,7 @@ static void tasklet_hi_action(struct sof
local_irq_disable();
list = __this_cpu_read(tasklet_hi_vec.head);
__this_cpu_write(tasklet_hi_vec.head, NULL);
- __this_cpu_write(tasklet_hi_vec.tail, &__get_cpu_var(tasklet_hi_vec).head);
+ __this_cpu_write(tasklet_hi_vec.tail, this_cpu_ptr(&tasklet_hi_vec.head));
local_irq_enable();
while (list) {
@@ -618,7 +618,7 @@ EXPORT_PER_CPU_SYMBOL(softirq_work_list)
static void __local_trigger(struct call_single_data *cp, int softirq)
{
- struct list_head *head = &__get_cpu_var(softirq_work_list[softirq]);
+ struct list_head *head = this_cpu_ptr(&softirq_work_list[softirq]);
list_add_tail(&cp->list, head);
@@ -718,7 +718,7 @@ static int remote_softirq_cpu_notify(str
if (list_empty(head))
continue;
- local_head = &__get_cpu_var(softirq_work_list[i]);
+ local_head = this_cpu_ptr(&softirq_work_list[i]);
list_splice_init(head, local_head);
raise_softirq_irqoff(i);
}
Index: linux/kernel/time/tick-common.c
===================================================================
--- linux.orig/kernel/time/tick-common.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/time/tick-common.c 2013-08-26 14:21:24.544227649 -0500
@@ -208,7 +208,7 @@ static void tick_setup_device(struct tic
void tick_install_replacement(struct clock_event_device *newdev)
{
- struct tick_device *td = &__get_cpu_var(tick_cpu_device);
+ struct tick_device *td = this_cpu_ptr(&tick_cpu_device);
int cpu = smp_processor_id();
clockevents_exchange_device(td->evtdev, newdev);
@@ -358,14 +358,14 @@ void tick_shutdown(unsigned int *cpup)
void tick_suspend(void)
{
- struct tick_device *td = &__get_cpu_var(tick_cpu_device);
+ struct tick_device *td = this_cpu_ptr(&tick_cpu_device);
clockevents_shutdown(td->evtdev);
}
void tick_resume(void)
{
- struct tick_device *td = &__get_cpu_var(tick_cpu_device);
+ struct tick_device *td = this_cpu_ptr(&tick_cpu_device);
int broadcast = tick_resume_broadcast();
clockevents_set_mode(td->evtdev, CLOCK_EVT_MODE_RESUME);
Index: linux/kernel/time/tick-oneshot.c
===================================================================
--- linux.orig/kernel/time/tick-oneshot.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/time/tick-oneshot.c 2013-08-26 14:21:24.544227649 -0500
@@ -59,7 +59,7 @@ void tick_setup_oneshot(struct clock_eve
*/
int tick_switch_to_oneshot(void (*handler)(struct clock_event_device *))
{
- struct tick_device *td = &__get_cpu_var(tick_cpu_device);
+ struct tick_device *td = this_cpu_ptr(&tick_cpu_device);
struct clock_event_device *dev = td->evtdev;
if (!dev || !(dev->features & CLOCK_EVT_FEAT_ONESHOT) ||
Index: linux/kernel/time/tick-sched.c
===================================================================
--- linux.orig/kernel/time/tick-sched.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/time/tick-sched.c 2013-08-26 14:21:24.540227691 -0500
@@ -199,7 +199,7 @@ static void tick_nohz_restart_sched_tick
*/
void tick_nohz_full_check(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
if (tick_nohz_full_cpu(smp_processor_id())) {
if (ts->tick_stopped && !is_idle_task(current)) {
@@ -225,7 +225,7 @@ static DEFINE_PER_CPU(struct irq_work, n
void tick_nohz_full_kick(void)
{
if (tick_nohz_full_cpu(smp_processor_id()))
- irq_work_queue(&__get_cpu_var(nohz_full_kick_work));
+ irq_work_queue(this_cpu_ptr(&nohz_full_kick_work));
}
static void nohz_full_kick_ipi(void *info)
@@ -536,7 +536,7 @@ static ktime_t tick_nohz_stop_sched_tick
unsigned long seq, last_jiffies, next_jiffies, delta_jiffies;
ktime_t last_update, expires, ret = { .tv64 = 0 };
unsigned long rcu_delta_jiffies;
- struct clock_event_device *dev = __get_cpu_var(tick_cpu_device).evtdev;
+ struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
u64 time_delta;
/* Read jiffies and the time when jiffies were updated last */
@@ -801,7 +801,7 @@ void tick_nohz_idle_enter(void)
local_irq_disable();
- ts = &__get_cpu_var(tick_cpu_sched);
+ ts = this_cpu_ptr(&tick_cpu_sched);
/*
* set ts->inidle unconditionally. even if the system did not
* switch to nohz mode the cpu frequency governers rely on the
@@ -824,7 +824,7 @@ EXPORT_SYMBOL_GPL(tick_nohz_idle_enter);
*/
void tick_nohz_irq_exit(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
if (ts->inidle)
__tick_nohz_idle_enter(ts);
@@ -839,7 +839,7 @@ void tick_nohz_irq_exit(void)
*/
ktime_t tick_nohz_get_sleep_length(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
return ts->sleep_length;
}
@@ -953,7 +953,7 @@ static int tick_nohz_reprogram(struct ti
*/
static void tick_nohz_handler(struct clock_event_device *dev)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
struct pt_regs *regs = get_irq_regs();
ktime_t now = ktime_get();
@@ -973,7 +973,7 @@ static void tick_nohz_handler(struct clo
*/
static void tick_nohz_switch_to_nohz(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
ktime_t next;
if (!tick_nohz_enabled)
@@ -1111,7 +1111,7 @@ early_param("skew_tick", skew_tick);
*/
void tick_setup_sched_timer(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
ktime_t now = ktime_get();
/*
@@ -1178,7 +1178,7 @@ void tick_clock_notify(void)
*/
void tick_oneshot_notify(void)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
set_bit(0, &ts->check_clocks);
}
@@ -1193,7 +1193,7 @@ void tick_oneshot_notify(void)
*/
int tick_check_oneshot_change(int allow_nohz)
{
- struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
if (!test_and_clear_bit(0, &ts->check_clocks))
return 0;
Index: linux/kernel/timer.c
===================================================================
--- linux.orig/kernel/timer.c 2013-08-26 14:19:14.000000000 -0500
+++ linux/kernel/timer.c 2013-08-26 14:21:24.544227649 -0500
@@ -621,7 +621,7 @@ static inline void debug_assert_init(str
static void do_init_timer(struct timer_list *timer, unsigned int flags,
const char *name, struct lock_class_key *key)
{
- struct tvec_base *base = __raw_get_cpu_var(tvec_bases);
+ struct tvec_base *base = __this_cpu_read(tvec_bases);
timer->entry.next = NULL;
timer->base = (void *)((unsigned long)base | flags);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 06/31] scheduler: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Ingo Molnar, Peter Zijlstra, linux-arch, Steven Rostedt,
linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/kernel_stat.h
===================================================================
--- linux.orig/include/linux/kernel_stat.h 2013-08-26 14:21:39.000000000 -0500
+++ linux/include/linux/kernel_stat.h 2013-08-26 14:23:08.883120293 -0500
@@ -47,8 +47,8 @@ DECLARE_PER_CPU(struct kernel_stat, ksta
DECLARE_PER_CPU(struct kernel_cpustat, kernel_cpustat);
/* Must have preemption disabled for this to be meaningful. */
-#define kstat_this_cpu (&__get_cpu_var(kstat))
-#define kcpustat_this_cpu (&__get_cpu_var(kernel_cpustat))
+#define kstat_this_cpu this_cpu_ptr(&kstat)
+#define kcpustat_this_cpu this_cpu_ptr(&kernel_cpustat)
#define kstat_cpu(cpu) per_cpu(kstat, cpu)
#define kcpustat_cpu(cpu) per_cpu(kernel_cpustat, cpu)
Index: linux/kernel/events/callchain.c
===================================================================
--- linux.orig/kernel/events/callchain.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/events/callchain.c 2013-08-26 14:23:08.875120379 -0500
@@ -134,7 +134,7 @@ static struct perf_callchain_entry *get_
int cpu;
struct callchain_cpus_entries *entries;
- *rctx = get_recursion_context(__get_cpu_var(callchain_recursion));
+ *rctx = get_recursion_context(this_cpu_ptr(callchain_recursion));
if (*rctx == -1)
return NULL;
@@ -150,7 +150,7 @@ static struct perf_callchain_entry *get_
static void
put_callchain_entry(int rctx)
{
- put_recursion_context(__get_cpu_var(callchain_recursion), rctx);
+ put_recursion_context(this_cpu_ptr(callchain_recursion), rctx);
}
struct perf_callchain_entry *
Index: linux/kernel/events/core.c
===================================================================
--- linux.orig/kernel/events/core.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/events/core.c 2013-08-26 14:23:08.875120379 -0500
@@ -238,10 +238,10 @@ void perf_sample_event_took(u64 sample_l
return;
/* decay the counter by 1 average sample */
- local_samples_len = __get_cpu_var(running_sample_length);
+ local_samples_len = __this_cpu_read(running_sample_length);
local_samples_len -= local_samples_len/NR_ACCUMULATED_SAMPLES;
local_samples_len += sample_len_ns;
- __get_cpu_var(running_sample_length) = local_samples_len;
+ __this_cpu_write(running_sample_length, local_samples_len);
/*
* note: this will be biased artifically low until we have
@@ -865,7 +865,7 @@ static DEFINE_PER_CPU(struct list_head,
static void perf_pmu_rotate_start(struct pmu *pmu)
{
struct perf_cpu_context *cpuctx = this_cpu_ptr(pmu->pmu_cpu_context);
- struct list_head *head = &__get_cpu_var(rotation_list);
+ struct list_head *head = this_cpu_ptr(&rotation_list);
WARN_ON(!irqs_disabled());
@@ -2321,7 +2321,7 @@ void __perf_event_task_sched_out(struct
* to check if we have to switch out PMU state.
* cgroup event are system-wide mode only
*/
- if (atomic_read(&__get_cpu_var(perf_cgroup_events)))
+ if (atomic_read(this_cpu_ptr(&perf_cgroup_events)))
perf_cgroup_sched_out(task, next);
}
@@ -2566,11 +2566,11 @@ void __perf_event_task_sched_in(struct t
* to check if we have to switch in PMU state.
* cgroup event are system-wide mode only
*/
- if (atomic_read(&__get_cpu_var(perf_cgroup_events)))
+ if (atomic_read(this_cpu_ptr(&perf_cgroup_events)))
perf_cgroup_sched_in(prev, task);
/* check for system-wide branch_stack events */
- if (atomic_read(&__get_cpu_var(perf_branch_stack_events)))
+ if (atomic_read(this_cpu_ptr(&perf_branch_stack_events)))
perf_branch_stack_sched_in(prev, task);
}
@@ -2811,7 +2811,7 @@ done:
#ifdef CONFIG_NO_HZ_FULL
bool perf_event_can_stop_tick(void)
{
- if (list_empty(&__get_cpu_var(rotation_list)))
+ if (list_empty(this_cpu_ptr(&rotation_list)))
return true;
else
return false;
@@ -2820,7 +2820,7 @@ bool perf_event_can_stop_tick(void)
void perf_event_task_tick(void)
{
- struct list_head *head = &__get_cpu_var(rotation_list);
+ struct list_head *head = this_cpu_ptr(&rotation_list);
struct perf_cpu_context *cpuctx, *tmp;
struct perf_event_context *ctx;
int throttled;
@@ -5414,7 +5414,7 @@ static void do_perf_sw_event(enum perf_t
struct perf_sample_data *data,
struct pt_regs *regs)
{
- struct swevent_htable *swhash = &__get_cpu_var(swevent_htable);
+ struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
struct perf_event *event;
struct hlist_head *head;
@@ -5433,7 +5433,7 @@ end:
int perf_swevent_get_recursion_context(void)
{
- struct swevent_htable *swhash = &__get_cpu_var(swevent_htable);
+ struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
return get_recursion_context(swhash->recursion);
}
@@ -5441,7 +5441,7 @@ EXPORT_SYMBOL_GPL(perf_swevent_get_recur
inline void perf_swevent_put_recursion_context(int rctx)
{
- struct swevent_htable *swhash = &__get_cpu_var(swevent_htable);
+ struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
put_recursion_context(swhash->recursion, rctx);
}
@@ -5470,7 +5470,7 @@ static void perf_swevent_read(struct per
static int perf_swevent_add(struct perf_event *event, int flags)
{
- struct swevent_htable *swhash = &__get_cpu_var(swevent_htable);
+ struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
struct hw_perf_event *hwc = &event->hw;
struct hlist_head *head;
Index: linux/kernel/sched/cputime.c
===================================================================
--- linux.orig/kernel/sched/cputime.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/sched/cputime.c 2013-08-26 14:23:08.879120335 -0500
@@ -121,7 +121,7 @@ static inline void task_group_account_fi
* is the only cgroup, then nothing else should be necessary.
*
*/
- __get_cpu_var(kernel_cpustat).cpustat[index] += tmp;
+ __this_cpu_add(kernel_cpustat.cpustat[index], tmp);
cpuacct_account_field(p, index, tmp);
}
Index: linux/kernel/sched/fair.c
===================================================================
--- linux.orig/kernel/sched/fair.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/sched/fair.c 2013-08-26 14:23:08.879120335 -0500
@@ -5057,7 +5057,7 @@ static int load_balance(int this_cpu, st
struct sched_group *group;
struct rq *busiest;
unsigned long flags;
- struct cpumask *cpus = __get_cpu_var(load_balance_mask);
+ struct cpumask *cpus = this_cpu_ptr(load_balance_mask);
struct lb_env env = {
.sd = sd,
Index: linux/kernel/sched/rt.c
===================================================================
--- linux.orig/kernel/sched/rt.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/sched/rt.c 2013-08-26 14:23:08.883120293 -0500
@@ -1389,7 +1389,7 @@ static DEFINE_PER_CPU(cpumask_var_t, loc
static int find_lowest_rq(struct task_struct *task)
{
struct sched_domain *sd;
- struct cpumask *lowest_mask = __get_cpu_var(local_cpu_mask);
+ struct cpumask *lowest_mask = this_cpu_ptr(local_cpu_mask);
int this_cpu = smp_processor_id();
int cpu = task_cpu(task);
Index: linux/kernel/sched/sched.h
===================================================================
--- linux.orig/kernel/sched/sched.h 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/sched/sched.h 2013-08-26 14:23:08.883120293 -0500
@@ -538,10 +538,10 @@ static inline int cpu_of(struct rq *rq)
DECLARE_PER_CPU(struct rq, runqueues);
#define cpu_rq(cpu) (&per_cpu(runqueues, (cpu)))
-#define this_rq() (&__get_cpu_var(runqueues))
+#define this_rq() this_cpu_ptr(&runqueues)
#define task_rq(p) cpu_rq(task_cpu(p))
#define cpu_curr(cpu) (cpu_rq(cpu)->curr)
-#define raw_rq() (&__raw_get_cpu_var(runqueues))
+#define raw_rq() __this_cpu_ptr(&runqueues)
static inline u64 rq_clock(struct rq *rq)
{
Index: linux/kernel/user-return-notifier.c
===================================================================
--- linux.orig/kernel/user-return-notifier.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/user-return-notifier.c 2013-08-26 14:23:08.883120293 -0500
@@ -14,7 +14,7 @@ static DEFINE_PER_CPU(struct hlist_head,
void user_return_notifier_register(struct user_return_notifier *urn)
{
set_tsk_thread_flag(current, TIF_USER_RETURN_NOTIFY);
- hlist_add_head(&urn->link, &__get_cpu_var(return_notifier_list));
+ hlist_add_head(&urn->link, this_cpu_ptr(&return_notifier_list));
}
EXPORT_SYMBOL_GPL(user_return_notifier_register);
@@ -25,7 +25,7 @@ EXPORT_SYMBOL_GPL(user_return_notifier_r
void user_return_notifier_unregister(struct user_return_notifier *urn)
{
hlist_del(&urn->link);
- if (hlist_empty(&__get_cpu_var(return_notifier_list)))
+ if (hlist_empty(this_cpu_ptr(&return_notifier_list)))
clear_tsk_thread_flag(current, TIF_USER_RETURN_NOTIFY);
}
EXPORT_SYMBOL_GPL(user_return_notifier_unregister);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 06/31] scheduler: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Ingo Molnar, Peter Zijlstra, linux-arch, Steven Rostedt,
linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/kernel_stat.h
===================================================================
--- linux.orig/include/linux/kernel_stat.h 2013-08-26 14:21:39.000000000 -0500
+++ linux/include/linux/kernel_stat.h 2013-08-26 14:23:08.883120293 -0500
@@ -47,8 +47,8 @@ DECLARE_PER_CPU(struct kernel_stat, ksta
DECLARE_PER_CPU(struct kernel_cpustat, kernel_cpustat);
/* Must have preemption disabled for this to be meaningful. */
-#define kstat_this_cpu (&__get_cpu_var(kstat))
-#define kcpustat_this_cpu (&__get_cpu_var(kernel_cpustat))
+#define kstat_this_cpu this_cpu_ptr(&kstat)
+#define kcpustat_this_cpu this_cpu_ptr(&kernel_cpustat)
#define kstat_cpu(cpu) per_cpu(kstat, cpu)
#define kcpustat_cpu(cpu) per_cpu(kernel_cpustat, cpu)
Index: linux/kernel/events/callchain.c
===================================================================
--- linux.orig/kernel/events/callchain.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/events/callchain.c 2013-08-26 14:23:08.875120379 -0500
@@ -134,7 +134,7 @@ static struct perf_callchain_entry *get_
int cpu;
struct callchain_cpus_entries *entries;
- *rctx = get_recursion_context(__get_cpu_var(callchain_recursion));
+ *rctx = get_recursion_context(this_cpu_ptr(callchain_recursion));
if (*rctx == -1)
return NULL;
@@ -150,7 +150,7 @@ static struct perf_callchain_entry *get_
static void
put_callchain_entry(int rctx)
{
- put_recursion_context(__get_cpu_var(callchain_recursion), rctx);
+ put_recursion_context(this_cpu_ptr(callchain_recursion), rctx);
}
struct perf_callchain_entry *
Index: linux/kernel/events/core.c
===================================================================
--- linux.orig/kernel/events/core.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/events/core.c 2013-08-26 14:23:08.875120379 -0500
@@ -238,10 +238,10 @@ void perf_sample_event_took(u64 sample_l
return;
/* decay the counter by 1 average sample */
- local_samples_len = __get_cpu_var(running_sample_length);
+ local_samples_len = __this_cpu_read(running_sample_length);
local_samples_len -= local_samples_len/NR_ACCUMULATED_SAMPLES;
local_samples_len += sample_len_ns;
- __get_cpu_var(running_sample_length) = local_samples_len;
+ __this_cpu_write(running_sample_length, local_samples_len);
/*
* note: this will be biased artifically low until we have
@@ -865,7 +865,7 @@ static DEFINE_PER_CPU(struct list_head,
static void perf_pmu_rotate_start(struct pmu *pmu)
{
struct perf_cpu_context *cpuctx = this_cpu_ptr(pmu->pmu_cpu_context);
- struct list_head *head = &__get_cpu_var(rotation_list);
+ struct list_head *head = this_cpu_ptr(&rotation_list);
WARN_ON(!irqs_disabled());
@@ -2321,7 +2321,7 @@ void __perf_event_task_sched_out(struct
* to check if we have to switch out PMU state.
* cgroup event are system-wide mode only
*/
- if (atomic_read(&__get_cpu_var(perf_cgroup_events)))
+ if (atomic_read(this_cpu_ptr(&perf_cgroup_events)))
perf_cgroup_sched_out(task, next);
}
@@ -2566,11 +2566,11 @@ void __perf_event_task_sched_in(struct t
* to check if we have to switch in PMU state.
* cgroup event are system-wide mode only
*/
- if (atomic_read(&__get_cpu_var(perf_cgroup_events)))
+ if (atomic_read(this_cpu_ptr(&perf_cgroup_events)))
perf_cgroup_sched_in(prev, task);
/* check for system-wide branch_stack events */
- if (atomic_read(&__get_cpu_var(perf_branch_stack_events)))
+ if (atomic_read(this_cpu_ptr(&perf_branch_stack_events)))
perf_branch_stack_sched_in(prev, task);
}
@@ -2811,7 +2811,7 @@ done:
#ifdef CONFIG_NO_HZ_FULL
bool perf_event_can_stop_tick(void)
{
- if (list_empty(&__get_cpu_var(rotation_list)))
+ if (list_empty(this_cpu_ptr(&rotation_list)))
return true;
else
return false;
@@ -2820,7 +2820,7 @@ bool perf_event_can_stop_tick(void)
void perf_event_task_tick(void)
{
- struct list_head *head = &__get_cpu_var(rotation_list);
+ struct list_head *head = this_cpu_ptr(&rotation_list);
struct perf_cpu_context *cpuctx, *tmp;
struct perf_event_context *ctx;
int throttled;
@@ -5414,7 +5414,7 @@ static void do_perf_sw_event(enum perf_t
struct perf_sample_data *data,
struct pt_regs *regs)
{
- struct swevent_htable *swhash = &__get_cpu_var(swevent_htable);
+ struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
struct perf_event *event;
struct hlist_head *head;
@@ -5433,7 +5433,7 @@ end:
int perf_swevent_get_recursion_context(void)
{
- struct swevent_htable *swhash = &__get_cpu_var(swevent_htable);
+ struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
return get_recursion_context(swhash->recursion);
}
@@ -5441,7 +5441,7 @@ EXPORT_SYMBOL_GPL(perf_swevent_get_recur
inline void perf_swevent_put_recursion_context(int rctx)
{
- struct swevent_htable *swhash = &__get_cpu_var(swevent_htable);
+ struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
put_recursion_context(swhash->recursion, rctx);
}
@@ -5470,7 +5470,7 @@ static void perf_swevent_read(struct per
static int perf_swevent_add(struct perf_event *event, int flags)
{
- struct swevent_htable *swhash = &__get_cpu_var(swevent_htable);
+ struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
struct hw_perf_event *hwc = &event->hw;
struct hlist_head *head;
Index: linux/kernel/sched/cputime.c
===================================================================
--- linux.orig/kernel/sched/cputime.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/sched/cputime.c 2013-08-26 14:23:08.879120335 -0500
@@ -121,7 +121,7 @@ static inline void task_group_account_fi
* is the only cgroup, then nothing else should be necessary.
*
*/
- __get_cpu_var(kernel_cpustat).cpustat[index] += tmp;
+ __this_cpu_add(kernel_cpustat.cpustat[index], tmp);
cpuacct_account_field(p, index, tmp);
}
Index: linux/kernel/sched/fair.c
===================================================================
--- linux.orig/kernel/sched/fair.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/sched/fair.c 2013-08-26 14:23:08.879120335 -0500
@@ -5057,7 +5057,7 @@ static int load_balance(int this_cpu, st
struct sched_group *group;
struct rq *busiest;
unsigned long flags;
- struct cpumask *cpus = __get_cpu_var(load_balance_mask);
+ struct cpumask *cpus = this_cpu_ptr(load_balance_mask);
struct lb_env env = {
.sd = sd,
Index: linux/kernel/sched/rt.c
===================================================================
--- linux.orig/kernel/sched/rt.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/sched/rt.c 2013-08-26 14:23:08.883120293 -0500
@@ -1389,7 +1389,7 @@ static DEFINE_PER_CPU(cpumask_var_t, loc
static int find_lowest_rq(struct task_struct *task)
{
struct sched_domain *sd;
- struct cpumask *lowest_mask = __get_cpu_var(local_cpu_mask);
+ struct cpumask *lowest_mask = this_cpu_ptr(local_cpu_mask);
int this_cpu = smp_processor_id();
int cpu = task_cpu(task);
Index: linux/kernel/sched/sched.h
===================================================================
--- linux.orig/kernel/sched/sched.h 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/sched/sched.h 2013-08-26 14:23:08.883120293 -0500
@@ -538,10 +538,10 @@ static inline int cpu_of(struct rq *rq)
DECLARE_PER_CPU(struct rq, runqueues);
#define cpu_rq(cpu) (&per_cpu(runqueues, (cpu)))
-#define this_rq() (&__get_cpu_var(runqueues))
+#define this_rq() this_cpu_ptr(&runqueues)
#define task_rq(p) cpu_rq(task_cpu(p))
#define cpu_curr(cpu) (cpu_rq(cpu)->curr)
-#define raw_rq() (&__raw_get_cpu_var(runqueues))
+#define raw_rq() __this_cpu_ptr(&runqueues)
static inline u64 rq_clock(struct rq *rq)
{
Index: linux/kernel/user-return-notifier.c
===================================================================
--- linux.orig/kernel/user-return-notifier.c 2013-08-26 14:21:39.000000000 -0500
+++ linux/kernel/user-return-notifier.c 2013-08-26 14:23:08.883120293 -0500
@@ -14,7 +14,7 @@ static DEFINE_PER_CPU(struct hlist_head,
void user_return_notifier_register(struct user_return_notifier *urn)
{
set_tsk_thread_flag(current, TIF_USER_RETURN_NOTIFY);
- hlist_add_head(&urn->link, &__get_cpu_var(return_notifier_list));
+ hlist_add_head(&urn->link, this_cpu_ptr(&return_notifier_list));
}
EXPORT_SYMBOL_GPL(user_return_notifier_register);
@@ -25,7 +25,7 @@ EXPORT_SYMBOL_GPL(user_return_notifier_r
void user_return_notifier_unregister(struct user_return_notifier *urn)
{
hlist_del(&urn->link);
- if (hlist_empty(&__get_cpu_var(return_notifier_list)))
+ if (hlist_empty(this_cpu_ptr(&return_notifier_list)))
clear_tsk_thread_flag(current, TIF_USER_RETURN_NOTIFY);
}
EXPORT_SYMBOL_GPL(user_return_notifier_unregister);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 09/31] block: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Jens Axboe, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/block/blk-iopoll.c
===================================================================
--- linux.orig/block/blk-iopoll.c 2013-08-26 14:36:07.858875667 -0500
+++ linux/block/blk-iopoll.c 2013-08-26 14:36:07.850875751 -0500
@@ -35,7 +35,7 @@ void blk_iopoll_sched(struct blk_iopoll
unsigned long flags;
local_irq_save(flags);
- list_add_tail(&iop->list, &__get_cpu_var(blk_cpu_iopoll));
+ list_add_tail(&iop->list, this_cpu_ptr(&blk_cpu_iopoll));
__raise_softirq_irqoff(BLOCK_IOPOLL_SOFTIRQ);
local_irq_restore(flags);
}
@@ -79,7 +79,7 @@ EXPORT_SYMBOL(blk_iopoll_complete);
static void blk_iopoll_softirq(struct softirq_action *h)
{
- struct list_head *list = &__get_cpu_var(blk_cpu_iopoll);
+ struct list_head *list = this_cpu_ptr(&blk_cpu_iopoll);
int rearm = 0, budget = blk_iopoll_budget;
unsigned long start_time = jiffies;
@@ -201,7 +201,7 @@ static int blk_iopoll_cpu_notify(struct
local_irq_disable();
list_splice_init(&per_cpu(blk_cpu_iopoll, cpu),
- &__get_cpu_var(blk_cpu_iopoll));
+ this_cpu_ptr(&blk_cpu_iopoll));
__raise_softirq_irqoff(BLOCK_IOPOLL_SOFTIRQ);
local_irq_enable();
}
Index: linux/block/blk-softirq.c
===================================================================
--- linux.orig/block/blk-softirq.c 2013-08-26 14:36:07.858875667 -0500
+++ linux/block/blk-softirq.c 2013-08-26 14:36:07.854875709 -0500
@@ -23,7 +23,7 @@ static void blk_done_softirq(struct soft
struct list_head *cpu_list, local_list;
local_irq_disable();
- cpu_list = &__get_cpu_var(blk_cpu_done);
+ cpu_list = this_cpu_ptr(&blk_cpu_done);
list_replace_init(cpu_list, &local_list);
local_irq_enable();
@@ -44,7 +44,7 @@ static void trigger_softirq(void *data)
struct list_head *list;
local_irq_save(flags);
- list = &__get_cpu_var(blk_cpu_done);
+ list = this_cpu_ptr(&blk_cpu_done);
list_add_tail(&rq->csd.list, list);
if (list->next == &rq->csd.list)
@@ -90,7 +90,7 @@ static int blk_cpu_notify(struct notifie
local_irq_disable();
list_splice_init(&per_cpu(blk_cpu_done, cpu),
- &__get_cpu_var(blk_cpu_done));
+ this_cpu_ptr(&blk_cpu_done));
raise_softirq_irqoff(BLOCK_SOFTIRQ);
local_irq_enable();
}
@@ -135,7 +135,7 @@ void __blk_complete_request(struct reque
if (ccpu == cpu || shared) {
struct list_head *list;
do_local:
- list = &__get_cpu_var(blk_cpu_done);
+ list = this_cpu_ptr(&blk_cpu_done);
list_add_tail(&req->csd.list, list);
/*
Index: linux/fs/fscache/object.c
===================================================================
--- linux.orig/fs/fscache/object.c 2013-08-26 14:36:07.858875667 -0500
+++ linux/fs/fscache/object.c 2013-08-26 14:36:07.854875709 -0500
@@ -796,7 +796,7 @@ void fscache_enqueue_object(struct fscac
*/
bool fscache_object_sleep_till_congested(signed long *timeoutp)
{
- wait_queue_head_t *cong_wq = &__get_cpu_var(fscache_object_cong_wait);
+ wait_queue_head_t *cong_wq = this_cpu_ptr(&fscache_object_cong_wait);
DEFINE_WAIT(wait);
if (fscache_object_congested())
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 08/31] tracing: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Steven Rostedt, Frederic Weisbecker, Ingo Molnar,
linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/kprobes.h
===================================================================
--- linux.orig/include/linux/kprobes.h 2013-08-26 14:25:53.000000000 -0500
+++ linux/include/linux/kprobes.h 2013-08-26 14:26:29.460993659 -0500
@@ -329,7 +329,7 @@ static inline void reset_current_kprobe(
static inline struct kprobe_ctlblk *get_kprobe_ctlblk(void)
{
- return (&__get_cpu_var(kprobe_ctlblk));
+ return this_cpu_ptr(&kprobe_ctlblk);
}
int register_kprobe(struct kprobe *p);
Index: linux/kernel/trace/ftrace.c
===================================================================
--- linux.orig/kernel/trace/ftrace.c 2013-08-26 14:25:53.000000000 -0500
+++ linux/kernel/trace/ftrace.c 2013-08-26 14:26:29.460993659 -0500
@@ -870,7 +870,7 @@ function_profile_call(unsigned long ip,
local_irq_save(flags);
- stat = &__get_cpu_var(ftrace_profile_stats);
+ stat = this_cpu_ptr(&ftrace_profile_stats);
if (!stat->hash || !ftrace_profile_enabled)
goto out;
@@ -901,7 +901,7 @@ static void profile_graph_return(struct
unsigned long flags;
local_irq_save(flags);
- stat = &__get_cpu_var(ftrace_profile_stats);
+ stat = this_cpu_ptr(&ftrace_profile_stats);
if (!stat->hash || !ftrace_profile_enabled)
goto out;
Index: linux/kernel/trace/trace.c
===================================================================
--- linux.orig/kernel/trace/trace.c 2013-08-26 14:25:53.000000000 -0500
+++ linux/kernel/trace/trace.c 2013-08-26 14:26:29.464993617 -0500
@@ -1676,7 +1676,7 @@ static void __ftrace_trace_stack(struct
*/
barrier();
if (use_stack == 1) {
- trace.entries = &__get_cpu_var(ftrace_stack).calls[0];
+ trace.entries = this_cpu_ptr(ftrace_stack.calls);
trace.max_entries = FTRACE_STACK_MAX_ENTRIES;
if (regs)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 09/31] block: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Jens Axboe, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/block/blk-iopoll.c
===================================================================
--- linux.orig/block/blk-iopoll.c 2013-08-26 14:36:07.858875667 -0500
+++ linux/block/blk-iopoll.c 2013-08-26 14:36:07.850875751 -0500
@@ -35,7 +35,7 @@ void blk_iopoll_sched(struct blk_iopoll
unsigned long flags;
local_irq_save(flags);
- list_add_tail(&iop->list, &__get_cpu_var(blk_cpu_iopoll));
+ list_add_tail(&iop->list, this_cpu_ptr(&blk_cpu_iopoll));
__raise_softirq_irqoff(BLOCK_IOPOLL_SOFTIRQ);
local_irq_restore(flags);
}
@@ -79,7 +79,7 @@ EXPORT_SYMBOL(blk_iopoll_complete);
static void blk_iopoll_softirq(struct softirq_action *h)
{
- struct list_head *list = &__get_cpu_var(blk_cpu_iopoll);
+ struct list_head *list = this_cpu_ptr(&blk_cpu_iopoll);
int rearm = 0, budget = blk_iopoll_budget;
unsigned long start_time = jiffies;
@@ -201,7 +201,7 @@ static int blk_iopoll_cpu_notify(struct
local_irq_disable();
list_splice_init(&per_cpu(blk_cpu_iopoll, cpu),
- &__get_cpu_var(blk_cpu_iopoll));
+ this_cpu_ptr(&blk_cpu_iopoll));
__raise_softirq_irqoff(BLOCK_IOPOLL_SOFTIRQ);
local_irq_enable();
}
Index: linux/block/blk-softirq.c
===================================================================
--- linux.orig/block/blk-softirq.c 2013-08-26 14:36:07.858875667 -0500
+++ linux/block/blk-softirq.c 2013-08-26 14:36:07.854875709 -0500
@@ -23,7 +23,7 @@ static void blk_done_softirq(struct soft
struct list_head *cpu_list, local_list;
local_irq_disable();
- cpu_list = &__get_cpu_var(blk_cpu_done);
+ cpu_list = this_cpu_ptr(&blk_cpu_done);
list_replace_init(cpu_list, &local_list);
local_irq_enable();
@@ -44,7 +44,7 @@ static void trigger_softirq(void *data)
struct list_head *list;
local_irq_save(flags);
- list = &__get_cpu_var(blk_cpu_done);
+ list = this_cpu_ptr(&blk_cpu_done);
list_add_tail(&rq->csd.list, list);
if (list->next == &rq->csd.list)
@@ -90,7 +90,7 @@ static int blk_cpu_notify(struct notifie
local_irq_disable();
list_splice_init(&per_cpu(blk_cpu_done, cpu),
- &__get_cpu_var(blk_cpu_done));
+ this_cpu_ptr(&blk_cpu_done));
raise_softirq_irqoff(BLOCK_SOFTIRQ);
local_irq_enable();
}
@@ -135,7 +135,7 @@ void __blk_complete_request(struct reque
if (ccpu == cpu || shared) {
struct list_head *list;
do_local:
- list = &__get_cpu_var(blk_cpu_done);
+ list = this_cpu_ptr(&blk_cpu_done);
list_add_tail(&req->csd.list, list);
/*
Index: linux/fs/fscache/object.c
===================================================================
--- linux.orig/fs/fscache/object.c 2013-08-26 14:36:07.858875667 -0500
+++ linux/fs/fscache/object.c 2013-08-26 14:36:07.854875709 -0500
@@ -796,7 +796,7 @@ void fscache_enqueue_object(struct fscac
*/
bool fscache_object_sleep_till_congested(signed long *timeoutp)
{
- wait_queue_head_t *cong_wq = &__get_cpu_var(fscache_object_cong_wait);
+ wait_queue_head_t *cong_wq = this_cpu_ptr(&fscache_object_cong_wait);
DEFINE_WAIT(wait);
if (fscache_object_congested())
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 08/31] tracing: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Steven Rostedt, Frederic Weisbecker, Ingo Molnar,
linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/kprobes.h
===================================================================
--- linux.orig/include/linux/kprobes.h 2013-08-26 14:25:53.000000000 -0500
+++ linux/include/linux/kprobes.h 2013-08-26 14:26:29.460993659 -0500
@@ -329,7 +329,7 @@ static inline void reset_current_kprobe(
static inline struct kprobe_ctlblk *get_kprobe_ctlblk(void)
{
- return (&__get_cpu_var(kprobe_ctlblk));
+ return this_cpu_ptr(&kprobe_ctlblk);
}
int register_kprobe(struct kprobe *p);
Index: linux/kernel/trace/ftrace.c
===================================================================
--- linux.orig/kernel/trace/ftrace.c 2013-08-26 14:25:53.000000000 -0500
+++ linux/kernel/trace/ftrace.c 2013-08-26 14:26:29.460993659 -0500
@@ -870,7 +870,7 @@ function_profile_call(unsigned long ip,
local_irq_save(flags);
- stat = &__get_cpu_var(ftrace_profile_stats);
+ stat = this_cpu_ptr(&ftrace_profile_stats);
if (!stat->hash || !ftrace_profile_enabled)
goto out;
@@ -901,7 +901,7 @@ static void profile_graph_return(struct
unsigned long flags;
local_irq_save(flags);
- stat = &__get_cpu_var(ftrace_profile_stats);
+ stat = this_cpu_ptr(&ftrace_profile_stats);
if (!stat->hash || !ftrace_profile_enabled)
goto out;
Index: linux/kernel/trace/trace.c
===================================================================
--- linux.orig/kernel/trace/trace.c 2013-08-26 14:25:53.000000000 -0500
+++ linux/kernel/trace/trace.c 2013-08-26 14:26:29.464993617 -0500
@@ -1676,7 +1676,7 @@ static void __ftrace_trace_stack(struct
*/
barrier();
if (use_stack == 1) {
- trace.entries = &__get_cpu_var(ftrace_stack).calls[0];
+ trace.entries = this_cpu_ptr(ftrace_stack.calls);
trace.max_entries = FTRACE_STACK_MAX_ENTRIES;
if (regs)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 11/31] percpu: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/percpu.h
===================================================================
--- linux.orig/include/linux/percpu.h 2013-08-26 14:29:49.000000000 -0500
+++ linux/include/linux/percpu.h 2013-08-26 14:30:18.302570608 -0500
@@ -28,7 +28,7 @@
*/
#define get_cpu_var(var) (*({ \
preempt_disable(); \
- &__get_cpu_var(var); }))
+ this_cpu_ptr(&var); }))
/*
* The weird & is necessary because sparse considers (void)(var) to be
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 10/31] rcu: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Dipankar Sarma, Paul E. McKenney, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/kernel/rcutree.c
===================================================================
--- linux.orig/kernel/rcutree.c 2013-08-26 14:27:35.000000000 -0500
+++ linux/kernel/rcutree.c 2013-08-26 14:28:13.959886770 -0500
@@ -383,7 +383,7 @@ static void rcu_eqs_enter(bool user)
long long oldval;
struct rcu_dynticks *rdtp;
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
oldval = rdtp->dynticks_nesting;
WARN_ON_ONCE((oldval & DYNTICK_TASK_NEST_MASK) == 0);
if ((oldval & DYNTICK_TASK_NEST_MASK) == DYNTICK_TASK_NEST_VALUE)
@@ -443,7 +443,7 @@ void rcu_user_enter_after_irq(void)
struct rcu_dynticks *rdtp;
local_irq_save(flags);
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
/* Ensure this irq is interrupting a non-idle RCU state. */
WARN_ON_ONCE(!(rdtp->dynticks_nesting & DYNTICK_TASK_MASK));
rdtp->dynticks_nesting = 1;
@@ -474,7 +474,7 @@ void rcu_irq_exit(void)
struct rcu_dynticks *rdtp;
local_irq_save(flags);
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
oldval = rdtp->dynticks_nesting;
rdtp->dynticks_nesting--;
WARN_ON_ONCE(rdtp->dynticks_nesting < 0);
@@ -523,7 +523,7 @@ static void rcu_eqs_exit(bool user)
struct rcu_dynticks *rdtp;
long long oldval;
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
oldval = rdtp->dynticks_nesting;
WARN_ON_ONCE(oldval < 0);
if (oldval & DYNTICK_TASK_NEST_MASK)
@@ -581,7 +581,7 @@ void rcu_user_exit_after_irq(void)
struct rcu_dynticks *rdtp;
local_irq_save(flags);
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
/* Ensure we are interrupting an RCU idle mode. */
WARN_ON_ONCE(rdtp->dynticks_nesting & DYNTICK_TASK_NEST_MASK);
rdtp->dynticks_nesting += DYNTICK_TASK_EXIT_IDLE;
@@ -615,7 +615,7 @@ void rcu_irq_enter(void)
long long oldval;
local_irq_save(flags);
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
oldval = rdtp->dynticks_nesting;
rdtp->dynticks_nesting++;
WARN_ON_ONCE(rdtp->dynticks_nesting == 0);
@@ -635,7 +635,7 @@ void rcu_irq_enter(void)
*/
void rcu_nmi_enter(void)
{
- struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);
+ struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
if (rdtp->dynticks_nmi_nesting == 0 &&
(atomic_read(&rdtp->dynticks) & 0x1))
@@ -657,7 +657,7 @@ void rcu_nmi_enter(void)
*/
void rcu_nmi_exit(void)
{
- struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);
+ struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
if (rdtp->dynticks_nmi_nesting == 0 ||
--rdtp->dynticks_nmi_nesting != 0)
@@ -680,7 +680,7 @@ int rcu_is_cpu_idle(void)
int ret;
preempt_disable();
- ret = (atomic_read(&__get_cpu_var(rcu_dynticks).dynticks) & 0x1) == 0;
+ ret = (atomic_read(this_cpu_ptr(&rcu_dynticks.dynticks)) & 0x1) == 0;
preempt_enable();
return ret;
}
@@ -718,7 +718,7 @@ bool rcu_lockdep_current_cpu_online(void
if (in_nmi())
return 1;
preempt_disable();
- rdp = &__get_cpu_var(rcu_sched_data);
+ rdp = this_cpu_ptr(&rcu_sched_data);
rnp = rdp->mynode;
ret = (rdp->grpmask & rnp->qsmaskinit) ||
!rcu_scheduler_fully_active;
@@ -738,7 +738,7 @@ EXPORT_SYMBOL_GPL(rcu_lockdep_current_cp
*/
static int rcu_is_cpu_rrupt_from_idle(void)
{
- return __get_cpu_var(rcu_dynticks).dynticks_nesting <= 1;
+ return __this_cpu_read(rcu_dynticks.dynticks_nesting) <= 1;
}
/*
Index: linux/kernel/rcutree_plugin.h
===================================================================
--- linux.orig/kernel/rcutree_plugin.h 2013-08-26 14:27:35.000000000 -0500
+++ linux/kernel/rcutree_plugin.h 2013-08-26 14:28:13.963886728 -0500
@@ -662,7 +662,7 @@ static void rcu_preempt_check_callbacks(
static void rcu_preempt_do_callbacks(void)
{
- rcu_do_batch(&rcu_preempt_state, &__get_cpu_var(rcu_preempt_data));
+ rcu_do_batch(&rcu_preempt_state, this_cpu_ptr(&rcu_preempt_data));
}
#endif /* #ifdef CONFIG_RCU_BOOST */
@@ -1334,7 +1334,7 @@ static void invoke_rcu_callbacks_kthread
*/
static bool rcu_is_callbacks_kthread(void)
{
- return __get_cpu_var(rcu_cpu_kthread_task) == current;
+ return __this_cpu_read(rcu_cpu_kthread_task) == current;
}
#define RCU_BOOST_DELAY_JIFFIES DIV_ROUND_UP(CONFIG_RCU_BOOST_DELAY * HZ, 1000)
@@ -1384,8 +1384,8 @@ static int rcu_spawn_one_boost_kthread(s
static void rcu_kthread_do_work(void)
{
- rcu_do_batch(&rcu_sched_state, &__get_cpu_var(rcu_sched_data));
- rcu_do_batch(&rcu_bh_state, &__get_cpu_var(rcu_bh_data));
+ rcu_do_batch(&rcu_sched_state, this_cpu_ptr(&rcu_sched_data));
+ rcu_do_batch(&rcu_bh_state, this_cpu_ptr(&rcu_bh_data));
rcu_preempt_do_callbacks();
}
@@ -1404,7 +1404,7 @@ static void rcu_cpu_kthread_park(unsigne
static int rcu_cpu_kthread_should_run(unsigned int cpu)
{
- return __get_cpu_var(rcu_cpu_has_work);
+ return __this_cpu_read(rcu_cpu_has_work);
}
/*
@@ -1414,8 +1414,8 @@ static int rcu_cpu_kthread_should_run(un
*/
static void rcu_cpu_kthread(unsigned int cpu)
{
- unsigned int *statusp = &__get_cpu_var(rcu_cpu_kthread_status);
- char work, *workp = &__get_cpu_var(rcu_cpu_has_work);
+ unsigned int *statusp = this_cpu_ptr(&rcu_cpu_kthread_status);
+ char work, *workp = this_cpu_ptr(&rcu_cpu_has_work);
int spincnt;
for (spincnt = 0; spincnt < 10; spincnt++) {
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 07/31] mm: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
2013-08-26 20:44 ` Christoph Lameter
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (29 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, akpm, linux-mm, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/lib/radix-tree.c
===================================================================
--- linux.orig/lib/radix-tree.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/lib/radix-tree.c 2013-08-26 14:25:30.709616290 -0500
@@ -215,7 +215,7 @@ radix_tree_node_alloc(struct radix_tree_
* succeed in getting a node here (and never reach
* kmem_cache_alloc)
*/
- rtp = &__get_cpu_var(radix_tree_preloads);
+ rtp = this_cpu_ptr(&radix_tree_preloads);
if (rtp->nr) {
ret = rtp->nodes[rtp->nr - 1];
rtp->nodes[rtp->nr - 1] = NULL;
@@ -271,14 +271,14 @@ int radix_tree_preload(gfp_t gfp_mask)
int ret = -ENOMEM;
preempt_disable();
- rtp = &__get_cpu_var(radix_tree_preloads);
+ rtp = this_cpu_ptr(&radix_tree_preloads);
while (rtp->nr < ARRAY_SIZE(rtp->nodes)) {
preempt_enable();
node = kmem_cache_alloc(radix_tree_node_cachep, gfp_mask);
if (node == NULL)
goto out;
preempt_disable();
- rtp = &__get_cpu_var(radix_tree_preloads);
+ rtp = this_cpu_ptr(&radix_tree_preloads);
if (rtp->nr < ARRAY_SIZE(rtp->nodes))
rtp->nodes[rtp->nr++] = node;
else
Index: linux/mm/memcontrol.c
===================================================================
--- linux.orig/mm/memcontrol.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/memcontrol.c 2013-08-26 14:25:30.713616248 -0500
@@ -2398,7 +2398,7 @@ static void drain_stock(struct memcg_sto
*/
static void drain_local_stock(struct work_struct *dummy)
{
- struct memcg_stock_pcp *stock = &__get_cpu_var(memcg_stock);
+ struct memcg_stock_pcp *stock = this_cpu_ptr(&memcg_stock);
drain_stock(stock);
clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags);
}
Index: linux/mm/memory-failure.c
===================================================================
--- linux.orig/mm/memory-failure.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/memory-failure.c 2013-08-26 14:25:30.713616248 -0500
@@ -1279,7 +1279,7 @@ static void memory_failure_work_func(str
unsigned long proc_flags;
int gotten;
- mf_cpu = &__get_cpu_var(memory_failure_cpu);
+ mf_cpu = this_cpu_ptr(&memory_failure_cpu);
for (;;) {
spin_lock_irqsave(&mf_cpu->lock, proc_flags);
gotten = kfifo_get(&mf_cpu->fifo, &entry);
Index: linux/mm/page-writeback.c
===================================================================
--- linux.orig/mm/page-writeback.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/page-writeback.c 2013-08-26 14:25:30.713616248 -0500
@@ -1487,7 +1487,7 @@ void balance_dirty_pages_ratelimited(str
* 1000+ tasks, all of them start dirtying pages at exactly the same
* time, hence all honoured too large initial task->nr_dirtied_pause.
*/
- p = &__get_cpu_var(bdp_ratelimits);
+ p = this_cpu_ptr(&bdp_ratelimits);
if (unlikely(current->nr_dirtied >= ratelimit))
*p = 0;
else if (unlikely(*p >= ratelimit_pages)) {
@@ -1499,7 +1499,7 @@ void balance_dirty_pages_ratelimited(str
* short-lived tasks (eg. gcc invocations in a kernel build) escaping
* the dirty throttling and livelock other long-run dirtiers.
*/
- p = &__get_cpu_var(dirty_throttle_leaks);
+ p = this_cpu_ptr(&dirty_throttle_leaks);
if (*p > 0 && current->nr_dirtied < ratelimit) {
unsigned long nr_pages_dirtied;
nr_pages_dirtied = min(*p, ratelimit - current->nr_dirtied);
Index: linux/mm/swap.c
===================================================================
--- linux.orig/mm/swap.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/swap.c 2013-08-26 14:25:30.717616206 -0500
@@ -359,7 +359,7 @@ void rotate_reclaimable_page(struct page
page_cache_get(page);
local_irq_save(flags);
- pvec = &__get_cpu_var(lru_rotate_pvecs);
+ pvec = this_cpu_ptr(&lru_rotate_pvecs);
if (!pagevec_add(pvec, page))
pagevec_move_tail(pvec);
local_irq_restore(flags);
Index: linux/mm/vmalloc.c
===================================================================
--- linux.orig/mm/vmalloc.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/vmalloc.c 2013-08-26 14:25:30.717616206 -0500
@@ -1487,7 +1487,7 @@ void vfree(const void *addr)
if (!addr)
return;
if (unlikely(in_interrupt())) {
- struct vfree_deferred *p = &__get_cpu_var(vfree_deferred);
+ struct vfree_deferred *p = this_cpu_ptr(&vfree_deferred);
if (llist_add((struct llist_node *)addr, &p->list))
schedule_work(&p->wq);
} else
Index: linux/mm/vmstat.c
===================================================================
--- linux.orig/mm/vmstat.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/vmstat.c 2013-08-26 14:25:30.717616206 -0500
@@ -1178,7 +1178,7 @@ int sysctl_stat_interval __read_mostly =
static void vmstat_update(struct work_struct *w)
{
refresh_cpu_vm_stats(smp_processor_id());
- schedule_delayed_work(&__get_cpu_var(vmstat_work),
+ schedule_delayed_work(this_cpu_ptr(&vmstat_work),
round_jiffies_relative(sysctl_stat_interval));
}
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 07/31] mm: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, akpm, linux-mm, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/lib/radix-tree.c
===================================================================
--- linux.orig/lib/radix-tree.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/lib/radix-tree.c 2013-08-26 14:25:30.709616290 -0500
@@ -215,7 +215,7 @@ radix_tree_node_alloc(struct radix_tree_
* succeed in getting a node here (and never reach
* kmem_cache_alloc)
*/
- rtp = &__get_cpu_var(radix_tree_preloads);
+ rtp = this_cpu_ptr(&radix_tree_preloads);
if (rtp->nr) {
ret = rtp->nodes[rtp->nr - 1];
rtp->nodes[rtp->nr - 1] = NULL;
@@ -271,14 +271,14 @@ int radix_tree_preload(gfp_t gfp_mask)
int ret = -ENOMEM;
preempt_disable();
- rtp = &__get_cpu_var(radix_tree_preloads);
+ rtp = this_cpu_ptr(&radix_tree_preloads);
while (rtp->nr < ARRAY_SIZE(rtp->nodes)) {
preempt_enable();
node = kmem_cache_alloc(radix_tree_node_cachep, gfp_mask);
if (node == NULL)
goto out;
preempt_disable();
- rtp = &__get_cpu_var(radix_tree_preloads);
+ rtp = this_cpu_ptr(&radix_tree_preloads);
if (rtp->nr < ARRAY_SIZE(rtp->nodes))
rtp->nodes[rtp->nr++] = node;
else
Index: linux/mm/memcontrol.c
===================================================================
--- linux.orig/mm/memcontrol.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/memcontrol.c 2013-08-26 14:25:30.713616248 -0500
@@ -2398,7 +2398,7 @@ static void drain_stock(struct memcg_sto
*/
static void drain_local_stock(struct work_struct *dummy)
{
- struct memcg_stock_pcp *stock = &__get_cpu_var(memcg_stock);
+ struct memcg_stock_pcp *stock = this_cpu_ptr(&memcg_stock);
drain_stock(stock);
clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags);
}
Index: linux/mm/memory-failure.c
===================================================================
--- linux.orig/mm/memory-failure.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/memory-failure.c 2013-08-26 14:25:30.713616248 -0500
@@ -1279,7 +1279,7 @@ static void memory_failure_work_func(str
unsigned long proc_flags;
int gotten;
- mf_cpu = &__get_cpu_var(memory_failure_cpu);
+ mf_cpu = this_cpu_ptr(&memory_failure_cpu);
for (;;) {
spin_lock_irqsave(&mf_cpu->lock, proc_flags);
gotten = kfifo_get(&mf_cpu->fifo, &entry);
Index: linux/mm/page-writeback.c
===================================================================
--- linux.orig/mm/page-writeback.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/page-writeback.c 2013-08-26 14:25:30.713616248 -0500
@@ -1487,7 +1487,7 @@ void balance_dirty_pages_ratelimited(str
* 1000+ tasks, all of them start dirtying pages at exactly the same
* time, hence all honoured too large initial task->nr_dirtied_pause.
*/
- p = &__get_cpu_var(bdp_ratelimits);
+ p = this_cpu_ptr(&bdp_ratelimits);
if (unlikely(current->nr_dirtied >= ratelimit))
*p = 0;
else if (unlikely(*p >= ratelimit_pages)) {
@@ -1499,7 +1499,7 @@ void balance_dirty_pages_ratelimited(str
* short-lived tasks (eg. gcc invocations in a kernel build) escaping
* the dirty throttling and livelock other long-run dirtiers.
*/
- p = &__get_cpu_var(dirty_throttle_leaks);
+ p = this_cpu_ptr(&dirty_throttle_leaks);
if (*p > 0 && current->nr_dirtied < ratelimit) {
unsigned long nr_pages_dirtied;
nr_pages_dirtied = min(*p, ratelimit - current->nr_dirtied);
Index: linux/mm/swap.c
===================================================================
--- linux.orig/mm/swap.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/swap.c 2013-08-26 14:25:30.717616206 -0500
@@ -359,7 +359,7 @@ void rotate_reclaimable_page(struct page
page_cache_get(page);
local_irq_save(flags);
- pvec = &__get_cpu_var(lru_rotate_pvecs);
+ pvec = this_cpu_ptr(&lru_rotate_pvecs);
if (!pagevec_add(pvec, page))
pagevec_move_tail(pvec);
local_irq_restore(flags);
Index: linux/mm/vmalloc.c
===================================================================
--- linux.orig/mm/vmalloc.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/vmalloc.c 2013-08-26 14:25:30.717616206 -0500
@@ -1487,7 +1487,7 @@ void vfree(const void *addr)
if (!addr)
return;
if (unlikely(in_interrupt())) {
- struct vfree_deferred *p = &__get_cpu_var(vfree_deferred);
+ struct vfree_deferred *p = this_cpu_ptr(&vfree_deferred);
if (llist_add((struct llist_node *)addr, &p->list))
schedule_work(&p->wq);
} else
Index: linux/mm/vmstat.c
===================================================================
--- linux.orig/mm/vmstat.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/vmstat.c 2013-08-26 14:25:30.717616206 -0500
@@ -1178,7 +1178,7 @@ int sysctl_stat_interval __read_mostly =
static void vmstat_update(struct work_struct *w)
{
refresh_cpu_vm_stats(smp_processor_id());
- schedule_delayed_work(&__get_cpu_var(vmstat_work),
+ schedule_delayed_work(this_cpu_ptr(&vmstat_work),
round_jiffies_relative(sysctl_stat_interval));
}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 11/31] percpu: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/percpu.h
===================================================================
--- linux.orig/include/linux/percpu.h 2013-08-26 14:29:49.000000000 -0500
+++ linux/include/linux/percpu.h 2013-08-26 14:30:18.302570608 -0500
@@ -28,7 +28,7 @@
*/
#define get_cpu_var(var) (*({ \
preempt_disable(); \
- &__get_cpu_var(var); }))
+ this_cpu_ptr(&var); }))
/*
* The weird & is necessary because sparse considers (void)(var) to be
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 10/31] rcu: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Dipankar Sarma, Paul E. McKenney, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/kernel/rcutree.c
===================================================================
--- linux.orig/kernel/rcutree.c 2013-08-26 14:27:35.000000000 -0500
+++ linux/kernel/rcutree.c 2013-08-26 14:28:13.959886770 -0500
@@ -383,7 +383,7 @@ static void rcu_eqs_enter(bool user)
long long oldval;
struct rcu_dynticks *rdtp;
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
oldval = rdtp->dynticks_nesting;
WARN_ON_ONCE((oldval & DYNTICK_TASK_NEST_MASK) == 0);
if ((oldval & DYNTICK_TASK_NEST_MASK) == DYNTICK_TASK_NEST_VALUE)
@@ -443,7 +443,7 @@ void rcu_user_enter_after_irq(void)
struct rcu_dynticks *rdtp;
local_irq_save(flags);
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
/* Ensure this irq is interrupting a non-idle RCU state. */
WARN_ON_ONCE(!(rdtp->dynticks_nesting & DYNTICK_TASK_MASK));
rdtp->dynticks_nesting = 1;
@@ -474,7 +474,7 @@ void rcu_irq_exit(void)
struct rcu_dynticks *rdtp;
local_irq_save(flags);
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
oldval = rdtp->dynticks_nesting;
rdtp->dynticks_nesting--;
WARN_ON_ONCE(rdtp->dynticks_nesting < 0);
@@ -523,7 +523,7 @@ static void rcu_eqs_exit(bool user)
struct rcu_dynticks *rdtp;
long long oldval;
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
oldval = rdtp->dynticks_nesting;
WARN_ON_ONCE(oldval < 0);
if (oldval & DYNTICK_TASK_NEST_MASK)
@@ -581,7 +581,7 @@ void rcu_user_exit_after_irq(void)
struct rcu_dynticks *rdtp;
local_irq_save(flags);
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
/* Ensure we are interrupting an RCU idle mode. */
WARN_ON_ONCE(rdtp->dynticks_nesting & DYNTICK_TASK_NEST_MASK);
rdtp->dynticks_nesting += DYNTICK_TASK_EXIT_IDLE;
@@ -615,7 +615,7 @@ void rcu_irq_enter(void)
long long oldval;
local_irq_save(flags);
- rdtp = &__get_cpu_var(rcu_dynticks);
+ rdtp = this_cpu_ptr(&rcu_dynticks);
oldval = rdtp->dynticks_nesting;
rdtp->dynticks_nesting++;
WARN_ON_ONCE(rdtp->dynticks_nesting == 0);
@@ -635,7 +635,7 @@ void rcu_irq_enter(void)
*/
void rcu_nmi_enter(void)
{
- struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);
+ struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
if (rdtp->dynticks_nmi_nesting == 0 &&
(atomic_read(&rdtp->dynticks) & 0x1))
@@ -657,7 +657,7 @@ void rcu_nmi_enter(void)
*/
void rcu_nmi_exit(void)
{
- struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);
+ struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
if (rdtp->dynticks_nmi_nesting == 0 ||
--rdtp->dynticks_nmi_nesting != 0)
@@ -680,7 +680,7 @@ int rcu_is_cpu_idle(void)
int ret;
preempt_disable();
- ret = (atomic_read(&__get_cpu_var(rcu_dynticks).dynticks) & 0x1) == 0;
+ ret = (atomic_read(this_cpu_ptr(&rcu_dynticks.dynticks)) & 0x1) == 0;
preempt_enable();
return ret;
}
@@ -718,7 +718,7 @@ bool rcu_lockdep_current_cpu_online(void
if (in_nmi())
return 1;
preempt_disable();
- rdp = &__get_cpu_var(rcu_sched_data);
+ rdp = this_cpu_ptr(&rcu_sched_data);
rnp = rdp->mynode;
ret = (rdp->grpmask & rnp->qsmaskinit) ||
!rcu_scheduler_fully_active;
@@ -738,7 +738,7 @@ EXPORT_SYMBOL_GPL(rcu_lockdep_current_cp
*/
static int rcu_is_cpu_rrupt_from_idle(void)
{
- return __get_cpu_var(rcu_dynticks).dynticks_nesting <= 1;
+ return __this_cpu_read(rcu_dynticks.dynticks_nesting) <= 1;
}
/*
Index: linux/kernel/rcutree_plugin.h
===================================================================
--- linux.orig/kernel/rcutree_plugin.h 2013-08-26 14:27:35.000000000 -0500
+++ linux/kernel/rcutree_plugin.h 2013-08-26 14:28:13.963886728 -0500
@@ -662,7 +662,7 @@ static void rcu_preempt_check_callbacks(
static void rcu_preempt_do_callbacks(void)
{
- rcu_do_batch(&rcu_preempt_state, &__get_cpu_var(rcu_preempt_data));
+ rcu_do_batch(&rcu_preempt_state, this_cpu_ptr(&rcu_preempt_data));
}
#endif /* #ifdef CONFIG_RCU_BOOST */
@@ -1334,7 +1334,7 @@ static void invoke_rcu_callbacks_kthread
*/
static bool rcu_is_callbacks_kthread(void)
{
- return __get_cpu_var(rcu_cpu_kthread_task) == current;
+ return __this_cpu_read(rcu_cpu_kthread_task) == current;
}
#define RCU_BOOST_DELAY_JIFFIES DIV_ROUND_UP(CONFIG_RCU_BOOST_DELAY * HZ, 1000)
@@ -1384,8 +1384,8 @@ static int rcu_spawn_one_boost_kthread(s
static void rcu_kthread_do_work(void)
{
- rcu_do_batch(&rcu_sched_state, &__get_cpu_var(rcu_sched_data));
- rcu_do_batch(&rcu_bh_state, &__get_cpu_var(rcu_bh_data));
+ rcu_do_batch(&rcu_sched_state, this_cpu_ptr(&rcu_sched_data));
+ rcu_do_batch(&rcu_bh_state, this_cpu_ptr(&rcu_bh_data));
rcu_preempt_do_callbacks();
}
@@ -1404,7 +1404,7 @@ static void rcu_cpu_kthread_park(unsigne
static int rcu_cpu_kthread_should_run(unsigned int cpu)
{
- return __get_cpu_var(rcu_cpu_has_work);
+ return __this_cpu_read(rcu_cpu_has_work);
}
/*
@@ -1414,8 +1414,8 @@ static int rcu_cpu_kthread_should_run(un
*/
static void rcu_cpu_kthread(unsigned int cpu)
{
- unsigned int *statusp = &__get_cpu_var(rcu_cpu_kthread_status);
- char work, *workp = &__get_cpu_var(rcu_cpu_has_work);
+ unsigned int *statusp = this_cpu_ptr(&rcu_cpu_kthread_status);
+ char work, *workp = this_cpu_ptr(&rcu_cpu_has_work);
int spincnt;
for (spincnt = 0; spincnt < 10; spincnt++) {
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 07/31] mm: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, akpm, linux-mm, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/lib/radix-tree.c
===================================================================
--- linux.orig/lib/radix-tree.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/lib/radix-tree.c 2013-08-26 14:25:30.709616290 -0500
@@ -215,7 +215,7 @@ radix_tree_node_alloc(struct radix_tree_
* succeed in getting a node here (and never reach
* kmem_cache_alloc)
*/
- rtp = &__get_cpu_var(radix_tree_preloads);
+ rtp = this_cpu_ptr(&radix_tree_preloads);
if (rtp->nr) {
ret = rtp->nodes[rtp->nr - 1];
rtp->nodes[rtp->nr - 1] = NULL;
@@ -271,14 +271,14 @@ int radix_tree_preload(gfp_t gfp_mask)
int ret = -ENOMEM;
preempt_disable();
- rtp = &__get_cpu_var(radix_tree_preloads);
+ rtp = this_cpu_ptr(&radix_tree_preloads);
while (rtp->nr < ARRAY_SIZE(rtp->nodes)) {
preempt_enable();
node = kmem_cache_alloc(radix_tree_node_cachep, gfp_mask);
if (node == NULL)
goto out;
preempt_disable();
- rtp = &__get_cpu_var(radix_tree_preloads);
+ rtp = this_cpu_ptr(&radix_tree_preloads);
if (rtp->nr < ARRAY_SIZE(rtp->nodes))
rtp->nodes[rtp->nr++] = node;
else
Index: linux/mm/memcontrol.c
===================================================================
--- linux.orig/mm/memcontrol.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/memcontrol.c 2013-08-26 14:25:30.713616248 -0500
@@ -2398,7 +2398,7 @@ static void drain_stock(struct memcg_sto
*/
static void drain_local_stock(struct work_struct *dummy)
{
- struct memcg_stock_pcp *stock = &__get_cpu_var(memcg_stock);
+ struct memcg_stock_pcp *stock = this_cpu_ptr(&memcg_stock);
drain_stock(stock);
clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags);
}
Index: linux/mm/memory-failure.c
===================================================================
--- linux.orig/mm/memory-failure.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/memory-failure.c 2013-08-26 14:25:30.713616248 -0500
@@ -1279,7 +1279,7 @@ static void memory_failure_work_func(str
unsigned long proc_flags;
int gotten;
- mf_cpu = &__get_cpu_var(memory_failure_cpu);
+ mf_cpu = this_cpu_ptr(&memory_failure_cpu);
for (;;) {
spin_lock_irqsave(&mf_cpu->lock, proc_flags);
gotten = kfifo_get(&mf_cpu->fifo, &entry);
Index: linux/mm/page-writeback.c
===================================================================
--- linux.orig/mm/page-writeback.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/page-writeback.c 2013-08-26 14:25:30.713616248 -0500
@@ -1487,7 +1487,7 @@ void balance_dirty_pages_ratelimited(str
* 1000+ tasks, all of them start dirtying pages at exactly the same
* time, hence all honoured too large initial task->nr_dirtied_pause.
*/
- p = &__get_cpu_var(bdp_ratelimits);
+ p = this_cpu_ptr(&bdp_ratelimits);
if (unlikely(current->nr_dirtied >= ratelimit))
*p = 0;
else if (unlikely(*p >= ratelimit_pages)) {
@@ -1499,7 +1499,7 @@ void balance_dirty_pages_ratelimited(str
* short-lived tasks (eg. gcc invocations in a kernel build) escaping
* the dirty throttling and livelock other long-run dirtiers.
*/
- p = &__get_cpu_var(dirty_throttle_leaks);
+ p = this_cpu_ptr(&dirty_throttle_leaks);
if (*p > 0 && current->nr_dirtied < ratelimit) {
unsigned long nr_pages_dirtied;
nr_pages_dirtied = min(*p, ratelimit - current->nr_dirtied);
Index: linux/mm/swap.c
===================================================================
--- linux.orig/mm/swap.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/swap.c 2013-08-26 14:25:30.717616206 -0500
@@ -359,7 +359,7 @@ void rotate_reclaimable_page(struct page
page_cache_get(page);
local_irq_save(flags);
- pvec = &__get_cpu_var(lru_rotate_pvecs);
+ pvec = this_cpu_ptr(&lru_rotate_pvecs);
if (!pagevec_add(pvec, page))
pagevec_move_tail(pvec);
local_irq_restore(flags);
Index: linux/mm/vmalloc.c
===================================================================
--- linux.orig/mm/vmalloc.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/vmalloc.c 2013-08-26 14:25:30.717616206 -0500
@@ -1487,7 +1487,7 @@ void vfree(const void *addr)
if (!addr)
return;
if (unlikely(in_interrupt())) {
- struct vfree_deferred *p = &__get_cpu_var(vfree_deferred);
+ struct vfree_deferred *p = this_cpu_ptr(&vfree_deferred);
if (llist_add((struct llist_node *)addr, &p->list))
schedule_work(&p->wq);
} else
Index: linux/mm/vmstat.c
===================================================================
--- linux.orig/mm/vmstat.c 2013-08-26 14:24:48.000000000 -0500
+++ linux/mm/vmstat.c 2013-08-26 14:25:30.717616206 -0500
@@ -1178,7 +1178,7 @@ int sysctl_stat_interval __read_mostly =
static void vmstat_update(struct work_struct *w)
{
refresh_cpu_vm_stats(smp_processor_id());
- schedule_delayed_work(&__get_cpu_var(vmstat_work),
+ schedule_delayed_work(this_cpu_ptr(&vmstat_work),
round_jiffies_relative(sysctl_stat_interval));
}
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 13/31] kernel misc: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/kernel/printk/printk.c
===================================================================
--- linux.orig/kernel/printk/printk.c 2013-08-26 14:30:25.846490787 -0500
+++ linux/kernel/printk/printk.c 2013-08-26 14:30:25.846490787 -0500
@@ -2441,7 +2441,7 @@ static void wake_up_klogd_work_func(stru
int pending = __this_cpu_xchg(printk_pending, 0);
if (pending & PRINTK_PENDING_SCHED) {
- char *buf = __get_cpu_var(printk_sched_buf);
+ char *buf = this_cpu_ptr(printk_sched_buf);
printk(KERN_WARNING "[sched_delayed] %s", buf);
}
@@ -2459,7 +2459,7 @@ void wake_up_klogd(void)
preempt_disable();
if (waitqueue_active(&log_wait)) {
this_cpu_or(printk_pending, PRINTK_PENDING_WAKEUP);
- irq_work_queue(&__get_cpu_var(wake_up_klogd_work));
+ irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
}
preempt_enable();
}
@@ -2472,14 +2472,14 @@ int printk_sched(const char *fmt, ...)
int r;
local_irq_save(flags);
- buf = __get_cpu_var(printk_sched_buf);
+ buf = this_cpu_ptr(printk_sched_buf);
va_start(args, fmt);
r = vsnprintf(buf, PRINTK_BUF_SIZE, fmt, args);
va_end(args);
__this_cpu_or(printk_pending, PRINTK_PENDING_SCHED);
- irq_work_queue(&__get_cpu_var(wake_up_klogd_work));
+ irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
local_irq_restore(flags);
return r;
Index: linux/kernel/smp.c
===================================================================
--- linux.orig/kernel/smp.c 2013-08-26 14:30:25.846490787 -0500
+++ linux/kernel/smp.c 2013-08-26 14:30:25.846490787 -0500
@@ -172,7 +172,7 @@ void generic_exec_single(int cpu, struct
*/
void generic_smp_call_function_single_interrupt(void)
{
- struct call_single_queue *q = &__get_cpu_var(call_single_queue);
+ struct call_single_queue *q = this_cpu_ptr(&call_single_queue);
LIST_HEAD(list);
/*
@@ -252,7 +252,7 @@ int smp_call_function_single(int cpu, sm
struct call_single_data *csd = &d;
if (!wait)
- csd = &__get_cpu_var(csd_data);
+ csd = this_cpu_ptr(&csd_data);
csd_lock(csd);
@@ -401,7 +401,7 @@ void smp_call_function_many(const struct
return;
}
- cfd = &__get_cpu_var(cfd_data);
+ cfd = this_cpu_ptr(&cfd_data);
cpumask_and(cfd->cpumask, mask, cpu_online_mask);
cpumask_clear_cpu(this_cpu, cfd->cpumask);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 14/31] drivers: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/drivers/char/random.c
===================================================================
--- linux.orig/drivers/char/random.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/char/random.c 2013-08-22 14:16:11.319914096 -0500
@@ -744,7 +744,7 @@ static DEFINE_PER_CPU(struct fast_pool,
void add_interrupt_randomness(int irq, int irq_flags)
{
struct entropy_store *r;
- struct fast_pool *fast_pool = &__get_cpu_var(irq_randomness);
+ struct fast_pool *fast_pool = this_cpu_ptr(&irq_randomness);
struct pt_regs *regs = get_irq_regs();
unsigned long now = jiffies;
__u32 input[4], cycles = get_cycles();
Index: linux/drivers/clocksource/metag_generic.c
===================================================================
--- linux.orig/drivers/clocksource/metag_generic.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/clocksource/metag_generic.c 2013-08-22 14:16:11.323914055 -0500
@@ -90,7 +90,7 @@ static struct clocksource clocksource_me
static irqreturn_t metag_timer_interrupt(int irq, void *dummy)
{
- struct clock_event_device *evt = &__get_cpu_var(local_clockevent);
+ struct clock_event_device *evt = this_cpu_ptr(&local_clockevent);
evt->event_handler(evt);
Index: linux/drivers/cpuidle/governors/ladder.c
===================================================================
--- linux.orig/drivers/cpuidle/governors/ladder.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/cpuidle/governors/ladder.c 2013-08-22 14:16:11.323914055 -0500
@@ -66,7 +66,7 @@ static inline void ladder_do_selection(s
static int ladder_select_state(struct cpuidle_driver *drv,
struct cpuidle_device *dev)
{
- struct ladder_device *ldev = &__get_cpu_var(ladder_devices);
+ struct ladder_device *ldev = this_cpu_ptr(&ladder_devices);
struct ladder_device_state *last_state;
int last_residency, last_idx = ldev->last_state_idx;
int latency_req = pm_qos_request(PM_QOS_CPU_DMA_LATENCY);
@@ -170,7 +170,7 @@ static int ladder_enable_device(struct c
*/
static void ladder_reflect(struct cpuidle_device *dev, int index)
{
- struct ladder_device *ldev = &__get_cpu_var(ladder_devices);
+ struct ladder_device *ldev = this_cpu_ptr(&ladder_devices);
if (index > 0)
ldev->last_state_idx = index;
}
Index: linux/drivers/cpuidle/governors/menu.c
===================================================================
--- linux.orig/drivers/cpuidle/governors/menu.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/cpuidle/governors/menu.c 2013-08-22 14:16:11.323914055 -0500
@@ -258,7 +258,7 @@ again:
*/
static int menu_select(struct cpuidle_driver *drv, struct cpuidle_device *dev)
{
- struct menu_device *data = &__get_cpu_var(menu_devices);
+ struct menu_device *data = this_cpu_ptr(&menu_devices);
int latency_req = pm_qos_request(PM_QOS_CPU_DMA_LATENCY);
int i;
int multiplier;
@@ -342,7 +342,7 @@ static int menu_select(struct cpuidle_dr
*/
static void menu_reflect(struct cpuidle_device *dev, int index)
{
- struct menu_device *data = &__get_cpu_var(menu_devices);
+ struct menu_device *data = this_cpu_ptr(&menu_devices);
data->last_state_idx = index;
if (index >= 0)
data->needs_update = 1;
@@ -355,7 +355,7 @@ static void menu_reflect(struct cpuidle_
*/
static void menu_update(struct cpuidle_driver *drv, struct cpuidle_device *dev)
{
- struct menu_device *data = &__get_cpu_var(menu_devices);
+ struct menu_device *data = this_cpu_ptr(&menu_devices);
int last_idx = data->last_state_idx;
unsigned int last_idle_us = cpuidle_get_last_residency(dev);
struct cpuidle_state *target = &drv->states[last_idx];
Index: linux/drivers/leds/trigger/ledtrig-cpu.c
===================================================================
--- linux.orig/drivers/leds/trigger/ledtrig-cpu.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/leds/trigger/ledtrig-cpu.c 2013-08-22 14:16:11.323914055 -0500
@@ -46,7 +46,7 @@ static DEFINE_PER_CPU(struct led_trigger
*/
void ledtrig_cpu(enum cpu_led_event ledevt)
{
- struct led_trigger_cpu *trig = &__get_cpu_var(cpu_trig);
+ struct led_trigger_cpu *trig = this_cpu_ptr(&cpu_trig);
/* Locate the correct CPU LED */
switch (ledevt) {
Index: linux/drivers/net/ethernet/tile/tilegx.c
===================================================================
--- linux.orig/drivers/net/ethernet/tile/tilegx.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/net/ethernet/tile/tilegx.c 2013-08-22 14:16:11.323914055 -0500
@@ -368,7 +368,7 @@ static void tile_net_pop_all_buffers(int
/* Provide linux buffers to mPIPE. */
static void tile_net_provide_needed_buffers(void)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
while (info->num_needed_small_buffers != 0) {
if (!tile_net_provide_buffer(true))
@@ -407,7 +407,7 @@ static inline bool filter_packet(struct
static void tile_net_receive_skb(struct net_device *dev, struct sk_buff *skb,
gxio_mpipe_idesc_t *idesc, unsigned long len)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct tile_net_priv *priv = netdev_priv(dev);
/* Encode the actual packet length. */
@@ -435,7 +435,7 @@ static void tile_net_receive_skb(struct
/* Handle a packet. Return true if "processed", false if "filtered". */
static bool tile_net_handle_packet(gxio_mpipe_idesc_t *idesc)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct net_device *dev = tile_net_devs_for_channel[idesc->channel];
uint8_t l2_offset;
void *va;
@@ -498,7 +498,7 @@ static bool tile_net_handle_packet(gxio_
*/
static int tile_net_poll(struct napi_struct *napi, int budget)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
unsigned int work = 0;
gxio_mpipe_idesc_t *idesc;
int i, n;
@@ -536,7 +536,7 @@ done:
/* Handle an ingress interrupt on the current cpu. */
static irqreturn_t tile_net_handle_ingress_irq(int irq, void *unused)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
napi_schedule(&info->napi);
return IRQ_HANDLED;
}
@@ -597,7 +597,7 @@ static enum hrtimer_restart tile_net_han
/* Make sure the egress timer is scheduled. */
static void tile_net_schedule_egress_timer(void)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
if (!info->egress_timer_scheduled) {
hrtimer_start(&info->egress_timer,
@@ -614,7 +614,7 @@ static void tile_net_schedule_egress_tim
*/
static enum hrtimer_restart tile_net_handle_egress_timer(struct hrtimer *t)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
unsigned long irqflags;
bool pending = false;
int i;
@@ -649,7 +649,7 @@ static enum hrtimer_restart tile_net_han
*/
static void tile_net_update_cpu(void *arg)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct net_device *dev = arg;
if (!info->has_iqueue)
@@ -1572,7 +1572,7 @@ static void tso_egress(struct net_device
*/
static int tile_net_tx_tso(struct sk_buff *skb, struct net_device *dev)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct tile_net_priv *priv = netdev_priv(dev);
int channel = priv->echannel;
struct tile_net_egress *egress = &egress_for_echannel[channel];
@@ -1638,7 +1638,7 @@ static unsigned int tile_net_tx_frags(st
/* Help the kernel transmit a packet. */
static int tile_net_tx(struct sk_buff *skb, struct net_device *dev)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct tile_net_priv *priv = netdev_priv(dev);
struct tile_net_egress *egress = &egress_for_echannel[priv->echannel];
gxio_mpipe_equeue_t *equeue = egress->equeue;
@@ -1869,7 +1869,7 @@ static void tile_net_dev_init(const char
/* Per-cpu module initialization. */
static void tile_net_init_module_percpu(void *unused)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
int my_cpu = smp_processor_id();
info->has_iqueue = false;
Index: linux/drivers/net/ethernet/tile/tilepro.c
===================================================================
--- linux.orig/drivers/net/ethernet/tile/tilepro.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/net/ethernet/tile/tilepro.c 2013-08-22 14:16:11.327914014 -0500
@@ -1078,13 +1078,13 @@ static void tile_net_register(void *dev_
PDEBUG("tile_net_register(queue_id %d)\n", queue_id);
if (!strcmp(dev->name, "xgbe0"))
- info = &__get_cpu_var(hv_xgbe0);
+ info = this_cpu_ptr(&hv_xgbe0);
else if (!strcmp(dev->name, "xgbe1"))
- info = &__get_cpu_var(hv_xgbe1);
+ info = this_cpu_ptr(&hv_xgbe1);
else if (!strcmp(dev->name, "gbe0"))
- info = &__get_cpu_var(hv_gbe0);
+ info = this_cpu_ptr(&hv_gbe0);
else if (!strcmp(dev->name, "gbe1"))
- info = &__get_cpu_var(hv_gbe1);
+ info = this_cpu_ptr(&hv_gbe1);
else
BUG();
Index: linux/drivers/oprofile/cpu_buffer.c
===================================================================
--- linux.orig/drivers/oprofile/cpu_buffer.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/oprofile/cpu_buffer.c 2013-08-22 14:16:11.327914014 -0500
@@ -45,7 +45,7 @@ unsigned long oprofile_get_cpu_buffer_si
void oprofile_cpu_buffer_inc_smpl_lost(void)
{
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
cpu_buf->sample_lost_overflow++;
}
@@ -297,7 +297,7 @@ __oprofile_add_ext_sample(unsigned long
unsigned long event, int is_kernel,
struct task_struct *task)
{
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
unsigned long backtrace = oprofile_backtrace_depth;
/*
@@ -357,7 +357,7 @@ oprofile_write_reserve(struct op_entry *
{
struct op_sample *sample;
int is_kernel = !user_mode(regs);
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
cpu_buf->sample_received++;
@@ -412,13 +412,13 @@ int oprofile_write_commit(struct op_entr
void oprofile_add_pc(unsigned long pc, int is_kernel, unsigned long event)
{
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
log_sample(cpu_buf, pc, 0, is_kernel, event, NULL);
}
void oprofile_add_trace(unsigned long pc)
{
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
if (!cpu_buf->tracing)
return;
Index: linux/drivers/oprofile/timer_int.c
===================================================================
--- linux.orig/drivers/oprofile/timer_int.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/oprofile/timer_int.c 2013-08-22 14:16:11.327914014 -0500
@@ -32,7 +32,7 @@ static enum hrtimer_restart oprofile_hrt
static void __oprofile_hrtimer_start(void *unused)
{
- struct hrtimer *hrtimer = &__get_cpu_var(oprofile_hrtimer);
+ struct hrtimer *hrtimer = this_cpu_ptr(&oprofile_hrtimer);
if (!ctr_running)
return;
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 12/31] watchdog: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Wim Van Sebroeck, linux-watchdog, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/kernel/watchdog.c
===================================================================
--- linux.orig/kernel/watchdog.c 2013-08-26 14:28:39.000000000 -0500
+++ linux/kernel/watchdog.c 2013-08-26 14:29:34.471034451 -0500
@@ -174,8 +174,8 @@ EXPORT_SYMBOL(touch_nmi_watchdog);
void touch_softlockup_watchdog_sync(void)
{
- __raw_get_cpu_var(softlockup_touch_sync) = true;
- __raw_get_cpu_var(watchdog_touch_ts) = 0;
+ __this_cpu_write(softlockup_touch_sync, 1);
+ __this_cpu_write(watchdog_touch_ts, 0);
}
#ifdef CONFIG_HARDLOCKUP_DETECTOR
@@ -341,7 +341,7 @@ static void watchdog_set_prio(unsigned i
static void watchdog_enable(unsigned int cpu)
{
- struct hrtimer *hrtimer = &__raw_get_cpu_var(watchdog_hrtimer);
+ struct hrtimer *hrtimer = __this_cpu_ptr(&watchdog_hrtimer);
/* kick off the timer for the hardlockup detector */
hrtimer_init(hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
@@ -361,7 +361,7 @@ static void watchdog_enable(unsigned int
static void watchdog_disable(unsigned int cpu)
{
- struct hrtimer *hrtimer = &__raw_get_cpu_var(watchdog_hrtimer);
+ struct hrtimer *hrtimer = __this_cpu_ptr(&watchdog_hrtimer);
watchdog_set_prio(SCHED_NORMAL, 0);
hrtimer_cancel(hrtimer);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 14/31] drivers: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/drivers/char/random.c
===================================================================
--- linux.orig/drivers/char/random.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/char/random.c 2013-08-22 14:16:11.319914096 -0500
@@ -744,7 +744,7 @@ static DEFINE_PER_CPU(struct fast_pool,
void add_interrupt_randomness(int irq, int irq_flags)
{
struct entropy_store *r;
- struct fast_pool *fast_pool = &__get_cpu_var(irq_randomness);
+ struct fast_pool *fast_pool = this_cpu_ptr(&irq_randomness);
struct pt_regs *regs = get_irq_regs();
unsigned long now = jiffies;
__u32 input[4], cycles = get_cycles();
Index: linux/drivers/clocksource/metag_generic.c
===================================================================
--- linux.orig/drivers/clocksource/metag_generic.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/clocksource/metag_generic.c 2013-08-22 14:16:11.323914055 -0500
@@ -90,7 +90,7 @@ static struct clocksource clocksource_me
static irqreturn_t metag_timer_interrupt(int irq, void *dummy)
{
- struct clock_event_device *evt = &__get_cpu_var(local_clockevent);
+ struct clock_event_device *evt = this_cpu_ptr(&local_clockevent);
evt->event_handler(evt);
Index: linux/drivers/cpuidle/governors/ladder.c
===================================================================
--- linux.orig/drivers/cpuidle/governors/ladder.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/cpuidle/governors/ladder.c 2013-08-22 14:16:11.323914055 -0500
@@ -66,7 +66,7 @@ static inline void ladder_do_selection(s
static int ladder_select_state(struct cpuidle_driver *drv,
struct cpuidle_device *dev)
{
- struct ladder_device *ldev = &__get_cpu_var(ladder_devices);
+ struct ladder_device *ldev = this_cpu_ptr(&ladder_devices);
struct ladder_device_state *last_state;
int last_residency, last_idx = ldev->last_state_idx;
int latency_req = pm_qos_request(PM_QOS_CPU_DMA_LATENCY);
@@ -170,7 +170,7 @@ static int ladder_enable_device(struct c
*/
static void ladder_reflect(struct cpuidle_device *dev, int index)
{
- struct ladder_device *ldev = &__get_cpu_var(ladder_devices);
+ struct ladder_device *ldev = this_cpu_ptr(&ladder_devices);
if (index > 0)
ldev->last_state_idx = index;
}
Index: linux/drivers/cpuidle/governors/menu.c
===================================================================
--- linux.orig/drivers/cpuidle/governors/menu.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/cpuidle/governors/menu.c 2013-08-22 14:16:11.323914055 -0500
@@ -258,7 +258,7 @@ again:
*/
static int menu_select(struct cpuidle_driver *drv, struct cpuidle_device *dev)
{
- struct menu_device *data = &__get_cpu_var(menu_devices);
+ struct menu_device *data = this_cpu_ptr(&menu_devices);
int latency_req = pm_qos_request(PM_QOS_CPU_DMA_LATENCY);
int i;
int multiplier;
@@ -342,7 +342,7 @@ static int menu_select(struct cpuidle_dr
*/
static void menu_reflect(struct cpuidle_device *dev, int index)
{
- struct menu_device *data = &__get_cpu_var(menu_devices);
+ struct menu_device *data = this_cpu_ptr(&menu_devices);
data->last_state_idx = index;
if (index >= 0)
data->needs_update = 1;
@@ -355,7 +355,7 @@ static void menu_reflect(struct cpuidle_
*/
static void menu_update(struct cpuidle_driver *drv, struct cpuidle_device *dev)
{
- struct menu_device *data = &__get_cpu_var(menu_devices);
+ struct menu_device *data = this_cpu_ptr(&menu_devices);
int last_idx = data->last_state_idx;
unsigned int last_idle_us = cpuidle_get_last_residency(dev);
struct cpuidle_state *target = &drv->states[last_idx];
Index: linux/drivers/leds/trigger/ledtrig-cpu.c
===================================================================
--- linux.orig/drivers/leds/trigger/ledtrig-cpu.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/leds/trigger/ledtrig-cpu.c 2013-08-22 14:16:11.323914055 -0500
@@ -46,7 +46,7 @@ static DEFINE_PER_CPU(struct led_trigger
*/
void ledtrig_cpu(enum cpu_led_event ledevt)
{
- struct led_trigger_cpu *trig = &__get_cpu_var(cpu_trig);
+ struct led_trigger_cpu *trig = this_cpu_ptr(&cpu_trig);
/* Locate the correct CPU LED */
switch (ledevt) {
Index: linux/drivers/net/ethernet/tile/tilegx.c
===================================================================
--- linux.orig/drivers/net/ethernet/tile/tilegx.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/net/ethernet/tile/tilegx.c 2013-08-22 14:16:11.323914055 -0500
@@ -368,7 +368,7 @@ static void tile_net_pop_all_buffers(int
/* Provide linux buffers to mPIPE. */
static void tile_net_provide_needed_buffers(void)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
while (info->num_needed_small_buffers != 0) {
if (!tile_net_provide_buffer(true))
@@ -407,7 +407,7 @@ static inline bool filter_packet(struct
static void tile_net_receive_skb(struct net_device *dev, struct sk_buff *skb,
gxio_mpipe_idesc_t *idesc, unsigned long len)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct tile_net_priv *priv = netdev_priv(dev);
/* Encode the actual packet length. */
@@ -435,7 +435,7 @@ static void tile_net_receive_skb(struct
/* Handle a packet. Return true if "processed", false if "filtered". */
static bool tile_net_handle_packet(gxio_mpipe_idesc_t *idesc)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct net_device *dev = tile_net_devs_for_channel[idesc->channel];
uint8_t l2_offset;
void *va;
@@ -498,7 +498,7 @@ static bool tile_net_handle_packet(gxio_
*/
static int tile_net_poll(struct napi_struct *napi, int budget)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
unsigned int work = 0;
gxio_mpipe_idesc_t *idesc;
int i, n;
@@ -536,7 +536,7 @@ done:
/* Handle an ingress interrupt on the current cpu. */
static irqreturn_t tile_net_handle_ingress_irq(int irq, void *unused)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
napi_schedule(&info->napi);
return IRQ_HANDLED;
}
@@ -597,7 +597,7 @@ static enum hrtimer_restart tile_net_han
/* Make sure the egress timer is scheduled. */
static void tile_net_schedule_egress_timer(void)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
if (!info->egress_timer_scheduled) {
hrtimer_start(&info->egress_timer,
@@ -614,7 +614,7 @@ static void tile_net_schedule_egress_tim
*/
static enum hrtimer_restart tile_net_handle_egress_timer(struct hrtimer *t)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
unsigned long irqflags;
bool pending = false;
int i;
@@ -649,7 +649,7 @@ static enum hrtimer_restart tile_net_han
*/
static void tile_net_update_cpu(void *arg)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct net_device *dev = arg;
if (!info->has_iqueue)
@@ -1572,7 +1572,7 @@ static void tso_egress(struct net_device
*/
static int tile_net_tx_tso(struct sk_buff *skb, struct net_device *dev)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct tile_net_priv *priv = netdev_priv(dev);
int channel = priv->echannel;
struct tile_net_egress *egress = &egress_for_echannel[channel];
@@ -1638,7 +1638,7 @@ static unsigned int tile_net_tx_frags(st
/* Help the kernel transmit a packet. */
static int tile_net_tx(struct sk_buff *skb, struct net_device *dev)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
struct tile_net_priv *priv = netdev_priv(dev);
struct tile_net_egress *egress = &egress_for_echannel[priv->echannel];
gxio_mpipe_equeue_t *equeue = egress->equeue;
@@ -1869,7 +1869,7 @@ static void tile_net_dev_init(const char
/* Per-cpu module initialization. */
static void tile_net_init_module_percpu(void *unused)
{
- struct tile_net_info *info = &__get_cpu_var(per_cpu_info);
+ struct tile_net_info *info = this_cpu_ptr(&per_cpu_info);
int my_cpu = smp_processor_id();
info->has_iqueue = false;
Index: linux/drivers/net/ethernet/tile/tilepro.c
===================================================================
--- linux.orig/drivers/net/ethernet/tile/tilepro.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/net/ethernet/tile/tilepro.c 2013-08-22 14:16:11.327914014 -0500
@@ -1078,13 +1078,13 @@ static void tile_net_register(void *dev_
PDEBUG("tile_net_register(queue_id %d)\n", queue_id);
if (!strcmp(dev->name, "xgbe0"))
- info = &__get_cpu_var(hv_xgbe0);
+ info = this_cpu_ptr(&hv_xgbe0);
else if (!strcmp(dev->name, "xgbe1"))
- info = &__get_cpu_var(hv_xgbe1);
+ info = this_cpu_ptr(&hv_xgbe1);
else if (!strcmp(dev->name, "gbe0"))
- info = &__get_cpu_var(hv_gbe0);
+ info = this_cpu_ptr(&hv_gbe0);
else if (!strcmp(dev->name, "gbe1"))
- info = &__get_cpu_var(hv_gbe1);
+ info = this_cpu_ptr(&hv_gbe1);
else
BUG();
Index: linux/drivers/oprofile/cpu_buffer.c
===================================================================
--- linux.orig/drivers/oprofile/cpu_buffer.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/oprofile/cpu_buffer.c 2013-08-22 14:16:11.327914014 -0500
@@ -45,7 +45,7 @@ unsigned long oprofile_get_cpu_buffer_si
void oprofile_cpu_buffer_inc_smpl_lost(void)
{
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
cpu_buf->sample_lost_overflow++;
}
@@ -297,7 +297,7 @@ __oprofile_add_ext_sample(unsigned long
unsigned long event, int is_kernel,
struct task_struct *task)
{
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
unsigned long backtrace = oprofile_backtrace_depth;
/*
@@ -357,7 +357,7 @@ oprofile_write_reserve(struct op_entry *
{
struct op_sample *sample;
int is_kernel = !user_mode(regs);
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
cpu_buf->sample_received++;
@@ -412,13 +412,13 @@ int oprofile_write_commit(struct op_entr
void oprofile_add_pc(unsigned long pc, int is_kernel, unsigned long event)
{
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
log_sample(cpu_buf, pc, 0, is_kernel, event, NULL);
}
void oprofile_add_trace(unsigned long pc)
{
- struct oprofile_cpu_buffer *cpu_buf = &__get_cpu_var(op_cpu_buffer);
+ struct oprofile_cpu_buffer *cpu_buf = this_cpu_ptr(&op_cpu_buffer);
if (!cpu_buf->tracing)
return;
Index: linux/drivers/oprofile/timer_int.c
===================================================================
--- linux.orig/drivers/oprofile/timer_int.c 2013-08-22 14:16:11.335913933 -0500
+++ linux/drivers/oprofile/timer_int.c 2013-08-22 14:16:11.327914014 -0500
@@ -32,7 +32,7 @@ static enum hrtimer_restart oprofile_hrt
static void __oprofile_hrtimer_start(void *unused)
{
- struct hrtimer *hrtimer = &__get_cpu_var(oprofile_hrtimer);
+ struct hrtimer *hrtimer = this_cpu_ptr(&oprofile_hrtimer);
if (!ctr_running)
return;
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 13/31] kernel misc: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/kernel/printk/printk.c
===================================================================
--- linux.orig/kernel/printk/printk.c 2013-08-26 14:30:25.846490787 -0500
+++ linux/kernel/printk/printk.c 2013-08-26 14:30:25.846490787 -0500
@@ -2441,7 +2441,7 @@ static void wake_up_klogd_work_func(stru
int pending = __this_cpu_xchg(printk_pending, 0);
if (pending & PRINTK_PENDING_SCHED) {
- char *buf = __get_cpu_var(printk_sched_buf);
+ char *buf = this_cpu_ptr(printk_sched_buf);
printk(KERN_WARNING "[sched_delayed] %s", buf);
}
@@ -2459,7 +2459,7 @@ void wake_up_klogd(void)
preempt_disable();
if (waitqueue_active(&log_wait)) {
this_cpu_or(printk_pending, PRINTK_PENDING_WAKEUP);
- irq_work_queue(&__get_cpu_var(wake_up_klogd_work));
+ irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
}
preempt_enable();
}
@@ -2472,14 +2472,14 @@ int printk_sched(const char *fmt, ...)
int r;
local_irq_save(flags);
- buf = __get_cpu_var(printk_sched_buf);
+ buf = this_cpu_ptr(printk_sched_buf);
va_start(args, fmt);
r = vsnprintf(buf, PRINTK_BUF_SIZE, fmt, args);
va_end(args);
__this_cpu_or(printk_pending, PRINTK_PENDING_SCHED);
- irq_work_queue(&__get_cpu_var(wake_up_klogd_work));
+ irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
local_irq_restore(flags);
return r;
Index: linux/kernel/smp.c
===================================================================
--- linux.orig/kernel/smp.c 2013-08-26 14:30:25.846490787 -0500
+++ linux/kernel/smp.c 2013-08-26 14:30:25.846490787 -0500
@@ -172,7 +172,7 @@ void generic_exec_single(int cpu, struct
*/
void generic_smp_call_function_single_interrupt(void)
{
- struct call_single_queue *q = &__get_cpu_var(call_single_queue);
+ struct call_single_queue *q = this_cpu_ptr(&call_single_queue);
LIST_HEAD(list);
/*
@@ -252,7 +252,7 @@ int smp_call_function_single(int cpu, sm
struct call_single_data *csd = &d;
if (!wait)
- csd = &__get_cpu_var(csd_data);
+ csd = this_cpu_ptr(&csd_data);
csd_lock(csd);
@@ -401,7 +401,7 @@ void smp_call_function_many(const struct
return;
}
- cfd = &__get_cpu_var(cfd_data);
+ cfd = this_cpu_ptr(&cfd_data);
cpumask_and(cfd->cpumask, mask, cpu_online_mask);
cpumask_clear_cpu(this_cpu, cfd->cpumask);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 12/31] watchdog: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Wim Van Sebroeck, linux-watchdog, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/kernel/watchdog.c
===================================================================
--- linux.orig/kernel/watchdog.c 2013-08-26 14:28:39.000000000 -0500
+++ linux/kernel/watchdog.c 2013-08-26 14:29:34.471034451 -0500
@@ -174,8 +174,8 @@ EXPORT_SYMBOL(touch_nmi_watchdog);
void touch_softlockup_watchdog_sync(void)
{
- __raw_get_cpu_var(softlockup_touch_sync) = true;
- __raw_get_cpu_var(watchdog_touch_ts) = 0;
+ __this_cpu_write(softlockup_touch_sync, 1);
+ __this_cpu_write(watchdog_touch_ts, 0);
}
#ifdef CONFIG_HARDLOCKUP_DETECTOR
@@ -341,7 +341,7 @@ static void watchdog_set_prio(unsigned i
static void watchdog_enable(unsigned int cpu)
{
- struct hrtimer *hrtimer = &__raw_get_cpu_var(watchdog_hrtimer);
+ struct hrtimer *hrtimer = __this_cpu_ptr(&watchdog_hrtimer);
/* kick off the timer for the hardlockup detector */
hrtimer_init(hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
@@ -361,7 +361,7 @@ static void watchdog_enable(unsigned int
static void watchdog_disable(unsigned int cpu)
{
- struct hrtimer *hrtimer = &__raw_get_cpu_var(watchdog_hrtimer);
+ struct hrtimer *hrtimer = __this_cpu_ptr(&watchdog_hrtimer);
watchdog_set_prio(SCHED_NORMAL, 0);
hrtimer_cancel(hrtimer);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 15/31] zcache/zsmalloc: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Dan Magenheimer, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/drivers/staging/zcache/ramster/ramster.c
===================================================================
--- linux.orig/drivers/staging/zcache/ramster/ramster.c 2013-08-22 14:14:40.400841754 -0500
+++ linux/drivers/staging/zcache/ramster/ramster.c 2013-08-22 14:15:51.000000000 -0500
@@ -82,7 +82,7 @@ static struct flushlist_node *ramster_fl
struct flushlist_node *flnode = NULL;
struct ramster_preload *kp;
- kp = &__get_cpu_var(ramster_preloads);
+ kp = this_cpu_ptr(&ramster_preloads);
flnode = kp->flnode;
BUG_ON(flnode == NULL);
kp->flnode = NULL;
@@ -109,7 +109,7 @@ int ramster_do_preload_flnode(struct tme
BUG_ON(!irqs_disabled());
if (unlikely(ramster_flnode_cache == NULL))
BUG();
- kp = &__get_cpu_var(ramster_preloads);
+ kp = this_cpu_ptr(&ramster_preloads);
flnode = kmem_cache_alloc(ramster_flnode_cache, GFP_ATOMIC);
if (unlikely(flnode == NULL) && kp->flnode == NULL)
BUG(); /* FIXME handle more gracefully, but how??? */
@@ -478,8 +478,8 @@ int ramster_remotify_pageframe(bool eph)
struct tmem_handle th[2];
unsigned int zsize[2];
- tmpmem[0] = __get_cpu_var(ramster_remoteputmem1);
- tmpmem[1] = __get_cpu_var(ramster_remoteputmem2);
+ tmpmem[0] = __this_cpu_read(ramster_remoteputmem1);
+ tmpmem[1] = __this_cpu_read(ramster_remoteputmem2);
local_bh_disable();
zbuds = zbud_make_zombie_lru(&th[0], &tmpmem[0], &zsize[0], eph);
/* now OK to release lock set in caller */
Index: linux/drivers/staging/zcache/zcache-main.c
===================================================================
--- linux.orig/drivers/staging/zcache/zcache-main.c 2013-08-22 14:14:40.400841754 -0500
+++ linux/drivers/staging/zcache/zcache-main.c 2013-08-22 14:15:51.000000000 -0500
@@ -250,7 +250,7 @@ static struct tmem_objnode *zcache_objno
struct zcache_preload *kp;
int i;
- kp = &__get_cpu_var(zcache_preloads);
+ kp = this_cpu_ptr(&zcache_preloads);
for (i = 0; i < ARRAY_SIZE(kp->objnodes); i++) {
objnode = kp->objnodes[i];
if (objnode != NULL) {
@@ -275,7 +275,7 @@ static struct tmem_obj *zcache_obj_alloc
struct tmem_obj *obj = NULL;
struct zcache_preload *kp;
- kp = &__get_cpu_var(zcache_preloads);
+ kp = this_cpu_ptr(&zcache_preloads);
obj = kp->obj;
BUG_ON(obj == NULL);
kp->obj = NULL;
@@ -531,7 +531,7 @@ void *zcache_pampd_create(char *data, un
/* pre-allocate per-cpu metadata */
BUG_ON(zcache_objnode_cache == NULL);
BUG_ON(zcache_obj_cache == NULL);
- kp = &__get_cpu_var(zcache_preloads);
+ kp = this_cpu_ptr(&zcache_preloads);
for (i = 0; i < ARRAY_SIZE(kp->objnodes); i++) {
objnode = kp->objnodes[i];
if (objnode == NULL) {
@@ -761,7 +761,7 @@ static DEFINE_PER_CPU(unsigned char *, z
static void zcache_compress(struct page *from, void **out_va, unsigned *out_len)
{
int ret;
- unsigned char *dmem = __get_cpu_var(zcache_dstmem);
+ unsigned char *dmem = __this_cpu_read(zcache_dstmem);
char *from_va;
BUG_ON(!irqs_disabled());
Index: linux/drivers/staging/zsmalloc/zsmalloc-main.c
===================================================================
--- linux.orig/drivers/staging/zsmalloc/zsmalloc-main.c 2013-08-22 14:14:29.000000000 -0500
+++ linux/drivers/staging/zsmalloc/zsmalloc-main.c 2013-08-22 14:15:51.000000000 -0500
@@ -1028,7 +1028,7 @@ void zs_unmap_object(struct zs_pool *poo
class = &pool->size_class[class_idx];
off = obj_idx_to_offset(page, obj_idx, class->size);
- area = &__get_cpu_var(zs_map_area);
+ area = this_cpu_ptr(&zs_map_area);
if (off + class->size <= PAGE_SIZE)
kunmap_atomic(area->vm_addr);
else {
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 16/31] x86: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86,
linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/x86/kernel/apic/x2apic_uv_x.c
===================================================================
--- linux.orig/arch/x86/kernel/apic/x2apic_uv_x.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/apic/x2apic_uv_x.c 2013-08-22 15:01:56.715952554 -0500
@@ -874,10 +874,10 @@ int uv_handle_nmi(unsigned int reason, s
spin_unlock(&uv_blade_info[bid].nmi_lock);
}
- if (likely(__get_cpu_var(cpu_last_nmi_count) == uv_blade_info[bid].nmi_count))
+ if (likely(__this_cpu_read(cpu_last_nmi_count) == uv_blade_info[bid].nmi_count))
return NMI_DONE;
- __get_cpu_var(cpu_last_nmi_count) = uv_blade_info[bid].nmi_count;
+ __this_cpu_write(cpu_last_nmi_count, uv_blade_info[bid].nmi_count);
/*
* Use a lock so only one cpu prints at a time.
Index: linux/arch/x86/kernel/cpu/mcheck/mce_intel.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/mcheck/mce_intel.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/mcheck/mce_intel.c 2013-08-22 16:16:54.974785742 -0500
@@ -86,7 +86,7 @@ void mce_intel_cmci_poll(void)
{
if (__this_cpu_read(cmci_storm_state) == CMCI_STORM_NONE)
return;
- machine_check_poll(MCP_TIMESTAMP, &__get_cpu_var(mce_banks_owned));
+ machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned));
}
void mce_intel_hcpu_update(unsigned long cpu)
@@ -179,7 +179,7 @@ static void intel_threshold_interrupt(vo
{
if (cmci_storm_detect())
return;
- machine_check_poll(MCP_TIMESTAMP, &__get_cpu_var(mce_banks_owned));
+ machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned));
mce_notify_irq();
}
@@ -190,7 +190,7 @@ static void intel_threshold_interrupt(vo
*/
static void cmci_discover(int banks)
{
- unsigned long *owned = (void *)&__get_cpu_var(mce_banks_owned);
+ unsigned long *owned = (void *)this_cpu_ptr(&mce_banks_owned);
unsigned long flags;
int i;
int bios_wrong_thresh = 0;
@@ -208,7 +208,7 @@ static void cmci_discover(int banks)
/* Already owned by someone else? */
if (val & MCI_CTL2_CMCI_EN) {
clear_bit(i, owned);
- __clear_bit(i, __get_cpu_var(mce_poll_banks));
+ __clear_bit(i, this_cpu_ptr(mce_poll_banks));
continue;
}
@@ -232,7 +232,7 @@ static void cmci_discover(int banks)
/* Did the enable bit stick? -- the bank supports CMCI */
if (val & MCI_CTL2_CMCI_EN) {
set_bit(i, owned);
- __clear_bit(i, __get_cpu_var(mce_poll_banks));
+ __clear_bit(i, this_cpu_ptr(mce_poll_banks));
/*
* We are able to set thresholds for some banks that
* had a threshold of 0. This means the BIOS has not
@@ -243,7 +243,7 @@ static void cmci_discover(int banks)
(val & MCI_CTL2_CMCI_THRESHOLD_MASK))
bios_wrong_thresh = 1;
} else {
- WARN_ON(!test_bit(i, __get_cpu_var(mce_poll_banks)));
+ WARN_ON(!test_bit(i, this_cpu_ptr(mce_poll_banks)));
}
}
raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
@@ -267,7 +267,7 @@ void cmci_recheck(void)
if (!mce_available(__this_cpu_ptr(&cpu_info)) || !cmci_supported(&banks))
return;
local_irq_save(flags);
- machine_check_poll(MCP_TIMESTAMP, &__get_cpu_var(mce_banks_owned));
+ machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned));
local_irq_restore(flags);
}
@@ -286,13 +286,13 @@ void cmci_clear(void)
return;
raw_spin_lock_irqsave(&cmci_discover_lock, flags);
for (i = 0; i < banks; i++) {
- if (!test_bit(i, __get_cpu_var(mce_banks_owned)))
+ if (!test_bit(i, this_cpu_ptr(mce_banks_owned)))
continue;
/* Disable CMCI */
rdmsrl(MSR_IA32_MCx_CTL2(i), val);
val &= ~MCI_CTL2_CMCI_EN;
wrmsrl(MSR_IA32_MCx_CTL2(i), val);
- __clear_bit(i, __get_cpu_var(mce_banks_owned));
+ __clear_bit(i, this_cpu_ptr(mce_banks_owned));
}
raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
}
Index: linux/arch/x86/kernel/irq_64.c
===================================================================
--- linux.orig/arch/x86/kernel/irq_64.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/irq_64.c 2013-08-22 16:16:54.974785742 -0500
@@ -52,13 +52,13 @@ static inline void stack_overflow_check(
regs->sp <= curbase + THREAD_SIZE)
return;
- irq_stack_top = (u64)__get_cpu_var(irq_stack_union.irq_stack) +
+ irq_stack_top = (u64)this_cpu_ptr(irq_stack_union.irq_stack) +
STACK_TOP_MARGIN;
- irq_stack_bottom = (u64)__get_cpu_var(irq_stack_ptr);
+ irq_stack_bottom = (u64)__this_cpu_read(irq_stack_ptr);
if (regs->sp >= irq_stack_top && regs->sp <= irq_stack_bottom)
return;
- oist = &__get_cpu_var(orig_ist);
+ oist = this_cpu_ptr(&orig_ist);
estack_top = (u64)oist->ist[0] - EXCEPTION_STKSZ + STACK_TOP_MARGIN;
estack_bottom = (u64)oist->ist[N_EXCEPTION_STACKS - 1];
if (regs->sp >= estack_top && regs->sp <= estack_bottom)
Index: linux/arch/x86/kernel/kvm.c
===================================================================
--- linux.orig/arch/x86/kernel/kvm.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/kvm.c 2013-08-22 16:16:54.974785742 -0500
@@ -242,9 +242,9 @@ u32 kvm_read_and_reset_pf_reason(void)
{
u32 reason = 0;
- if (__get_cpu_var(apf_reason).enabled) {
- reason = __get_cpu_var(apf_reason).reason;
- __get_cpu_var(apf_reason).reason = 0;
+ if (__this_cpu_read(apf_reason.enabled)) {
+ reason = __this_cpu_read(apf_reason.reason);
+ __this_cpu_write(apf_reason.reason, 0);
}
return reason;
@@ -315,7 +315,7 @@ static void kvm_guest_apic_eoi_write(u32
* there's no need for lock or memory barriers.
* An optimization barrier is implied in apic write.
*/
- if (__test_and_clear_bit(KVM_PV_EOI_BIT, &__get_cpu_var(kvm_apic_eoi)))
+ if (__test_and_clear_bit(KVM_PV_EOI_BIT, this_cpu_ptr(&kvm_apic_eoi)))
return;
apic_write(APIC_EOI, APIC_EOI_ACK);
}
@@ -326,13 +326,13 @@ void kvm_guest_cpu_init(void)
return;
if (kvm_para_has_feature(KVM_FEATURE_ASYNC_PF) && kvmapf) {
- u64 pa = slow_virt_to_phys(&__get_cpu_var(apf_reason));
+ u64 pa = slow_virt_to_phys(this_cpu_ptr(&apf_reason));
#ifdef CONFIG_PREEMPT
pa |= KVM_ASYNC_PF_SEND_ALWAYS;
#endif
wrmsrl(MSR_KVM_ASYNC_PF_EN, pa | KVM_ASYNC_PF_ENABLED);
- __get_cpu_var(apf_reason).enabled = 1;
+ __this_cpu_write(apf_reason.enabled, 1);
printk(KERN_INFO"KVM setup async PF for cpu %d\n",
smp_processor_id());
}
@@ -341,8 +341,8 @@ void kvm_guest_cpu_init(void)
unsigned long pa;
/* Size alignment is implied but just to make it explicit. */
BUILD_BUG_ON(__alignof__(kvm_apic_eoi) < 4);
- __get_cpu_var(kvm_apic_eoi) = 0;
- pa = slow_virt_to_phys(&__get_cpu_var(kvm_apic_eoi))
+ __this_cpu_write(kvm_apic_eoi, 0);
+ pa = slow_virt_to_phys(this_cpu_ptr(&kvm_apic_eoi))
| KVM_MSR_ENABLED;
wrmsrl(MSR_KVM_PV_EOI_EN, pa);
}
@@ -353,11 +353,11 @@ void kvm_guest_cpu_init(void)
static void kvm_pv_disable_apf(void)
{
- if (!__get_cpu_var(apf_reason).enabled)
+ if (!__this_cpu_read(apf_reason.enabled))
return;
wrmsrl(MSR_KVM_ASYNC_PF_EN, 0);
- __get_cpu_var(apf_reason).enabled = 0;
+ __this_cpu_write(apf_reason.enabled, 0);
printk(KERN_INFO"Unregister pv shared memory for cpu %d\n",
smp_processor_id());
Index: linux/arch/x86/kvm/svm.c
===================================================================
--- linux.orig/arch/x86/kvm/svm.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kvm/svm.c 2013-08-22 16:16:54.974785742 -0500
@@ -654,7 +654,7 @@ static int svm_hardware_enable(void *gar
if (static_cpu_has(X86_FEATURE_TSCRATEMSR)) {
wrmsrl(MSR_AMD64_TSC_RATIO, TSC_RATIO_DEFAULT);
- __get_cpu_var(current_tsc_ratio) = TSC_RATIO_DEFAULT;
+ __this_cpu_write(current_tsc_ratio, TSC_RATIO_DEFAULT);
}
@@ -1312,8 +1312,8 @@ static void svm_vcpu_load(struct kvm_vcp
rdmsrl(host_save_user_msrs[i], svm->host_user_msrs[i]);
if (static_cpu_has(X86_FEATURE_TSCRATEMSR) &&
- svm->tsc_ratio != __get_cpu_var(current_tsc_ratio)) {
- __get_cpu_var(current_tsc_ratio) = svm->tsc_ratio;
+ svm->tsc_ratio != __this_cpu_read(current_tsc_ratio)) {
+ __this_cpu_write(current_tsc_ratio, svm->tsc_ratio);
wrmsrl(MSR_AMD64_TSC_RATIO, svm->tsc_ratio);
}
}
Index: linux/arch/x86/kvm/x86.c
===================================================================
--- linux.orig/arch/x86/kvm/x86.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kvm/x86.c 2013-08-22 15:01:56.715952554 -0500
@@ -1485,7 +1485,7 @@ static int kvm_guest_time_update(struct
/* Keep irq disabled to prevent changes to the clock */
local_irq_save(flags);
- this_tsc_khz = __get_cpu_var(cpu_tsc_khz);
+ this_tsc_khz = __this_cpu_read(cpu_tsc_khz);
if (unlikely(this_tsc_khz == 0)) {
local_irq_restore(flags);
kvm_make_request(KVM_REQ_CLOCK_UPDATE, v);
Index: linux/arch/x86/oprofile/op_model_p4.c
===================================================================
--- linux.orig/arch/x86/oprofile/op_model_p4.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/oprofile/op_model_p4.c 2013-08-22 15:01:56.715952554 -0500
@@ -372,7 +372,7 @@ static unsigned int get_stagger(void)
{
#ifdef CONFIG_SMP
int cpu = smp_processor_id();
- return cpu != cpumask_first(__get_cpu_var(cpu_sibling_map));
+ return cpu != cpumask_first(this_cpu_ptr(cpu_sibling_map));
#endif
return 0;
}
Index: linux/arch/x86/xen/enlighten.c
===================================================================
--- linux.orig/arch/x86/xen/enlighten.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/xen/enlighten.c 2013-08-22 16:16:54.974785742 -0500
@@ -817,7 +817,7 @@ static void xen_convert_trap_info(const
void xen_copy_trap_info(struct trap_info *traps)
{
- const struct desc_ptr *desc = &__get_cpu_var(idt_desc);
+ const struct desc_ptr *desc = this_cpu_ptr(&idt_desc);
xen_convert_trap_info(desc, traps);
}
@@ -834,7 +834,7 @@ static void xen_load_idt(const struct de
spin_lock(&lock);
- __get_cpu_var(idt_desc) = *desc;
+ memcpy(this_cpu_ptr(&idt_desc), desc, sizeof(idt_desc));
xen_convert_trap_info(desc, traps);
Index: linux/arch/x86/kernel/apb_timer.c
===================================================================
--- linux.orig/arch/x86/kernel/apb_timer.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/apb_timer.c 2013-08-22 15:01:56.715952554 -0500
@@ -146,7 +146,7 @@ static inline int is_apbt_capable(void)
static int __init apbt_clockevent_register(void)
{
struct sfi_timer_table_entry *mtmr;
- struct apbt_dev *adev = &__get_cpu_var(cpu_apbt_dev);
+ struct apbt_dev *adev = this_cpu_ptr(&cpu_apbt_dev);
mtmr = sfi_get_mtmr(APBT_CLOCKEVENT0_NUM);
if (mtmr == NULL) {
@@ -200,7 +200,7 @@ void apbt_setup_secondary_clock(void)
if (!cpu)
return;
- adev = &__get_cpu_var(cpu_apbt_dev);
+ adev = this_cpu_ptr(&cpu_apbt_dev);
if (!adev->timer) {
adev->timer = dw_apb_clockevent_init(cpu, adev->name,
APBT_CLOCKEVENT_RATING, adev_virt_addr(adev),
Index: linux/arch/x86/kernel/apic/apic.c
===================================================================
--- linux.orig/arch/x86/kernel/apic/apic.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/apic/apic.c 2013-08-22 15:01:56.715952554 -0500
@@ -546,7 +546,7 @@ static DEFINE_PER_CPU(struct clock_event
*/
static void setup_APIC_timer(void)
{
- struct clock_event_device *levt = &__get_cpu_var(lapic_events);
+ struct clock_event_device *levt = this_cpu_ptr(&lapic_events);
if (this_cpu_has(X86_FEATURE_ARAT)) {
lapic_clockevent.features &= ~CLOCK_EVT_FEAT_C3STOP;
@@ -681,7 +681,7 @@ calibrate_by_pmtimer(long deltapm, long
static int __init calibrate_APIC_clock(void)
{
- struct clock_event_device *levt = &__get_cpu_var(lapic_events);
+ struct clock_event_device *levt = this_cpu_ptr(&lapic_events);
void (*real_handler)(struct clock_event_device *dev);
unsigned long deltaj;
long delta, deltatsc;
Index: linux/arch/x86/kernel/cpu/mcheck/mce-inject.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/mcheck/mce-inject.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/mcheck/mce-inject.c 2013-08-22 15:01:56.715952554 -0500
@@ -83,7 +83,7 @@ static DEFINE_MUTEX(mce_inject_mutex);
static int mce_raise_notify(unsigned int cmd, struct pt_regs *regs)
{
int cpu = smp_processor_id();
- struct mce *m = &__get_cpu_var(injectm);
+ struct mce *m = this_cpu_ptr(&injectm);
if (!cpumask_test_cpu(cpu, mce_inject_cpumask))
return NMI_DONE;
cpumask_clear_cpu(cpu, mce_inject_cpumask);
@@ -97,7 +97,7 @@ static int mce_raise_notify(unsigned int
static void mce_irq_ipi(void *info)
{
int cpu = smp_processor_id();
- struct mce *m = &__get_cpu_var(injectm);
+ struct mce *m = this_cpu_ptr(&injectm);
if (cpumask_test_cpu(cpu, mce_inject_cpumask) &&
m->inject_flags & MCJ_EXCEPTION) {
@@ -109,7 +109,7 @@ static void mce_irq_ipi(void *info)
/* Inject mce on current CPU */
static int raise_local(void)
{
- struct mce *m = &__get_cpu_var(injectm);
+ struct mce *m = this_cpu_ptr(&injectm);
int context = MCJ_CTX(m->inject_flags);
int ret = 0;
int cpu = m->extcpu;
Index: linux/arch/x86/kernel/cpu/mcheck/mce.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/mcheck/mce.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/mcheck/mce.c 2013-08-22 15:01:56.719952513 -0500
@@ -390,7 +390,7 @@ static u64 mce_rdmsrl(u32 msr)
if (offset < 0)
return 0;
- return *(u64 *)((char *)&__get_cpu_var(injectm) + offset);
+ return *(u64 *)((char *)this_cpu_ptr(&injectm) + offset);
}
if (rdmsrl_safe(msr, &v)) {
@@ -412,7 +412,7 @@ static void mce_wrmsrl(u32 msr, u64 v)
int offset = msr_to_offset(msr);
if (offset >= 0)
- *(u64 *)((char *)&__get_cpu_var(injectm) + offset) = v;
+ *(u64 *)((char *)this_cpu_ptr(&injectm) + offset) = v;
return;
}
wrmsrl(msr, v);
@@ -468,7 +468,7 @@ static DEFINE_PER_CPU(struct mce_ring, m
/* Runs with CPU affinity in workqueue */
static int mce_ring_empty(void)
{
- struct mce_ring *r = &__get_cpu_var(mce_ring);
+ struct mce_ring *r = this_cpu_ptr(&mce_ring);
return r->start == r->end;
}
@@ -480,7 +480,7 @@ static int mce_ring_get(unsigned long *p
*pfn = 0;
get_cpu();
- r = &__get_cpu_var(mce_ring);
+ r = this_cpu_ptr(&mce_ring);
if (r->start == r->end)
goto out;
*pfn = r->ring[r->start];
@@ -494,7 +494,7 @@ out:
/* Always runs in MCE context with preempt off */
static int mce_ring_add(unsigned long pfn)
{
- struct mce_ring *r = &__get_cpu_var(mce_ring);
+ struct mce_ring *r = this_cpu_ptr(&mce_ring);
unsigned next;
next = (r->end + 1) % MCE_RING_SIZE;
@@ -516,7 +516,7 @@ int mce_available(struct cpuinfo_x86 *c)
static void mce_schedule_work(void)
{
if (!mce_ring_empty())
- schedule_work(&__get_cpu_var(mce_work));
+ schedule_work(this_cpu_ptr(&mce_work));
}
DEFINE_PER_CPU(struct irq_work, mce_irq_work);
@@ -541,7 +541,7 @@ static void mce_report_event(struct pt_r
return;
}
- irq_work_queue(&__get_cpu_var(mce_irq_work));
+ irq_work_queue(this_cpu_ptr(&mce_irq_work));
}
/*
@@ -1037,7 +1037,7 @@ void do_machine_check(struct pt_regs *re
mce_gather_info(&m, regs);
- final = &__get_cpu_var(mces_seen);
+ final = this_cpu_ptr(&mces_seen);
*final = m;
memset(valid_banks, 0, sizeof(valid_banks));
@@ -1271,14 +1271,14 @@ static unsigned long (*mce_adjust_timer)
static void mce_timer_fn(unsigned long data)
{
- struct timer_list *t = &__get_cpu_var(mce_timer);
+ struct timer_list *t = this_cpu_ptr(&mce_timer);
unsigned long iv;
WARN_ON(smp_processor_id() != data);
if (mce_available(__this_cpu_ptr(&cpu_info))) {
machine_check_poll(MCP_TIMESTAMP,
- &__get_cpu_var(mce_poll_banks));
+ this_cpu_ptr(&mce_poll_banks));
mce_intel_cmci_poll();
}
@@ -1306,7 +1306,7 @@ static void mce_timer_fn(unsigned long d
*/
void mce_timer_kick(unsigned long interval)
{
- struct timer_list *t = &__get_cpu_var(mce_timer);
+ struct timer_list *t = this_cpu_ptr(&mce_timer);
unsigned long when = jiffies + interval;
unsigned long iv = __this_cpu_read(mce_next_interval);
@@ -1642,7 +1642,7 @@ static void mce_start_timer(unsigned int
static void __mcheck_cpu_init_timer(void)
{
- struct timer_list *t = &__get_cpu_var(mce_timer);
+ struct timer_list *t = this_cpu_ptr(&mce_timer);
unsigned int cpu = smp_processor_id();
setup_timer(t, mce_timer_fn, cpu);
@@ -1685,8 +1685,8 @@ void mcheck_cpu_init(struct cpuinfo_x86
__mcheck_cpu_init_generic();
__mcheck_cpu_init_vendor(c);
__mcheck_cpu_init_timer();
- INIT_WORK(&__get_cpu_var(mce_work), mce_process_work);
- init_irq_work(&__get_cpu_var(mce_irq_work), &mce_irq_work_cb);
+ INIT_WORK(this_cpu_ptr(&mce_work), mce_process_work);
+ init_irq_work(this_cpu_ptr(&mce_irq_work), &mce_irq_work_cb);
}
/*
Index: linux/arch/x86/kernel/cpu/mcheck/mce_amd.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/mcheck/mce_amd.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/mcheck/mce_amd.c 2013-08-22 15:01:56.719952513 -0500
@@ -310,7 +310,7 @@ static void amd_threshold_interrupt(void
* event.
*/
machine_check_poll(MCP_TIMESTAMP,
- &__get_cpu_var(mce_poll_banks));
+ this_cpu_ptr(&mce_poll_banks));
if (high & MASK_OVERFLOW_HI) {
rdmsrl(address, m.misc);
Index: linux/arch/x86/kernel/cpu/perf_event.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event.c 2013-08-22 15:01:56.719952513 -0500
@@ -493,7 +493,7 @@ static int __x86_pmu_event_init(struct p
void x86_pmu_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
@@ -511,7 +511,7 @@ void x86_pmu_disable_all(void)
static void x86_pmu_disable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!x86_pmu_initialized())
return;
@@ -528,7 +528,7 @@ static void x86_pmu_disable(struct pmu *
void x86_pmu_enable_all(int added)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
@@ -874,7 +874,7 @@ static void x86_pmu_start(struct perf_ev
static void x86_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event;
struct hw_perf_event *hwc;
int i, added = cpuc->n_added;
@@ -1023,7 +1023,7 @@ void x86_pmu_enable_event(struct perf_ev
*/
static int x86_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc;
int assign[X86_PMC_IDX_MAX];
int n, n0, ret;
@@ -1070,7 +1070,7 @@ out:
static void x86_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = event->hw.idx;
if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED)))
@@ -1149,7 +1149,7 @@ void perf_event_print_debug(void)
void x86_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
if (__test_and_clear_bit(hwc->idx, cpuc->active_mask)) {
@@ -1171,7 +1171,7 @@ void x86_pmu_stop(struct perf_event *eve
static void x86_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
/*
@@ -1213,7 +1213,7 @@ int x86_pmu_handle_irq(struct pt_regs *r
int idx, handled = 0;
u64 val;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
/*
* Some chipsets need to unmask the LVTPC in a particular spot
@@ -1608,7 +1608,7 @@ static void x86_pmu_cancel_txn(struct pm
*/
static int x86_pmu_commit_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int assign[X86_PMC_IDX_MAX];
int n, ret;
Index: linux/arch/x86/kernel/cpu/perf_event_amd.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_amd.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_amd.c 2013-08-22 15:01:56.719952513 -0500
@@ -700,7 +700,7 @@ __init int amd_pmu_init(void)
void amd_pmu_enable_virt(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
cpuc->perf_ctr_virt_mask = 0;
@@ -712,7 +712,7 @@ EXPORT_SYMBOL_GPL(amd_pmu_enable_virt);
void amd_pmu_disable_virt(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
/*
* We only mask out the Host-only bit so that host-only counting works
Index: linux/arch/x86/kernel/cpu/perf_event_intel.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_intel.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_intel.c 2013-08-22 15:01:56.719952513 -0500
@@ -898,7 +898,7 @@ static inline bool intel_pmu_needs_lbr_s
static void intel_pmu_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL, 0);
@@ -911,7 +911,7 @@ static void intel_pmu_disable_all(void)
static void intel_pmu_enable_all(int added)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
intel_pmu_pebs_enable_all();
intel_pmu_lbr_enable_all();
@@ -945,7 +945,7 @@ static void intel_pmu_enable_all(int add
*/
static void intel_pmu_nhm_workaround(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
static const unsigned long nhm_magic[4] = {
0x4300B5,
0x4300D2,
@@ -1039,7 +1039,7 @@ static void intel_pmu_disable_fixed(stru
static void intel_pmu_disable_event(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (unlikely(hwc->idx == INTEL_PMC_IDX_FIXED_BTS)) {
intel_pmu_disable_bts();
@@ -1102,7 +1102,7 @@ static void intel_pmu_enable_fixed(struc
static void intel_pmu_enable_event(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (unlikely(hwc->idx == INTEL_PMC_IDX_FIXED_BTS)) {
if (!__this_cpu_read(cpu_hw_events.enabled))
@@ -1182,7 +1182,7 @@ static int intel_pmu_handle_irq(struct p
u64 status;
int handled;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
/*
* No known reason to not always do late ACK,
@@ -1600,7 +1600,7 @@ EXPORT_SYMBOL_GPL(perf_guest_get_msrs);
static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_guest_switch_msr *arr = cpuc->guest_switch_msrs;
arr[0].msr = MSR_CORE_PERF_GLOBAL_CTRL;
@@ -1621,7 +1621,7 @@ static struct perf_guest_switch_msr *int
static struct perf_guest_switch_msr *core_guest_get_msrs(int *nr)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_guest_switch_msr *arr = cpuc->guest_switch_msrs;
int idx;
@@ -1655,7 +1655,7 @@ static void core_pmu_enable_event(struct
static void core_pmu_enable_all(int added)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
Index: linux/arch/x86/kernel/cpu/perf_event_intel_ds.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_intel_ds.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_intel_ds.c 2013-08-22 15:01:56.719952513 -0500
@@ -424,7 +424,7 @@ void intel_pmu_enable_bts(u64 config)
void intel_pmu_disable_bts(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
unsigned long debugctlmsr;
if (!cpuc->ds)
@@ -441,7 +441,7 @@ void intel_pmu_disable_bts(void)
int intel_pmu_drain_bts_buffer(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct bts_record {
u64 from;
@@ -634,7 +634,7 @@ struct event_constraint *intel_pebs_cons
void intel_pmu_pebs_enable(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
hwc->config &= ~ARCH_PERFMON_EVENTSEL_INT;
@@ -649,7 +649,7 @@ void intel_pmu_pebs_enable(struct perf_e
void intel_pmu_pebs_disable(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
cpuc->pebs_enabled &= ~(1ULL << hwc->idx);
@@ -667,7 +667,7 @@ void intel_pmu_pebs_disable(struct perf_
void intel_pmu_pebs_enable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->pebs_enabled)
wrmsrl(MSR_IA32_PEBS_ENABLE, cpuc->pebs_enabled);
@@ -675,7 +675,7 @@ void intel_pmu_pebs_enable_all(void)
void intel_pmu_pebs_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->pebs_enabled)
wrmsrl(MSR_IA32_PEBS_ENABLE, 0);
@@ -683,7 +683,7 @@ void intel_pmu_pebs_disable_all(void)
static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
unsigned long from = cpuc->lbr_entries[0].from;
unsigned long old_to, to = cpuc->lbr_entries[0].to;
unsigned long ip = regs->ip;
@@ -766,7 +766,7 @@ static void __intel_pmu_pebs_event(struc
* We cast to pebs_record_nhm to get the load latency data
* if extra_reg MSR_PEBS_LD_LAT_THRESHOLD used
*/
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct pebs_record_nhm *pebs = __pebs;
struct pebs_record_hsw *pebs_hsw = __pebs;
struct perf_sample_data data;
@@ -847,7 +847,7 @@ static void __intel_pmu_pebs_event(struc
static void intel_pmu_drain_pebs_core(struct pt_regs *iregs)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct perf_event *event = cpuc->events[0]; /* PMC0 only */
struct pebs_record_core *at, *top;
@@ -889,7 +889,7 @@ static void intel_pmu_drain_pebs_core(st
static void __intel_pmu_drain_pebs_nhm(struct pt_regs *iregs, void *at,
void *top)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct perf_event *event = NULL;
u64 status = 0;
@@ -926,7 +926,7 @@ static void __intel_pmu_drain_pebs_nhm(s
static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct pebs_record_nhm *at, *top;
int n;
@@ -955,7 +955,7 @@ static void intel_pmu_drain_pebs_nhm(str
static void intel_pmu_drain_pebs_hsw(struct pt_regs *iregs)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct pebs_record_hsw *at, *top;
int n;
Index: linux/arch/x86/kernel/cpu/perf_event_intel_lbr.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_intel_lbr.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_intel_lbr.c 2013-08-22 15:01:56.719952513 -0500
@@ -133,7 +133,7 @@ static void intel_pmu_lbr_filter(struct
static void __intel_pmu_lbr_enable(void)
{
u64 debugctl;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->lbr_sel)
wrmsrl(MSR_LBR_SELECT, cpuc->lbr_sel->config);
@@ -183,7 +183,7 @@ void intel_pmu_lbr_reset(void)
void intel_pmu_lbr_enable(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!x86_pmu.lbr_nr)
return;
@@ -203,7 +203,7 @@ void intel_pmu_lbr_enable(struct perf_ev
void intel_pmu_lbr_disable(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!x86_pmu.lbr_nr)
return;
@@ -220,7 +220,7 @@ void intel_pmu_lbr_disable(struct perf_e
void intel_pmu_lbr_enable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->lbr_users)
__intel_pmu_lbr_enable();
@@ -228,7 +228,7 @@ void intel_pmu_lbr_enable_all(void)
void intel_pmu_lbr_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->lbr_users)
__intel_pmu_lbr_disable();
@@ -319,7 +319,7 @@ static void intel_pmu_lbr_read_64(struct
void intel_pmu_lbr_read(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!cpuc->lbr_users)
return;
Index: linux/arch/x86/kernel/cpu/perf_event_knc.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_knc.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_knc.c 2013-08-22 15:01:56.719952513 -0500
@@ -217,7 +217,7 @@ static int knc_pmu_handle_irq(struct pt_
int bit, loops;
u64 status;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
knc_pmu_disable_all();
Index: linux/arch/x86/kernel/cpu/perf_event_p4.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_p4.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_p4.c 2013-08-22 15:01:56.719952513 -0500
@@ -915,7 +915,7 @@ static inline void p4_pmu_disable_event(
static void p4_pmu_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
@@ -984,7 +984,7 @@ static void p4_pmu_enable_event(struct p
static void p4_pmu_enable_all(int added)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
@@ -1004,7 +1004,7 @@ static int p4_pmu_handle_irq(struct pt_r
int idx, handled = 0;
u64 val;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
int overflow;
Index: linux/arch/x86/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/x86/kernel/hw_breakpoint.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/hw_breakpoint.c 2013-08-22 15:01:56.719952513 -0500
@@ -110,7 +110,7 @@ int arch_install_hw_breakpoint(struct pe
int i;
for (i = 0; i < HBP_NUM; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (!*slot) {
*slot = bp;
@@ -124,7 +124,7 @@ int arch_install_hw_breakpoint(struct pe
set_debugreg(info->address, i);
__this_cpu_write(cpu_debugreg[i], info->address);
- dr7 = &__get_cpu_var(cpu_dr7);
+ dr7 = this_cpu_ptr(&cpu_dr7);
*dr7 |= encode_dr7(i, info->len, info->type);
set_debugreg(*dr7, 7);
@@ -148,7 +148,7 @@ void arch_uninstall_hw_breakpoint(struct
int i;
for (i = 0; i < HBP_NUM; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (*slot == bp) {
*slot = NULL;
@@ -159,7 +159,7 @@ void arch_uninstall_hw_breakpoint(struct
if (WARN_ONCE(i == HBP_NUM, "Can't find any breakpoint slot"))
return;
- dr7 = &__get_cpu_var(cpu_dr7);
+ dr7 = this_cpu_ptr(&cpu_dr7);
*dr7 &= ~__encode_dr7(i, info->len, info->type);
set_debugreg(*dr7, 7);
Index: linux/arch/x86/kvm/vmx.c
===================================================================
--- linux.orig/arch/x86/kvm/vmx.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kvm/vmx.c 2013-08-22 15:01:56.719952513 -0500
@@ -1513,7 +1513,7 @@ static void reload_tss(void)
/*
* VT restores TR but not its size. Useless.
*/
- struct desc_ptr *gdt = &__get_cpu_var(host_gdt);
+ struct desc_ptr *gdt = this_cpu_ptr(&host_gdt);
struct desc_struct *descs;
descs = (void *)gdt->address;
@@ -1559,7 +1559,7 @@ static bool update_transition_efer(struc
static unsigned long segment_base(u16 selector)
{
- struct desc_ptr *gdt = &__get_cpu_var(host_gdt);
+ struct desc_ptr *gdt = this_cpu_ptr(&host_gdt);
struct desc_struct *d;
unsigned long table_base;
unsigned long v;
@@ -1685,7 +1685,7 @@ static void __vmx_load_host_state(struct
*/
if (!user_has_fpu() && !vmx->vcpu.guest_fpu_loaded)
stts();
- load_gdt(&__get_cpu_var(host_gdt));
+ load_gdt(this_cpu_ptr(&host_gdt));
}
static void vmx_load_host_state(struct vcpu_vmx *vmx)
@@ -1715,7 +1715,7 @@ static void vmx_vcpu_load(struct kvm_vcp
}
if (vmx->loaded_vmcs->cpu != cpu) {
- struct desc_ptr *gdt = &__get_cpu_var(host_gdt);
+ struct desc_ptr *gdt = this_cpu_ptr(&host_gdt);
unsigned long sysenter_esp;
kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
@@ -2611,7 +2611,7 @@ static int hardware_enable(void *garbage
ept_sync_global();
}
- native_store_gdt(&__get_cpu_var(host_gdt));
+ native_store_gdt(this_cpu_ptr(&host_gdt));
return 0;
}
Index: linux/arch/x86/mm/kmemcheck/kmemcheck.c
===================================================================
--- linux.orig/arch/x86/mm/kmemcheck/kmemcheck.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/mm/kmemcheck/kmemcheck.c 2013-08-22 15:01:56.719952513 -0500
@@ -134,7 +134,7 @@ static DEFINE_PER_CPU(struct kmemcheck_c
bool kmemcheck_active(struct pt_regs *regs)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
return data->balance > 0;
}
@@ -142,7 +142,7 @@ bool kmemcheck_active(struct pt_regs *re
/* Save an address that needs to be shown/hidden */
static void kmemcheck_save_addr(unsigned long addr)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
BUG_ON(data->n_addrs >= ARRAY_SIZE(data->addr));
data->addr[data->n_addrs++] = addr;
@@ -150,7 +150,7 @@ static void kmemcheck_save_addr(unsigned
static unsigned int kmemcheck_show_all(void)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
unsigned int i;
unsigned int n;
@@ -163,7 +163,7 @@ static unsigned int kmemcheck_show_all(v
static unsigned int kmemcheck_hide_all(void)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
unsigned int i;
unsigned int n;
@@ -179,7 +179,7 @@ static unsigned int kmemcheck_hide_all(v
*/
void kmemcheck_show(struct pt_regs *regs)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
BUG_ON(!irqs_disabled());
@@ -220,7 +220,7 @@ void kmemcheck_show(struct pt_regs *regs
*/
void kmemcheck_hide(struct pt_regs *regs)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
int n;
BUG_ON(!irqs_disabled());
@@ -522,7 +522,7 @@ static void kmemcheck_access(struct pt_r
const uint8_t *insn_primary;
unsigned int size;
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
/* Recursive fault -- ouch. */
if (data->busy) {
Index: linux/arch/x86/oprofile/nmi_int.c
===================================================================
--- linux.orig/arch/x86/oprofile/nmi_int.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/oprofile/nmi_int.c 2013-08-22 15:01:56.719952513 -0500
@@ -64,11 +64,11 @@ u64 op_x86_get_ctrl(struct op_x86_model_
static int profile_exceptions_notify(unsigned int val, struct pt_regs *regs)
{
if (ctr_running)
- model->check_ctrs(regs, &__get_cpu_var(cpu_msrs));
+ model->check_ctrs(regs, this_cpu_ptr(&cpu_msrs));
else if (!nmi_enabled)
return NMI_DONE;
else
- model->stop(&__get_cpu_var(cpu_msrs));
+ model->stop(this_cpu_ptr(&cpu_msrs));
return NMI_HANDLED;
}
@@ -91,7 +91,7 @@ static void nmi_cpu_save_registers(struc
static void nmi_cpu_start(void *dummy)
{
- struct op_msrs const *msrs = &__get_cpu_var(cpu_msrs);
+ struct op_msrs const *msrs = this_cpu_ptr(&cpu_msrs);
if (!msrs->controls)
WARN_ON_ONCE(1);
else
@@ -111,7 +111,7 @@ static int nmi_start(void)
static void nmi_cpu_stop(void *dummy)
{
- struct op_msrs const *msrs = &__get_cpu_var(cpu_msrs);
+ struct op_msrs const *msrs = this_cpu_ptr(&cpu_msrs);
if (!msrs->controls)
WARN_ON_ONCE(1);
else
Index: linux/arch/x86/platform/uv/uv_time.c
===================================================================
--- linux.orig/arch/x86/platform/uv/uv_time.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/platform/uv/uv_time.c 2013-08-22 15:01:56.719952513 -0500
@@ -365,7 +365,7 @@ __setup("uvrtcevt", uv_enable_evt_rtc);
static __init void uv_rtc_register_clockevents(struct work_struct *dummy)
{
- struct clock_event_device *ced = &__get_cpu_var(cpu_ced);
+ struct clock_event_device *ced = this_cpu_ptr(&cpu_ced);
*ced = clock_event_device_uv;
ced->cpumask = cpumask_of(smp_processor_id());
Index: linux/arch/x86/xen/multicalls.c
===================================================================
--- linux.orig/arch/x86/xen/multicalls.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/xen/multicalls.c 2013-08-22 15:01:56.719952513 -0500
@@ -54,7 +54,7 @@ DEFINE_PER_CPU(unsigned long, xen_mc_irq
void xen_mc_flush(void)
{
- struct mc_buffer *b = &__get_cpu_var(mc_buffer);
+ struct mc_buffer *b = this_cpu_ptr(&mc_buffer);
struct multicall_entry *mc;
int ret = 0;
unsigned long flags;
@@ -131,7 +131,7 @@ void xen_mc_flush(void)
struct multicall_space __xen_mc_entry(size_t args)
{
- struct mc_buffer *b = &__get_cpu_var(mc_buffer);
+ struct mc_buffer *b = this_cpu_ptr(&mc_buffer);
struct multicall_space ret;
unsigned argidx = roundup(b->argidx, sizeof(u64));
@@ -162,7 +162,7 @@ struct multicall_space __xen_mc_entry(si
struct multicall_space xen_mc_extend_args(unsigned long op, size_t size)
{
- struct mc_buffer *b = &__get_cpu_var(mc_buffer);
+ struct mc_buffer *b = this_cpu_ptr(&mc_buffer);
struct multicall_space ret = { NULL, NULL };
BUG_ON(preemptible());
@@ -192,7 +192,7 @@ out:
void xen_mc_callback(void (*fn)(void *), void *data)
{
- struct mc_buffer *b = &__get_cpu_var(mc_buffer);
+ struct mc_buffer *b = this_cpu_ptr(&mc_buffer);
struct callback *cb;
if (b->cbidx == MC_BATCH) {
Index: linux/arch/x86/xen/time.c
===================================================================
--- linux.orig/arch/x86/xen/time.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/xen/time.c 2013-08-22 15:01:56.719952513 -0500
@@ -80,7 +80,7 @@ static void get_runstate_snapshot(struct
BUG_ON(preemptible());
- state = &__get_cpu_var(xen_runstate);
+ state = this_cpu_ptr(&xen_runstate);
/*
* The runstate info is always updated by the hypervisor on
@@ -123,7 +123,7 @@ static void do_stolen_accounting(void)
WARN_ON(state.state != RUNSTATE_running);
- snap = &__get_cpu_var(xen_runstate_snapshot);
+ snap = this_cpu_ptr(&xen_runstate_snapshot);
/* work out how much time the VCPU has not been runn*ing* */
runnable = state.time[RUNSTATE_runnable] - snap->time[RUNSTATE_runnable];
@@ -158,7 +158,7 @@ cycle_t xen_clocksource_read(void)
cycle_t ret;
preempt_disable_notrace();
- src = &__get_cpu_var(xen_vcpu)->time;
+ src = this_cpu_ptr(&xen_vcpu->time);
ret = pvclock_clocksource_read(src);
preempt_enable_notrace();
return ret;
@@ -397,7 +397,7 @@ static DEFINE_PER_CPU(struct xen_clock_e
static irqreturn_t xen_timer_interrupt(int irq, void *dev_id)
{
- struct clock_event_device *evt = &__get_cpu_var(xen_clock_events).evt;
+ struct clock_event_device *evt = this_cpu_ptr(&xen_clock_events.evt);
irqreturn_t ret;
ret = IRQ_NONE;
@@ -460,7 +460,7 @@ void xen_setup_cpu_clockevents(void)
{
BUG_ON(preemptible());
- clockevents_register_device(&__get_cpu_var(xen_clock_events).evt);
+ clockevents_register_device(this_cpu_ptr(&xen_clock_events.evt));
}
void xen_timer_resume(void)
Index: linux/arch/x86/include/asm/uv/uv_hub.h
===================================================================
--- linux.orig/arch/x86/include/asm/uv/uv_hub.h 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/include/asm/uv/uv_hub.h 2013-08-22 15:01:56.719952513 -0500
@@ -164,7 +164,7 @@ struct uv_hub_info_s {
};
DECLARE_PER_CPU(struct uv_hub_info_s, __uv_hub_info);
-#define uv_hub_info (&__get_cpu_var(__uv_hub_info))
+#define uv_hub_info this_cpu_ptr(&__uv_hub_info)
#define uv_cpu_hub_info(cpu) (&per_cpu(__uv_hub_info, cpu))
/*
Index: linux/arch/x86/include/asm/perf_event_p4.h
===================================================================
--- linux.orig/arch/x86/include/asm/perf_event_p4.h 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/include/asm/perf_event_p4.h 2013-08-22 15:01:56.719952513 -0500
@@ -189,7 +189,7 @@ static inline int p4_ht_thread(int cpu)
{
#ifdef CONFIG_SMP
if (smp_num_siblings == 2)
- return cpu != cpumask_first(__this_cpu_ptr(&cpu_sibling_map[0]));
+ return cpu != cpumask_first(this_cpu_ptr(cpu_sibling_map));
#endif
return 0;
}
Index: linux/arch/x86/kernel/apic/x2apic_cluster.c
===================================================================
--- linux.orig/arch/x86/kernel/apic/x2apic_cluster.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/apic/x2apic_cluster.c 2013-08-22 15:01:56.719952513 -0500
@@ -43,7 +43,7 @@ __x2apic_send_IPI_mask(const struct cpum
* We are to modify mask, so we need an own copy
* and be sure it's manipulated with irq off.
*/
- ipi_mask_ptr = __raw_get_cpu_var(ipi_mask);
+ ipi_mask_ptr = this_cpu_ptr(ipi_mask);
cpumask_copy(ipi_mask_ptr, mask);
/*
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 17/31] mips: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Ralf Baechle, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/mips/include/asm/fpu_emulator.h
===================================================================
--- linux.orig/arch/mips/include/asm/fpu_emulator.h 2013-08-22 14:18:23.138568266 -0500
+++ linux/arch/mips/include/asm/fpu_emulator.h 2013-08-22 14:18:23.130568348 -0500
@@ -43,7 +43,7 @@ DECLARE_PER_CPU(struct mips_fpu_emulator
#define MIPS_FPU_EMU_INC_STATS(M) \
do { \
preempt_disable(); \
- __local_inc(&__get_cpu_var(fpuemustats).M); \
+ __this_cpu_inc(fpuemustats.M); \
preempt_enable(); \
} while (0)
Index: linux/arch/mips/cavium-octeon/octeon-irq.c
===================================================================
--- linux.orig/arch/mips/cavium-octeon/octeon-irq.c 2013-08-22 14:17:32.000000000 -0500
+++ linux/arch/mips/cavium-octeon/octeon-irq.c 2013-08-22 14:18:54.210250887 -0500
@@ -264,13 +264,13 @@ static void octeon_irq_ciu_enable_local(
unsigned long *pen;
unsigned long flags;
union octeon_ciu_chip_data cd;
- raw_spinlock_t *lock = &__get_cpu_var(octeon_irq_ciu_spinlock);
+ raw_spinlock_t *lock = this_cpu_ptr(&octeon_irq_ciu_spinlock);
cd.p = irq_data_get_irq_chip_data(data);
raw_spin_lock_irqsave(lock, flags);
if (cd.s.line == 0) {
- pen = &__get_cpu_var(octeon_irq_ciu0_en_mirror);
+ pen = this_cpu_ptr(&octeon_irq_ciu0_en_mirror);
__set_bit(cd.s.bit, pen);
/*
* Must be visible to octeon_irq_ip{2,3}_ciu() before
@@ -279,7 +279,7 @@ static void octeon_irq_ciu_enable_local(
wmb();
cvmx_write_csr(CVMX_CIU_INTX_EN0(cvmx_get_core_num() * 2), *pen);
} else {
- pen = &__get_cpu_var(octeon_irq_ciu1_en_mirror);
+ pen = this_cpu_ptr(&octeon_irq_ciu1_en_mirror);
__set_bit(cd.s.bit, pen);
/*
* Must be visible to octeon_irq_ip{2,3}_ciu() before
@@ -296,13 +296,13 @@ static void octeon_irq_ciu_disable_local
unsigned long *pen;
unsigned long flags;
union octeon_ciu_chip_data cd;
- raw_spinlock_t *lock = &__get_cpu_var(octeon_irq_ciu_spinlock);
+ raw_spinlock_t *lock = this_cpu_ptr(&octeon_irq_ciu_spinlock);
cd.p = irq_data_get_irq_chip_data(data);
raw_spin_lock_irqsave(lock, flags);
if (cd.s.line == 0) {
- pen = &__get_cpu_var(octeon_irq_ciu0_en_mirror);
+ pen = this_cpu_ptr(&octeon_irq_ciu0_en_mirror);
__clear_bit(cd.s.bit, pen);
/*
* Must be visible to octeon_irq_ip{2,3}_ciu() before
@@ -311,7 +311,7 @@ static void octeon_irq_ciu_disable_local
wmb();
cvmx_write_csr(CVMX_CIU_INTX_EN0(cvmx_get_core_num() * 2), *pen);
} else {
- pen = &__get_cpu_var(octeon_irq_ciu1_en_mirror);
+ pen = this_cpu_ptr(&octeon_irq_ciu1_en_mirror);
__clear_bit(cd.s.bit, pen);
/*
* Must be visible to octeon_irq_ip{2,3}_ciu() before
@@ -431,11 +431,11 @@ static void octeon_irq_ciu_enable_local_
if (cd.s.line == 0) {
int index = cvmx_get_core_num() * 2;
- set_bit(cd.s.bit, &__get_cpu_var(octeon_irq_ciu0_en_mirror));
+ set_bit(cd.s.bit, this_cpu_ptr(&octeon_irq_ciu0_en_mirror));
cvmx_write_csr(CVMX_CIU_INTX_EN0_W1S(index), mask);
} else {
int index = cvmx_get_core_num() * 2 + 1;
- set_bit(cd.s.bit, &__get_cpu_var(octeon_irq_ciu1_en_mirror));
+ set_bit(cd.s.bit, this_cpu_ptr(&octeon_irq_ciu1_en_mirror));
cvmx_write_csr(CVMX_CIU_INTX_EN1_W1S(index), mask);
}
}
@@ -450,11 +450,11 @@ static void octeon_irq_ciu_disable_local
if (cd.s.line == 0) {
int index = cvmx_get_core_num() * 2;
- clear_bit(cd.s.bit, &__get_cpu_var(octeon_irq_ciu0_en_mirror));
+ clear_bit(cd.s.bit, this_cpu_ptr(&octeon_irq_ciu0_en_mirror));
cvmx_write_csr(CVMX_CIU_INTX_EN0_W1C(index), mask);
} else {
int index = cvmx_get_core_num() * 2 + 1;
- clear_bit(cd.s.bit, &__get_cpu_var(octeon_irq_ciu1_en_mirror));
+ clear_bit(cd.s.bit, this_cpu_ptr(&octeon_irq_ciu1_en_mirror));
cvmx_write_csr(CVMX_CIU_INTX_EN1_W1C(index), mask);
}
}
@@ -1063,7 +1063,7 @@ static void octeon_irq_ip2_ciu(void)
const unsigned long core_id = cvmx_get_core_num();
u64 ciu_sum = cvmx_read_csr(CVMX_CIU_INTX_SUM0(core_id * 2));
- ciu_sum &= __get_cpu_var(octeon_irq_ciu0_en_mirror);
+ ciu_sum &= __this_cpu_read(octeon_irq_ciu0_en_mirror);
if (likely(ciu_sum)) {
int bit = fls64(ciu_sum) - 1;
int irq = octeon_irq_ciu_to_irq[0][bit];
@@ -1080,7 +1080,7 @@ static void octeon_irq_ip3_ciu(void)
{
u64 ciu_sum = cvmx_read_csr(CVMX_CIU_INT_SUM1);
- ciu_sum &= __get_cpu_var(octeon_irq_ciu1_en_mirror);
+ ciu_sum &= __this_cpu_read(octeon_irq_ciu1_en_mirror);
if (likely(ciu_sum)) {
int bit = fls64(ciu_sum) - 1;
int irq = octeon_irq_ciu_to_irq[1][bit];
@@ -1129,10 +1129,10 @@ static void octeon_irq_init_ciu_percpu(v
int coreid = cvmx_get_core_num();
- __get_cpu_var(octeon_irq_ciu0_en_mirror) = 0;
- __get_cpu_var(octeon_irq_ciu1_en_mirror) = 0;
+ __this_cpu_write(octeon_irq_ciu0_en_mirror, 0);
+ __this_cpu_write(octeon_irq_ciu1_en_mirror, 0);
wmb();
- raw_spin_lock_init(&__get_cpu_var(octeon_irq_ciu_spinlock));
+ raw_spin_lock_init(this_cpu_ptr(&octeon_irq_ciu_spinlock));
/*
* Disable All CIU Interrupts. The ones we need will be
* enabled later. Read the SUM register so we know the write
Index: linux/arch/mips/kernel/kprobes.c
===================================================================
--- linux.orig/arch/mips/kernel/kprobes.c 2013-08-22 14:17:32.000000000 -0500
+++ linux/arch/mips/kernel/kprobes.c 2013-08-22 14:18:54.210250887 -0500
@@ -224,7 +224,7 @@ static void save_previous_kprobe(struct
static void restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
kcb->kprobe_old_SR = kcb->prev_kprobe.old_SR;
kcb->kprobe_saved_SR = kcb->prev_kprobe.saved_SR;
@@ -234,7 +234,7 @@ static void restore_previous_kprobe(stru
static void set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
kcb->kprobe_saved_SR = kcb->kprobe_old_SR = (regs->cp0_status & ST0_IE);
kcb->kprobe_saved_epc = regs->cp0_epc;
}
@@ -385,7 +385,7 @@ static int __kprobes kprobe_handler(stru
ret = 1;
goto no_kprobe;
}
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs))
goto ss_probe;
}
Index: linux/arch/mips/kernel/perf_event_mipsxx.c
===================================================================
--- linux.orig/arch/mips/kernel/perf_event_mipsxx.c 2013-08-22 14:17:32.000000000 -0500
+++ linux/arch/mips/kernel/perf_event_mipsxx.c 2013-08-22 14:18:54.210250887 -0500
@@ -340,7 +340,7 @@ static int mipsxx_pmu_alloc_counter(stru
static void mipsxx_pmu_enable_event(struct hw_perf_event *evt, int idx)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
WARN_ON(idx < 0 || idx >= mipspmu.num_counters);
@@ -360,7 +360,7 @@ static void mipsxx_pmu_enable_event(stru
static void mipsxx_pmu_disable_event(int idx)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
unsigned long flags;
WARN_ON(idx < 0 || idx >= mipspmu.num_counters);
@@ -460,7 +460,7 @@ static void mipspmu_stop(struct perf_eve
static int mipspmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx;
int err = 0;
@@ -496,7 +496,7 @@ out:
static void mipspmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -1270,7 +1270,7 @@ static int __hw_perf_event_init(struct p
static void pause_local_counters(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int ctr = mipspmu.num_counters;
unsigned long flags;
@@ -1286,7 +1286,7 @@ static void pause_local_counters(void)
static void resume_local_counters(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int ctr = mipspmu.num_counters;
do {
@@ -1297,7 +1297,7 @@ static void resume_local_counters(void)
static int mipsxx_pmu_handle_shared_irq(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_sample_data data;
unsigned int counters = mipspmu.num_counters;
u64 counter;
Index: linux/arch/mips/kernel/smp-bmips.c
===================================================================
--- linux.orig/arch/mips/kernel/smp-bmips.c 2013-08-22 14:17:32.000000000 -0500
+++ linux/arch/mips/kernel/smp-bmips.c 2013-08-22 14:18:54.214250846 -0500
@@ -304,7 +304,7 @@ static irqreturn_t bmips_ipi_interrupt(i
int action, cpu = irq - IPI0_IRQ;
spin_lock_irqsave(&ipi_lock, flags);
- action = __get_cpu_var(ipi_action_mask);
+ action = __this_cpu_read(ipi_action_mask);
per_cpu(ipi_action_mask, cpu) = 0;
clear_c0_cause(cpu ? C_SW1 : C_SW0);
spin_unlock_irqrestore(&ipi_lock, flags);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 18/31] s390: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Martin Schwidefsky, Heiko Carstens, linux390, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/s390/include/asm/irq.h
===================================================================
--- linux.orig/arch/s390/include/asm/irq.h 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/include/asm/irq.h 2013-08-22 14:49:17.327583548 -0500
@@ -55,7 +55,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct ir
static __always_inline void inc_irq_stat(enum interruption_class irq)
{
- __get_cpu_var(irq_stat).irqs[irq]++;
+ __this_cpu_inc(irq_stat.irqs[irq]);
}
struct ext_code {
Index: linux/arch/s390/include/asm/cputime.h
===================================================================
--- linux.orig/arch/s390/include/asm/cputime.h 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/include/asm/cputime.h 2013-08-22 14:49:17.327583548 -0500
@@ -187,7 +187,7 @@ cputime64_t s390_get_idle_time(int cpu);
static inline int s390_nohz_delay(int cpu)
{
- return __get_cpu_var(s390_idle).nohz_delay != 0;
+ return __this_cpu_read(s390_idle.nohz_delay) != 0;
}
#define arch_needs_cpu(cpu) s390_nohz_delay(cpu)
Index: linux/arch/s390/kernel/kprobes.c
===================================================================
--- linux.orig/arch/s390/kernel/kprobes.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/kprobes.c 2013-08-22 14:49:17.327583548 -0500
@@ -212,9 +212,9 @@ static void __kprobes disable_singlestep
*/
static void __kprobes push_kprobe(struct kprobe_ctlblk *kcb, struct kprobe *p)
{
- kcb->prev_kprobe.kp = __get_cpu_var(current_kprobe);
+ kcb->prev_kprobe.kp = __this_cpu_read(current_kprobe);
kcb->prev_kprobe.status = kcb->kprobe_status;
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
/*
@@ -224,7 +224,7 @@ static void __kprobes push_kprobe(struct
*/
static void __kprobes pop_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
@@ -305,7 +305,7 @@ static int __kprobes kprobe_handler(stru
enable_singlestep(kcb, regs, (unsigned long) p->ainsn.insn);
return 1;
} else if (kprobe_running()) {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
/*
* Continuation after the jprobe completed and
Index: linux/arch/s390/kernel/nmi.c
===================================================================
--- linux.orig/arch/s390/kernel/nmi.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/nmi.c 2013-08-22 14:49:17.327583548 -0500
@@ -53,8 +53,8 @@ void s390_handle_mcck(void)
*/
local_irq_save(flags);
local_mcck_disable();
- mcck = __get_cpu_var(cpu_mcck);
- memset(&__get_cpu_var(cpu_mcck), 0, sizeof(struct mcck_struct));
+ mcck = __this_cpu_read(cpu_mcck);
+ memset(this_cpu_ptr(&cpu_mcck), 0, sizeof(struct mcck_struct));
clear_thread_flag(TIF_MCCK_PENDING);
local_mcck_enable();
local_irq_restore(flags);
@@ -256,7 +256,7 @@ void notrace s390_do_machine_check(struc
nmi_enter();
inc_irq_stat(NMI_NMI);
mci = (struct mci *) &S390_lowcore.mcck_interruption_code;
- mcck = &__get_cpu_var(cpu_mcck);
+ mcck = this_cpu_ptr(&cpu_mcck);
umode = user_mode(regs);
if (mci->sd) {
Index: linux/arch/s390/kernel/perf_cpum_cf.c
===================================================================
--- linux.orig/arch/s390/kernel/perf_cpum_cf.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/perf_cpum_cf.c 2013-08-22 14:49:17.331583508 -0500
@@ -173,7 +173,7 @@ static int validate_ctr_auth(const struc
*/
static void cpumf_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
int err;
if (cpuhw->flags & PMU_F_ENABLED)
@@ -196,7 +196,7 @@ static void cpumf_pmu_enable(struct pmu
*/
static void cpumf_pmu_disable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
int err;
u64 inactive;
@@ -230,7 +230,7 @@ static void cpumf_measurement_alert(stru
return;
inc_irq_stat(IRQEXT_CMC);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
/* Measurement alerts are shared and might happen when the PMU
* is not reserved. Ignore these alerts in this case. */
@@ -250,7 +250,7 @@ static void cpumf_measurement_alert(stru
#define PMC_RELEASE 1
static void setup_pmc_cpu(void *flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
switch (*((int *) flags)) {
case PMC_INIT:
@@ -481,7 +481,7 @@ static void cpumf_pmu_read(struct perf_e
static void cpumf_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
@@ -512,7 +512,7 @@ static void cpumf_pmu_start(struct perf_
static void cpumf_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
if (!(hwc->state & PERF_HES_STOPPED)) {
@@ -533,7 +533,7 @@ static void cpumf_pmu_stop(struct perf_e
static int cpumf_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
/* Check authorization for the counter set to which this
* counter belongs.
@@ -557,7 +557,7 @@ static int cpumf_pmu_add(struct perf_eve
static void cpumf_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
cpumf_pmu_stop(event, PERF_EF_UPDATE);
@@ -581,7 +581,7 @@ static void cpumf_pmu_del(struct perf_ev
*/
static void cpumf_pmu_start_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
perf_pmu_disable(pmu);
cpuhw->flags |= PERF_EVENT_TXN;
@@ -595,7 +595,7 @@ static void cpumf_pmu_start_txn(struct p
*/
static void cpumf_pmu_cancel_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
WARN_ON(cpuhw->tx_state != cpuhw->state);
@@ -610,7 +610,7 @@ static void cpumf_pmu_cancel_txn(struct
*/
static int cpumf_pmu_commit_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
u64 state;
/* check if the updated state can be scheduled */
Index: linux/arch/s390/kernel/processor.c
===================================================================
--- linux.orig/arch/s390/kernel/processor.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/processor.c 2013-08-22 14:49:17.331583508 -0500
@@ -23,8 +23,8 @@ static DEFINE_PER_CPU(struct cpuid, cpu_
*/
void cpu_init(void)
{
- struct s390_idle_data *idle = &__get_cpu_var(s390_idle);
- struct cpuid *id = &__get_cpu_var(cpu_id);
+ struct s390_idle_data *idle = this_cpu_ptr(&s390_idle);
+ struct cpuid *id = this_cpu_ptr(&cpu_id);
get_cpu_id(id);
atomic_inc(&init_mm.mm_count);
Index: linux/arch/s390/kernel/time.c
===================================================================
--- linux.orig/arch/s390/kernel/time.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/time.c 2013-08-22 14:49:17.331583508 -0500
@@ -93,7 +93,7 @@ void clock_comparator_work(void)
S390_lowcore.clock_comparator = -1ULL;
set_clock_comparator(S390_lowcore.clock_comparator);
- cd = &__get_cpu_var(comparators);
+ cd = this_cpu_ptr(&comparators);
cd->event_handler(cd);
}
@@ -363,7 +363,7 @@ EXPORT_SYMBOL(get_sync_clock);
*/
static void disable_sync_clock(void *dummy)
{
- atomic_t *sw_ptr = &__get_cpu_var(clock_sync_word);
+ atomic_t *sw_ptr = this_cpu_ptr(&clock_sync_word);
/*
* Clear the in-sync bit 2^31. All get_sync_clock calls will
* fail until the sync bit is turned back on. In addition
@@ -380,7 +380,7 @@ static void disable_sync_clock(void *dum
*/
static void enable_sync_clock(void)
{
- atomic_t *sw_ptr = &__get_cpu_var(clock_sync_word);
+ atomic_t *sw_ptr = this_cpu_ptr(&clock_sync_word);
atomic_set_mask(0x80000000, sw_ptr);
}
Index: linux/arch/s390/kernel/vtime.c
===================================================================
--- linux.orig/arch/s390/kernel/vtime.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/vtime.c 2013-08-22 14:49:17.331583508 -0500
@@ -153,7 +153,7 @@ EXPORT_SYMBOL_GPL(vtime_account_system);
void __kprobes vtime_stop_cpu(void)
{
- struct s390_idle_data *idle = &__get_cpu_var(s390_idle);
+ struct s390_idle_data *idle = this_cpu_ptr(&s390_idle);
unsigned long long idle_time;
unsigned long psw_mask;
Index: linux/arch/s390/oprofile/hwsampler.c
===================================================================
--- linux.orig/arch/s390/oprofile/hwsampler.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/oprofile/hwsampler.c 2013-08-22 14:49:17.331583508 -0500
@@ -228,7 +228,7 @@ static inline unsigned long *trailer_ent
static void hws_ext_handler(struct ext_code ext_code,
unsigned int param32, unsigned long param64)
{
- struct hws_cpu_buffer *cb = &__get_cpu_var(sampler_cpu_buffer);
+ struct hws_cpu_buffer *cb = this_cpu_ptr(&sampler_cpu_buffer);
if (!(param32 & CPU_MF_INT_SF_MASK))
return;
Index: linux/arch/s390/pci/pci.c
===================================================================
--- linux.orig/arch/s390/pci/pci.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/pci/pci.c 2013-08-22 14:49:17.335583467 -0500
@@ -409,7 +409,7 @@ static DEFINE_PER_CPU(unsigned long, nex
static void zpci_irq_handler(struct airq_struct *airq)
{
- unsigned long sbit, mbit, last = 0, start = __get_cpu_var(next_sbit);
+ unsigned long sbit, mbit, last = 0, start = __this_cpu_read(next_sbit);
int rescan = 0, max = aisb_max;
struct zdev_irq_map *imap;
@@ -460,7 +460,7 @@ scan:
}
out:
/* store next device bit to scan */
- __get_cpu_var(next_sbit) = (++last >= aisb_max) ? 0 : last;
+ __this_cpu_write(next_sbit, (++last >= aisb_max) ? 0 : last);
}
/* msi_vecs - number of requested interrupts, 0 place function to error state */
Index: linux/arch/s390/kernel/irq.c
===================================================================
--- linux.orig/arch/s390/kernel/irq.c 2013-08-08 02:54:34.344984334 -0500
+++ linux/arch/s390/kernel/irq.c 2013-08-22 14:53:32.025028108 -0500
@@ -250,7 +250,7 @@ void __irq_entry do_extint(struct pt_reg
kstat_incr_irqs_this_cpu(EXTERNAL_INTERRUPT, NULL);
ext_code = *(struct ext_code *) ®s->int_code;
if (ext_code.code != 0x1004)
- __get_cpu_var(s390_idle).nohz_delay = 1;
+ __this_cpu_write(s390_idle.nohz_delay, 1);
index = ext_hash(ext_code.code);
rcu_read_lock();
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 15/31] zcache/zsmalloc: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Dan Magenheimer, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/drivers/staging/zcache/ramster/ramster.c
===================================================================
--- linux.orig/drivers/staging/zcache/ramster/ramster.c 2013-08-22 14:14:40.400841754 -0500
+++ linux/drivers/staging/zcache/ramster/ramster.c 2013-08-22 14:15:51.000000000 -0500
@@ -82,7 +82,7 @@ static struct flushlist_node *ramster_fl
struct flushlist_node *flnode = NULL;
struct ramster_preload *kp;
- kp = &__get_cpu_var(ramster_preloads);
+ kp = this_cpu_ptr(&ramster_preloads);
flnode = kp->flnode;
BUG_ON(flnode == NULL);
kp->flnode = NULL;
@@ -109,7 +109,7 @@ int ramster_do_preload_flnode(struct tme
BUG_ON(!irqs_disabled());
if (unlikely(ramster_flnode_cache == NULL))
BUG();
- kp = &__get_cpu_var(ramster_preloads);
+ kp = this_cpu_ptr(&ramster_preloads);
flnode = kmem_cache_alloc(ramster_flnode_cache, GFP_ATOMIC);
if (unlikely(flnode == NULL) && kp->flnode == NULL)
BUG(); /* FIXME handle more gracefully, but how??? */
@@ -478,8 +478,8 @@ int ramster_remotify_pageframe(bool eph)
struct tmem_handle th[2];
unsigned int zsize[2];
- tmpmem[0] = __get_cpu_var(ramster_remoteputmem1);
- tmpmem[1] = __get_cpu_var(ramster_remoteputmem2);
+ tmpmem[0] = __this_cpu_read(ramster_remoteputmem1);
+ tmpmem[1] = __this_cpu_read(ramster_remoteputmem2);
local_bh_disable();
zbuds = zbud_make_zombie_lru(&th[0], &tmpmem[0], &zsize[0], eph);
/* now OK to release lock set in caller */
Index: linux/drivers/staging/zcache/zcache-main.c
===================================================================
--- linux.orig/drivers/staging/zcache/zcache-main.c 2013-08-22 14:14:40.400841754 -0500
+++ linux/drivers/staging/zcache/zcache-main.c 2013-08-22 14:15:51.000000000 -0500
@@ -250,7 +250,7 @@ static struct tmem_objnode *zcache_objno
struct zcache_preload *kp;
int i;
- kp = &__get_cpu_var(zcache_preloads);
+ kp = this_cpu_ptr(&zcache_preloads);
for (i = 0; i < ARRAY_SIZE(kp->objnodes); i++) {
objnode = kp->objnodes[i];
if (objnode != NULL) {
@@ -275,7 +275,7 @@ static struct tmem_obj *zcache_obj_alloc
struct tmem_obj *obj = NULL;
struct zcache_preload *kp;
- kp = &__get_cpu_var(zcache_preloads);
+ kp = this_cpu_ptr(&zcache_preloads);
obj = kp->obj;
BUG_ON(obj == NULL);
kp->obj = NULL;
@@ -531,7 +531,7 @@ void *zcache_pampd_create(char *data, un
/* pre-allocate per-cpu metadata */
BUG_ON(zcache_objnode_cache == NULL);
BUG_ON(zcache_obj_cache == NULL);
- kp = &__get_cpu_var(zcache_preloads);
+ kp = this_cpu_ptr(&zcache_preloads);
for (i = 0; i < ARRAY_SIZE(kp->objnodes); i++) {
objnode = kp->objnodes[i];
if (objnode == NULL) {
@@ -761,7 +761,7 @@ static DEFINE_PER_CPU(unsigned char *, z
static void zcache_compress(struct page *from, void **out_va, unsigned *out_len)
{
int ret;
- unsigned char *dmem = __get_cpu_var(zcache_dstmem);
+ unsigned char *dmem = __this_cpu_read(zcache_dstmem);
char *from_va;
BUG_ON(!irqs_disabled());
Index: linux/drivers/staging/zsmalloc/zsmalloc-main.c
===================================================================
--- linux.orig/drivers/staging/zsmalloc/zsmalloc-main.c 2013-08-22 14:14:29.000000000 -0500
+++ linux/drivers/staging/zsmalloc/zsmalloc-main.c 2013-08-22 14:15:51.000000000 -0500
@@ -1028,7 +1028,7 @@ void zs_unmap_object(struct zs_pool *poo
class = &pool->size_class[class_idx];
off = obj_idx_to_offset(page, obj_idx, class->size);
- area = &__get_cpu_var(zs_map_area);
+ area = this_cpu_ptr(&zs_map_area);
if (off + class->size <= PAGE_SIZE)
kunmap_atomic(area->vm_addr);
else {
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 16/31] x86: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86,
linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/x86/kernel/apic/x2apic_uv_x.c
===================================================================
--- linux.orig/arch/x86/kernel/apic/x2apic_uv_x.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/apic/x2apic_uv_x.c 2013-08-22 15:01:56.715952554 -0500
@@ -874,10 +874,10 @@ int uv_handle_nmi(unsigned int reason, s
spin_unlock(&uv_blade_info[bid].nmi_lock);
}
- if (likely(__get_cpu_var(cpu_last_nmi_count) == uv_blade_info[bid].nmi_count))
+ if (likely(__this_cpu_read(cpu_last_nmi_count) == uv_blade_info[bid].nmi_count))
return NMI_DONE;
- __get_cpu_var(cpu_last_nmi_count) = uv_blade_info[bid].nmi_count;
+ __this_cpu_write(cpu_last_nmi_count, uv_blade_info[bid].nmi_count);
/*
* Use a lock so only one cpu prints at a time.
Index: linux/arch/x86/kernel/cpu/mcheck/mce_intel.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/mcheck/mce_intel.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/mcheck/mce_intel.c 2013-08-22 16:16:54.974785742 -0500
@@ -86,7 +86,7 @@ void mce_intel_cmci_poll(void)
{
if (__this_cpu_read(cmci_storm_state) == CMCI_STORM_NONE)
return;
- machine_check_poll(MCP_TIMESTAMP, &__get_cpu_var(mce_banks_owned));
+ machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned));
}
void mce_intel_hcpu_update(unsigned long cpu)
@@ -179,7 +179,7 @@ static void intel_threshold_interrupt(vo
{
if (cmci_storm_detect())
return;
- machine_check_poll(MCP_TIMESTAMP, &__get_cpu_var(mce_banks_owned));
+ machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned));
mce_notify_irq();
}
@@ -190,7 +190,7 @@ static void intel_threshold_interrupt(vo
*/
static void cmci_discover(int banks)
{
- unsigned long *owned = (void *)&__get_cpu_var(mce_banks_owned);
+ unsigned long *owned = (void *)this_cpu_ptr(&mce_banks_owned);
unsigned long flags;
int i;
int bios_wrong_thresh = 0;
@@ -208,7 +208,7 @@ static void cmci_discover(int banks)
/* Already owned by someone else? */
if (val & MCI_CTL2_CMCI_EN) {
clear_bit(i, owned);
- __clear_bit(i, __get_cpu_var(mce_poll_banks));
+ __clear_bit(i, this_cpu_ptr(mce_poll_banks));
continue;
}
@@ -232,7 +232,7 @@ static void cmci_discover(int banks)
/* Did the enable bit stick? -- the bank supports CMCI */
if (val & MCI_CTL2_CMCI_EN) {
set_bit(i, owned);
- __clear_bit(i, __get_cpu_var(mce_poll_banks));
+ __clear_bit(i, this_cpu_ptr(mce_poll_banks));
/*
* We are able to set thresholds for some banks that
* had a threshold of 0. This means the BIOS has not
@@ -243,7 +243,7 @@ static void cmci_discover(int banks)
(val & MCI_CTL2_CMCI_THRESHOLD_MASK))
bios_wrong_thresh = 1;
} else {
- WARN_ON(!test_bit(i, __get_cpu_var(mce_poll_banks)));
+ WARN_ON(!test_bit(i, this_cpu_ptr(mce_poll_banks)));
}
}
raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
@@ -267,7 +267,7 @@ void cmci_recheck(void)
if (!mce_available(__this_cpu_ptr(&cpu_info)) || !cmci_supported(&banks))
return;
local_irq_save(flags);
- machine_check_poll(MCP_TIMESTAMP, &__get_cpu_var(mce_banks_owned));
+ machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned));
local_irq_restore(flags);
}
@@ -286,13 +286,13 @@ void cmci_clear(void)
return;
raw_spin_lock_irqsave(&cmci_discover_lock, flags);
for (i = 0; i < banks; i++) {
- if (!test_bit(i, __get_cpu_var(mce_banks_owned)))
+ if (!test_bit(i, this_cpu_ptr(mce_banks_owned)))
continue;
/* Disable CMCI */
rdmsrl(MSR_IA32_MCx_CTL2(i), val);
val &= ~MCI_CTL2_CMCI_EN;
wrmsrl(MSR_IA32_MCx_CTL2(i), val);
- __clear_bit(i, __get_cpu_var(mce_banks_owned));
+ __clear_bit(i, this_cpu_ptr(mce_banks_owned));
}
raw_spin_unlock_irqrestore(&cmci_discover_lock, flags);
}
Index: linux/arch/x86/kernel/irq_64.c
===================================================================
--- linux.orig/arch/x86/kernel/irq_64.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/irq_64.c 2013-08-22 16:16:54.974785742 -0500
@@ -52,13 +52,13 @@ static inline void stack_overflow_check(
regs->sp <= curbase + THREAD_SIZE)
return;
- irq_stack_top = (u64)__get_cpu_var(irq_stack_union.irq_stack) +
+ irq_stack_top = (u64)this_cpu_ptr(irq_stack_union.irq_stack) +
STACK_TOP_MARGIN;
- irq_stack_bottom = (u64)__get_cpu_var(irq_stack_ptr);
+ irq_stack_bottom = (u64)__this_cpu_read(irq_stack_ptr);
if (regs->sp >= irq_stack_top && regs->sp <= irq_stack_bottom)
return;
- oist = &__get_cpu_var(orig_ist);
+ oist = this_cpu_ptr(&orig_ist);
estack_top = (u64)oist->ist[0] - EXCEPTION_STKSZ + STACK_TOP_MARGIN;
estack_bottom = (u64)oist->ist[N_EXCEPTION_STACKS - 1];
if (regs->sp >= estack_top && regs->sp <= estack_bottom)
Index: linux/arch/x86/kernel/kvm.c
===================================================================
--- linux.orig/arch/x86/kernel/kvm.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/kvm.c 2013-08-22 16:16:54.974785742 -0500
@@ -242,9 +242,9 @@ u32 kvm_read_and_reset_pf_reason(void)
{
u32 reason = 0;
- if (__get_cpu_var(apf_reason).enabled) {
- reason = __get_cpu_var(apf_reason).reason;
- __get_cpu_var(apf_reason).reason = 0;
+ if (__this_cpu_read(apf_reason.enabled)) {
+ reason = __this_cpu_read(apf_reason.reason);
+ __this_cpu_write(apf_reason.reason, 0);
}
return reason;
@@ -315,7 +315,7 @@ static void kvm_guest_apic_eoi_write(u32
* there's no need for lock or memory barriers.
* An optimization barrier is implied in apic write.
*/
- if (__test_and_clear_bit(KVM_PV_EOI_BIT, &__get_cpu_var(kvm_apic_eoi)))
+ if (__test_and_clear_bit(KVM_PV_EOI_BIT, this_cpu_ptr(&kvm_apic_eoi)))
return;
apic_write(APIC_EOI, APIC_EOI_ACK);
}
@@ -326,13 +326,13 @@ void kvm_guest_cpu_init(void)
return;
if (kvm_para_has_feature(KVM_FEATURE_ASYNC_PF) && kvmapf) {
- u64 pa = slow_virt_to_phys(&__get_cpu_var(apf_reason));
+ u64 pa = slow_virt_to_phys(this_cpu_ptr(&apf_reason));
#ifdef CONFIG_PREEMPT
pa |= KVM_ASYNC_PF_SEND_ALWAYS;
#endif
wrmsrl(MSR_KVM_ASYNC_PF_EN, pa | KVM_ASYNC_PF_ENABLED);
- __get_cpu_var(apf_reason).enabled = 1;
+ __this_cpu_write(apf_reason.enabled, 1);
printk(KERN_INFO"KVM setup async PF for cpu %d\n",
smp_processor_id());
}
@@ -341,8 +341,8 @@ void kvm_guest_cpu_init(void)
unsigned long pa;
/* Size alignment is implied but just to make it explicit. */
BUILD_BUG_ON(__alignof__(kvm_apic_eoi) < 4);
- __get_cpu_var(kvm_apic_eoi) = 0;
- pa = slow_virt_to_phys(&__get_cpu_var(kvm_apic_eoi))
+ __this_cpu_write(kvm_apic_eoi, 0);
+ pa = slow_virt_to_phys(this_cpu_ptr(&kvm_apic_eoi))
| KVM_MSR_ENABLED;
wrmsrl(MSR_KVM_PV_EOI_EN, pa);
}
@@ -353,11 +353,11 @@ void kvm_guest_cpu_init(void)
static void kvm_pv_disable_apf(void)
{
- if (!__get_cpu_var(apf_reason).enabled)
+ if (!__this_cpu_read(apf_reason.enabled))
return;
wrmsrl(MSR_KVM_ASYNC_PF_EN, 0);
- __get_cpu_var(apf_reason).enabled = 0;
+ __this_cpu_write(apf_reason.enabled, 0);
printk(KERN_INFO"Unregister pv shared memory for cpu %d\n",
smp_processor_id());
Index: linux/arch/x86/kvm/svm.c
===================================================================
--- linux.orig/arch/x86/kvm/svm.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kvm/svm.c 2013-08-22 16:16:54.974785742 -0500
@@ -654,7 +654,7 @@ static int svm_hardware_enable(void *gar
if (static_cpu_has(X86_FEATURE_TSCRATEMSR)) {
wrmsrl(MSR_AMD64_TSC_RATIO, TSC_RATIO_DEFAULT);
- __get_cpu_var(current_tsc_ratio) = TSC_RATIO_DEFAULT;
+ __this_cpu_write(current_tsc_ratio, TSC_RATIO_DEFAULT);
}
@@ -1312,8 +1312,8 @@ static void svm_vcpu_load(struct kvm_vcp
rdmsrl(host_save_user_msrs[i], svm->host_user_msrs[i]);
if (static_cpu_has(X86_FEATURE_TSCRATEMSR) &&
- svm->tsc_ratio != __get_cpu_var(current_tsc_ratio)) {
- __get_cpu_var(current_tsc_ratio) = svm->tsc_ratio;
+ svm->tsc_ratio != __this_cpu_read(current_tsc_ratio)) {
+ __this_cpu_write(current_tsc_ratio, svm->tsc_ratio);
wrmsrl(MSR_AMD64_TSC_RATIO, svm->tsc_ratio);
}
}
Index: linux/arch/x86/kvm/x86.c
===================================================================
--- linux.orig/arch/x86/kvm/x86.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kvm/x86.c 2013-08-22 15:01:56.715952554 -0500
@@ -1485,7 +1485,7 @@ static int kvm_guest_time_update(struct
/* Keep irq disabled to prevent changes to the clock */
local_irq_save(flags);
- this_tsc_khz = __get_cpu_var(cpu_tsc_khz);
+ this_tsc_khz = __this_cpu_read(cpu_tsc_khz);
if (unlikely(this_tsc_khz == 0)) {
local_irq_restore(flags);
kvm_make_request(KVM_REQ_CLOCK_UPDATE, v);
Index: linux/arch/x86/oprofile/op_model_p4.c
===================================================================
--- linux.orig/arch/x86/oprofile/op_model_p4.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/oprofile/op_model_p4.c 2013-08-22 15:01:56.715952554 -0500
@@ -372,7 +372,7 @@ static unsigned int get_stagger(void)
{
#ifdef CONFIG_SMP
int cpu = smp_processor_id();
- return cpu != cpumask_first(__get_cpu_var(cpu_sibling_map));
+ return cpu != cpumask_first(this_cpu_ptr(cpu_sibling_map));
#endif
return 0;
}
Index: linux/arch/x86/xen/enlighten.c
===================================================================
--- linux.orig/arch/x86/xen/enlighten.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/xen/enlighten.c 2013-08-22 16:16:54.974785742 -0500
@@ -817,7 +817,7 @@ static void xen_convert_trap_info(const
void xen_copy_trap_info(struct trap_info *traps)
{
- const struct desc_ptr *desc = &__get_cpu_var(idt_desc);
+ const struct desc_ptr *desc = this_cpu_ptr(&idt_desc);
xen_convert_trap_info(desc, traps);
}
@@ -834,7 +834,7 @@ static void xen_load_idt(const struct de
spin_lock(&lock);
- __get_cpu_var(idt_desc) = *desc;
+ memcpy(this_cpu_ptr(&idt_desc), desc, sizeof(idt_desc));
xen_convert_trap_info(desc, traps);
Index: linux/arch/x86/kernel/apb_timer.c
===================================================================
--- linux.orig/arch/x86/kernel/apb_timer.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/apb_timer.c 2013-08-22 15:01:56.715952554 -0500
@@ -146,7 +146,7 @@ static inline int is_apbt_capable(void)
static int __init apbt_clockevent_register(void)
{
struct sfi_timer_table_entry *mtmr;
- struct apbt_dev *adev = &__get_cpu_var(cpu_apbt_dev);
+ struct apbt_dev *adev = this_cpu_ptr(&cpu_apbt_dev);
mtmr = sfi_get_mtmr(APBT_CLOCKEVENT0_NUM);
if (mtmr == NULL) {
@@ -200,7 +200,7 @@ void apbt_setup_secondary_clock(void)
if (!cpu)
return;
- adev = &__get_cpu_var(cpu_apbt_dev);
+ adev = this_cpu_ptr(&cpu_apbt_dev);
if (!adev->timer) {
adev->timer = dw_apb_clockevent_init(cpu, adev->name,
APBT_CLOCKEVENT_RATING, adev_virt_addr(adev),
Index: linux/arch/x86/kernel/apic/apic.c
===================================================================
--- linux.orig/arch/x86/kernel/apic/apic.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/apic/apic.c 2013-08-22 15:01:56.715952554 -0500
@@ -546,7 +546,7 @@ static DEFINE_PER_CPU(struct clock_event
*/
static void setup_APIC_timer(void)
{
- struct clock_event_device *levt = &__get_cpu_var(lapic_events);
+ struct clock_event_device *levt = this_cpu_ptr(&lapic_events);
if (this_cpu_has(X86_FEATURE_ARAT)) {
lapic_clockevent.features &= ~CLOCK_EVT_FEAT_C3STOP;
@@ -681,7 +681,7 @@ calibrate_by_pmtimer(long deltapm, long
static int __init calibrate_APIC_clock(void)
{
- struct clock_event_device *levt = &__get_cpu_var(lapic_events);
+ struct clock_event_device *levt = this_cpu_ptr(&lapic_events);
void (*real_handler)(struct clock_event_device *dev);
unsigned long deltaj;
long delta, deltatsc;
Index: linux/arch/x86/kernel/cpu/mcheck/mce-inject.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/mcheck/mce-inject.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/mcheck/mce-inject.c 2013-08-22 15:01:56.715952554 -0500
@@ -83,7 +83,7 @@ static DEFINE_MUTEX(mce_inject_mutex);
static int mce_raise_notify(unsigned int cmd, struct pt_regs *regs)
{
int cpu = smp_processor_id();
- struct mce *m = &__get_cpu_var(injectm);
+ struct mce *m = this_cpu_ptr(&injectm);
if (!cpumask_test_cpu(cpu, mce_inject_cpumask))
return NMI_DONE;
cpumask_clear_cpu(cpu, mce_inject_cpumask);
@@ -97,7 +97,7 @@ static int mce_raise_notify(unsigned int
static void mce_irq_ipi(void *info)
{
int cpu = smp_processor_id();
- struct mce *m = &__get_cpu_var(injectm);
+ struct mce *m = this_cpu_ptr(&injectm);
if (cpumask_test_cpu(cpu, mce_inject_cpumask) &&
m->inject_flags & MCJ_EXCEPTION) {
@@ -109,7 +109,7 @@ static void mce_irq_ipi(void *info)
/* Inject mce on current CPU */
static int raise_local(void)
{
- struct mce *m = &__get_cpu_var(injectm);
+ struct mce *m = this_cpu_ptr(&injectm);
int context = MCJ_CTX(m->inject_flags);
int ret = 0;
int cpu = m->extcpu;
Index: linux/arch/x86/kernel/cpu/mcheck/mce.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/mcheck/mce.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/mcheck/mce.c 2013-08-22 15:01:56.719952513 -0500
@@ -390,7 +390,7 @@ static u64 mce_rdmsrl(u32 msr)
if (offset < 0)
return 0;
- return *(u64 *)((char *)&__get_cpu_var(injectm) + offset);
+ return *(u64 *)((char *)this_cpu_ptr(&injectm) + offset);
}
if (rdmsrl_safe(msr, &v)) {
@@ -412,7 +412,7 @@ static void mce_wrmsrl(u32 msr, u64 v)
int offset = msr_to_offset(msr);
if (offset >= 0)
- *(u64 *)((char *)&__get_cpu_var(injectm) + offset) = v;
+ *(u64 *)((char *)this_cpu_ptr(&injectm) + offset) = v;
return;
}
wrmsrl(msr, v);
@@ -468,7 +468,7 @@ static DEFINE_PER_CPU(struct mce_ring, m
/* Runs with CPU affinity in workqueue */
static int mce_ring_empty(void)
{
- struct mce_ring *r = &__get_cpu_var(mce_ring);
+ struct mce_ring *r = this_cpu_ptr(&mce_ring);
return r->start == r->end;
}
@@ -480,7 +480,7 @@ static int mce_ring_get(unsigned long *p
*pfn = 0;
get_cpu();
- r = &__get_cpu_var(mce_ring);
+ r = this_cpu_ptr(&mce_ring);
if (r->start == r->end)
goto out;
*pfn = r->ring[r->start];
@@ -494,7 +494,7 @@ out:
/* Always runs in MCE context with preempt off */
static int mce_ring_add(unsigned long pfn)
{
- struct mce_ring *r = &__get_cpu_var(mce_ring);
+ struct mce_ring *r = this_cpu_ptr(&mce_ring);
unsigned next;
next = (r->end + 1) % MCE_RING_SIZE;
@@ -516,7 +516,7 @@ int mce_available(struct cpuinfo_x86 *c)
static void mce_schedule_work(void)
{
if (!mce_ring_empty())
- schedule_work(&__get_cpu_var(mce_work));
+ schedule_work(this_cpu_ptr(&mce_work));
}
DEFINE_PER_CPU(struct irq_work, mce_irq_work);
@@ -541,7 +541,7 @@ static void mce_report_event(struct pt_r
return;
}
- irq_work_queue(&__get_cpu_var(mce_irq_work));
+ irq_work_queue(this_cpu_ptr(&mce_irq_work));
}
/*
@@ -1037,7 +1037,7 @@ void do_machine_check(struct pt_regs *re
mce_gather_info(&m, regs);
- final = &__get_cpu_var(mces_seen);
+ final = this_cpu_ptr(&mces_seen);
*final = m;
memset(valid_banks, 0, sizeof(valid_banks));
@@ -1271,14 +1271,14 @@ static unsigned long (*mce_adjust_timer)
static void mce_timer_fn(unsigned long data)
{
- struct timer_list *t = &__get_cpu_var(mce_timer);
+ struct timer_list *t = this_cpu_ptr(&mce_timer);
unsigned long iv;
WARN_ON(smp_processor_id() != data);
if (mce_available(__this_cpu_ptr(&cpu_info))) {
machine_check_poll(MCP_TIMESTAMP,
- &__get_cpu_var(mce_poll_banks));
+ this_cpu_ptr(&mce_poll_banks));
mce_intel_cmci_poll();
}
@@ -1306,7 +1306,7 @@ static void mce_timer_fn(unsigned long d
*/
void mce_timer_kick(unsigned long interval)
{
- struct timer_list *t = &__get_cpu_var(mce_timer);
+ struct timer_list *t = this_cpu_ptr(&mce_timer);
unsigned long when = jiffies + interval;
unsigned long iv = __this_cpu_read(mce_next_interval);
@@ -1642,7 +1642,7 @@ static void mce_start_timer(unsigned int
static void __mcheck_cpu_init_timer(void)
{
- struct timer_list *t = &__get_cpu_var(mce_timer);
+ struct timer_list *t = this_cpu_ptr(&mce_timer);
unsigned int cpu = smp_processor_id();
setup_timer(t, mce_timer_fn, cpu);
@@ -1685,8 +1685,8 @@ void mcheck_cpu_init(struct cpuinfo_x86
__mcheck_cpu_init_generic();
__mcheck_cpu_init_vendor(c);
__mcheck_cpu_init_timer();
- INIT_WORK(&__get_cpu_var(mce_work), mce_process_work);
- init_irq_work(&__get_cpu_var(mce_irq_work), &mce_irq_work_cb);
+ INIT_WORK(this_cpu_ptr(&mce_work), mce_process_work);
+ init_irq_work(this_cpu_ptr(&mce_irq_work), &mce_irq_work_cb);
}
/*
Index: linux/arch/x86/kernel/cpu/mcheck/mce_amd.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/mcheck/mce_amd.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/mcheck/mce_amd.c 2013-08-22 15:01:56.719952513 -0500
@@ -310,7 +310,7 @@ static void amd_threshold_interrupt(void
* event.
*/
machine_check_poll(MCP_TIMESTAMP,
- &__get_cpu_var(mce_poll_banks));
+ this_cpu_ptr(&mce_poll_banks));
if (high & MASK_OVERFLOW_HI) {
rdmsrl(address, m.misc);
Index: linux/arch/x86/kernel/cpu/perf_event.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event.c 2013-08-22 15:01:56.719952513 -0500
@@ -493,7 +493,7 @@ static int __x86_pmu_event_init(struct p
void x86_pmu_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
@@ -511,7 +511,7 @@ void x86_pmu_disable_all(void)
static void x86_pmu_disable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!x86_pmu_initialized())
return;
@@ -528,7 +528,7 @@ static void x86_pmu_disable(struct pmu *
void x86_pmu_enable_all(int added)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
@@ -874,7 +874,7 @@ static void x86_pmu_start(struct perf_ev
static void x86_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event;
struct hw_perf_event *hwc;
int i, added = cpuc->n_added;
@@ -1023,7 +1023,7 @@ void x86_pmu_enable_event(struct perf_ev
*/
static int x86_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc;
int assign[X86_PMC_IDX_MAX];
int n, n0, ret;
@@ -1070,7 +1070,7 @@ out:
static void x86_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = event->hw.idx;
if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED)))
@@ -1149,7 +1149,7 @@ void perf_event_print_debug(void)
void x86_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
if (__test_and_clear_bit(hwc->idx, cpuc->active_mask)) {
@@ -1171,7 +1171,7 @@ void x86_pmu_stop(struct perf_event *eve
static void x86_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
/*
@@ -1213,7 +1213,7 @@ int x86_pmu_handle_irq(struct pt_regs *r
int idx, handled = 0;
u64 val;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
/*
* Some chipsets need to unmask the LVTPC in a particular spot
@@ -1608,7 +1608,7 @@ static void x86_pmu_cancel_txn(struct pm
*/
static int x86_pmu_commit_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int assign[X86_PMC_IDX_MAX];
int n, ret;
Index: linux/arch/x86/kernel/cpu/perf_event_amd.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_amd.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_amd.c 2013-08-22 15:01:56.719952513 -0500
@@ -700,7 +700,7 @@ __init int amd_pmu_init(void)
void amd_pmu_enable_virt(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
cpuc->perf_ctr_virt_mask = 0;
@@ -712,7 +712,7 @@ EXPORT_SYMBOL_GPL(amd_pmu_enable_virt);
void amd_pmu_disable_virt(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
/*
* We only mask out the Host-only bit so that host-only counting works
Index: linux/arch/x86/kernel/cpu/perf_event_intel.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_intel.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_intel.c 2013-08-22 15:01:56.719952513 -0500
@@ -898,7 +898,7 @@ static inline bool intel_pmu_needs_lbr_s
static void intel_pmu_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL, 0);
@@ -911,7 +911,7 @@ static void intel_pmu_disable_all(void)
static void intel_pmu_enable_all(int added)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
intel_pmu_pebs_enable_all();
intel_pmu_lbr_enable_all();
@@ -945,7 +945,7 @@ static void intel_pmu_enable_all(int add
*/
static void intel_pmu_nhm_workaround(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
static const unsigned long nhm_magic[4] = {
0x4300B5,
0x4300D2,
@@ -1039,7 +1039,7 @@ static void intel_pmu_disable_fixed(stru
static void intel_pmu_disable_event(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (unlikely(hwc->idx == INTEL_PMC_IDX_FIXED_BTS)) {
intel_pmu_disable_bts();
@@ -1102,7 +1102,7 @@ static void intel_pmu_enable_fixed(struc
static void intel_pmu_enable_event(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (unlikely(hwc->idx == INTEL_PMC_IDX_FIXED_BTS)) {
if (!__this_cpu_read(cpu_hw_events.enabled))
@@ -1182,7 +1182,7 @@ static int intel_pmu_handle_irq(struct p
u64 status;
int handled;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
/*
* No known reason to not always do late ACK,
@@ -1600,7 +1600,7 @@ EXPORT_SYMBOL_GPL(perf_guest_get_msrs);
static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_guest_switch_msr *arr = cpuc->guest_switch_msrs;
arr[0].msr = MSR_CORE_PERF_GLOBAL_CTRL;
@@ -1621,7 +1621,7 @@ static struct perf_guest_switch_msr *int
static struct perf_guest_switch_msr *core_guest_get_msrs(int *nr)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_guest_switch_msr *arr = cpuc->guest_switch_msrs;
int idx;
@@ -1655,7 +1655,7 @@ static void core_pmu_enable_event(struct
static void core_pmu_enable_all(int added)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
Index: linux/arch/x86/kernel/cpu/perf_event_intel_ds.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_intel_ds.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_intel_ds.c 2013-08-22 15:01:56.719952513 -0500
@@ -424,7 +424,7 @@ void intel_pmu_enable_bts(u64 config)
void intel_pmu_disable_bts(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
unsigned long debugctlmsr;
if (!cpuc->ds)
@@ -441,7 +441,7 @@ void intel_pmu_disable_bts(void)
int intel_pmu_drain_bts_buffer(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct bts_record {
u64 from;
@@ -634,7 +634,7 @@ struct event_constraint *intel_pebs_cons
void intel_pmu_pebs_enable(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
hwc->config &= ~ARCH_PERFMON_EVENTSEL_INT;
@@ -649,7 +649,7 @@ void intel_pmu_pebs_enable(struct perf_e
void intel_pmu_pebs_disable(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
cpuc->pebs_enabled &= ~(1ULL << hwc->idx);
@@ -667,7 +667,7 @@ void intel_pmu_pebs_disable(struct perf_
void intel_pmu_pebs_enable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->pebs_enabled)
wrmsrl(MSR_IA32_PEBS_ENABLE, cpuc->pebs_enabled);
@@ -675,7 +675,7 @@ void intel_pmu_pebs_enable_all(void)
void intel_pmu_pebs_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->pebs_enabled)
wrmsrl(MSR_IA32_PEBS_ENABLE, 0);
@@ -683,7 +683,7 @@ void intel_pmu_pebs_disable_all(void)
static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
unsigned long from = cpuc->lbr_entries[0].from;
unsigned long old_to, to = cpuc->lbr_entries[0].to;
unsigned long ip = regs->ip;
@@ -766,7 +766,7 @@ static void __intel_pmu_pebs_event(struc
* We cast to pebs_record_nhm to get the load latency data
* if extra_reg MSR_PEBS_LD_LAT_THRESHOLD used
*/
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct pebs_record_nhm *pebs = __pebs;
struct pebs_record_hsw *pebs_hsw = __pebs;
struct perf_sample_data data;
@@ -847,7 +847,7 @@ static void __intel_pmu_pebs_event(struc
static void intel_pmu_drain_pebs_core(struct pt_regs *iregs)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct perf_event *event = cpuc->events[0]; /* PMC0 only */
struct pebs_record_core *at, *top;
@@ -889,7 +889,7 @@ static void intel_pmu_drain_pebs_core(st
static void __intel_pmu_drain_pebs_nhm(struct pt_regs *iregs, void *at,
void *top)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct perf_event *event = NULL;
u64 status = 0;
@@ -926,7 +926,7 @@ static void __intel_pmu_drain_pebs_nhm(s
static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct pebs_record_nhm *at, *top;
int n;
@@ -955,7 +955,7 @@ static void intel_pmu_drain_pebs_nhm(str
static void intel_pmu_drain_pebs_hsw(struct pt_regs *iregs)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct pebs_record_hsw *at, *top;
int n;
Index: linux/arch/x86/kernel/cpu/perf_event_intel_lbr.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_intel_lbr.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_intel_lbr.c 2013-08-22 15:01:56.719952513 -0500
@@ -133,7 +133,7 @@ static void intel_pmu_lbr_filter(struct
static void __intel_pmu_lbr_enable(void)
{
u64 debugctl;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->lbr_sel)
wrmsrl(MSR_LBR_SELECT, cpuc->lbr_sel->config);
@@ -183,7 +183,7 @@ void intel_pmu_lbr_reset(void)
void intel_pmu_lbr_enable(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!x86_pmu.lbr_nr)
return;
@@ -203,7 +203,7 @@ void intel_pmu_lbr_enable(struct perf_ev
void intel_pmu_lbr_disable(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!x86_pmu.lbr_nr)
return;
@@ -220,7 +220,7 @@ void intel_pmu_lbr_disable(struct perf_e
void intel_pmu_lbr_enable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->lbr_users)
__intel_pmu_lbr_enable();
@@ -228,7 +228,7 @@ void intel_pmu_lbr_enable_all(void)
void intel_pmu_lbr_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->lbr_users)
__intel_pmu_lbr_disable();
@@ -319,7 +319,7 @@ static void intel_pmu_lbr_read_64(struct
void intel_pmu_lbr_read(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!cpuc->lbr_users)
return;
Index: linux/arch/x86/kernel/cpu/perf_event_knc.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_knc.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_knc.c 2013-08-22 15:01:56.719952513 -0500
@@ -217,7 +217,7 @@ static int knc_pmu_handle_irq(struct pt_
int bit, loops;
u64 status;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
knc_pmu_disable_all();
Index: linux/arch/x86/kernel/cpu/perf_event_p4.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/perf_event_p4.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/cpu/perf_event_p4.c 2013-08-22 15:01:56.719952513 -0500
@@ -915,7 +915,7 @@ static inline void p4_pmu_disable_event(
static void p4_pmu_disable_all(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
@@ -984,7 +984,7 @@ static void p4_pmu_enable_event(struct p
static void p4_pmu_enable_all(int added)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
@@ -1004,7 +1004,7 @@ static int p4_pmu_handle_irq(struct pt_r
int idx, handled = 0;
u64 val;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
int overflow;
Index: linux/arch/x86/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/x86/kernel/hw_breakpoint.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/hw_breakpoint.c 2013-08-22 15:01:56.719952513 -0500
@@ -110,7 +110,7 @@ int arch_install_hw_breakpoint(struct pe
int i;
for (i = 0; i < HBP_NUM; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (!*slot) {
*slot = bp;
@@ -124,7 +124,7 @@ int arch_install_hw_breakpoint(struct pe
set_debugreg(info->address, i);
__this_cpu_write(cpu_debugreg[i], info->address);
- dr7 = &__get_cpu_var(cpu_dr7);
+ dr7 = this_cpu_ptr(&cpu_dr7);
*dr7 |= encode_dr7(i, info->len, info->type);
set_debugreg(*dr7, 7);
@@ -148,7 +148,7 @@ void arch_uninstall_hw_breakpoint(struct
int i;
for (i = 0; i < HBP_NUM; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (*slot == bp) {
*slot = NULL;
@@ -159,7 +159,7 @@ void arch_uninstall_hw_breakpoint(struct
if (WARN_ONCE(i == HBP_NUM, "Can't find any breakpoint slot"))
return;
- dr7 = &__get_cpu_var(cpu_dr7);
+ dr7 = this_cpu_ptr(&cpu_dr7);
*dr7 &= ~__encode_dr7(i, info->len, info->type);
set_debugreg(*dr7, 7);
Index: linux/arch/x86/kvm/vmx.c
===================================================================
--- linux.orig/arch/x86/kvm/vmx.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kvm/vmx.c 2013-08-22 15:01:56.719952513 -0500
@@ -1513,7 +1513,7 @@ static void reload_tss(void)
/*
* VT restores TR but not its size. Useless.
*/
- struct desc_ptr *gdt = &__get_cpu_var(host_gdt);
+ struct desc_ptr *gdt = this_cpu_ptr(&host_gdt);
struct desc_struct *descs;
descs = (void *)gdt->address;
@@ -1559,7 +1559,7 @@ static bool update_transition_efer(struc
static unsigned long segment_base(u16 selector)
{
- struct desc_ptr *gdt = &__get_cpu_var(host_gdt);
+ struct desc_ptr *gdt = this_cpu_ptr(&host_gdt);
struct desc_struct *d;
unsigned long table_base;
unsigned long v;
@@ -1685,7 +1685,7 @@ static void __vmx_load_host_state(struct
*/
if (!user_has_fpu() && !vmx->vcpu.guest_fpu_loaded)
stts();
- load_gdt(&__get_cpu_var(host_gdt));
+ load_gdt(this_cpu_ptr(&host_gdt));
}
static void vmx_load_host_state(struct vcpu_vmx *vmx)
@@ -1715,7 +1715,7 @@ static void vmx_vcpu_load(struct kvm_vcp
}
if (vmx->loaded_vmcs->cpu != cpu) {
- struct desc_ptr *gdt = &__get_cpu_var(host_gdt);
+ struct desc_ptr *gdt = this_cpu_ptr(&host_gdt);
unsigned long sysenter_esp;
kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
@@ -2611,7 +2611,7 @@ static int hardware_enable(void *garbage
ept_sync_global();
}
- native_store_gdt(&__get_cpu_var(host_gdt));
+ native_store_gdt(this_cpu_ptr(&host_gdt));
return 0;
}
Index: linux/arch/x86/mm/kmemcheck/kmemcheck.c
===================================================================
--- linux.orig/arch/x86/mm/kmemcheck/kmemcheck.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/mm/kmemcheck/kmemcheck.c 2013-08-22 15:01:56.719952513 -0500
@@ -134,7 +134,7 @@ static DEFINE_PER_CPU(struct kmemcheck_c
bool kmemcheck_active(struct pt_regs *regs)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
return data->balance > 0;
}
@@ -142,7 +142,7 @@ bool kmemcheck_active(struct pt_regs *re
/* Save an address that needs to be shown/hidden */
static void kmemcheck_save_addr(unsigned long addr)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
BUG_ON(data->n_addrs >= ARRAY_SIZE(data->addr));
data->addr[data->n_addrs++] = addr;
@@ -150,7 +150,7 @@ static void kmemcheck_save_addr(unsigned
static unsigned int kmemcheck_show_all(void)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
unsigned int i;
unsigned int n;
@@ -163,7 +163,7 @@ static unsigned int kmemcheck_show_all(v
static unsigned int kmemcheck_hide_all(void)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
unsigned int i;
unsigned int n;
@@ -179,7 +179,7 @@ static unsigned int kmemcheck_hide_all(v
*/
void kmemcheck_show(struct pt_regs *regs)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
BUG_ON(!irqs_disabled());
@@ -220,7 +220,7 @@ void kmemcheck_show(struct pt_regs *regs
*/
void kmemcheck_hide(struct pt_regs *regs)
{
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
int n;
BUG_ON(!irqs_disabled());
@@ -522,7 +522,7 @@ static void kmemcheck_access(struct pt_r
const uint8_t *insn_primary;
unsigned int size;
- struct kmemcheck_context *data = &__get_cpu_var(kmemcheck_context);
+ struct kmemcheck_context *data = this_cpu_ptr(&kmemcheck_context);
/* Recursive fault -- ouch. */
if (data->busy) {
Index: linux/arch/x86/oprofile/nmi_int.c
===================================================================
--- linux.orig/arch/x86/oprofile/nmi_int.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/oprofile/nmi_int.c 2013-08-22 15:01:56.719952513 -0500
@@ -64,11 +64,11 @@ u64 op_x86_get_ctrl(struct op_x86_model_
static int profile_exceptions_notify(unsigned int val, struct pt_regs *regs)
{
if (ctr_running)
- model->check_ctrs(regs, &__get_cpu_var(cpu_msrs));
+ model->check_ctrs(regs, this_cpu_ptr(&cpu_msrs));
else if (!nmi_enabled)
return NMI_DONE;
else
- model->stop(&__get_cpu_var(cpu_msrs));
+ model->stop(this_cpu_ptr(&cpu_msrs));
return NMI_HANDLED;
}
@@ -91,7 +91,7 @@ static void nmi_cpu_save_registers(struc
static void nmi_cpu_start(void *dummy)
{
- struct op_msrs const *msrs = &__get_cpu_var(cpu_msrs);
+ struct op_msrs const *msrs = this_cpu_ptr(&cpu_msrs);
if (!msrs->controls)
WARN_ON_ONCE(1);
else
@@ -111,7 +111,7 @@ static int nmi_start(void)
static void nmi_cpu_stop(void *dummy)
{
- struct op_msrs const *msrs = &__get_cpu_var(cpu_msrs);
+ struct op_msrs const *msrs = this_cpu_ptr(&cpu_msrs);
if (!msrs->controls)
WARN_ON_ONCE(1);
else
Index: linux/arch/x86/platform/uv/uv_time.c
===================================================================
--- linux.orig/arch/x86/platform/uv/uv_time.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/platform/uv/uv_time.c 2013-08-22 15:01:56.719952513 -0500
@@ -365,7 +365,7 @@ __setup("uvrtcevt", uv_enable_evt_rtc);
static __init void uv_rtc_register_clockevents(struct work_struct *dummy)
{
- struct clock_event_device *ced = &__get_cpu_var(cpu_ced);
+ struct clock_event_device *ced = this_cpu_ptr(&cpu_ced);
*ced = clock_event_device_uv;
ced->cpumask = cpumask_of(smp_processor_id());
Index: linux/arch/x86/xen/multicalls.c
===================================================================
--- linux.orig/arch/x86/xen/multicalls.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/xen/multicalls.c 2013-08-22 15:01:56.719952513 -0500
@@ -54,7 +54,7 @@ DEFINE_PER_CPU(unsigned long, xen_mc_irq
void xen_mc_flush(void)
{
- struct mc_buffer *b = &__get_cpu_var(mc_buffer);
+ struct mc_buffer *b = this_cpu_ptr(&mc_buffer);
struct multicall_entry *mc;
int ret = 0;
unsigned long flags;
@@ -131,7 +131,7 @@ void xen_mc_flush(void)
struct multicall_space __xen_mc_entry(size_t args)
{
- struct mc_buffer *b = &__get_cpu_var(mc_buffer);
+ struct mc_buffer *b = this_cpu_ptr(&mc_buffer);
struct multicall_space ret;
unsigned argidx = roundup(b->argidx, sizeof(u64));
@@ -162,7 +162,7 @@ struct multicall_space __xen_mc_entry(si
struct multicall_space xen_mc_extend_args(unsigned long op, size_t size)
{
- struct mc_buffer *b = &__get_cpu_var(mc_buffer);
+ struct mc_buffer *b = this_cpu_ptr(&mc_buffer);
struct multicall_space ret = { NULL, NULL };
BUG_ON(preemptible());
@@ -192,7 +192,7 @@ out:
void xen_mc_callback(void (*fn)(void *), void *data)
{
- struct mc_buffer *b = &__get_cpu_var(mc_buffer);
+ struct mc_buffer *b = this_cpu_ptr(&mc_buffer);
struct callback *cb;
if (b->cbidx == MC_BATCH) {
Index: linux/arch/x86/xen/time.c
===================================================================
--- linux.orig/arch/x86/xen/time.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/xen/time.c 2013-08-22 15:01:56.719952513 -0500
@@ -80,7 +80,7 @@ static void get_runstate_snapshot(struct
BUG_ON(preemptible());
- state = &__get_cpu_var(xen_runstate);
+ state = this_cpu_ptr(&xen_runstate);
/*
* The runstate info is always updated by the hypervisor on
@@ -123,7 +123,7 @@ static void do_stolen_accounting(void)
WARN_ON(state.state != RUNSTATE_running);
- snap = &__get_cpu_var(xen_runstate_snapshot);
+ snap = this_cpu_ptr(&xen_runstate_snapshot);
/* work out how much time the VCPU has not been runn*ing* */
runnable = state.time[RUNSTATE_runnable] - snap->time[RUNSTATE_runnable];
@@ -158,7 +158,7 @@ cycle_t xen_clocksource_read(void)
cycle_t ret;
preempt_disable_notrace();
- src = &__get_cpu_var(xen_vcpu)->time;
+ src = this_cpu_ptr(&xen_vcpu->time);
ret = pvclock_clocksource_read(src);
preempt_enable_notrace();
return ret;
@@ -397,7 +397,7 @@ static DEFINE_PER_CPU(struct xen_clock_e
static irqreturn_t xen_timer_interrupt(int irq, void *dev_id)
{
- struct clock_event_device *evt = &__get_cpu_var(xen_clock_events).evt;
+ struct clock_event_device *evt = this_cpu_ptr(&xen_clock_events.evt);
irqreturn_t ret;
ret = IRQ_NONE;
@@ -460,7 +460,7 @@ void xen_setup_cpu_clockevents(void)
{
BUG_ON(preemptible());
- clockevents_register_device(&__get_cpu_var(xen_clock_events).evt);
+ clockevents_register_device(this_cpu_ptr(&xen_clock_events.evt));
}
void xen_timer_resume(void)
Index: linux/arch/x86/include/asm/uv/uv_hub.h
===================================================================
--- linux.orig/arch/x86/include/asm/uv/uv_hub.h 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/include/asm/uv/uv_hub.h 2013-08-22 15:01:56.719952513 -0500
@@ -164,7 +164,7 @@ struct uv_hub_info_s {
};
DECLARE_PER_CPU(struct uv_hub_info_s, __uv_hub_info);
-#define uv_hub_info (&__get_cpu_var(__uv_hub_info))
+#define uv_hub_info this_cpu_ptr(&__uv_hub_info)
#define uv_cpu_hub_info(cpu) (&per_cpu(__uv_hub_info, cpu))
/*
Index: linux/arch/x86/include/asm/perf_event_p4.h
===================================================================
--- linux.orig/arch/x86/include/asm/perf_event_p4.h 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/include/asm/perf_event_p4.h 2013-08-22 15:01:56.719952513 -0500
@@ -189,7 +189,7 @@ static inline int p4_ht_thread(int cpu)
{
#ifdef CONFIG_SMP
if (smp_num_siblings == 2)
- return cpu != cpumask_first(__this_cpu_ptr(&cpu_sibling_map[0]));
+ return cpu != cpumask_first(this_cpu_ptr(cpu_sibling_map));
#endif
return 0;
}
Index: linux/arch/x86/kernel/apic/x2apic_cluster.c
===================================================================
--- linux.orig/arch/x86/kernel/apic/x2apic_cluster.c 2013-08-22 15:01:56.723952473 -0500
+++ linux/arch/x86/kernel/apic/x2apic_cluster.c 2013-08-22 15:01:56.719952513 -0500
@@ -43,7 +43,7 @@ __x2apic_send_IPI_mask(const struct cpum
* We are to modify mask, so we need an own copy
* and be sure it's manipulated with irq off.
*/
- ipi_mask_ptr = __raw_get_cpu_var(ipi_mask);
+ ipi_mask_ptr = this_cpu_ptr(ipi_mask);
cpumask_copy(ipi_mask_ptr, mask);
/*
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 17/31] mips: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Ralf Baechle, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/mips/include/asm/fpu_emulator.h
===================================================================
--- linux.orig/arch/mips/include/asm/fpu_emulator.h 2013-08-22 14:18:23.138568266 -0500
+++ linux/arch/mips/include/asm/fpu_emulator.h 2013-08-22 14:18:23.130568348 -0500
@@ -43,7 +43,7 @@ DECLARE_PER_CPU(struct mips_fpu_emulator
#define MIPS_FPU_EMU_INC_STATS(M) \
do { \
preempt_disable(); \
- __local_inc(&__get_cpu_var(fpuemustats).M); \
+ __this_cpu_inc(fpuemustats.M); \
preempt_enable(); \
} while (0)
Index: linux/arch/mips/cavium-octeon/octeon-irq.c
===================================================================
--- linux.orig/arch/mips/cavium-octeon/octeon-irq.c 2013-08-22 14:17:32.000000000 -0500
+++ linux/arch/mips/cavium-octeon/octeon-irq.c 2013-08-22 14:18:54.210250887 -0500
@@ -264,13 +264,13 @@ static void octeon_irq_ciu_enable_local(
unsigned long *pen;
unsigned long flags;
union octeon_ciu_chip_data cd;
- raw_spinlock_t *lock = &__get_cpu_var(octeon_irq_ciu_spinlock);
+ raw_spinlock_t *lock = this_cpu_ptr(&octeon_irq_ciu_spinlock);
cd.p = irq_data_get_irq_chip_data(data);
raw_spin_lock_irqsave(lock, flags);
if (cd.s.line == 0) {
- pen = &__get_cpu_var(octeon_irq_ciu0_en_mirror);
+ pen = this_cpu_ptr(&octeon_irq_ciu0_en_mirror);
__set_bit(cd.s.bit, pen);
/*
* Must be visible to octeon_irq_ip{2,3}_ciu() before
@@ -279,7 +279,7 @@ static void octeon_irq_ciu_enable_local(
wmb();
cvmx_write_csr(CVMX_CIU_INTX_EN0(cvmx_get_core_num() * 2), *pen);
} else {
- pen = &__get_cpu_var(octeon_irq_ciu1_en_mirror);
+ pen = this_cpu_ptr(&octeon_irq_ciu1_en_mirror);
__set_bit(cd.s.bit, pen);
/*
* Must be visible to octeon_irq_ip{2,3}_ciu() before
@@ -296,13 +296,13 @@ static void octeon_irq_ciu_disable_local
unsigned long *pen;
unsigned long flags;
union octeon_ciu_chip_data cd;
- raw_spinlock_t *lock = &__get_cpu_var(octeon_irq_ciu_spinlock);
+ raw_spinlock_t *lock = this_cpu_ptr(&octeon_irq_ciu_spinlock);
cd.p = irq_data_get_irq_chip_data(data);
raw_spin_lock_irqsave(lock, flags);
if (cd.s.line == 0) {
- pen = &__get_cpu_var(octeon_irq_ciu0_en_mirror);
+ pen = this_cpu_ptr(&octeon_irq_ciu0_en_mirror);
__clear_bit(cd.s.bit, pen);
/*
* Must be visible to octeon_irq_ip{2,3}_ciu() before
@@ -311,7 +311,7 @@ static void octeon_irq_ciu_disable_local
wmb();
cvmx_write_csr(CVMX_CIU_INTX_EN0(cvmx_get_core_num() * 2), *pen);
} else {
- pen = &__get_cpu_var(octeon_irq_ciu1_en_mirror);
+ pen = this_cpu_ptr(&octeon_irq_ciu1_en_mirror);
__clear_bit(cd.s.bit, pen);
/*
* Must be visible to octeon_irq_ip{2,3}_ciu() before
@@ -431,11 +431,11 @@ static void octeon_irq_ciu_enable_local_
if (cd.s.line == 0) {
int index = cvmx_get_core_num() * 2;
- set_bit(cd.s.bit, &__get_cpu_var(octeon_irq_ciu0_en_mirror));
+ set_bit(cd.s.bit, this_cpu_ptr(&octeon_irq_ciu0_en_mirror));
cvmx_write_csr(CVMX_CIU_INTX_EN0_W1S(index), mask);
} else {
int index = cvmx_get_core_num() * 2 + 1;
- set_bit(cd.s.bit, &__get_cpu_var(octeon_irq_ciu1_en_mirror));
+ set_bit(cd.s.bit, this_cpu_ptr(&octeon_irq_ciu1_en_mirror));
cvmx_write_csr(CVMX_CIU_INTX_EN1_W1S(index), mask);
}
}
@@ -450,11 +450,11 @@ static void octeon_irq_ciu_disable_local
if (cd.s.line == 0) {
int index = cvmx_get_core_num() * 2;
- clear_bit(cd.s.bit, &__get_cpu_var(octeon_irq_ciu0_en_mirror));
+ clear_bit(cd.s.bit, this_cpu_ptr(&octeon_irq_ciu0_en_mirror));
cvmx_write_csr(CVMX_CIU_INTX_EN0_W1C(index), mask);
} else {
int index = cvmx_get_core_num() * 2 + 1;
- clear_bit(cd.s.bit, &__get_cpu_var(octeon_irq_ciu1_en_mirror));
+ clear_bit(cd.s.bit, this_cpu_ptr(&octeon_irq_ciu1_en_mirror));
cvmx_write_csr(CVMX_CIU_INTX_EN1_W1C(index), mask);
}
}
@@ -1063,7 +1063,7 @@ static void octeon_irq_ip2_ciu(void)
const unsigned long core_id = cvmx_get_core_num();
u64 ciu_sum = cvmx_read_csr(CVMX_CIU_INTX_SUM0(core_id * 2));
- ciu_sum &= __get_cpu_var(octeon_irq_ciu0_en_mirror);
+ ciu_sum &= __this_cpu_read(octeon_irq_ciu0_en_mirror);
if (likely(ciu_sum)) {
int bit = fls64(ciu_sum) - 1;
int irq = octeon_irq_ciu_to_irq[0][bit];
@@ -1080,7 +1080,7 @@ static void octeon_irq_ip3_ciu(void)
{
u64 ciu_sum = cvmx_read_csr(CVMX_CIU_INT_SUM1);
- ciu_sum &= __get_cpu_var(octeon_irq_ciu1_en_mirror);
+ ciu_sum &= __this_cpu_read(octeon_irq_ciu1_en_mirror);
if (likely(ciu_sum)) {
int bit = fls64(ciu_sum) - 1;
int irq = octeon_irq_ciu_to_irq[1][bit];
@@ -1129,10 +1129,10 @@ static void octeon_irq_init_ciu_percpu(v
int coreid = cvmx_get_core_num();
- __get_cpu_var(octeon_irq_ciu0_en_mirror) = 0;
- __get_cpu_var(octeon_irq_ciu1_en_mirror) = 0;
+ __this_cpu_write(octeon_irq_ciu0_en_mirror, 0);
+ __this_cpu_write(octeon_irq_ciu1_en_mirror, 0);
wmb();
- raw_spin_lock_init(&__get_cpu_var(octeon_irq_ciu_spinlock));
+ raw_spin_lock_init(this_cpu_ptr(&octeon_irq_ciu_spinlock));
/*
* Disable All CIU Interrupts. The ones we need will be
* enabled later. Read the SUM register so we know the write
Index: linux/arch/mips/kernel/kprobes.c
===================================================================
--- linux.orig/arch/mips/kernel/kprobes.c 2013-08-22 14:17:32.000000000 -0500
+++ linux/arch/mips/kernel/kprobes.c 2013-08-22 14:18:54.210250887 -0500
@@ -224,7 +224,7 @@ static void save_previous_kprobe(struct
static void restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
kcb->kprobe_old_SR = kcb->prev_kprobe.old_SR;
kcb->kprobe_saved_SR = kcb->prev_kprobe.saved_SR;
@@ -234,7 +234,7 @@ static void restore_previous_kprobe(stru
static void set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
kcb->kprobe_saved_SR = kcb->kprobe_old_SR = (regs->cp0_status & ST0_IE);
kcb->kprobe_saved_epc = regs->cp0_epc;
}
@@ -385,7 +385,7 @@ static int __kprobes kprobe_handler(stru
ret = 1;
goto no_kprobe;
}
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs))
goto ss_probe;
}
Index: linux/arch/mips/kernel/perf_event_mipsxx.c
===================================================================
--- linux.orig/arch/mips/kernel/perf_event_mipsxx.c 2013-08-22 14:17:32.000000000 -0500
+++ linux/arch/mips/kernel/perf_event_mipsxx.c 2013-08-22 14:18:54.210250887 -0500
@@ -340,7 +340,7 @@ static int mipsxx_pmu_alloc_counter(stru
static void mipsxx_pmu_enable_event(struct hw_perf_event *evt, int idx)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
WARN_ON(idx < 0 || idx >= mipspmu.num_counters);
@@ -360,7 +360,7 @@ static void mipsxx_pmu_enable_event(stru
static void mipsxx_pmu_disable_event(int idx)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
unsigned long flags;
WARN_ON(idx < 0 || idx >= mipspmu.num_counters);
@@ -460,7 +460,7 @@ static void mipspmu_stop(struct perf_eve
static int mipspmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx;
int err = 0;
@@ -496,7 +496,7 @@ out:
static void mipspmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -1270,7 +1270,7 @@ static int __hw_perf_event_init(struct p
static void pause_local_counters(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int ctr = mipspmu.num_counters;
unsigned long flags;
@@ -1286,7 +1286,7 @@ static void pause_local_counters(void)
static void resume_local_counters(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int ctr = mipspmu.num_counters;
do {
@@ -1297,7 +1297,7 @@ static void resume_local_counters(void)
static int mipsxx_pmu_handle_shared_irq(void)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_sample_data data;
unsigned int counters = mipspmu.num_counters;
u64 counter;
Index: linux/arch/mips/kernel/smp-bmips.c
===================================================================
--- linux.orig/arch/mips/kernel/smp-bmips.c 2013-08-22 14:17:32.000000000 -0500
+++ linux/arch/mips/kernel/smp-bmips.c 2013-08-22 14:18:54.214250846 -0500
@@ -304,7 +304,7 @@ static irqreturn_t bmips_ipi_interrupt(i
int action, cpu = irq - IPI0_IRQ;
spin_lock_irqsave(&ipi_lock, flags);
- action = __get_cpu_var(ipi_action_mask);
+ action = __this_cpu_read(ipi_action_mask);
per_cpu(ipi_action_mask, cpu) = 0;
clear_c0_cause(cpu ? C_SW1 : C_SW0);
spin_unlock_irqrestore(&ipi_lock, flags);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 18/31] s390: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Martin Schwidefsky, Heiko Carstens, linux390, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/s390/include/asm/irq.h
===================================================================
--- linux.orig/arch/s390/include/asm/irq.h 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/include/asm/irq.h 2013-08-22 14:49:17.327583548 -0500
@@ -55,7 +55,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct ir
static __always_inline void inc_irq_stat(enum interruption_class irq)
{
- __get_cpu_var(irq_stat).irqs[irq]++;
+ __this_cpu_inc(irq_stat.irqs[irq]);
}
struct ext_code {
Index: linux/arch/s390/include/asm/cputime.h
===================================================================
--- linux.orig/arch/s390/include/asm/cputime.h 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/include/asm/cputime.h 2013-08-22 14:49:17.327583548 -0500
@@ -187,7 +187,7 @@ cputime64_t s390_get_idle_time(int cpu);
static inline int s390_nohz_delay(int cpu)
{
- return __get_cpu_var(s390_idle).nohz_delay != 0;
+ return __this_cpu_read(s390_idle.nohz_delay) != 0;
}
#define arch_needs_cpu(cpu) s390_nohz_delay(cpu)
Index: linux/arch/s390/kernel/kprobes.c
===================================================================
--- linux.orig/arch/s390/kernel/kprobes.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/kprobes.c 2013-08-22 14:49:17.327583548 -0500
@@ -212,9 +212,9 @@ static void __kprobes disable_singlestep
*/
static void __kprobes push_kprobe(struct kprobe_ctlblk *kcb, struct kprobe *p)
{
- kcb->prev_kprobe.kp = __get_cpu_var(current_kprobe);
+ kcb->prev_kprobe.kp = __this_cpu_read(current_kprobe);
kcb->prev_kprobe.status = kcb->kprobe_status;
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
/*
@@ -224,7 +224,7 @@ static void __kprobes push_kprobe(struct
*/
static void __kprobes pop_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
@@ -305,7 +305,7 @@ static int __kprobes kprobe_handler(stru
enable_singlestep(kcb, regs, (unsigned long) p->ainsn.insn);
return 1;
} else if (kprobe_running()) {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
/*
* Continuation after the jprobe completed and
Index: linux/arch/s390/kernel/nmi.c
===================================================================
--- linux.orig/arch/s390/kernel/nmi.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/nmi.c 2013-08-22 14:49:17.327583548 -0500
@@ -53,8 +53,8 @@ void s390_handle_mcck(void)
*/
local_irq_save(flags);
local_mcck_disable();
- mcck = __get_cpu_var(cpu_mcck);
- memset(&__get_cpu_var(cpu_mcck), 0, sizeof(struct mcck_struct));
+ mcck = __this_cpu_read(cpu_mcck);
+ memset(this_cpu_ptr(&cpu_mcck), 0, sizeof(struct mcck_struct));
clear_thread_flag(TIF_MCCK_PENDING);
local_mcck_enable();
local_irq_restore(flags);
@@ -256,7 +256,7 @@ void notrace s390_do_machine_check(struc
nmi_enter();
inc_irq_stat(NMI_NMI);
mci = (struct mci *) &S390_lowcore.mcck_interruption_code;
- mcck = &__get_cpu_var(cpu_mcck);
+ mcck = this_cpu_ptr(&cpu_mcck);
umode = user_mode(regs);
if (mci->sd) {
Index: linux/arch/s390/kernel/perf_cpum_cf.c
===================================================================
--- linux.orig/arch/s390/kernel/perf_cpum_cf.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/perf_cpum_cf.c 2013-08-22 14:49:17.331583508 -0500
@@ -173,7 +173,7 @@ static int validate_ctr_auth(const struc
*/
static void cpumf_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
int err;
if (cpuhw->flags & PMU_F_ENABLED)
@@ -196,7 +196,7 @@ static void cpumf_pmu_enable(struct pmu
*/
static void cpumf_pmu_disable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
int err;
u64 inactive;
@@ -230,7 +230,7 @@ static void cpumf_measurement_alert(stru
return;
inc_irq_stat(IRQEXT_CMC);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
/* Measurement alerts are shared and might happen when the PMU
* is not reserved. Ignore these alerts in this case. */
@@ -250,7 +250,7 @@ static void cpumf_measurement_alert(stru
#define PMC_RELEASE 1
static void setup_pmc_cpu(void *flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
switch (*((int *) flags)) {
case PMC_INIT:
@@ -481,7 +481,7 @@ static void cpumf_pmu_read(struct perf_e
static void cpumf_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
@@ -512,7 +512,7 @@ static void cpumf_pmu_start(struct perf_
static void cpumf_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
if (!(hwc->state & PERF_HES_STOPPED)) {
@@ -533,7 +533,7 @@ static void cpumf_pmu_stop(struct perf_e
static int cpumf_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
/* Check authorization for the counter set to which this
* counter belongs.
@@ -557,7 +557,7 @@ static int cpumf_pmu_add(struct perf_eve
static void cpumf_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
cpumf_pmu_stop(event, PERF_EF_UPDATE);
@@ -581,7 +581,7 @@ static void cpumf_pmu_del(struct perf_ev
*/
static void cpumf_pmu_start_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
perf_pmu_disable(pmu);
cpuhw->flags |= PERF_EVENT_TXN;
@@ -595,7 +595,7 @@ static void cpumf_pmu_start_txn(struct p
*/
static void cpumf_pmu_cancel_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
WARN_ON(cpuhw->tx_state != cpuhw->state);
@@ -610,7 +610,7 @@ static void cpumf_pmu_cancel_txn(struct
*/
static int cpumf_pmu_commit_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
u64 state;
/* check if the updated state can be scheduled */
Index: linux/arch/s390/kernel/processor.c
===================================================================
--- linux.orig/arch/s390/kernel/processor.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/processor.c 2013-08-22 14:49:17.331583508 -0500
@@ -23,8 +23,8 @@ static DEFINE_PER_CPU(struct cpuid, cpu_
*/
void cpu_init(void)
{
- struct s390_idle_data *idle = &__get_cpu_var(s390_idle);
- struct cpuid *id = &__get_cpu_var(cpu_id);
+ struct s390_idle_data *idle = this_cpu_ptr(&s390_idle);
+ struct cpuid *id = this_cpu_ptr(&cpu_id);
get_cpu_id(id);
atomic_inc(&init_mm.mm_count);
Index: linux/arch/s390/kernel/time.c
===================================================================
--- linux.orig/arch/s390/kernel/time.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/time.c 2013-08-22 14:49:17.331583508 -0500
@@ -93,7 +93,7 @@ void clock_comparator_work(void)
S390_lowcore.clock_comparator = -1ULL;
set_clock_comparator(S390_lowcore.clock_comparator);
- cd = &__get_cpu_var(comparators);
+ cd = this_cpu_ptr(&comparators);
cd->event_handler(cd);
}
@@ -363,7 +363,7 @@ EXPORT_SYMBOL(get_sync_clock);
*/
static void disable_sync_clock(void *dummy)
{
- atomic_t *sw_ptr = &__get_cpu_var(clock_sync_word);
+ atomic_t *sw_ptr = this_cpu_ptr(&clock_sync_word);
/*
* Clear the in-sync bit 2^31. All get_sync_clock calls will
* fail until the sync bit is turned back on. In addition
@@ -380,7 +380,7 @@ static void disable_sync_clock(void *dum
*/
static void enable_sync_clock(void)
{
- atomic_t *sw_ptr = &__get_cpu_var(clock_sync_word);
+ atomic_t *sw_ptr = this_cpu_ptr(&clock_sync_word);
atomic_set_mask(0x80000000, sw_ptr);
}
Index: linux/arch/s390/kernel/vtime.c
===================================================================
--- linux.orig/arch/s390/kernel/vtime.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/kernel/vtime.c 2013-08-22 14:49:17.331583508 -0500
@@ -153,7 +153,7 @@ EXPORT_SYMBOL_GPL(vtime_account_system);
void __kprobes vtime_stop_cpu(void)
{
- struct s390_idle_data *idle = &__get_cpu_var(s390_idle);
+ struct s390_idle_data *idle = this_cpu_ptr(&s390_idle);
unsigned long long idle_time;
unsigned long psw_mask;
Index: linux/arch/s390/oprofile/hwsampler.c
===================================================================
--- linux.orig/arch/s390/oprofile/hwsampler.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/oprofile/hwsampler.c 2013-08-22 14:49:17.331583508 -0500
@@ -228,7 +228,7 @@ static inline unsigned long *trailer_ent
static void hws_ext_handler(struct ext_code ext_code,
unsigned int param32, unsigned long param64)
{
- struct hws_cpu_buffer *cb = &__get_cpu_var(sampler_cpu_buffer);
+ struct hws_cpu_buffer *cb = this_cpu_ptr(&sampler_cpu_buffer);
if (!(param32 & CPU_MF_INT_SF_MASK))
return;
Index: linux/arch/s390/pci/pci.c
===================================================================
--- linux.orig/arch/s390/pci/pci.c 2013-08-22 14:49:17.339583428 -0500
+++ linux/arch/s390/pci/pci.c 2013-08-22 14:49:17.335583467 -0500
@@ -409,7 +409,7 @@ static DEFINE_PER_CPU(unsigned long, nex
static void zpci_irq_handler(struct airq_struct *airq)
{
- unsigned long sbit, mbit, last = 0, start = __get_cpu_var(next_sbit);
+ unsigned long sbit, mbit, last = 0, start = __this_cpu_read(next_sbit);
int rescan = 0, max = aisb_max;
struct zdev_irq_map *imap;
@@ -460,7 +460,7 @@ scan:
}
out:
/* store next device bit to scan */
- __get_cpu_var(next_sbit) = (++last >= aisb_max) ? 0 : last;
+ __this_cpu_write(next_sbit, (++last >= aisb_max) ? 0 : last);
}
/* msi_vecs - number of requested interrupts, 0 place function to error state */
Index: linux/arch/s390/kernel/irq.c
===================================================================
--- linux.orig/arch/s390/kernel/irq.c 2013-08-08 02:54:34.344984334 -0500
+++ linux/arch/s390/kernel/irq.c 2013-08-22 14:53:32.025028108 -0500
@@ -250,7 +250,7 @@ void __irq_entry do_extint(struct pt_reg
kstat_incr_irqs_this_cpu(EXTERNAL_INTERRUPT, NULL);
ext_code = *(struct ext_code *) ®s->int_code;
if (ext_code.code != 0x1004)
- __get_cpu_var(s390_idle).nohz_delay = 1;
+ __this_cpu_write(s390_idle.nohz_delay, 1);
index = ext_hash(ext_code.code);
rcu_read_lock();
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 01/31] x86: Use this_cpu_inc/dec for debug registers
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
I think this patch already exists in one form or another in some x86 tree.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/x86/include/asm/debugreg.h
===================================================================
--- linux.orig/arch/x86/include/asm/debugreg.h 2013-08-06 13:52:23.740092873 -0500
+++ linux/arch/x86/include/asm/debugreg.h 2013-08-06 13:52:23.740092873 -0500
@@ -97,11 +97,11 @@ extern void hw_breakpoint_restore(void);
DECLARE_PER_CPU(int, debug_stack_usage);
static inline void debug_stack_usage_inc(void)
{
- __get_cpu_var(debug_stack_usage)++;
+ __this_cpu_inc(debug_stack_usage);
}
static inline void debug_stack_usage_dec(void)
{
- __get_cpu_var(debug_stack_usage)--;
+ __this_cpu_dec(debug_stack_usage);
}
int is_debug_stack(unsigned long addr);
void debug_stack_set_zero(void);
Index: linux/arch/x86/kernel/cpu/common.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/common.c 2013-07-19 09:06:36.850047837 -0500
+++ linux/arch/x86/kernel/cpu/common.c 2013-08-06 13:54:36.426384889 -0500
@@ -1144,9 +1144,9 @@ DEFINE_PER_CPU(int, debug_stack_usage);
int is_debug_stack(unsigned long addr)
{
- return __get_cpu_var(debug_stack_usage) ||
- (addr <= __get_cpu_var(debug_stack_addr) &&
- addr > (__get_cpu_var(debug_stack_addr) - DEBUG_STKSZ));
+ return __this_cpu_read(debug_stack_usage) ||
+ (addr <= __this_cpu_read(debug_stack_addr) &&
+ addr > (__this_cpu_read(debug_stack_addr) - DEBUG_STKSZ));
}
DEFINE_PER_CPU(u32, debug_idt_ctr);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 19/31] ia64: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
2013-08-26 20:44 ` Christoph Lameter
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (29 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Tony Luck, Fenghua Yu, linux-ia64, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/alpha/kernel/perf_event.c
===================================================================
--- linux.orig/arch/alpha/kernel/perf_event.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/alpha/kernel/perf_event.c 2013-08-23 09:10:22.665620124 -0500
@@ -803,7 +803,7 @@ static void alpha_perf_event_irq_handler
struct hw_perf_event *hwc;
int idx, j;
- __get_cpu_var(irq_pmi_count)++;
+ __this_cpu_inc(irq_pmi_count);
cpuc = &__get_cpu_var(cpu_hw_events);
/* Completely counting through the PMC's period to trigger a new PMC
Index: linux/arch/ia64/kernel/irq.c
===================================================================
--- linux.orig/arch/ia64/kernel/irq.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/irq.c 2013-08-23 09:10:22.665620124 -0500
@@ -42,7 +42,7 @@ ia64_vector __ia64_irq_to_vector(int irq
unsigned int __ia64_local_vector_to_irq (ia64_vector vec)
{
- return __get_cpu_var(vector_irq)[vec];
+ return __this_cpu_read(vector_irq[vec]);
}
#endif
Index: linux/arch/ia64/kernel/irq_ia64.c
===================================================================
--- linux.orig/arch/ia64/kernel/irq_ia64.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/irq_ia64.c 2013-08-23 09:10:22.669620084 -0500
@@ -338,7 +338,7 @@ static irqreturn_t smp_irq_move_cleanup_
int irq;
struct irq_desc *desc;
struct irq_cfg *cfg;
- irq = __get_cpu_var(vector_irq)[vector];
+ irq = __this_cpu_read(vector_irq[vector]);
if (irq < 0)
continue;
@@ -352,7 +352,7 @@ static irqreturn_t smp_irq_move_cleanup_
goto unlock;
spin_lock_irqsave(&vector_lock, flags);
- __get_cpu_var(vector_irq)[vector] = -1;
+ __this_cpu_write(vector_irq[vector], -1);
cpu_clear(me, vector_table[vector]);
spin_unlock_irqrestore(&vector_lock, flags);
cfg->move_cleanup_count--;
Index: linux/arch/ia64/kernel/kprobes.c
===================================================================
--- linux.orig/arch/ia64/kernel/kprobes.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/kprobes.c 2013-08-23 09:10:22.669620084 -0500
@@ -396,7 +396,7 @@ static void __kprobes restore_previous_k
{
unsigned int i;
i = atomic_read(&kcb->prev_kprobe_index);
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe[i-1].kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe[i-1].kp);
kcb->kprobe_status = kcb->prev_kprobe[i-1].status;
atomic_sub(1, &kcb->prev_kprobe_index);
}
@@ -404,7 +404,7 @@ static void __kprobes restore_previous_k
static void __kprobes set_current_kprobe(struct kprobe *p,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static void kretprobe_trampoline(void)
@@ -823,7 +823,7 @@ static int __kprobes pre_kprobes_handler
/*
* jprobe instrumented function just completed
*/
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
goto ss_probe;
}
Index: linux/arch/ia64/kernel/mca.c
===================================================================
--- linux.orig/arch/ia64/kernel/mca.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/mca.c 2013-08-23 09:11:54.244715775 -0500
@@ -1341,7 +1341,7 @@ ia64_mca_handler(struct pt_regs *regs, s
ia64_mlogbuf_finish(1);
}
- if (__get_cpu_var(ia64_mca_tr_reload)) {
+ if (__this_cpu_read(ia64_mca_tr_reload)) {
mca_insert_tr(0x1); /*Reload dynamic itrs*/
mca_insert_tr(0x2); /*Reload dynamic itrs*/
}
@@ -1874,14 +1874,14 @@ ia64_mca_cpu_init(void *cpu_data)
"MCA", cpu);
format_mca_init_stack(data, offsetof(struct ia64_mca_cpu, init_stack),
"INIT", cpu);
- __get_cpu_var(ia64_mca_data) = __per_cpu_mca[cpu] = __pa(data);
+ __this_cpu_write(ia64_mca_data, __per_cpu_mca[cpu] = __pa(data));
/*
* Stash away a copy of the PTE needed to map the per-CPU page.
* We may need it during MCA recovery.
*/
- __get_cpu_var(ia64_mca_per_cpu_pte) =
- pte_val(mk_pte_phys(__pa(cpu_data), PAGE_KERNEL));
+ __this_cpu_write(ia64_mca_per_cpu_pte,
+ pte_val(mk_pte_phys(__pa(cpu_data), PAGE_KERNEL)));
/*
* Also, stash away a copy of the PAL address and the PTE
@@ -1890,10 +1890,10 @@ ia64_mca_cpu_init(void *cpu_data)
pal_vaddr = efi_get_pal_addr();
if (!pal_vaddr)
return;
- __get_cpu_var(ia64_mca_pal_base) =
- GRANULEROUNDDOWN((unsigned long) pal_vaddr);
- __get_cpu_var(ia64_mca_pal_pte) = pte_val(mk_pte_phys(__pa(pal_vaddr),
- PAGE_KERNEL));
+ __this_cpu_write(ia64_mca_pal_base,
+ GRANULEROUNDDOWN((unsigned long) pal_vaddr));
+ __this_cpu_write(ia64_mca_pal_pte, pte_val(mk_pte_phys(__pa(pal_vaddr),
+ PAGE_KERNEL)));
}
static void ia64_mca_cmc_vector_adjust(void *dummy)
Index: linux/arch/ia64/kernel/process.c
===================================================================
--- linux.orig/arch/ia64/kernel/process.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/process.c 2013-08-23 09:10:22.669620084 -0500
@@ -215,7 +215,7 @@ static inline void play_dead(void)
unsigned int this_cpu = smp_processor_id();
/* Ack it */
- __get_cpu_var(cpu_state) = CPU_DEAD;
+ __this_cpu_write(cpu_state, CPU_DEAD);
max_xtp();
local_irq_disable();
@@ -273,7 +273,7 @@ ia64_save_extra (struct task_struct *tas
if ((task->thread.flags & IA64_THREAD_PM_VALID) != 0)
pfm_save_regs(task);
- info = __get_cpu_var(pfm_syst_info);
+ info = __this_cpu_read(pfm_syst_info);
if (info & PFM_CPUINFO_SYST_WIDE)
pfm_syst_wide_update_task(task, info, 0);
#endif
@@ -293,7 +293,7 @@ ia64_load_extra (struct task_struct *tas
if ((task->thread.flags & IA64_THREAD_PM_VALID) != 0)
pfm_load_regs(task);
- info = __get_cpu_var(pfm_syst_info);
+ info = __this_cpu_read(pfm_syst_info);
if (info & PFM_CPUINFO_SYST_WIDE)
pfm_syst_wide_update_task(task, info, 1);
#endif
Index: linux/arch/ia64/sn/kernel/sn2/sn2_smp.c
===================================================================
--- linux.orig/arch/ia64/sn/kernel/sn2/sn2_smp.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/sn/kernel/sn2/sn2_smp.c 2013-08-23 09:10:22.669620084 -0500
@@ -134,8 +134,8 @@ sn2_ipi_flush_all_tlb(struct mm_struct *
itc = ia64_get_itc();
smp_flush_tlb_cpumask(*mm_cpumask(mm));
itc = ia64_get_itc() - itc;
- __get_cpu_var(ptcstats).shub_ipi_flushes_itc_clocks += itc;
- __get_cpu_var(ptcstats).shub_ipi_flushes++;
+ __this_cpu_add(ptcstats.shub_ipi_flushes_itc_clocks, itc);
+ __this_cpu_inc(ptcstats.shub_ipi_flushes);
}
/**
@@ -199,14 +199,14 @@ sn2_global_tlb_purge(struct mm_struct *m
start += (1UL << nbits);
} while (start < end);
ia64_srlz_i();
- __get_cpu_var(ptcstats).ptc_l++;
+ __this_cpu_inc(ptcstats.ptc_l);
preempt_enable();
return;
}
if (atomic_read(&mm->mm_users) == 1 && mymm) {
flush_tlb_mm(mm);
- __get_cpu_var(ptcstats).change_rid++;
+ __this_cpu_inc(ptcstats.change_rid);
preempt_enable();
return;
}
@@ -250,11 +250,11 @@ sn2_global_tlb_purge(struct mm_struct *m
spin_lock_irqsave(PTC_LOCK(shub1), flags);
itc2 = ia64_get_itc();
- __get_cpu_var(ptcstats).lock_itc_clocks += itc2 - itc;
- __get_cpu_var(ptcstats).shub_ptc_flushes++;
- __get_cpu_var(ptcstats).nodes_flushed += nix;
+ __this_cpu_add(ptcstats.lock_itc_clocks, itc2 - itc);
+ __this_cpu_inc(ptcstats.shub_ptc_flushes);
+ __this_cpu_add(ptcstats.nodes_flushed, nix);
if (!mymm)
- __get_cpu_var(ptcstats).shub_ptc_flushes_not_my_mm++;
+ __this_cpu_inc(ptcstats.shub_ptc_flushes_not_my_mm);
if (use_cpu_ptcga && !mymm) {
old_rr = ia64_get_rr(start);
@@ -299,9 +299,9 @@ sn2_global_tlb_purge(struct mm_struct *m
done:
itc2 = ia64_get_itc() - itc2;
- __get_cpu_var(ptcstats).shub_itc_clocks += itc2;
- if (itc2 > __get_cpu_var(ptcstats).shub_itc_clocks_max)
- __get_cpu_var(ptcstats).shub_itc_clocks_max = itc2;
+ __this_cpu_add(ptcstats)shub_itc_clocks, itc2);
+ if (itc2 > __this_cpu_read(ptcstats.shub_itc_clocks_max))
+ __this_cpu_write(ptcstats).shub_itc_clocks_max, itc2);
if (old_rr) {
ia64_set_rr(start, old_rr);
@@ -311,7 +311,7 @@ done:
spin_unlock_irqrestore(PTC_LOCK(shub1), flags);
if (flush_opt == 1 && deadlock) {
- __get_cpu_var(ptcstats).deadlocks++;
+ __this_cpu_inc(ptcstats.deadlocks);
sn2_ipi_flush_all_tlb(mm);
}
@@ -334,7 +334,7 @@ sn2_ptc_deadlock_recovery(short *nasids,
short nasid, i;
unsigned long *piows, zeroval, n;
- __get_cpu_var(ptcstats).deadlocks++;
+ __this_cpu_inc(ptcstats.deadlocks);
piows = (unsigned long *) pda->pio_write_status_addr;
zeroval = pda->pio_write_status_val;
@@ -349,7 +349,7 @@ sn2_ptc_deadlock_recovery(short *nasids,
ptc1 = CHANGE_NASID(nasid, ptc1);
n = sn2_ptc_deadlock_recovery_core(ptc0, data0, ptc1, data1, piows, zeroval);
- __get_cpu_var(ptcstats).deadlocks2 += n;
+ __this_cpu_add(ptcstats.deadlocks2, n);
}
}
Index: linux/arch/ia64/include/asm/hw_irq.h
===================================================================
--- linux.orig/arch/ia64/include/asm/hw_irq.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/hw_irq.h 2013-08-23 09:10:22.673620044 -0500
@@ -160,7 +160,7 @@ static inline ia64_vector __ia64_irq_to_
static inline unsigned int
__ia64_local_vector_to_irq (ia64_vector vec)
{
- return __get_cpu_var(vector_irq)[vec];
+ return __this_cpu_read(vector_irq[vec]);
}
#endif
Index: linux/arch/ia64/include/asm/sn/nodepda.h
===================================================================
--- linux.orig/arch/ia64/include/asm/sn/nodepda.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/sn/nodepda.h 2013-08-23 09:10:22.673620044 -0500
@@ -70,7 +70,7 @@ typedef struct nodepda_s nodepda_t;
*/
DECLARE_PER_CPU(struct nodepda_s *, __sn_nodepda);
-#define sn_nodepda (__get_cpu_var(__sn_nodepda))
+#define sn_nodepda __this_cpu_read(__sn_nodepda)
#define NODEPDA(cnodeid) (sn_nodepda->pernode_pdaindr[cnodeid])
/*
Index: linux/arch/ia64/include/asm/switch_to.h
===================================================================
--- linux.orig/arch/ia64/include/asm/switch_to.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/switch_to.h 2013-08-23 09:10:22.673620044 -0500
@@ -32,7 +32,7 @@ extern void ia64_load_extra (struct task
#ifdef CONFIG_PERFMON
DECLARE_PER_CPU(unsigned long, pfm_syst_info);
-# define PERFMON_IS_SYSWIDE() (__get_cpu_var(pfm_syst_info) & 0x1)
+# define PERFMON_IS_SYSWIDE() (__this_cpu_read(pfm_syst_info) & 0x1)
#else
# define PERFMON_IS_SYSWIDE() (0)
#endif
Index: linux/arch/ia64/include/asm/sn/arch.h
===================================================================
--- linux.orig/arch/ia64/include/asm/sn/arch.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/sn/arch.h 2013-08-23 09:10:22.673620044 -0500
@@ -57,7 +57,7 @@ struct sn_hub_info_s {
u16 nasid_bitmask;
};
DECLARE_PER_CPU(struct sn_hub_info_s, __sn_hub_info);
-#define sn_hub_info (&__get_cpu_var(__sn_hub_info))
+#define sn_hub_info this_cpu_ptr(&__sn_hub_info)
#define is_shub2() (sn_hub_info->shub2)
#define is_shub1() (sn_hub_info->shub2 == 0)
@@ -72,7 +72,7 @@ DECLARE_PER_CPU(struct sn_hub_info_s, __
* cpu.
*/
DECLARE_PER_CPU(short, __sn_cnodeid_to_nasid[MAX_COMPACT_NODES]);
-#define sn_cnodeid_to_nasid (&__get_cpu_var(__sn_cnodeid_to_nasid[0]))
+#define sn_cnodeid_to_nasid this_cpu_ptr(&__sn_cnodeid_to_nasid[0])
extern u8 sn_partition_id;
Index: linux/arch/ia64/include/asm/uv/uv_hub.h
===================================================================
--- linux.orig/arch/ia64/include/asm/uv/uv_hub.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/uv/uv_hub.h 2013-08-23 09:10:22.673620044 -0500
@@ -108,7 +108,7 @@ struct uv_hub_info_s {
unsigned char n_val;
};
DECLARE_PER_CPU(struct uv_hub_info_s, __uv_hub_info);
-#define uv_hub_info (&__get_cpu_var(__uv_hub_info))
+#define uv_hub_info this_cpu_ptr(&__uv_hub_info)
#define uv_cpu_hub_info(cpu) (&per_cpu(__uv_hub_info, cpu))
/*
Index: linux/arch/ia64/kernel/traps.c
===================================================================
--- linux.orig/arch/ia64/kernel/traps.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/traps.c 2013-08-23 09:10:22.673620044 -0500
@@ -299,7 +299,7 @@ handle_fpu_swa (int fp_fault, struct pt_
if (!(current->thread.flags & IA64_THREAD_FPEMU_NOPRINT)) {
unsigned long count, current_jiffies = jiffies;
- struct fpu_swa_msg *cp = &__get_cpu_var(cpulast);
+ struct fpu_swa_msg *cp = this_cpu_ptr(&cpulast);
if (unlikely(current_jiffies > cp->time))
cp->count = 0;
Index: linux/arch/ia64/xen/time.c
===================================================================
--- linux.orig/arch/ia64/xen/time.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/xen/time.c 2013-08-23 09:10:22.673620044 -0500
@@ -68,7 +68,7 @@ static void get_runstate_snapshot(struct
BUG_ON(preemptible());
- state = &__get_cpu_var(xen_runstate);
+ state = this_cpu_ptr(&xen_runstate);
/*
* The runstate info is always updated by the hypervisor on
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 01/31] x86: Use this_cpu_inc/dec for debug registers
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
I think this patch already exists in one form or another in some x86 tree.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/x86/include/asm/debugreg.h
===================================================================
--- linux.orig/arch/x86/include/asm/debugreg.h 2013-08-06 13:52:23.740092873 -0500
+++ linux/arch/x86/include/asm/debugreg.h 2013-08-06 13:52:23.740092873 -0500
@@ -97,11 +97,11 @@ extern void hw_breakpoint_restore(void);
DECLARE_PER_CPU(int, debug_stack_usage);
static inline void debug_stack_usage_inc(void)
{
- __get_cpu_var(debug_stack_usage)++;
+ __this_cpu_inc(debug_stack_usage);
}
static inline void debug_stack_usage_dec(void)
{
- __get_cpu_var(debug_stack_usage)--;
+ __this_cpu_dec(debug_stack_usage);
}
int is_debug_stack(unsigned long addr);
void debug_stack_set_zero(void);
Index: linux/arch/x86/kernel/cpu/common.c
===================================================================
--- linux.orig/arch/x86/kernel/cpu/common.c 2013-07-19 09:06:36.850047837 -0500
+++ linux/arch/x86/kernel/cpu/common.c 2013-08-06 13:54:36.426384889 -0500
@@ -1144,9 +1144,9 @@ DEFINE_PER_CPU(int, debug_stack_usage);
int is_debug_stack(unsigned long addr)
{
- return __get_cpu_var(debug_stack_usage) ||
- (addr <= __get_cpu_var(debug_stack_addr) &&
- addr > (__get_cpu_var(debug_stack_addr) - DEBUG_STKSZ));
+ return __this_cpu_read(debug_stack_usage) ||
+ (addr <= __this_cpu_read(debug_stack_addr) &&
+ addr > (__this_cpu_read(debug_stack_addr) - DEBUG_STKSZ));
}
DEFINE_PER_CPU(u32, debug_idt_ctr);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 19/31] ia64: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Tony Luck, Fenghua Yu, linux-ia64, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/alpha/kernel/perf_event.c
===================================================================
--- linux.orig/arch/alpha/kernel/perf_event.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/alpha/kernel/perf_event.c 2013-08-23 09:10:22.665620124 -0500
@@ -803,7 +803,7 @@ static void alpha_perf_event_irq_handler
struct hw_perf_event *hwc;
int idx, j;
- __get_cpu_var(irq_pmi_count)++;
+ __this_cpu_inc(irq_pmi_count);
cpuc = &__get_cpu_var(cpu_hw_events);
/* Completely counting through the PMC's period to trigger a new PMC
Index: linux/arch/ia64/kernel/irq.c
===================================================================
--- linux.orig/arch/ia64/kernel/irq.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/irq.c 2013-08-23 09:10:22.665620124 -0500
@@ -42,7 +42,7 @@ ia64_vector __ia64_irq_to_vector(int irq
unsigned int __ia64_local_vector_to_irq (ia64_vector vec)
{
- return __get_cpu_var(vector_irq)[vec];
+ return __this_cpu_read(vector_irq[vec]);
}
#endif
Index: linux/arch/ia64/kernel/irq_ia64.c
===================================================================
--- linux.orig/arch/ia64/kernel/irq_ia64.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/irq_ia64.c 2013-08-23 09:10:22.669620084 -0500
@@ -338,7 +338,7 @@ static irqreturn_t smp_irq_move_cleanup_
int irq;
struct irq_desc *desc;
struct irq_cfg *cfg;
- irq = __get_cpu_var(vector_irq)[vector];
+ irq = __this_cpu_read(vector_irq[vector]);
if (irq < 0)
continue;
@@ -352,7 +352,7 @@ static irqreturn_t smp_irq_move_cleanup_
goto unlock;
spin_lock_irqsave(&vector_lock, flags);
- __get_cpu_var(vector_irq)[vector] = -1;
+ __this_cpu_write(vector_irq[vector], -1);
cpu_clear(me, vector_table[vector]);
spin_unlock_irqrestore(&vector_lock, flags);
cfg->move_cleanup_count--;
Index: linux/arch/ia64/kernel/kprobes.c
===================================================================
--- linux.orig/arch/ia64/kernel/kprobes.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/kprobes.c 2013-08-23 09:10:22.669620084 -0500
@@ -396,7 +396,7 @@ static void __kprobes restore_previous_k
{
unsigned int i;
i = atomic_read(&kcb->prev_kprobe_index);
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe[i-1].kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe[i-1].kp);
kcb->kprobe_status = kcb->prev_kprobe[i-1].status;
atomic_sub(1, &kcb->prev_kprobe_index);
}
@@ -404,7 +404,7 @@ static void __kprobes restore_previous_k
static void __kprobes set_current_kprobe(struct kprobe *p,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static void kretprobe_trampoline(void)
@@ -823,7 +823,7 @@ static int __kprobes pre_kprobes_handler
/*
* jprobe instrumented function just completed
*/
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
goto ss_probe;
}
Index: linux/arch/ia64/kernel/mca.c
===================================================================
--- linux.orig/arch/ia64/kernel/mca.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/mca.c 2013-08-23 09:11:54.244715775 -0500
@@ -1341,7 +1341,7 @@ ia64_mca_handler(struct pt_regs *regs, s
ia64_mlogbuf_finish(1);
}
- if (__get_cpu_var(ia64_mca_tr_reload)) {
+ if (__this_cpu_read(ia64_mca_tr_reload)) {
mca_insert_tr(0x1); /*Reload dynamic itrs*/
mca_insert_tr(0x2); /*Reload dynamic itrs*/
}
@@ -1874,14 +1874,14 @@ ia64_mca_cpu_init(void *cpu_data)
"MCA", cpu);
format_mca_init_stack(data, offsetof(struct ia64_mca_cpu, init_stack),
"INIT", cpu);
- __get_cpu_var(ia64_mca_data) = __per_cpu_mca[cpu] = __pa(data);
+ __this_cpu_write(ia64_mca_data, __per_cpu_mca[cpu] = __pa(data));
/*
* Stash away a copy of the PTE needed to map the per-CPU page.
* We may need it during MCA recovery.
*/
- __get_cpu_var(ia64_mca_per_cpu_pte) =
- pte_val(mk_pte_phys(__pa(cpu_data), PAGE_KERNEL));
+ __this_cpu_write(ia64_mca_per_cpu_pte,
+ pte_val(mk_pte_phys(__pa(cpu_data), PAGE_KERNEL)));
/*
* Also, stash away a copy of the PAL address and the PTE
@@ -1890,10 +1890,10 @@ ia64_mca_cpu_init(void *cpu_data)
pal_vaddr = efi_get_pal_addr();
if (!pal_vaddr)
return;
- __get_cpu_var(ia64_mca_pal_base) =
- GRANULEROUNDDOWN((unsigned long) pal_vaddr);
- __get_cpu_var(ia64_mca_pal_pte) = pte_val(mk_pte_phys(__pa(pal_vaddr),
- PAGE_KERNEL));
+ __this_cpu_write(ia64_mca_pal_base,
+ GRANULEROUNDDOWN((unsigned long) pal_vaddr));
+ __this_cpu_write(ia64_mca_pal_pte, pte_val(mk_pte_phys(__pa(pal_vaddr),
+ PAGE_KERNEL)));
}
static void ia64_mca_cmc_vector_adjust(void *dummy)
Index: linux/arch/ia64/kernel/process.c
===================================================================
--- linux.orig/arch/ia64/kernel/process.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/process.c 2013-08-23 09:10:22.669620084 -0500
@@ -215,7 +215,7 @@ static inline void play_dead(void)
unsigned int this_cpu = smp_processor_id();
/* Ack it */
- __get_cpu_var(cpu_state) = CPU_DEAD;
+ __this_cpu_write(cpu_state, CPU_DEAD);
max_xtp();
local_irq_disable();
@@ -273,7 +273,7 @@ ia64_save_extra (struct task_struct *tas
if ((task->thread.flags & IA64_THREAD_PM_VALID) != 0)
pfm_save_regs(task);
- info = __get_cpu_var(pfm_syst_info);
+ info = __this_cpu_read(pfm_syst_info);
if (info & PFM_CPUINFO_SYST_WIDE)
pfm_syst_wide_update_task(task, info, 0);
#endif
@@ -293,7 +293,7 @@ ia64_load_extra (struct task_struct *tas
if ((task->thread.flags & IA64_THREAD_PM_VALID) != 0)
pfm_load_regs(task);
- info = __get_cpu_var(pfm_syst_info);
+ info = __this_cpu_read(pfm_syst_info);
if (info & PFM_CPUINFO_SYST_WIDE)
pfm_syst_wide_update_task(task, info, 1);
#endif
Index: linux/arch/ia64/sn/kernel/sn2/sn2_smp.c
===================================================================
--- linux.orig/arch/ia64/sn/kernel/sn2/sn2_smp.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/sn/kernel/sn2/sn2_smp.c 2013-08-23 09:10:22.669620084 -0500
@@ -134,8 +134,8 @@ sn2_ipi_flush_all_tlb(struct mm_struct *
itc = ia64_get_itc();
smp_flush_tlb_cpumask(*mm_cpumask(mm));
itc = ia64_get_itc() - itc;
- __get_cpu_var(ptcstats).shub_ipi_flushes_itc_clocks += itc;
- __get_cpu_var(ptcstats).shub_ipi_flushes++;
+ __this_cpu_add(ptcstats.shub_ipi_flushes_itc_clocks, itc);
+ __this_cpu_inc(ptcstats.shub_ipi_flushes);
}
/**
@@ -199,14 +199,14 @@ sn2_global_tlb_purge(struct mm_struct *m
start += (1UL << nbits);
} while (start < end);
ia64_srlz_i();
- __get_cpu_var(ptcstats).ptc_l++;
+ __this_cpu_inc(ptcstats.ptc_l);
preempt_enable();
return;
}
if (atomic_read(&mm->mm_users) == 1 && mymm) {
flush_tlb_mm(mm);
- __get_cpu_var(ptcstats).change_rid++;
+ __this_cpu_inc(ptcstats.change_rid);
preempt_enable();
return;
}
@@ -250,11 +250,11 @@ sn2_global_tlb_purge(struct mm_struct *m
spin_lock_irqsave(PTC_LOCK(shub1), flags);
itc2 = ia64_get_itc();
- __get_cpu_var(ptcstats).lock_itc_clocks += itc2 - itc;
- __get_cpu_var(ptcstats).shub_ptc_flushes++;
- __get_cpu_var(ptcstats).nodes_flushed += nix;
+ __this_cpu_add(ptcstats.lock_itc_clocks, itc2 - itc);
+ __this_cpu_inc(ptcstats.shub_ptc_flushes);
+ __this_cpu_add(ptcstats.nodes_flushed, nix);
if (!mymm)
- __get_cpu_var(ptcstats).shub_ptc_flushes_not_my_mm++;
+ __this_cpu_inc(ptcstats.shub_ptc_flushes_not_my_mm);
if (use_cpu_ptcga && !mymm) {
old_rr = ia64_get_rr(start);
@@ -299,9 +299,9 @@ sn2_global_tlb_purge(struct mm_struct *m
done:
itc2 = ia64_get_itc() - itc2;
- __get_cpu_var(ptcstats).shub_itc_clocks += itc2;
- if (itc2 > __get_cpu_var(ptcstats).shub_itc_clocks_max)
- __get_cpu_var(ptcstats).shub_itc_clocks_max = itc2;
+ __this_cpu_add(ptcstats)shub_itc_clocks, itc2);
+ if (itc2 > __this_cpu_read(ptcstats.shub_itc_clocks_max))
+ __this_cpu_write(ptcstats).shub_itc_clocks_max, itc2);
if (old_rr) {
ia64_set_rr(start, old_rr);
@@ -311,7 +311,7 @@ done:
spin_unlock_irqrestore(PTC_LOCK(shub1), flags);
if (flush_opt == 1 && deadlock) {
- __get_cpu_var(ptcstats).deadlocks++;
+ __this_cpu_inc(ptcstats.deadlocks);
sn2_ipi_flush_all_tlb(mm);
}
@@ -334,7 +334,7 @@ sn2_ptc_deadlock_recovery(short *nasids,
short nasid, i;
unsigned long *piows, zeroval, n;
- __get_cpu_var(ptcstats).deadlocks++;
+ __this_cpu_inc(ptcstats.deadlocks);
piows = (unsigned long *) pda->pio_write_status_addr;
zeroval = pda->pio_write_status_val;
@@ -349,7 +349,7 @@ sn2_ptc_deadlock_recovery(short *nasids,
ptc1 = CHANGE_NASID(nasid, ptc1);
n = sn2_ptc_deadlock_recovery_core(ptc0, data0, ptc1, data1, piows, zeroval);
- __get_cpu_var(ptcstats).deadlocks2 += n;
+ __this_cpu_add(ptcstats.deadlocks2, n);
}
}
Index: linux/arch/ia64/include/asm/hw_irq.h
===================================================================
--- linux.orig/arch/ia64/include/asm/hw_irq.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/hw_irq.h 2013-08-23 09:10:22.673620044 -0500
@@ -160,7 +160,7 @@ static inline ia64_vector __ia64_irq_to_
static inline unsigned int
__ia64_local_vector_to_irq (ia64_vector vec)
{
- return __get_cpu_var(vector_irq)[vec];
+ return __this_cpu_read(vector_irq[vec]);
}
#endif
Index: linux/arch/ia64/include/asm/sn/nodepda.h
===================================================================
--- linux.orig/arch/ia64/include/asm/sn/nodepda.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/sn/nodepda.h 2013-08-23 09:10:22.673620044 -0500
@@ -70,7 +70,7 @@ typedef struct nodepda_s nodepda_t;
*/
DECLARE_PER_CPU(struct nodepda_s *, __sn_nodepda);
-#define sn_nodepda (__get_cpu_var(__sn_nodepda))
+#define sn_nodepda __this_cpu_read(__sn_nodepda)
#define NODEPDA(cnodeid) (sn_nodepda->pernode_pdaindr[cnodeid])
/*
Index: linux/arch/ia64/include/asm/switch_to.h
===================================================================
--- linux.orig/arch/ia64/include/asm/switch_to.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/switch_to.h 2013-08-23 09:10:22.673620044 -0500
@@ -32,7 +32,7 @@ extern void ia64_load_extra (struct task
#ifdef CONFIG_PERFMON
DECLARE_PER_CPU(unsigned long, pfm_syst_info);
-# define PERFMON_IS_SYSWIDE() (__get_cpu_var(pfm_syst_info) & 0x1)
+# define PERFMON_IS_SYSWIDE() (__this_cpu_read(pfm_syst_info) & 0x1)
#else
# define PERFMON_IS_SYSWIDE() (0)
#endif
Index: linux/arch/ia64/include/asm/sn/arch.h
===================================================================
--- linux.orig/arch/ia64/include/asm/sn/arch.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/sn/arch.h 2013-08-23 09:10:22.673620044 -0500
@@ -57,7 +57,7 @@ struct sn_hub_info_s {
u16 nasid_bitmask;
};
DECLARE_PER_CPU(struct sn_hub_info_s, __sn_hub_info);
-#define sn_hub_info (&__get_cpu_var(__sn_hub_info))
+#define sn_hub_info this_cpu_ptr(&__sn_hub_info)
#define is_shub2() (sn_hub_info->shub2)
#define is_shub1() (sn_hub_info->shub2 == 0)
@@ -72,7 +72,7 @@ DECLARE_PER_CPU(struct sn_hub_info_s, __
* cpu.
*/
DECLARE_PER_CPU(short, __sn_cnodeid_to_nasid[MAX_COMPACT_NODES]);
-#define sn_cnodeid_to_nasid (&__get_cpu_var(__sn_cnodeid_to_nasid[0]))
+#define sn_cnodeid_to_nasid this_cpu_ptr(&__sn_cnodeid_to_nasid[0])
extern u8 sn_partition_id;
Index: linux/arch/ia64/include/asm/uv/uv_hub.h
===================================================================
--- linux.orig/arch/ia64/include/asm/uv/uv_hub.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/uv/uv_hub.h 2013-08-23 09:10:22.673620044 -0500
@@ -108,7 +108,7 @@ struct uv_hub_info_s {
unsigned char n_val;
};
DECLARE_PER_CPU(struct uv_hub_info_s, __uv_hub_info);
-#define uv_hub_info (&__get_cpu_var(__uv_hub_info))
+#define uv_hub_info this_cpu_ptr(&__uv_hub_info)
#define uv_cpu_hub_info(cpu) (&per_cpu(__uv_hub_info, cpu))
/*
Index: linux/arch/ia64/kernel/traps.c
===================================================================
--- linux.orig/arch/ia64/kernel/traps.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/traps.c 2013-08-23 09:10:22.673620044 -0500
@@ -299,7 +299,7 @@ handle_fpu_swa (int fp_fault, struct pt_
if (!(current->thread.flags & IA64_THREAD_FPEMU_NOPRINT)) {
unsigned long count, current_jiffies = jiffies;
- struct fpu_swa_msg *cp = &__get_cpu_var(cpulast);
+ struct fpu_swa_msg *cp = this_cpu_ptr(&cpulast);
if (unlikely(current_jiffies > cp->time))
cp->count = 0;
Index: linux/arch/ia64/xen/time.c
===================================================================
--- linux.orig/arch/ia64/xen/time.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/xen/time.c 2013-08-23 09:10:22.673620044 -0500
@@ -68,7 +68,7 @@ static void get_runstate_snapshot(struct
BUG_ON(preemptible());
- state = &__get_cpu_var(xen_runstate);
+ state = this_cpu_ptr(&xen_runstate);
/*
* The runstate info is always updated by the hypervisor on
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 19/31] ia64: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Tony Luck, Fenghua Yu, linux-ia64
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/alpha/kernel/perf_event.c
=================================--- linux.orig/arch/alpha/kernel/perf_event.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/alpha/kernel/perf_event.c 2013-08-23 09:10:22.665620124 -0500
@@ -803,7 +803,7 @@ static void alpha_perf_event_irq_handler
struct hw_perf_event *hwc;
int idx, j;
- __get_cpu_var(irq_pmi_count)++;
+ __this_cpu_inc(irq_pmi_count);
cpuc = &__get_cpu_var(cpu_hw_events);
/* Completely counting through the PMC's period to trigger a new PMC
Index: linux/arch/ia64/kernel/irq.c
=================================--- linux.orig/arch/ia64/kernel/irq.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/irq.c 2013-08-23 09:10:22.665620124 -0500
@@ -42,7 +42,7 @@ ia64_vector __ia64_irq_to_vector(int irq
unsigned int __ia64_local_vector_to_irq (ia64_vector vec)
{
- return __get_cpu_var(vector_irq)[vec];
+ return __this_cpu_read(vector_irq[vec]);
}
#endif
Index: linux/arch/ia64/kernel/irq_ia64.c
=================================--- linux.orig/arch/ia64/kernel/irq_ia64.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/irq_ia64.c 2013-08-23 09:10:22.669620084 -0500
@@ -338,7 +338,7 @@ static irqreturn_t smp_irq_move_cleanup_
int irq;
struct irq_desc *desc;
struct irq_cfg *cfg;
- irq = __get_cpu_var(vector_irq)[vector];
+ irq = __this_cpu_read(vector_irq[vector]);
if (irq < 0)
continue;
@@ -352,7 +352,7 @@ static irqreturn_t smp_irq_move_cleanup_
goto unlock;
spin_lock_irqsave(&vector_lock, flags);
- __get_cpu_var(vector_irq)[vector] = -1;
+ __this_cpu_write(vector_irq[vector], -1);
cpu_clear(me, vector_table[vector]);
spin_unlock_irqrestore(&vector_lock, flags);
cfg->move_cleanup_count--;
Index: linux/arch/ia64/kernel/kprobes.c
=================================--- linux.orig/arch/ia64/kernel/kprobes.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/kprobes.c 2013-08-23 09:10:22.669620084 -0500
@@ -396,7 +396,7 @@ static void __kprobes restore_previous_k
{
unsigned int i;
i = atomic_read(&kcb->prev_kprobe_index);
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe[i-1].kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe[i-1].kp);
kcb->kprobe_status = kcb->prev_kprobe[i-1].status;
atomic_sub(1, &kcb->prev_kprobe_index);
}
@@ -404,7 +404,7 @@ static void __kprobes restore_previous_k
static void __kprobes set_current_kprobe(struct kprobe *p,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static void kretprobe_trampoline(void)
@@ -823,7 +823,7 @@ static int __kprobes pre_kprobes_handler
/*
* jprobe instrumented function just completed
*/
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
goto ss_probe;
}
Index: linux/arch/ia64/kernel/mca.c
=================================--- linux.orig/arch/ia64/kernel/mca.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/mca.c 2013-08-23 09:11:54.244715775 -0500
@@ -1341,7 +1341,7 @@ ia64_mca_handler(struct pt_regs *regs, s
ia64_mlogbuf_finish(1);
}
- if (__get_cpu_var(ia64_mca_tr_reload)) {
+ if (__this_cpu_read(ia64_mca_tr_reload)) {
mca_insert_tr(0x1); /*Reload dynamic itrs*/
mca_insert_tr(0x2); /*Reload dynamic itrs*/
}
@@ -1874,14 +1874,14 @@ ia64_mca_cpu_init(void *cpu_data)
"MCA", cpu);
format_mca_init_stack(data, offsetof(struct ia64_mca_cpu, init_stack),
"INIT", cpu);
- __get_cpu_var(ia64_mca_data) = __per_cpu_mca[cpu] = __pa(data);
+ __this_cpu_write(ia64_mca_data, __per_cpu_mca[cpu] = __pa(data));
/*
* Stash away a copy of the PTE needed to map the per-CPU page.
* We may need it during MCA recovery.
*/
- __get_cpu_var(ia64_mca_per_cpu_pte) - pte_val(mk_pte_phys(__pa(cpu_data), PAGE_KERNEL));
+ __this_cpu_write(ia64_mca_per_cpu_pte,
+ pte_val(mk_pte_phys(__pa(cpu_data), PAGE_KERNEL)));
/*
* Also, stash away a copy of the PAL address and the PTE
@@ -1890,10 +1890,10 @@ ia64_mca_cpu_init(void *cpu_data)
pal_vaddr = efi_get_pal_addr();
if (!pal_vaddr)
return;
- __get_cpu_var(ia64_mca_pal_base) - GRANULEROUNDDOWN((unsigned long) pal_vaddr);
- __get_cpu_var(ia64_mca_pal_pte) = pte_val(mk_pte_phys(__pa(pal_vaddr),
- PAGE_KERNEL));
+ __this_cpu_write(ia64_mca_pal_base,
+ GRANULEROUNDDOWN((unsigned long) pal_vaddr));
+ __this_cpu_write(ia64_mca_pal_pte, pte_val(mk_pte_phys(__pa(pal_vaddr),
+ PAGE_KERNEL)));
}
static void ia64_mca_cmc_vector_adjust(void *dummy)
Index: linux/arch/ia64/kernel/process.c
=================================--- linux.orig/arch/ia64/kernel/process.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/process.c 2013-08-23 09:10:22.669620084 -0500
@@ -215,7 +215,7 @@ static inline void play_dead(void)
unsigned int this_cpu = smp_processor_id();
/* Ack it */
- __get_cpu_var(cpu_state) = CPU_DEAD;
+ __this_cpu_write(cpu_state, CPU_DEAD);
max_xtp();
local_irq_disable();
@@ -273,7 +273,7 @@ ia64_save_extra (struct task_struct *tas
if ((task->thread.flags & IA64_THREAD_PM_VALID) != 0)
pfm_save_regs(task);
- info = __get_cpu_var(pfm_syst_info);
+ info = __this_cpu_read(pfm_syst_info);
if (info & PFM_CPUINFO_SYST_WIDE)
pfm_syst_wide_update_task(task, info, 0);
#endif
@@ -293,7 +293,7 @@ ia64_load_extra (struct task_struct *tas
if ((task->thread.flags & IA64_THREAD_PM_VALID) != 0)
pfm_load_regs(task);
- info = __get_cpu_var(pfm_syst_info);
+ info = __this_cpu_read(pfm_syst_info);
if (info & PFM_CPUINFO_SYST_WIDE)
pfm_syst_wide_update_task(task, info, 1);
#endif
Index: linux/arch/ia64/sn/kernel/sn2/sn2_smp.c
=================================--- linux.orig/arch/ia64/sn/kernel/sn2/sn2_smp.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/sn/kernel/sn2/sn2_smp.c 2013-08-23 09:10:22.669620084 -0500
@@ -134,8 +134,8 @@ sn2_ipi_flush_all_tlb(struct mm_struct *
itc = ia64_get_itc();
smp_flush_tlb_cpumask(*mm_cpumask(mm));
itc = ia64_get_itc() - itc;
- __get_cpu_var(ptcstats).shub_ipi_flushes_itc_clocks += itc;
- __get_cpu_var(ptcstats).shub_ipi_flushes++;
+ __this_cpu_add(ptcstats.shub_ipi_flushes_itc_clocks, itc);
+ __this_cpu_inc(ptcstats.shub_ipi_flushes);
}
/**
@@ -199,14 +199,14 @@ sn2_global_tlb_purge(struct mm_struct *m
start += (1UL << nbits);
} while (start < end);
ia64_srlz_i();
- __get_cpu_var(ptcstats).ptc_l++;
+ __this_cpu_inc(ptcstats.ptc_l);
preempt_enable();
return;
}
if (atomic_read(&mm->mm_users) = 1 && mymm) {
flush_tlb_mm(mm);
- __get_cpu_var(ptcstats).change_rid++;
+ __this_cpu_inc(ptcstats.change_rid);
preempt_enable();
return;
}
@@ -250,11 +250,11 @@ sn2_global_tlb_purge(struct mm_struct *m
spin_lock_irqsave(PTC_LOCK(shub1), flags);
itc2 = ia64_get_itc();
- __get_cpu_var(ptcstats).lock_itc_clocks += itc2 - itc;
- __get_cpu_var(ptcstats).shub_ptc_flushes++;
- __get_cpu_var(ptcstats).nodes_flushed += nix;
+ __this_cpu_add(ptcstats.lock_itc_clocks, itc2 - itc);
+ __this_cpu_inc(ptcstats.shub_ptc_flushes);
+ __this_cpu_add(ptcstats.nodes_flushed, nix);
if (!mymm)
- __get_cpu_var(ptcstats).shub_ptc_flushes_not_my_mm++;
+ __this_cpu_inc(ptcstats.shub_ptc_flushes_not_my_mm);
if (use_cpu_ptcga && !mymm) {
old_rr = ia64_get_rr(start);
@@ -299,9 +299,9 @@ sn2_global_tlb_purge(struct mm_struct *m
done:
itc2 = ia64_get_itc() - itc2;
- __get_cpu_var(ptcstats).shub_itc_clocks += itc2;
- if (itc2 > __get_cpu_var(ptcstats).shub_itc_clocks_max)
- __get_cpu_var(ptcstats).shub_itc_clocks_max = itc2;
+ __this_cpu_add(ptcstats)shub_itc_clocks, itc2);
+ if (itc2 > __this_cpu_read(ptcstats.shub_itc_clocks_max))
+ __this_cpu_write(ptcstats).shub_itc_clocks_max, itc2);
if (old_rr) {
ia64_set_rr(start, old_rr);
@@ -311,7 +311,7 @@ done:
spin_unlock_irqrestore(PTC_LOCK(shub1), flags);
if (flush_opt = 1 && deadlock) {
- __get_cpu_var(ptcstats).deadlocks++;
+ __this_cpu_inc(ptcstats.deadlocks);
sn2_ipi_flush_all_tlb(mm);
}
@@ -334,7 +334,7 @@ sn2_ptc_deadlock_recovery(short *nasids,
short nasid, i;
unsigned long *piows, zeroval, n;
- __get_cpu_var(ptcstats).deadlocks++;
+ __this_cpu_inc(ptcstats.deadlocks);
piows = (unsigned long *) pda->pio_write_status_addr;
zeroval = pda->pio_write_status_val;
@@ -349,7 +349,7 @@ sn2_ptc_deadlock_recovery(short *nasids,
ptc1 = CHANGE_NASID(nasid, ptc1);
n = sn2_ptc_deadlock_recovery_core(ptc0, data0, ptc1, data1, piows, zeroval);
- __get_cpu_var(ptcstats).deadlocks2 += n;
+ __this_cpu_add(ptcstats.deadlocks2, n);
}
}
Index: linux/arch/ia64/include/asm/hw_irq.h
=================================--- linux.orig/arch/ia64/include/asm/hw_irq.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/hw_irq.h 2013-08-23 09:10:22.673620044 -0500
@@ -160,7 +160,7 @@ static inline ia64_vector __ia64_irq_to_
static inline unsigned int
__ia64_local_vector_to_irq (ia64_vector vec)
{
- return __get_cpu_var(vector_irq)[vec];
+ return __this_cpu_read(vector_irq[vec]);
}
#endif
Index: linux/arch/ia64/include/asm/sn/nodepda.h
=================================--- linux.orig/arch/ia64/include/asm/sn/nodepda.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/sn/nodepda.h 2013-08-23 09:10:22.673620044 -0500
@@ -70,7 +70,7 @@ typedef struct nodepda_s nodepda_t;
*/
DECLARE_PER_CPU(struct nodepda_s *, __sn_nodepda);
-#define sn_nodepda (__get_cpu_var(__sn_nodepda))
+#define sn_nodepda __this_cpu_read(__sn_nodepda)
#define NODEPDA(cnodeid) (sn_nodepda->pernode_pdaindr[cnodeid])
/*
Index: linux/arch/ia64/include/asm/switch_to.h
=================================--- linux.orig/arch/ia64/include/asm/switch_to.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/switch_to.h 2013-08-23 09:10:22.673620044 -0500
@@ -32,7 +32,7 @@ extern void ia64_load_extra (struct task
#ifdef CONFIG_PERFMON
DECLARE_PER_CPU(unsigned long, pfm_syst_info);
-# define PERFMON_IS_SYSWIDE() (__get_cpu_var(pfm_syst_info) & 0x1)
+# define PERFMON_IS_SYSWIDE() (__this_cpu_read(pfm_syst_info) & 0x1)
#else
# define PERFMON_IS_SYSWIDE() (0)
#endif
Index: linux/arch/ia64/include/asm/sn/arch.h
=================================--- linux.orig/arch/ia64/include/asm/sn/arch.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/sn/arch.h 2013-08-23 09:10:22.673620044 -0500
@@ -57,7 +57,7 @@ struct sn_hub_info_s {
u16 nasid_bitmask;
};
DECLARE_PER_CPU(struct sn_hub_info_s, __sn_hub_info);
-#define sn_hub_info (&__get_cpu_var(__sn_hub_info))
+#define sn_hub_info this_cpu_ptr(&__sn_hub_info)
#define is_shub2() (sn_hub_info->shub2)
#define is_shub1() (sn_hub_info->shub2 = 0)
@@ -72,7 +72,7 @@ DECLARE_PER_CPU(struct sn_hub_info_s, __
* cpu.
*/
DECLARE_PER_CPU(short, __sn_cnodeid_to_nasid[MAX_COMPACT_NODES]);
-#define sn_cnodeid_to_nasid (&__get_cpu_var(__sn_cnodeid_to_nasid[0]))
+#define sn_cnodeid_to_nasid this_cpu_ptr(&__sn_cnodeid_to_nasid[0])
extern u8 sn_partition_id;
Index: linux/arch/ia64/include/asm/uv/uv_hub.h
=================================--- linux.orig/arch/ia64/include/asm/uv/uv_hub.h 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/include/asm/uv/uv_hub.h 2013-08-23 09:10:22.673620044 -0500
@@ -108,7 +108,7 @@ struct uv_hub_info_s {
unsigned char n_val;
};
DECLARE_PER_CPU(struct uv_hub_info_s, __uv_hub_info);
-#define uv_hub_info (&__get_cpu_var(__uv_hub_info))
+#define uv_hub_info this_cpu_ptr(&__uv_hub_info)
#define uv_cpu_hub_info(cpu) (&per_cpu(__uv_hub_info, cpu))
/*
Index: linux/arch/ia64/kernel/traps.c
=================================--- linux.orig/arch/ia64/kernel/traps.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/kernel/traps.c 2013-08-23 09:10:22.673620044 -0500
@@ -299,7 +299,7 @@ handle_fpu_swa (int fp_fault, struct pt_
if (!(current->thread.flags & IA64_THREAD_FPEMU_NOPRINT)) {
unsigned long count, current_jiffies = jiffies;
- struct fpu_swa_msg *cp = &__get_cpu_var(cpulast);
+ struct fpu_swa_msg *cp = this_cpu_ptr(&cpulast);
if (unlikely(current_jiffies > cp->time))
cp->count = 0;
Index: linux/arch/ia64/xen/time.c
=================================--- linux.orig/arch/ia64/xen/time.c 2013-08-23 09:10:22.673620044 -0500
+++ linux/arch/ia64/xen/time.c 2013-08-23 09:10:22.673620044 -0500
@@ -68,7 +68,7 @@ static void get_runstate_snapshot(struct
BUG_ON(preemptible());
- state = &__get_cpu_var(xen_runstate);
+ state = this_cpu_ptr(&xen_runstate);
/*
* The runstate info is always updated by the hypervisor on
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 20/31] powerpc: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Benjamin Herrenschmidt, Paul Mackerras, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/powerpc/include/asm/cputime.h
===================================================================
--- linux.orig/arch/powerpc/include/asm/cputime.h 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/include/asm/cputime.h 2013-08-22 14:56:47.935059693 -0500
@@ -56,10 +56,10 @@ static inline unsigned long cputime_to_j
static inline cputime_t cputime_to_scaled(const cputime_t ct)
{
if (cpu_has_feature(CPU_FTR_SPURR) &&
- __get_cpu_var(cputime_last_delta))
+ __this_cpu_read(cputime_last_delta))
return (__force u64) ct *
- __get_cpu_var(cputime_scaled_last_delta) /
- __get_cpu_var(cputime_last_delta);
+ __this_cpu_read(cputime_scaled_last_delta) /
+ __this_cpu_read(cputime_last_delta);
return ct;
}
Index: linux/arch/powerpc/include/asm/hardirq.h
===================================================================
--- linux.orig/arch/powerpc/include/asm/hardirq.h 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/include/asm/hardirq.h 2013-08-22 14:56:47.935059693 -0500
@@ -19,7 +19,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpust
#define __ARCH_IRQ_STAT
-#define local_softirq_pending() __get_cpu_var(irq_stat).__softirq_pending
+#define local_softirq_pending() __this_cpu_read(irq_stat.__softirq_pending)
static inline void ack_bad_irq(unsigned int irq)
{
Index: linux/arch/powerpc/include/asm/tlbflush.h
===================================================================
--- linux.orig/arch/powerpc/include/asm/tlbflush.h 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/include/asm/tlbflush.h 2013-08-22 14:56:47.935059693 -0500
@@ -107,14 +107,14 @@ extern void __flush_tlb_pending(struct p
static inline void arch_enter_lazy_mmu_mode(void)
{
- struct ppc64_tlb_batch *batch = &__get_cpu_var(ppc64_tlb_batch);
+ struct ppc64_tlb_batch *batch = this_cpu_ptr(&ppc64_tlb_batch);
batch->active = 1;
}
static inline void arch_leave_lazy_mmu_mode(void)
{
- struct ppc64_tlb_batch *batch = &__get_cpu_var(ppc64_tlb_batch);
+ struct ppc64_tlb_batch *batch = this_cpu_ptr(&ppc64_tlb_batch);
if (batch->index)
__flush_tlb_pending(batch);
Index: linux/arch/powerpc/include/asm/xics.h
===================================================================
--- linux.orig/arch/powerpc/include/asm/xics.h 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/include/asm/xics.h 2013-08-22 14:56:47.935059693 -0500
@@ -97,7 +97,7 @@ DECLARE_PER_CPU(struct xics_cppr, xics_c
static inline void xics_push_cppr(unsigned int vec)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
if (WARN_ON(os_cppr->index >= MAX_NUM_PRIORITIES - 1))
return;
@@ -110,7 +110,7 @@ static inline void xics_push_cppr(unsign
static inline unsigned char xics_pop_cppr(void)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
if (WARN_ON(os_cppr->index < 1))
return LOWEST_PRIORITY;
@@ -120,7 +120,7 @@ static inline unsigned char xics_pop_cpp
static inline void xics_set_base_cppr(unsigned char cppr)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
/* we only really want to set the priority when there's
* just one cppr value on the stack
@@ -132,7 +132,7 @@ static inline void xics_set_base_cppr(un
static inline unsigned char xics_cppr_top(void)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
return os_cppr->stack[os_cppr->index];
}
Index: linux/arch/powerpc/kernel/dbell.c
===================================================================
--- linux.orig/arch/powerpc/kernel/dbell.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/dbell.c 2013-08-22 14:56:47.935059693 -0500
@@ -41,7 +41,7 @@ void doorbell_exception(struct pt_regs *
may_hard_irq_enable();
- __get_cpu_var(irq_stat).doorbell_irqs++;
+ __this_cpu_inc(irq_stat.doorbell_irqs);
smp_ipi_demux();
Index: linux/arch/powerpc/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/powerpc/kernel/hw_breakpoint.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/hw_breakpoint.c 2013-08-22 14:56:47.935059693 -0500
@@ -64,7 +64,7 @@ int hw_breakpoint_slots(int type)
int arch_install_hw_breakpoint(struct perf_event *bp)
{
struct arch_hw_breakpoint *info = counter_arch_bp(bp);
- struct perf_event **slot = &__get_cpu_var(bp_per_reg);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg);
*slot = bp;
@@ -89,7 +89,7 @@ int arch_install_hw_breakpoint(struct pe
*/
void arch_uninstall_hw_breakpoint(struct perf_event *bp)
{
- struct perf_event **slot = &__get_cpu_var(bp_per_reg);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg);
if (*slot != bp) {
WARN_ONCE(1, "Can't find the breakpoint");
@@ -227,7 +227,7 @@ int __kprobes hw_breakpoint_handler(stru
*/
rcu_read_lock();
- bp = __get_cpu_var(bp_per_reg);
+ bp = __this_cpu_read(bp_per_reg);
if (!bp)
goto out;
info = counter_arch_bp(bp);
Index: linux/arch/powerpc/kernel/irq.c
===================================================================
--- linux.orig/arch/powerpc/kernel/irq.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/irq.c 2013-08-22 14:56:47.935059693 -0500
@@ -114,7 +114,7 @@ static inline notrace void set_soft_enab
static inline notrace int decrementer_check_overflow(void)
{
u64 now = get_tb_or_rtc();
- u64 *next_tb = &__get_cpu_var(decrementers_next_tb);
+ u64 *next_tb = this_cpu_ptr(&decrementers_next_tb);
return now >= *next_tb;
}
@@ -526,7 +526,7 @@ void do_IRQ(struct pt_regs *regs)
if (irq != NO_IRQ)
handle_one_irq(irq);
else
- __get_cpu_var(irq_stat).spurious_irqs++;
+ __this_cpu_inc(irq_stat.spurious_irqs);
trace_irq_exit(regs);
Index: linux/arch/powerpc/kernel/kprobes.c
===================================================================
--- linux.orig/arch/powerpc/kernel/kprobes.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/kprobes.c 2013-08-22 14:56:47.935059693 -0500
@@ -118,7 +118,7 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
kcb->kprobe_saved_msr = kcb->prev_kprobe.saved_msr;
}
@@ -126,7 +126,7 @@ static void __kprobes restore_previous_k
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
kcb->kprobe_saved_msr = regs->msr;
}
@@ -191,7 +191,7 @@ static int __kprobes kprobe_handler(stru
ret = 1;
goto no_kprobe;
}
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
goto ss_probe;
}
Index: linux/arch/powerpc/kernel/process.c
===================================================================
--- linux.orig/arch/powerpc/kernel/process.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/process.c 2013-08-22 14:56:47.935059693 -0500
@@ -453,7 +453,7 @@ static inline int set_dawr(struct arch_h
int set_breakpoint(struct arch_hw_breakpoint *brk)
{
- __get_cpu_var(current_brk) = *brk;
+ __this_cpu_write(current_brk, *brk);
if (cpu_has_feature(CPU_FTR_DAWR))
return set_dawr(brk);
@@ -686,7 +686,7 @@ struct task_struct *__switch_to(struct t
* schedule DABR
*/
#ifndef CONFIG_HAVE_HW_BREAKPOINT
- if (unlikely(hw_brk_match(&__get_cpu_var(current_brk), &new->thread.hw_brk)))
+ if (unlikely(hw_brk_match(this_cpu_ptr(¤t_brk), &new->thread.hw_brk)))
set_breakpoint(&new->thread.hw_brk);
#endif /* CONFIG_HAVE_HW_BREAKPOINT */
#endif
@@ -700,7 +700,7 @@ struct task_struct *__switch_to(struct t
* Collect processor utilization data per process
*/
if (firmware_has_feature(FW_FEATURE_SPLPAR)) {
- struct cpu_usage *cu = &__get_cpu_var(cpu_usage_array);
+ struct cpu_usage *cu = this_cpu_ptr(&cpu_usage_array);
long unsigned start_tb, current_tb;
start_tb = old_thread->start_tb;
cu->current_tb = current_tb = mfspr(SPRN_PURR);
@@ -710,7 +710,7 @@ struct task_struct *__switch_to(struct t
#endif /* CONFIG_PPC64 */
#ifdef CONFIG_PPC_BOOK3S_64
- batch = &__get_cpu_var(ppc64_tlb_batch);
+ batch = this_cpu_ptr(&ppc64_tlb_batch);
if (batch->active) {
current_thread_info()->local_flags |= _TLF_LAZY_MMU;
if (batch->index)
@@ -735,7 +735,7 @@ struct task_struct *__switch_to(struct t
#ifdef CONFIG_PPC_BOOK3S_64
if (current_thread_info()->local_flags & _TLF_LAZY_MMU) {
current_thread_info()->local_flags &= ~_TLF_LAZY_MMU;
- batch = &__get_cpu_var(ppc64_tlb_batch);
+ batch = this_cpu_ptr(&ppc64_tlb_batch);
batch->active = 1;
}
#endif /* CONFIG_PPC_BOOK3S_64 */
Index: linux/arch/powerpc/kernel/smp.c
===================================================================
--- linux.orig/arch/powerpc/kernel/smp.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/smp.c 2013-08-22 14:56:47.935059693 -0500
@@ -212,7 +212,7 @@ void smp_muxed_ipi_message_pass(int cpu,
irqreturn_t smp_ipi_demux(void)
{
- struct cpu_messages *info = &__get_cpu_var(ipi_message);
+ struct cpu_messages *info = this_cpu_ptr(&ipi_message);
unsigned int all;
mb(); /* order any irq clear */
@@ -402,9 +402,9 @@ void generic_mach_cpu_die(void)
idle_task_exit();
cpu = smp_processor_id();
printk(KERN_DEBUG "CPU%d offline\n", cpu);
- __get_cpu_var(cpu_state) = CPU_DEAD;
+ __this_cpu_write(cpu_state, CPU_DEAD);
smp_wmb();
- while (__get_cpu_var(cpu_state) != CPU_UP_PREPARE)
+ while (__this_cpu_read(cpu_state) != CPU_UP_PREPARE)
cpu_relax();
}
Index: linux/arch/powerpc/kernel/sysfs.c
===================================================================
--- linux.orig/arch/powerpc/kernel/sysfs.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/sysfs.c 2013-08-22 14:56:47.935059693 -0500
@@ -97,10 +97,10 @@ void ppc_enable_pmcs(void)
ppc_set_pmu_inuse(1);
/* Only need to enable them once */
- if (__get_cpu_var(pmcs_enabled))
+ if (__this_cpu_read(pmcs_enabled))
return;
- __get_cpu_var(pmcs_enabled) = 1;
+ __this_cpu_write(pmcs_enabled, 1);
if (ppc_md.enable_pmcs)
ppc_md.enable_pmcs();
Index: linux/arch/powerpc/kernel/time.c
===================================================================
--- linux.orig/arch/powerpc/kernel/time.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/time.c 2013-08-22 14:56:47.935059693 -0500
@@ -457,9 +457,9 @@ static inline void clear_irq_work_pendin
DEFINE_PER_CPU(u8, irq_work_pending);
-#define set_irq_work_pending_flag() __get_cpu_var(irq_work_pending) = 1
-#define test_irq_work_pending() __get_cpu_var(irq_work_pending)
-#define clear_irq_work_pending() __get_cpu_var(irq_work_pending) = 0
+#define set_irq_work_pending_flag() __this_cpu_write(irq_work_pending, 1)
+#define test_irq_work_pending() __this_cpu_read(irq_work_pending)
+#define clear_irq_work_pending() __this_cpu_write(irq_work_pending, 0)
#endif /* 32 vs 64 bit */
@@ -485,8 +485,8 @@ void arch_irq_work_raise(void)
void timer_interrupt(struct pt_regs * regs)
{
struct pt_regs *old_regs;
- u64 *next_tb = &__get_cpu_var(decrementers_next_tb);
- struct clock_event_device *evt = &__get_cpu_var(decrementers);
+ u64 *next_tb = this_cpu_ptr(&decrementers_next_tb);
+ struct clock_event_device *evt = this_cpu_ptr(&decrementers);
u64 now;
/* Ensure a positive value is written to the decrementer, or else
@@ -510,7 +510,7 @@ void timer_interrupt(struct pt_regs * re
*/
may_hard_irq_enable();
- __get_cpu_var(irq_stat).timer_irqs++;
+ __this_cpu_inc(irq_stat.timer_irqs);
#if defined(CONFIG_PPC32) && defined(CONFIG_PMAC)
if (atomic_read(&ppc_n_lost_interrupts) != 0)
@@ -541,7 +541,7 @@ void timer_interrupt(struct pt_regs * re
#ifdef CONFIG_PPC64
/* collect purr register values often, for accurate calculations */
if (firmware_has_feature(FW_FEATURE_SPLPAR)) {
- struct cpu_usage *cu = &__get_cpu_var(cpu_usage_array);
+ struct cpu_usage *cu = this_cpu_ptr(&cpu_usage_array);
cu->current_tb = mfspr(SPRN_PURR);
}
#endif
@@ -801,7 +801,7 @@ static void __init clocksource_init(void
static int decrementer_set_next_event(unsigned long evt,
struct clock_event_device *dev)
{
- __get_cpu_var(decrementers_next_tb) = get_tb_or_rtc() + evt;
+ __this_cpu_write(decrementers_next_tb, get_tb_or_rtc() + evt);
set_dec(evt);
return 0;
}
Index: linux/arch/powerpc/kernel/traps.c
===================================================================
--- linux.orig/arch/powerpc/kernel/traps.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/traps.c 2013-08-22 14:56:47.935059693 -0500
@@ -670,7 +670,7 @@ void machine_check_exception(struct pt_r
enum ctx_state prev_state = exception_enter();
int recover = 0;
- __get_cpu_var(irq_stat).mce_exceptions++;
+ __this_cpu_inc(irq_stat.mce_exceptions);
/* See if any machine dependent calls. In theory, we would want
* to call the CPU first, and call the ppc_md. one if the CPU
@@ -1436,7 +1436,7 @@ void vsx_unavailable_tm(struct pt_regs *
void performance_monitor_exception(struct pt_regs *regs)
{
- __get_cpu_var(irq_stat).pmu_irqs++;
+ __this_cpu_inc(irq_stat.pmu_irqs);
perf_irq(regs);
}
Index: linux/arch/powerpc/kvm/e500.c
===================================================================
--- linux.orig/arch/powerpc/kvm/e500.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kvm/e500.c 2013-08-22 14:56:47.935059693 -0500
@@ -74,11 +74,11 @@ static inline int local_sid_setup_one(st
unsigned long sid;
int ret = -1;
- sid = ++(__get_cpu_var(pcpu_last_used_sid));
+ sid = __this_cpu_inc_return(pcpu_last_used_sid);
if (sid < NUM_TIDS) {
- __get_cpu_var(pcpu_sids).entry[sid] = entry;
+ __this_cpu_write(pcpu_sids)entry[sid], entry);
entry->val = sid;
- entry->pentry = &__get_cpu_var(pcpu_sids).entry[sid];
+ entry->pentry = this_cpu_ptr(&pcpu_sids.entry[sid]);
ret = sid;
}
@@ -106,8 +106,8 @@ static inline int local_sid_setup_one(st
static inline int local_sid_lookup(struct id *entry)
{
if (entry && entry->val != 0 &&
- __get_cpu_var(pcpu_sids).entry[entry->val] == entry &&
- entry->pentry == &__get_cpu_var(pcpu_sids).entry[entry->val])
+ __this_cpu_read(pcpu_sids.entry[entry->val]) == entry &&
+ entry->pentry == this_cpu_ptr(&pcpu_sids.entry[entry->val]))
return entry->val;
return -1;
}
@@ -115,8 +115,8 @@ static inline int local_sid_lookup(struc
/* Invalidate all id mappings on local core -- call with preempt disabled */
static inline void local_sid_destroy_all(void)
{
- __get_cpu_var(pcpu_last_used_sid) = 0;
- memset(&__get_cpu_var(pcpu_sids), 0, sizeof(__get_cpu_var(pcpu_sids)));
+ __this_cpu_write(pcpu_last_used_sid, 0);
+ memset(this_cpu_ptr(&pcpu_sids), 0, sizeof(pcpu_sids));
}
static void *kvmppc_e500_id_table_alloc(struct kvmppc_vcpu_e500 *vcpu_e500)
Index: linux/arch/powerpc/kvm/e500mc.c
===================================================================
--- linux.orig/arch/powerpc/kvm/e500mc.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kvm/e500mc.c 2013-08-22 14:56:47.935059693 -0500
@@ -139,9 +139,9 @@ void kvmppc_core_vcpu_load(struct kvm_vc
mtspr(SPRN_GESR, vcpu->arch.shared->esr);
if (vcpu->arch.oldpir != mfspr(SPRN_PIR) ||
- __get_cpu_var(last_vcpu_on_cpu) != vcpu) {
+ __this_cpu_read(last_vcpu_on_cpu) != vcpu) {
kvmppc_e500_tlbil_all(vcpu_e500);
- __get_cpu_var(last_vcpu_on_cpu) = vcpu;
+ __this_cpu_read(last_vcpu_on_cpu) = vcpu;
}
kvmppc_load_guest_fp(vcpu);
Index: linux/arch/powerpc/mm/hash_native_64.c
===================================================================
--- linux.orig/arch/powerpc/mm/hash_native_64.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/hash_native_64.c 2013-08-22 14:56:47.935059693 -0500
@@ -643,7 +643,7 @@ static void native_flush_hash_range(unsi
unsigned long want_v;
unsigned long flags;
real_pte_t pte;
- struct ppc64_tlb_batch *batch = &__get_cpu_var(ppc64_tlb_batch);
+ struct ppc64_tlb_batch *batch = this_cpu_ptr(&ppc64_tlb_batch);
unsigned long psize = batch->psize;
int ssize = batch->ssize;
int i;
Index: linux/arch/powerpc/mm/hash_utils_64.c
===================================================================
--- linux.orig/arch/powerpc/mm/hash_utils_64.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/hash_utils_64.c 2013-08-22 14:56:47.935059693 -0500
@@ -1285,7 +1285,7 @@ void flush_hash_range(unsigned long numb
else {
int i;
struct ppc64_tlb_batch *batch =
- &__get_cpu_var(ppc64_tlb_batch);
+ this_cpu_ptr(&ppc64_tlb_batch);
for (i = 0; i < number; i++)
flush_hash_page(batch->vpn[i], batch->pte[i],
Index: linux/arch/powerpc/mm/hugetlbpage-book3e.c
===================================================================
--- linux.orig/arch/powerpc/mm/hugetlbpage-book3e.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/hugetlbpage-book3e.c 2013-08-22 14:56:47.935059693 -0500
@@ -80,14 +80,14 @@ void book3e_hugetlb_preload(struct vm_ar
ncams = mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY;
/* We have to use the CAM(TLB1) on FSL parts for hugepages */
- index = __get_cpu_var(next_tlbcam_idx);
+ index = __this_cpu_read(next_tlbcam_idx);
mtspr(SPRN_MAS0, MAS0_ESEL(index) | MAS0_TLBSEL(1));
/* Just round-robin the entries and wrap when we hit the end */
if (unlikely(index == ncams - 1))
- __get_cpu_var(next_tlbcam_idx) = tlbcam_index;
+ __this_cpu_write(next_tlbcam_idx, tlbcam_index);
else
- __get_cpu_var(next_tlbcam_idx)++;
+ __this_cpu_inc(next_tlbcam_idx);
#endif
mas1 = MAS1_VALID | MAS1_TID(mm->context.id) | MAS1_TSIZE(tsize);
mas2 = ea & ~((1UL << shift) - 1);
Index: linux/arch/powerpc/mm/hugetlbpage.c
===================================================================
--- linux.orig/arch/powerpc/mm/hugetlbpage.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/hugetlbpage.c 2013-08-22 14:56:47.935059693 -0500
@@ -462,7 +462,7 @@ static void hugepd_free(struct mmu_gathe
{
struct hugepd_freelist **batchp;
- batchp = &__get_cpu_var(hugepd_freelist_cur);
+ batchp = this_cpu_ptr(&hugepd_freelist_cur);
if (atomic_read(&tlb->mm->mm_users) < 2 ||
cpumask_equal(mm_cpumask(tlb->mm),
Index: linux/arch/powerpc/mm/stab.c
===================================================================
--- linux.orig/arch/powerpc/mm/stab.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/stab.c 2013-08-22 14:56:47.935059693 -0500
@@ -133,12 +133,12 @@ static int __ste_allocate(unsigned long
stab_entry = make_ste(get_paca()->stab_addr, GET_ESID(ea), vsid);
if (!is_kernel_addr(ea)) {
- offset = __get_cpu_var(stab_cache_ptr);
+ offset = __this_cpu_read(stab_cache_ptr);
if (offset < NR_STAB_CACHE_ENTRIES)
- __get_cpu_var(stab_cache[offset++]) = stab_entry;
+ __this_cpu_read(stab_cache[offset++]) = stab_entry;
else
offset = NR_STAB_CACHE_ENTRIES+1;
- __get_cpu_var(stab_cache_ptr) = offset;
+ __this_cpu_write(stab_cache_ptr, offset);
/* Order update */
asm volatile("sync":::"memory");
@@ -177,12 +177,12 @@ void switch_stab(struct task_struct *tsk
*/
hard_irq_disable();
- offset = __get_cpu_var(stab_cache_ptr);
+ offset = __this_cpu_read(stab_cache_ptr);
if (offset <= NR_STAB_CACHE_ENTRIES) {
int i;
for (i = 0; i < offset; i++) {
- ste = stab + __get_cpu_var(stab_cache[i]);
+ ste = stab + __this_cpu_read(stab_cache[i]);
ste->esid_data = 0; /* invalidate entry */
}
} else {
@@ -206,7 +206,7 @@ void switch_stab(struct task_struct *tsk
asm volatile("sync; slbia; sync":::"memory");
- __get_cpu_var(stab_cache_ptr) = 0;
+ __this_cpu_write(stab_cache_ptr, 0);
/* Now preload some entries for the new task */
if (test_tsk_thread_flag(tsk, TIF_32BIT))
Index: linux/arch/powerpc/perf/core-book3s.c
===================================================================
--- linux.orig/arch/powerpc/perf/core-book3s.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/perf/core-book3s.c 2013-08-22 14:56:47.935059693 -0500
@@ -332,7 +332,7 @@ static void power_pmu_bhrb_reset(void)
static void power_pmu_bhrb_enable(struct perf_event *event)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!ppmu->bhrb_nr)
return;
@@ -347,7 +347,7 @@ static void power_pmu_bhrb_enable(struct
static void power_pmu_bhrb_disable(struct perf_event *event)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!ppmu->bhrb_nr)
return;
@@ -961,7 +961,7 @@ static void power_pmu_disable(struct pmu
if (!ppmu)
return;
local_irq_save(flags);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!cpuhw->disabled) {
/*
@@ -1027,7 +1027,7 @@ static void power_pmu_enable(struct pmu
return;
local_irq_save(flags);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!cpuhw->disabled)
goto out;
@@ -1211,7 +1211,7 @@ static int power_pmu_add(struct perf_eve
* Add the event to the list (if there is room)
* and check whether the total set is still feasible.
*/
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
n0 = cpuhw->n_events;
if (n0 >= ppmu->n_counter)
goto out;
@@ -1277,7 +1277,7 @@ static void power_pmu_del(struct perf_ev
power_pmu_read(event);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
for (i = 0; i < cpuhw->n_events; ++i) {
if (event == cpuhw->event[i]) {
while (++i < cpuhw->n_events) {
@@ -1383,7 +1383,7 @@ static void power_pmu_stop(struct perf_e
*/
void power_pmu_start_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
perf_pmu_disable(pmu);
cpuhw->group_flag |= PERF_EVENT_TXN;
@@ -1397,7 +1397,7 @@ void power_pmu_start_txn(struct pmu *pmu
*/
void power_pmu_cancel_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
cpuhw->group_flag &= ~PERF_EVENT_TXN;
perf_pmu_enable(pmu);
@@ -1415,7 +1415,7 @@ int power_pmu_commit_txn(struct pmu *pmu
if (!ppmu)
return -EAGAIN;
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
n = cpuhw->n_events;
if (check_excludes(cpuhw->event, cpuhw->flags, 0, n))
return -EAGAIN;
@@ -1772,7 +1772,7 @@ static void record_and_restart(struct pe
if (event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK) {
struct cpu_hw_events *cpuhw;
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
power_pmu_bhrb_read(cpuhw);
data.br_stack = &cpuhw->bhrb_stack;
}
@@ -1845,7 +1845,7 @@ static bool pmc_overflow(unsigned long v
static void perf_event_interrupt(struct pt_regs *regs)
{
int i, j;
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event;
unsigned long val[8];
int found, active;
Index: linux/arch/powerpc/perf/core-fsl-emb.c
===================================================================
--- linux.orig/arch/powerpc/perf/core-fsl-emb.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/perf/core-fsl-emb.c 2013-08-22 14:56:47.935059693 -0500
@@ -186,7 +186,7 @@ static void fsl_emb_pmu_disable(struct p
unsigned long flags;
local_irq_save(flags);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!cpuhw->disabled) {
cpuhw->disabled = 1;
@@ -225,7 +225,7 @@ static void fsl_emb_pmu_enable(struct pm
unsigned long flags;
local_irq_save(flags);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!cpuhw->disabled)
goto out;
@@ -623,7 +623,7 @@ static void record_and_restart(struct pe
static void perf_event_interrupt(struct pt_regs *regs)
{
int i;
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event;
unsigned long val;
int found = 0;
Index: linux/arch/powerpc/platforms/cell/interrupt.c
===================================================================
--- linux.orig/arch/powerpc/platforms/cell/interrupt.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/cell/interrupt.c 2013-08-22 14:56:47.935059693 -0500
@@ -82,7 +82,7 @@ static void iic_unmask(struct irq_data *
static void iic_eoi(struct irq_data *d)
{
- struct iic *iic = &__get_cpu_var(cpu_iic);
+ struct iic *iic = this_cpu_ptr(&cpu_iic);
out_be64(&iic->regs->prio, iic->eoi_stack[--iic->eoi_ptr]);
BUG_ON(iic->eoi_ptr < 0);
}
@@ -148,7 +148,7 @@ static unsigned int iic_get_irq(void)
struct iic *iic;
unsigned int virq;
- iic = &__get_cpu_var(cpu_iic);
+ iic = this_cpu_ptr(&cpu_iic);
*(unsigned long *) &pending =
in_be64((u64 __iomem *) &iic->regs->pending_destr);
if (!(pending.flags & CBE_IIC_IRQ_VALID))
@@ -163,7 +163,7 @@ static unsigned int iic_get_irq(void)
void iic_setup_cpu(void)
{
- out_be64(&__get_cpu_var(cpu_iic).regs->prio, 0xff);
+ out_be64(this_cpu_ptr(&cpu_iic.regs->prio), 0xff);
}
u8 iic_get_target_id(int cpu)
Index: linux/arch/powerpc/platforms/ps3/interrupt.c
===================================================================
--- linux.orig/arch/powerpc/platforms/ps3/interrupt.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/ps3/interrupt.c 2013-08-22 14:56:47.935059693 -0500
@@ -711,7 +711,7 @@ void __init ps3_register_ipi_irq(unsigne
static unsigned int ps3_get_irq(void)
{
- struct ps3_private *pd = &__get_cpu_var(ps3_private);
+ struct ps3_private *pd = this_cpu_ptr(&ps3_private);
u64 x = (pd->bmp.status & pd->bmp.mask);
unsigned int plug;
Index: linux/arch/powerpc/platforms/pseries/dtl.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/dtl.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/dtl.c 2013-08-22 14:56:47.935059693 -0500
@@ -76,7 +76,7 @@ static atomic_t dtl_count;
*/
static void consume_dtle(struct dtl_entry *dtle, u64 index)
{
- struct dtl_ring *dtlr = &__get_cpu_var(dtl_rings);
+ struct dtl_ring *dtlr = this_cpu_ptr(&dtl_rings);
struct dtl_entry *wp = dtlr->write_ptr;
struct lppaca *vpa = local_paca->lppaca_ptr;
Index: linux/arch/powerpc/platforms/pseries/hvCall_inst.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/hvCall_inst.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/hvCall_inst.c 2013-08-22 14:56:47.935059693 -0500
@@ -109,7 +109,7 @@ static void probe_hcall_entry(void *igno
if (opcode > MAX_HCALL_OPCODE)
return;
- h = &__get_cpu_var(hcall_stats)[opcode / 4];
+ h = this_cpu_ptr(&hcall_stats[opcode / 4]);
h->tb_start = mftb();
h->purr_start = mfspr(SPRN_PURR);
}
@@ -122,7 +122,7 @@ static void probe_hcall_exit(void *ignor
if (opcode > MAX_HCALL_OPCODE)
return;
- h = &__get_cpu_var(hcall_stats)[opcode / 4];
+ h = this_cpu_ptr(&hcall_stats[opcode / 4]);
h->num_calls++;
h->tb_total += mftb() - h->tb_start;
h->purr_total += mfspr(SPRN_PURR) - h->purr_start;
Index: linux/arch/powerpc/platforms/pseries/iommu.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/iommu.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/iommu.c 2013-08-22 14:56:47.935059693 -0500
@@ -201,7 +201,7 @@ static int tce_buildmulti_pSeriesLP(stru
local_irq_save(flags); /* to protect tcep and the page behind it */
- tcep = __get_cpu_var(tce_page);
+ tcep = __this_cpu_read(tce_page);
/* This is safe to do since interrupts are off when we're called
* from iommu_alloc{,_sg}()
@@ -214,7 +214,7 @@ static int tce_buildmulti_pSeriesLP(stru
return tce_build_pSeriesLP(tbl, tcenum, npages, uaddr,
direction, attrs);
}
- __get_cpu_var(tce_page) = tcep;
+ __this_cpu_write(tce_page, tcep);
}
rpn = __pa(uaddr) >> TCE_SHIFT;
@@ -399,7 +399,7 @@ static int tce_setrange_multi_pSeriesLP(
long l, limit;
local_irq_disable(); /* to protect tcep and the page behind it */
- tcep = __get_cpu_var(tce_page);
+ tcep = __this_cpu_read(tce_page);
if (!tcep) {
tcep = (u64 *)__get_free_page(GFP_ATOMIC);
@@ -407,7 +407,7 @@ static int tce_setrange_multi_pSeriesLP(
local_irq_enable();
return -ENOMEM;
}
- __get_cpu_var(tce_page) = tcep;
+ __this_cpu_write(tce_page, tcep);
}
proto_tce = TCE_PCI_READ | TCE_PCI_WRITE;
Index: linux/arch/powerpc/platforms/pseries/lpar.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/lpar.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/lpar.c 2013-08-22 14:56:47.935059693 -0500
@@ -490,7 +490,7 @@ static void pSeries_lpar_flush_hash_rang
unsigned long vpn;
unsigned long i, pix, rc;
unsigned long flags = 0;
- struct ppc64_tlb_batch *batch = &__get_cpu_var(ppc64_tlb_batch);
+ struct ppc64_tlb_batch *batch = this_cpu_ptr(&ppc64_tlb_batch);
int lock_tlbie = !mmu_has_feature(MMU_FTR_LOCKLESS_TLBIE);
unsigned long param[9];
unsigned long hash, index, shift, hidx, slot;
@@ -665,7 +665,7 @@ void __trace_hcall_entry(unsigned long o
local_irq_save(flags);
- depth = &__get_cpu_var(hcall_trace_depth);
+ depth = this_cpu_ptr(&hcall_trace_depth);
if (*depth)
goto out;
@@ -690,7 +690,7 @@ void __trace_hcall_exit(long opcode, uns
local_irq_save(flags);
- depth = &__get_cpu_var(hcall_trace_depth);
+ depth = this_cpu_ptr(&hcall_trace_depth);
if (*depth)
goto out;
Index: linux/arch/powerpc/platforms/pseries/ras.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/ras.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/ras.c 2013-08-22 14:56:47.935059693 -0500
@@ -301,8 +301,8 @@ static struct rtas_error_log *fwnmi_get_
/* If it isn't an extended log we can use the per cpu 64bit buffer */
h = (struct rtas_error_log *)&savep[1];
if (!h->extended) {
- memcpy(&__get_cpu_var(mce_data_buf), h, sizeof(__u64));
- errhdr = (struct rtas_error_log *)&__get_cpu_var(mce_data_buf);
+ memcpy(this_cpu_ptr(&mce_data_buf), h, sizeof(__u64));
+ errhdr = (struct rtas_error_log *)this_cpu_ptr(&mce_data_buf);
} else {
int len;
Index: linux/arch/powerpc/sysdev/xics/xics-common.c
===================================================================
--- linux.orig/arch/powerpc/sysdev/xics/xics-common.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/sysdev/xics/xics-common.c 2013-08-22 14:56:47.935059693 -0500
@@ -155,7 +155,7 @@ int __init xics_smp_probe(void)
void xics_teardown_cpu(void)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
/*
* we have to reset the cppr index to 0 because we're
Index: linux/arch/powerpc/kernel/iommu.c
===================================================================
--- linux.orig/arch/powerpc/kernel/iommu.c 2013-08-08 02:54:34.148984340 -0500
+++ linux/arch/powerpc/kernel/iommu.c 2013-08-22 14:57:22.922707906 -0500
@@ -208,7 +208,7 @@ static unsigned long iommu_range_alloc(s
* We don't need to disable preemption here because any CPU can
* safely use any IOMMU pool.
*/
- pool_nr = __raw_get_cpu_var(iommu_pool_hash) & (tbl->nr_pools - 1);
+ pool_nr = __this_cpu_read(iommu_pool_hash) & (tbl->nr_pools - 1);
if (largealloc)
pool = &(tbl->large_pool);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 20/31] powerpc: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Benjamin Herrenschmidt, Paul Mackerras, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/powerpc/include/asm/cputime.h
===================================================================
--- linux.orig/arch/powerpc/include/asm/cputime.h 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/include/asm/cputime.h 2013-08-22 14:56:47.935059693 -0500
@@ -56,10 +56,10 @@ static inline unsigned long cputime_to_j
static inline cputime_t cputime_to_scaled(const cputime_t ct)
{
if (cpu_has_feature(CPU_FTR_SPURR) &&
- __get_cpu_var(cputime_last_delta))
+ __this_cpu_read(cputime_last_delta))
return (__force u64) ct *
- __get_cpu_var(cputime_scaled_last_delta) /
- __get_cpu_var(cputime_last_delta);
+ __this_cpu_read(cputime_scaled_last_delta) /
+ __this_cpu_read(cputime_last_delta);
return ct;
}
Index: linux/arch/powerpc/include/asm/hardirq.h
===================================================================
--- linux.orig/arch/powerpc/include/asm/hardirq.h 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/include/asm/hardirq.h 2013-08-22 14:56:47.935059693 -0500
@@ -19,7 +19,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpust
#define __ARCH_IRQ_STAT
-#define local_softirq_pending() __get_cpu_var(irq_stat).__softirq_pending
+#define local_softirq_pending() __this_cpu_read(irq_stat.__softirq_pending)
static inline void ack_bad_irq(unsigned int irq)
{
Index: linux/arch/powerpc/include/asm/tlbflush.h
===================================================================
--- linux.orig/arch/powerpc/include/asm/tlbflush.h 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/include/asm/tlbflush.h 2013-08-22 14:56:47.935059693 -0500
@@ -107,14 +107,14 @@ extern void __flush_tlb_pending(struct p
static inline void arch_enter_lazy_mmu_mode(void)
{
- struct ppc64_tlb_batch *batch = &__get_cpu_var(ppc64_tlb_batch);
+ struct ppc64_tlb_batch *batch = this_cpu_ptr(&ppc64_tlb_batch);
batch->active = 1;
}
static inline void arch_leave_lazy_mmu_mode(void)
{
- struct ppc64_tlb_batch *batch = &__get_cpu_var(ppc64_tlb_batch);
+ struct ppc64_tlb_batch *batch = this_cpu_ptr(&ppc64_tlb_batch);
if (batch->index)
__flush_tlb_pending(batch);
Index: linux/arch/powerpc/include/asm/xics.h
===================================================================
--- linux.orig/arch/powerpc/include/asm/xics.h 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/include/asm/xics.h 2013-08-22 14:56:47.935059693 -0500
@@ -97,7 +97,7 @@ DECLARE_PER_CPU(struct xics_cppr, xics_c
static inline void xics_push_cppr(unsigned int vec)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
if (WARN_ON(os_cppr->index >= MAX_NUM_PRIORITIES - 1))
return;
@@ -110,7 +110,7 @@ static inline void xics_push_cppr(unsign
static inline unsigned char xics_pop_cppr(void)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
if (WARN_ON(os_cppr->index < 1))
return LOWEST_PRIORITY;
@@ -120,7 +120,7 @@ static inline unsigned char xics_pop_cpp
static inline void xics_set_base_cppr(unsigned char cppr)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
/* we only really want to set the priority when there's
* just one cppr value on the stack
@@ -132,7 +132,7 @@ static inline void xics_set_base_cppr(un
static inline unsigned char xics_cppr_top(void)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
return os_cppr->stack[os_cppr->index];
}
Index: linux/arch/powerpc/kernel/dbell.c
===================================================================
--- linux.orig/arch/powerpc/kernel/dbell.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/dbell.c 2013-08-22 14:56:47.935059693 -0500
@@ -41,7 +41,7 @@ void doorbell_exception(struct pt_regs *
may_hard_irq_enable();
- __get_cpu_var(irq_stat).doorbell_irqs++;
+ __this_cpu_inc(irq_stat.doorbell_irqs);
smp_ipi_demux();
Index: linux/arch/powerpc/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/powerpc/kernel/hw_breakpoint.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/hw_breakpoint.c 2013-08-22 14:56:47.935059693 -0500
@@ -64,7 +64,7 @@ int hw_breakpoint_slots(int type)
int arch_install_hw_breakpoint(struct perf_event *bp)
{
struct arch_hw_breakpoint *info = counter_arch_bp(bp);
- struct perf_event **slot = &__get_cpu_var(bp_per_reg);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg);
*slot = bp;
@@ -89,7 +89,7 @@ int arch_install_hw_breakpoint(struct pe
*/
void arch_uninstall_hw_breakpoint(struct perf_event *bp)
{
- struct perf_event **slot = &__get_cpu_var(bp_per_reg);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg);
if (*slot != bp) {
WARN_ONCE(1, "Can't find the breakpoint");
@@ -227,7 +227,7 @@ int __kprobes hw_breakpoint_handler(stru
*/
rcu_read_lock();
- bp = __get_cpu_var(bp_per_reg);
+ bp = __this_cpu_read(bp_per_reg);
if (!bp)
goto out;
info = counter_arch_bp(bp);
Index: linux/arch/powerpc/kernel/irq.c
===================================================================
--- linux.orig/arch/powerpc/kernel/irq.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/irq.c 2013-08-22 14:56:47.935059693 -0500
@@ -114,7 +114,7 @@ static inline notrace void set_soft_enab
static inline notrace int decrementer_check_overflow(void)
{
u64 now = get_tb_or_rtc();
- u64 *next_tb = &__get_cpu_var(decrementers_next_tb);
+ u64 *next_tb = this_cpu_ptr(&decrementers_next_tb);
return now >= *next_tb;
}
@@ -526,7 +526,7 @@ void do_IRQ(struct pt_regs *regs)
if (irq != NO_IRQ)
handle_one_irq(irq);
else
- __get_cpu_var(irq_stat).spurious_irqs++;
+ __this_cpu_inc(irq_stat.spurious_irqs);
trace_irq_exit(regs);
Index: linux/arch/powerpc/kernel/kprobes.c
===================================================================
--- linux.orig/arch/powerpc/kernel/kprobes.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/kprobes.c 2013-08-22 14:56:47.935059693 -0500
@@ -118,7 +118,7 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
kcb->kprobe_saved_msr = kcb->prev_kprobe.saved_msr;
}
@@ -126,7 +126,7 @@ static void __kprobes restore_previous_k
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
kcb->kprobe_saved_msr = regs->msr;
}
@@ -191,7 +191,7 @@ static int __kprobes kprobe_handler(stru
ret = 1;
goto no_kprobe;
}
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
goto ss_probe;
}
Index: linux/arch/powerpc/kernel/process.c
===================================================================
--- linux.orig/arch/powerpc/kernel/process.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/process.c 2013-08-22 14:56:47.935059693 -0500
@@ -453,7 +453,7 @@ static inline int set_dawr(struct arch_h
int set_breakpoint(struct arch_hw_breakpoint *brk)
{
- __get_cpu_var(current_brk) = *brk;
+ __this_cpu_write(current_brk, *brk);
if (cpu_has_feature(CPU_FTR_DAWR))
return set_dawr(brk);
@@ -686,7 +686,7 @@ struct task_struct *__switch_to(struct t
* schedule DABR
*/
#ifndef CONFIG_HAVE_HW_BREAKPOINT
- if (unlikely(hw_brk_match(&__get_cpu_var(current_brk), &new->thread.hw_brk)))
+ if (unlikely(hw_brk_match(this_cpu_ptr(¤t_brk), &new->thread.hw_brk)))
set_breakpoint(&new->thread.hw_brk);
#endif /* CONFIG_HAVE_HW_BREAKPOINT */
#endif
@@ -700,7 +700,7 @@ struct task_struct *__switch_to(struct t
* Collect processor utilization data per process
*/
if (firmware_has_feature(FW_FEATURE_SPLPAR)) {
- struct cpu_usage *cu = &__get_cpu_var(cpu_usage_array);
+ struct cpu_usage *cu = this_cpu_ptr(&cpu_usage_array);
long unsigned start_tb, current_tb;
start_tb = old_thread->start_tb;
cu->current_tb = current_tb = mfspr(SPRN_PURR);
@@ -710,7 +710,7 @@ struct task_struct *__switch_to(struct t
#endif /* CONFIG_PPC64 */
#ifdef CONFIG_PPC_BOOK3S_64
- batch = &__get_cpu_var(ppc64_tlb_batch);
+ batch = this_cpu_ptr(&ppc64_tlb_batch);
if (batch->active) {
current_thread_info()->local_flags |= _TLF_LAZY_MMU;
if (batch->index)
@@ -735,7 +735,7 @@ struct task_struct *__switch_to(struct t
#ifdef CONFIG_PPC_BOOK3S_64
if (current_thread_info()->local_flags & _TLF_LAZY_MMU) {
current_thread_info()->local_flags &= ~_TLF_LAZY_MMU;
- batch = &__get_cpu_var(ppc64_tlb_batch);
+ batch = this_cpu_ptr(&ppc64_tlb_batch);
batch->active = 1;
}
#endif /* CONFIG_PPC_BOOK3S_64 */
Index: linux/arch/powerpc/kernel/smp.c
===================================================================
--- linux.orig/arch/powerpc/kernel/smp.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/smp.c 2013-08-22 14:56:47.935059693 -0500
@@ -212,7 +212,7 @@ void smp_muxed_ipi_message_pass(int cpu,
irqreturn_t smp_ipi_demux(void)
{
- struct cpu_messages *info = &__get_cpu_var(ipi_message);
+ struct cpu_messages *info = this_cpu_ptr(&ipi_message);
unsigned int all;
mb(); /* order any irq clear */
@@ -402,9 +402,9 @@ void generic_mach_cpu_die(void)
idle_task_exit();
cpu = smp_processor_id();
printk(KERN_DEBUG "CPU%d offline\n", cpu);
- __get_cpu_var(cpu_state) = CPU_DEAD;
+ __this_cpu_write(cpu_state, CPU_DEAD);
smp_wmb();
- while (__get_cpu_var(cpu_state) != CPU_UP_PREPARE)
+ while (__this_cpu_read(cpu_state) != CPU_UP_PREPARE)
cpu_relax();
}
Index: linux/arch/powerpc/kernel/sysfs.c
===================================================================
--- linux.orig/arch/powerpc/kernel/sysfs.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/sysfs.c 2013-08-22 14:56:47.935059693 -0500
@@ -97,10 +97,10 @@ void ppc_enable_pmcs(void)
ppc_set_pmu_inuse(1);
/* Only need to enable them once */
- if (__get_cpu_var(pmcs_enabled))
+ if (__this_cpu_read(pmcs_enabled))
return;
- __get_cpu_var(pmcs_enabled) = 1;
+ __this_cpu_write(pmcs_enabled, 1);
if (ppc_md.enable_pmcs)
ppc_md.enable_pmcs();
Index: linux/arch/powerpc/kernel/time.c
===================================================================
--- linux.orig/arch/powerpc/kernel/time.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/time.c 2013-08-22 14:56:47.935059693 -0500
@@ -457,9 +457,9 @@ static inline void clear_irq_work_pendin
DEFINE_PER_CPU(u8, irq_work_pending);
-#define set_irq_work_pending_flag() __get_cpu_var(irq_work_pending) = 1
-#define test_irq_work_pending() __get_cpu_var(irq_work_pending)
-#define clear_irq_work_pending() __get_cpu_var(irq_work_pending) = 0
+#define set_irq_work_pending_flag() __this_cpu_write(irq_work_pending, 1)
+#define test_irq_work_pending() __this_cpu_read(irq_work_pending)
+#define clear_irq_work_pending() __this_cpu_write(irq_work_pending, 0)
#endif /* 32 vs 64 bit */
@@ -485,8 +485,8 @@ void arch_irq_work_raise(void)
void timer_interrupt(struct pt_regs * regs)
{
struct pt_regs *old_regs;
- u64 *next_tb = &__get_cpu_var(decrementers_next_tb);
- struct clock_event_device *evt = &__get_cpu_var(decrementers);
+ u64 *next_tb = this_cpu_ptr(&decrementers_next_tb);
+ struct clock_event_device *evt = this_cpu_ptr(&decrementers);
u64 now;
/* Ensure a positive value is written to the decrementer, or else
@@ -510,7 +510,7 @@ void timer_interrupt(struct pt_regs * re
*/
may_hard_irq_enable();
- __get_cpu_var(irq_stat).timer_irqs++;
+ __this_cpu_inc(irq_stat.timer_irqs);
#if defined(CONFIG_PPC32) && defined(CONFIG_PMAC)
if (atomic_read(&ppc_n_lost_interrupts) != 0)
@@ -541,7 +541,7 @@ void timer_interrupt(struct pt_regs * re
#ifdef CONFIG_PPC64
/* collect purr register values often, for accurate calculations */
if (firmware_has_feature(FW_FEATURE_SPLPAR)) {
- struct cpu_usage *cu = &__get_cpu_var(cpu_usage_array);
+ struct cpu_usage *cu = this_cpu_ptr(&cpu_usage_array);
cu->current_tb = mfspr(SPRN_PURR);
}
#endif
@@ -801,7 +801,7 @@ static void __init clocksource_init(void
static int decrementer_set_next_event(unsigned long evt,
struct clock_event_device *dev)
{
- __get_cpu_var(decrementers_next_tb) = get_tb_or_rtc() + evt;
+ __this_cpu_write(decrementers_next_tb, get_tb_or_rtc() + evt);
set_dec(evt);
return 0;
}
Index: linux/arch/powerpc/kernel/traps.c
===================================================================
--- linux.orig/arch/powerpc/kernel/traps.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kernel/traps.c 2013-08-22 14:56:47.935059693 -0500
@@ -670,7 +670,7 @@ void machine_check_exception(struct pt_r
enum ctx_state prev_state = exception_enter();
int recover = 0;
- __get_cpu_var(irq_stat).mce_exceptions++;
+ __this_cpu_inc(irq_stat.mce_exceptions);
/* See if any machine dependent calls. In theory, we would want
* to call the CPU first, and call the ppc_md. one if the CPU
@@ -1436,7 +1436,7 @@ void vsx_unavailable_tm(struct pt_regs *
void performance_monitor_exception(struct pt_regs *regs)
{
- __get_cpu_var(irq_stat).pmu_irqs++;
+ __this_cpu_inc(irq_stat.pmu_irqs);
perf_irq(regs);
}
Index: linux/arch/powerpc/kvm/e500.c
===================================================================
--- linux.orig/arch/powerpc/kvm/e500.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kvm/e500.c 2013-08-22 14:56:47.935059693 -0500
@@ -74,11 +74,11 @@ static inline int local_sid_setup_one(st
unsigned long sid;
int ret = -1;
- sid = ++(__get_cpu_var(pcpu_last_used_sid));
+ sid = __this_cpu_inc_return(pcpu_last_used_sid);
if (sid < NUM_TIDS) {
- __get_cpu_var(pcpu_sids).entry[sid] = entry;
+ __this_cpu_write(pcpu_sids)entry[sid], entry);
entry->val = sid;
- entry->pentry = &__get_cpu_var(pcpu_sids).entry[sid];
+ entry->pentry = this_cpu_ptr(&pcpu_sids.entry[sid]);
ret = sid;
}
@@ -106,8 +106,8 @@ static inline int local_sid_setup_one(st
static inline int local_sid_lookup(struct id *entry)
{
if (entry && entry->val != 0 &&
- __get_cpu_var(pcpu_sids).entry[entry->val] == entry &&
- entry->pentry == &__get_cpu_var(pcpu_sids).entry[entry->val])
+ __this_cpu_read(pcpu_sids.entry[entry->val]) == entry &&
+ entry->pentry == this_cpu_ptr(&pcpu_sids.entry[entry->val]))
return entry->val;
return -1;
}
@@ -115,8 +115,8 @@ static inline int local_sid_lookup(struc
/* Invalidate all id mappings on local core -- call with preempt disabled */
static inline void local_sid_destroy_all(void)
{
- __get_cpu_var(pcpu_last_used_sid) = 0;
- memset(&__get_cpu_var(pcpu_sids), 0, sizeof(__get_cpu_var(pcpu_sids)));
+ __this_cpu_write(pcpu_last_used_sid, 0);
+ memset(this_cpu_ptr(&pcpu_sids), 0, sizeof(pcpu_sids));
}
static void *kvmppc_e500_id_table_alloc(struct kvmppc_vcpu_e500 *vcpu_e500)
Index: linux/arch/powerpc/kvm/e500mc.c
===================================================================
--- linux.orig/arch/powerpc/kvm/e500mc.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/kvm/e500mc.c 2013-08-22 14:56:47.935059693 -0500
@@ -139,9 +139,9 @@ void kvmppc_core_vcpu_load(struct kvm_vc
mtspr(SPRN_GESR, vcpu->arch.shared->esr);
if (vcpu->arch.oldpir != mfspr(SPRN_PIR) ||
- __get_cpu_var(last_vcpu_on_cpu) != vcpu) {
+ __this_cpu_read(last_vcpu_on_cpu) != vcpu) {
kvmppc_e500_tlbil_all(vcpu_e500);
- __get_cpu_var(last_vcpu_on_cpu) = vcpu;
+ __this_cpu_read(last_vcpu_on_cpu) = vcpu;
}
kvmppc_load_guest_fp(vcpu);
Index: linux/arch/powerpc/mm/hash_native_64.c
===================================================================
--- linux.orig/arch/powerpc/mm/hash_native_64.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/hash_native_64.c 2013-08-22 14:56:47.935059693 -0500
@@ -643,7 +643,7 @@ static void native_flush_hash_range(unsi
unsigned long want_v;
unsigned long flags;
real_pte_t pte;
- struct ppc64_tlb_batch *batch = &__get_cpu_var(ppc64_tlb_batch);
+ struct ppc64_tlb_batch *batch = this_cpu_ptr(&ppc64_tlb_batch);
unsigned long psize = batch->psize;
int ssize = batch->ssize;
int i;
Index: linux/arch/powerpc/mm/hash_utils_64.c
===================================================================
--- linux.orig/arch/powerpc/mm/hash_utils_64.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/hash_utils_64.c 2013-08-22 14:56:47.935059693 -0500
@@ -1285,7 +1285,7 @@ void flush_hash_range(unsigned long numb
else {
int i;
struct ppc64_tlb_batch *batch =
- &__get_cpu_var(ppc64_tlb_batch);
+ this_cpu_ptr(&ppc64_tlb_batch);
for (i = 0; i < number; i++)
flush_hash_page(batch->vpn[i], batch->pte[i],
Index: linux/arch/powerpc/mm/hugetlbpage-book3e.c
===================================================================
--- linux.orig/arch/powerpc/mm/hugetlbpage-book3e.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/hugetlbpage-book3e.c 2013-08-22 14:56:47.935059693 -0500
@@ -80,14 +80,14 @@ void book3e_hugetlb_preload(struct vm_ar
ncams = mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY;
/* We have to use the CAM(TLB1) on FSL parts for hugepages */
- index = __get_cpu_var(next_tlbcam_idx);
+ index = __this_cpu_read(next_tlbcam_idx);
mtspr(SPRN_MAS0, MAS0_ESEL(index) | MAS0_TLBSEL(1));
/* Just round-robin the entries and wrap when we hit the end */
if (unlikely(index == ncams - 1))
- __get_cpu_var(next_tlbcam_idx) = tlbcam_index;
+ __this_cpu_write(next_tlbcam_idx, tlbcam_index);
else
- __get_cpu_var(next_tlbcam_idx)++;
+ __this_cpu_inc(next_tlbcam_idx);
#endif
mas1 = MAS1_VALID | MAS1_TID(mm->context.id) | MAS1_TSIZE(tsize);
mas2 = ea & ~((1UL << shift) - 1);
Index: linux/arch/powerpc/mm/hugetlbpage.c
===================================================================
--- linux.orig/arch/powerpc/mm/hugetlbpage.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/hugetlbpage.c 2013-08-22 14:56:47.935059693 -0500
@@ -462,7 +462,7 @@ static void hugepd_free(struct mmu_gathe
{
struct hugepd_freelist **batchp;
- batchp = &__get_cpu_var(hugepd_freelist_cur);
+ batchp = this_cpu_ptr(&hugepd_freelist_cur);
if (atomic_read(&tlb->mm->mm_users) < 2 ||
cpumask_equal(mm_cpumask(tlb->mm),
Index: linux/arch/powerpc/mm/stab.c
===================================================================
--- linux.orig/arch/powerpc/mm/stab.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/mm/stab.c 2013-08-22 14:56:47.935059693 -0500
@@ -133,12 +133,12 @@ static int __ste_allocate(unsigned long
stab_entry = make_ste(get_paca()->stab_addr, GET_ESID(ea), vsid);
if (!is_kernel_addr(ea)) {
- offset = __get_cpu_var(stab_cache_ptr);
+ offset = __this_cpu_read(stab_cache_ptr);
if (offset < NR_STAB_CACHE_ENTRIES)
- __get_cpu_var(stab_cache[offset++]) = stab_entry;
+ __this_cpu_read(stab_cache[offset++]) = stab_entry;
else
offset = NR_STAB_CACHE_ENTRIES+1;
- __get_cpu_var(stab_cache_ptr) = offset;
+ __this_cpu_write(stab_cache_ptr, offset);
/* Order update */
asm volatile("sync":::"memory");
@@ -177,12 +177,12 @@ void switch_stab(struct task_struct *tsk
*/
hard_irq_disable();
- offset = __get_cpu_var(stab_cache_ptr);
+ offset = __this_cpu_read(stab_cache_ptr);
if (offset <= NR_STAB_CACHE_ENTRIES) {
int i;
for (i = 0; i < offset; i++) {
- ste = stab + __get_cpu_var(stab_cache[i]);
+ ste = stab + __this_cpu_read(stab_cache[i]);
ste->esid_data = 0; /* invalidate entry */
}
} else {
@@ -206,7 +206,7 @@ void switch_stab(struct task_struct *tsk
asm volatile("sync; slbia; sync":::"memory");
- __get_cpu_var(stab_cache_ptr) = 0;
+ __this_cpu_write(stab_cache_ptr, 0);
/* Now preload some entries for the new task */
if (test_tsk_thread_flag(tsk, TIF_32BIT))
Index: linux/arch/powerpc/perf/core-book3s.c
===================================================================
--- linux.orig/arch/powerpc/perf/core-book3s.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/perf/core-book3s.c 2013-08-22 14:56:47.935059693 -0500
@@ -332,7 +332,7 @@ static void power_pmu_bhrb_reset(void)
static void power_pmu_bhrb_enable(struct perf_event *event)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!ppmu->bhrb_nr)
return;
@@ -347,7 +347,7 @@ static void power_pmu_bhrb_enable(struct
static void power_pmu_bhrb_disable(struct perf_event *event)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!ppmu->bhrb_nr)
return;
@@ -961,7 +961,7 @@ static void power_pmu_disable(struct pmu
if (!ppmu)
return;
local_irq_save(flags);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!cpuhw->disabled) {
/*
@@ -1027,7 +1027,7 @@ static void power_pmu_enable(struct pmu
return;
local_irq_save(flags);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!cpuhw->disabled)
goto out;
@@ -1211,7 +1211,7 @@ static int power_pmu_add(struct perf_eve
* Add the event to the list (if there is room)
* and check whether the total set is still feasible.
*/
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
n0 = cpuhw->n_events;
if (n0 >= ppmu->n_counter)
goto out;
@@ -1277,7 +1277,7 @@ static void power_pmu_del(struct perf_ev
power_pmu_read(event);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
for (i = 0; i < cpuhw->n_events; ++i) {
if (event == cpuhw->event[i]) {
while (++i < cpuhw->n_events) {
@@ -1383,7 +1383,7 @@ static void power_pmu_stop(struct perf_e
*/
void power_pmu_start_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
perf_pmu_disable(pmu);
cpuhw->group_flag |= PERF_EVENT_TXN;
@@ -1397,7 +1397,7 @@ void power_pmu_start_txn(struct pmu *pmu
*/
void power_pmu_cancel_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
cpuhw->group_flag &= ~PERF_EVENT_TXN;
perf_pmu_enable(pmu);
@@ -1415,7 +1415,7 @@ int power_pmu_commit_txn(struct pmu *pmu
if (!ppmu)
return -EAGAIN;
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
n = cpuhw->n_events;
if (check_excludes(cpuhw->event, cpuhw->flags, 0, n))
return -EAGAIN;
@@ -1772,7 +1772,7 @@ static void record_and_restart(struct pe
if (event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK) {
struct cpu_hw_events *cpuhw;
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
power_pmu_bhrb_read(cpuhw);
data.br_stack = &cpuhw->bhrb_stack;
}
@@ -1845,7 +1845,7 @@ static bool pmc_overflow(unsigned long v
static void perf_event_interrupt(struct pt_regs *regs)
{
int i, j;
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event;
unsigned long val[8];
int found, active;
Index: linux/arch/powerpc/perf/core-fsl-emb.c
===================================================================
--- linux.orig/arch/powerpc/perf/core-fsl-emb.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/perf/core-fsl-emb.c 2013-08-22 14:56:47.935059693 -0500
@@ -186,7 +186,7 @@ static void fsl_emb_pmu_disable(struct p
unsigned long flags;
local_irq_save(flags);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!cpuhw->disabled) {
cpuhw->disabled = 1;
@@ -225,7 +225,7 @@ static void fsl_emb_pmu_enable(struct pm
unsigned long flags;
local_irq_save(flags);
- cpuhw = &__get_cpu_var(cpu_hw_events);
+ cpuhw = this_cpu_ptr(&cpu_hw_events);
if (!cpuhw->disabled)
goto out;
@@ -623,7 +623,7 @@ static void record_and_restart(struct pe
static void perf_event_interrupt(struct pt_regs *regs)
{
int i;
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event;
unsigned long val;
int found = 0;
Index: linux/arch/powerpc/platforms/cell/interrupt.c
===================================================================
--- linux.orig/arch/powerpc/platforms/cell/interrupt.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/cell/interrupt.c 2013-08-22 14:56:47.935059693 -0500
@@ -82,7 +82,7 @@ static void iic_unmask(struct irq_data *
static void iic_eoi(struct irq_data *d)
{
- struct iic *iic = &__get_cpu_var(cpu_iic);
+ struct iic *iic = this_cpu_ptr(&cpu_iic);
out_be64(&iic->regs->prio, iic->eoi_stack[--iic->eoi_ptr]);
BUG_ON(iic->eoi_ptr < 0);
}
@@ -148,7 +148,7 @@ static unsigned int iic_get_irq(void)
struct iic *iic;
unsigned int virq;
- iic = &__get_cpu_var(cpu_iic);
+ iic = this_cpu_ptr(&cpu_iic);
*(unsigned long *) &pending =
in_be64((u64 __iomem *) &iic->regs->pending_destr);
if (!(pending.flags & CBE_IIC_IRQ_VALID))
@@ -163,7 +163,7 @@ static unsigned int iic_get_irq(void)
void iic_setup_cpu(void)
{
- out_be64(&__get_cpu_var(cpu_iic).regs->prio, 0xff);
+ out_be64(this_cpu_ptr(&cpu_iic.regs->prio), 0xff);
}
u8 iic_get_target_id(int cpu)
Index: linux/arch/powerpc/platforms/ps3/interrupt.c
===================================================================
--- linux.orig/arch/powerpc/platforms/ps3/interrupt.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/ps3/interrupt.c 2013-08-22 14:56:47.935059693 -0500
@@ -711,7 +711,7 @@ void __init ps3_register_ipi_irq(unsigne
static unsigned int ps3_get_irq(void)
{
- struct ps3_private *pd = &__get_cpu_var(ps3_private);
+ struct ps3_private *pd = this_cpu_ptr(&ps3_private);
u64 x = (pd->bmp.status & pd->bmp.mask);
unsigned int plug;
Index: linux/arch/powerpc/platforms/pseries/dtl.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/dtl.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/dtl.c 2013-08-22 14:56:47.935059693 -0500
@@ -76,7 +76,7 @@ static atomic_t dtl_count;
*/
static void consume_dtle(struct dtl_entry *dtle, u64 index)
{
- struct dtl_ring *dtlr = &__get_cpu_var(dtl_rings);
+ struct dtl_ring *dtlr = this_cpu_ptr(&dtl_rings);
struct dtl_entry *wp = dtlr->write_ptr;
struct lppaca *vpa = local_paca->lppaca_ptr;
Index: linux/arch/powerpc/platforms/pseries/hvCall_inst.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/hvCall_inst.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/hvCall_inst.c 2013-08-22 14:56:47.935059693 -0500
@@ -109,7 +109,7 @@ static void probe_hcall_entry(void *igno
if (opcode > MAX_HCALL_OPCODE)
return;
- h = &__get_cpu_var(hcall_stats)[opcode / 4];
+ h = this_cpu_ptr(&hcall_stats[opcode / 4]);
h->tb_start = mftb();
h->purr_start = mfspr(SPRN_PURR);
}
@@ -122,7 +122,7 @@ static void probe_hcall_exit(void *ignor
if (opcode > MAX_HCALL_OPCODE)
return;
- h = &__get_cpu_var(hcall_stats)[opcode / 4];
+ h = this_cpu_ptr(&hcall_stats[opcode / 4]);
h->num_calls++;
h->tb_total += mftb() - h->tb_start;
h->purr_total += mfspr(SPRN_PURR) - h->purr_start;
Index: linux/arch/powerpc/platforms/pseries/iommu.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/iommu.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/iommu.c 2013-08-22 14:56:47.935059693 -0500
@@ -201,7 +201,7 @@ static int tce_buildmulti_pSeriesLP(stru
local_irq_save(flags); /* to protect tcep and the page behind it */
- tcep = __get_cpu_var(tce_page);
+ tcep = __this_cpu_read(tce_page);
/* This is safe to do since interrupts are off when we're called
* from iommu_alloc{,_sg}()
@@ -214,7 +214,7 @@ static int tce_buildmulti_pSeriesLP(stru
return tce_build_pSeriesLP(tbl, tcenum, npages, uaddr,
direction, attrs);
}
- __get_cpu_var(tce_page) = tcep;
+ __this_cpu_write(tce_page, tcep);
}
rpn = __pa(uaddr) >> TCE_SHIFT;
@@ -399,7 +399,7 @@ static int tce_setrange_multi_pSeriesLP(
long l, limit;
local_irq_disable(); /* to protect tcep and the page behind it */
- tcep = __get_cpu_var(tce_page);
+ tcep = __this_cpu_read(tce_page);
if (!tcep) {
tcep = (u64 *)__get_free_page(GFP_ATOMIC);
@@ -407,7 +407,7 @@ static int tce_setrange_multi_pSeriesLP(
local_irq_enable();
return -ENOMEM;
}
- __get_cpu_var(tce_page) = tcep;
+ __this_cpu_write(tce_page, tcep);
}
proto_tce = TCE_PCI_READ | TCE_PCI_WRITE;
Index: linux/arch/powerpc/platforms/pseries/lpar.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/lpar.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/lpar.c 2013-08-22 14:56:47.935059693 -0500
@@ -490,7 +490,7 @@ static void pSeries_lpar_flush_hash_rang
unsigned long vpn;
unsigned long i, pix, rc;
unsigned long flags = 0;
- struct ppc64_tlb_batch *batch = &__get_cpu_var(ppc64_tlb_batch);
+ struct ppc64_tlb_batch *batch = this_cpu_ptr(&ppc64_tlb_batch);
int lock_tlbie = !mmu_has_feature(MMU_FTR_LOCKLESS_TLBIE);
unsigned long param[9];
unsigned long hash, index, shift, hidx, slot;
@@ -665,7 +665,7 @@ void __trace_hcall_entry(unsigned long o
local_irq_save(flags);
- depth = &__get_cpu_var(hcall_trace_depth);
+ depth = this_cpu_ptr(&hcall_trace_depth);
if (*depth)
goto out;
@@ -690,7 +690,7 @@ void __trace_hcall_exit(long opcode, uns
local_irq_save(flags);
- depth = &__get_cpu_var(hcall_trace_depth);
+ depth = this_cpu_ptr(&hcall_trace_depth);
if (*depth)
goto out;
Index: linux/arch/powerpc/platforms/pseries/ras.c
===================================================================
--- linux.orig/arch/powerpc/platforms/pseries/ras.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/platforms/pseries/ras.c 2013-08-22 14:56:47.935059693 -0500
@@ -301,8 +301,8 @@ static struct rtas_error_log *fwnmi_get_
/* If it isn't an extended log we can use the per cpu 64bit buffer */
h = (struct rtas_error_log *)&savep[1];
if (!h->extended) {
- memcpy(&__get_cpu_var(mce_data_buf), h, sizeof(__u64));
- errhdr = (struct rtas_error_log *)&__get_cpu_var(mce_data_buf);
+ memcpy(this_cpu_ptr(&mce_data_buf), h, sizeof(__u64));
+ errhdr = (struct rtas_error_log *)this_cpu_ptr(&mce_data_buf);
} else {
int len;
Index: linux/arch/powerpc/sysdev/xics/xics-common.c
===================================================================
--- linux.orig/arch/powerpc/sysdev/xics/xics-common.c 2013-08-22 14:56:47.939059652 -0500
+++ linux/arch/powerpc/sysdev/xics/xics-common.c 2013-08-22 14:56:47.935059693 -0500
@@ -155,7 +155,7 @@ int __init xics_smp_probe(void)
void xics_teardown_cpu(void)
{
- struct xics_cppr *os_cppr = &__get_cpu_var(xics_cppr);
+ struct xics_cppr *os_cppr = this_cpu_ptr(&xics_cppr);
/*
* we have to reset the cppr index to 0 because we're
Index: linux/arch/powerpc/kernel/iommu.c
===================================================================
--- linux.orig/arch/powerpc/kernel/iommu.c 2013-08-08 02:54:34.148984340 -0500
+++ linux/arch/powerpc/kernel/iommu.c 2013-08-22 14:57:22.922707906 -0500
@@ -208,7 +208,7 @@ static unsigned long iommu_range_alloc(s
* We don't need to disable preemption here because any CPU can
* safely use any IOMMU pool.
*/
- pool_nr = __raw_get_cpu_var(iommu_pool_hash) & (tbl->nr_pools - 1);
+ pool_nr = __this_cpu_read(iommu_pool_hash) & (tbl->nr_pools - 1);
if (largealloc)
pool = &(tbl->large_pool);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 21/31] sparc: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
2013-08-26 20:44 ` Christoph Lameter
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (29 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, David S. Miller, sparclinux, linux-arch, Steven Rostedt,
linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/sparc/include/asm/cpudata_32.h
===================================================================
--- linux.orig/arch/sparc/include/asm/cpudata_32.h 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/include/asm/cpudata_32.h 2013-08-22 14:37:15.942964723 -0500
@@ -26,6 +26,6 @@ typedef struct {
DECLARE_PER_CPU(cpuinfo_sparc, __cpu_data);
#define cpu_data(__cpu) per_cpu(__cpu_data, (__cpu))
-#define local_cpu_data() __get_cpu_var(__cpu_data)
+#define local_cpu_data() __this_cpu_read(__cpu_data)
#endif /* _SPARC_CPUDATA_H */
Index: linux/arch/sparc/include/asm/cpudata_64.h
===================================================================
--- linux.orig/arch/sparc/include/asm/cpudata_64.h 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/include/asm/cpudata_64.h 2013-08-22 14:37:15.942964723 -0500
@@ -33,7 +33,7 @@ typedef struct {
DECLARE_PER_CPU(cpuinfo_sparc, __cpu_data);
#define cpu_data(__cpu) per_cpu(__cpu_data, (__cpu))
-#define local_cpu_data() __get_cpu_var(__cpu_data)
+#define local_cpu_data() __this_cpu_read(__cpu_data)
extern const struct seq_operations cpuinfo_op;
Index: linux/arch/sparc/kernel/kprobes.c
===================================================================
--- linux.orig/arch/sparc/kernel/kprobes.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/kprobes.c 2013-08-22 14:37:15.942964723 -0500
@@ -82,7 +82,7 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
kcb->kprobe_orig_tnpc = kcb->prev_kprobe.orig_tnpc;
kcb->kprobe_orig_tstate_pil = kcb->prev_kprobe.orig_tstate_pil;
@@ -91,7 +91,7 @@ static void __kprobes restore_previous_k
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
kcb->kprobe_orig_tnpc = regs->tnpc;
kcb->kprobe_orig_tstate_pil = (regs->tstate & TSTATE_PIL);
}
@@ -154,7 +154,7 @@ static int __kprobes kprobe_handler(stru
ret = 1;
goto no_kprobe;
}
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs))
goto ss_probe;
}
Index: linux/arch/sparc/kernel/leon_smp.c
===================================================================
--- linux.orig/arch/sparc/kernel/leon_smp.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/leon_smp.c 2013-08-22 14:37:15.942964723 -0500
@@ -354,7 +354,7 @@ static void leon_ipi_resched(int cpu)
void leonsmp_ipi_interrupt(void)
{
- struct leon_ipi_work *work = &__get_cpu_var(leon_ipi_work);
+ struct leon_ipi_work *work = this_cpu_ptr(&leon_ipi_work);
if (work->single) {
work->single = 0;
Index: linux/arch/sparc/kernel/nmi.c
===================================================================
--- linux.orig/arch/sparc/kernel/nmi.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/nmi.c 2013-08-22 14:37:15.942964723 -0500
@@ -111,20 +111,20 @@ notrace __kprobes void perfctr_irq(int i
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
sum = local_cpu_data().irq0_irqs;
- if (__get_cpu_var(nmi_touch)) {
- __get_cpu_var(nmi_touch) = 0;
+ if (__this_cpu_read(nmi_touch)) {
+ __this_cpu_write(nmi_touch, 0);
touched = 1;
}
- if (!touched && __get_cpu_var(last_irq_sum) == sum) {
+ if (!touched && __this_cpu_read(last_irq_sum) == sum) {
__this_cpu_inc(alert_counter);
if (__this_cpu_read(alert_counter) == 30 * nmi_hz)
die_nmi("BUG: NMI Watchdog detected LOCKUP",
regs, panic_on_timeout);
} else {
- __get_cpu_var(last_irq_sum) = sum;
+ __this_cpu_write(last_irq_sum, sum);
__this_cpu_write(alert_counter, 0);
}
- if (__get_cpu_var(wd_enabled)) {
+ if (__this_cpu_read(wd_enabled)) {
pcr_ops->write_pic(0, pcr_ops->nmi_picl_value(nmi_hz));
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_enable);
}
@@ -166,7 +166,7 @@ static void report_broken_nmi(int cpu, i
void stop_nmi_watchdog(void *unused)
{
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
- __get_cpu_var(wd_enabled) = 0;
+ __this_cpu_write(wd_enabled, 0);
atomic_dec(&nmi_active);
}
@@ -219,7 +219,7 @@ error:
void start_nmi_watchdog(void *unused)
{
- __get_cpu_var(wd_enabled) = 1;
+ __this_cpu_write(wd_enabled, 1);
atomic_inc(&nmi_active);
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
@@ -230,7 +230,7 @@ void start_nmi_watchdog(void *unused)
static void nmi_adjust_hz_one(void *unused)
{
- if (!__get_cpu_var(wd_enabled))
+ if (!__this_cpu_read(wd_enabled))
return;
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
Index: linux/arch/sparc/kernel/pci_sun4v.c
===================================================================
--- linux.orig/arch/sparc/kernel/pci_sun4v.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/pci_sun4v.c 2013-08-22 14:37:15.942964723 -0500
@@ -48,7 +48,7 @@ static int iommu_batch_initialized;
/* Interrupts must be disabled. */
static inline void iommu_batch_start(struct device *dev, unsigned long prot, unsigned long entry)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
p->dev = dev;
p->prot = prot;
@@ -94,7 +94,7 @@ static long iommu_batch_flush(struct iom
static inline void iommu_batch_new_entry(unsigned long entry)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
if (p->entry + p->npages == entry)
return;
@@ -106,7 +106,7 @@ static inline void iommu_batch_new_entry
/* Interrupts must be disabled. */
static inline long iommu_batch_add(u64 phys_page)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
BUG_ON(p->npages >= PGLIST_NENTS);
@@ -120,7 +120,7 @@ static inline long iommu_batch_add(u64 p
/* Interrupts must be disabled. */
static inline long iommu_batch_end(void)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
BUG_ON(p->npages >= PGLIST_NENTS);
Index: linux/arch/sparc/kernel/perf_event.c
===================================================================
--- linux.orig/arch/sparc/kernel/perf_event.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/perf_event.c 2013-08-22 14:37:15.946964681 -0500
@@ -1013,7 +1013,7 @@ static void update_pcrs_for_enable(struc
static void sparc_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
if (cpuc->enabled)
@@ -1031,7 +1031,7 @@ static void sparc_pmu_enable(struct pmu
static void sparc_pmu_disable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
if (!cpuc->enabled)
@@ -1065,7 +1065,7 @@ static int active_event_index(struct cpu
static void sparc_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
if (flags & PERF_EF_RELOAD) {
@@ -1080,7 +1080,7 @@ static void sparc_pmu_start(struct perf_
static void sparc_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
if (!(event->hw.state & PERF_HES_STOPPED)) {
@@ -1096,7 +1096,7 @@ static void sparc_pmu_stop(struct perf_e
static void sparc_pmu_del(struct perf_event *event, int _flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
unsigned long flags;
int i;
@@ -1133,7 +1133,7 @@ static void sparc_pmu_del(struct perf_ev
static void sparc_pmu_read(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
struct hw_perf_event *hwc = &event->hw;
@@ -1145,7 +1145,7 @@ static DEFINE_MUTEX(pmc_grab_mutex);
static void perf_stop_nmi_watchdog(void *unused)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
stop_nmi_watchdog(NULL);
@@ -1356,7 +1356,7 @@ static int collect_events(struct perf_ev
static int sparc_pmu_add(struct perf_event *event, int ef_flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int n0, ret = -EAGAIN;
unsigned long flags;
@@ -1498,7 +1498,7 @@ static int sparc_pmu_event_init(struct p
*/
static void sparc_pmu_start_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
perf_pmu_disable(pmu);
cpuhw->group_flag |= PERF_EVENT_TXN;
@@ -1511,7 +1511,7 @@ static void sparc_pmu_start_txn(struct p
*/
static void sparc_pmu_cancel_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
cpuhw->group_flag &= ~PERF_EVENT_TXN;
perf_pmu_enable(pmu);
@@ -1524,13 +1524,13 @@ static void sparc_pmu_cancel_txn(struct
*/
static int sparc_pmu_commit_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int n;
if (!sparc_pmu)
return -EINVAL;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
n = cpuc->n_events;
if (check_excludes(cpuc->event, 0, n))
return -EINVAL;
@@ -1601,7 +1601,7 @@ static int __kprobes perf_event_nmi_hand
regs = args->regs;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
/* If the PMU has the TOE IRQ enable bits, we need to do a
* dummy write to the %pcr to clear the overflow bits and thus
Index: linux/arch/sparc/kernel/sun4d_smp.c
===================================================================
--- linux.orig/arch/sparc/kernel/sun4d_smp.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/sun4d_smp.c 2013-08-22 14:37:15.946964681 -0500
@@ -204,7 +204,7 @@ static void __init smp4d_ipi_init(void)
void sun4d_ipi_interrupt(void)
{
- struct sun4d_ipi_work *work = &__get_cpu_var(sun4d_ipi_work);
+ struct sun4d_ipi_work *work = this_cpu_ptr(&sun4d_ipi_work);
if (work->single) {
work->single = 0;
Index: linux/arch/sparc/kernel/time_64.c
===================================================================
--- linux.orig/arch/sparc/kernel/time_64.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/time_64.c 2013-08-22 14:37:15.946964681 -0500
@@ -766,7 +766,7 @@ void setup_sparc64_timer(void)
: /* no outputs */
: "r" (pstate));
- sevt = &__get_cpu_var(sparc64_events);
+ sevt = this_cpu_ptr(&sparc64_events);
memcpy(sevt, &sparc64_clockevent, sizeof(*sevt));
sevt->cpumask = cpumask_of(smp_processor_id());
Index: linux/arch/sparc/mm/tlb.c
===================================================================
--- linux.orig/arch/sparc/mm/tlb.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/mm/tlb.c 2013-08-22 14:37:15.946964681 -0500
@@ -53,14 +53,14 @@ out:
void arch_enter_lazy_mmu_mode(void)
{
- struct tlb_batch *tb = &__get_cpu_var(tlb_batch);
+ struct tlb_batch *tb = this_cpu_ptr(&tlb_batch);
tb->active = 1;
}
void arch_leave_lazy_mmu_mode(void)
{
- struct tlb_batch *tb = &__get_cpu_var(tlb_batch);
+ struct tlb_batch *tb = this_cpu_ptr(&tlb_batch);
if (tb->tlb_nr)
flush_tlb_pending();
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 21/31] sparc: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, David S. Miller, sparclinux, linux-arch, Steven Rostedt,
linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/sparc/include/asm/cpudata_32.h
===================================================================
--- linux.orig/arch/sparc/include/asm/cpudata_32.h 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/include/asm/cpudata_32.h 2013-08-22 14:37:15.942964723 -0500
@@ -26,6 +26,6 @@ typedef struct {
DECLARE_PER_CPU(cpuinfo_sparc, __cpu_data);
#define cpu_data(__cpu) per_cpu(__cpu_data, (__cpu))
-#define local_cpu_data() __get_cpu_var(__cpu_data)
+#define local_cpu_data() __this_cpu_read(__cpu_data)
#endif /* _SPARC_CPUDATA_H */
Index: linux/arch/sparc/include/asm/cpudata_64.h
===================================================================
--- linux.orig/arch/sparc/include/asm/cpudata_64.h 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/include/asm/cpudata_64.h 2013-08-22 14:37:15.942964723 -0500
@@ -33,7 +33,7 @@ typedef struct {
DECLARE_PER_CPU(cpuinfo_sparc, __cpu_data);
#define cpu_data(__cpu) per_cpu(__cpu_data, (__cpu))
-#define local_cpu_data() __get_cpu_var(__cpu_data)
+#define local_cpu_data() __this_cpu_read(__cpu_data)
extern const struct seq_operations cpuinfo_op;
Index: linux/arch/sparc/kernel/kprobes.c
===================================================================
--- linux.orig/arch/sparc/kernel/kprobes.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/kprobes.c 2013-08-22 14:37:15.942964723 -0500
@@ -82,7 +82,7 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
kcb->kprobe_orig_tnpc = kcb->prev_kprobe.orig_tnpc;
kcb->kprobe_orig_tstate_pil = kcb->prev_kprobe.orig_tstate_pil;
@@ -91,7 +91,7 @@ static void __kprobes restore_previous_k
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
kcb->kprobe_orig_tnpc = regs->tnpc;
kcb->kprobe_orig_tstate_pil = (regs->tstate & TSTATE_PIL);
}
@@ -154,7 +154,7 @@ static int __kprobes kprobe_handler(stru
ret = 1;
goto no_kprobe;
}
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs))
goto ss_probe;
}
Index: linux/arch/sparc/kernel/leon_smp.c
===================================================================
--- linux.orig/arch/sparc/kernel/leon_smp.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/leon_smp.c 2013-08-22 14:37:15.942964723 -0500
@@ -354,7 +354,7 @@ static void leon_ipi_resched(int cpu)
void leonsmp_ipi_interrupt(void)
{
- struct leon_ipi_work *work = &__get_cpu_var(leon_ipi_work);
+ struct leon_ipi_work *work = this_cpu_ptr(&leon_ipi_work);
if (work->single) {
work->single = 0;
Index: linux/arch/sparc/kernel/nmi.c
===================================================================
--- linux.orig/arch/sparc/kernel/nmi.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/nmi.c 2013-08-22 14:37:15.942964723 -0500
@@ -111,20 +111,20 @@ notrace __kprobes void perfctr_irq(int i
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
sum = local_cpu_data().irq0_irqs;
- if (__get_cpu_var(nmi_touch)) {
- __get_cpu_var(nmi_touch) = 0;
+ if (__this_cpu_read(nmi_touch)) {
+ __this_cpu_write(nmi_touch, 0);
touched = 1;
}
- if (!touched && __get_cpu_var(last_irq_sum) == sum) {
+ if (!touched && __this_cpu_read(last_irq_sum) == sum) {
__this_cpu_inc(alert_counter);
if (__this_cpu_read(alert_counter) == 30 * nmi_hz)
die_nmi("BUG: NMI Watchdog detected LOCKUP",
regs, panic_on_timeout);
} else {
- __get_cpu_var(last_irq_sum) = sum;
+ __this_cpu_write(last_irq_sum, sum);
__this_cpu_write(alert_counter, 0);
}
- if (__get_cpu_var(wd_enabled)) {
+ if (__this_cpu_read(wd_enabled)) {
pcr_ops->write_pic(0, pcr_ops->nmi_picl_value(nmi_hz));
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_enable);
}
@@ -166,7 +166,7 @@ static void report_broken_nmi(int cpu, i
void stop_nmi_watchdog(void *unused)
{
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
- __get_cpu_var(wd_enabled) = 0;
+ __this_cpu_write(wd_enabled, 0);
atomic_dec(&nmi_active);
}
@@ -219,7 +219,7 @@ error:
void start_nmi_watchdog(void *unused)
{
- __get_cpu_var(wd_enabled) = 1;
+ __this_cpu_write(wd_enabled, 1);
atomic_inc(&nmi_active);
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
@@ -230,7 +230,7 @@ void start_nmi_watchdog(void *unused)
static void nmi_adjust_hz_one(void *unused)
{
- if (!__get_cpu_var(wd_enabled))
+ if (!__this_cpu_read(wd_enabled))
return;
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
Index: linux/arch/sparc/kernel/pci_sun4v.c
===================================================================
--- linux.orig/arch/sparc/kernel/pci_sun4v.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/pci_sun4v.c 2013-08-22 14:37:15.942964723 -0500
@@ -48,7 +48,7 @@ static int iommu_batch_initialized;
/* Interrupts must be disabled. */
static inline void iommu_batch_start(struct device *dev, unsigned long prot, unsigned long entry)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
p->dev = dev;
p->prot = prot;
@@ -94,7 +94,7 @@ static long iommu_batch_flush(struct iom
static inline void iommu_batch_new_entry(unsigned long entry)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
if (p->entry + p->npages == entry)
return;
@@ -106,7 +106,7 @@ static inline void iommu_batch_new_entry
/* Interrupts must be disabled. */
static inline long iommu_batch_add(u64 phys_page)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
BUG_ON(p->npages >= PGLIST_NENTS);
@@ -120,7 +120,7 @@ static inline long iommu_batch_add(u64 p
/* Interrupts must be disabled. */
static inline long iommu_batch_end(void)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
BUG_ON(p->npages >= PGLIST_NENTS);
Index: linux/arch/sparc/kernel/perf_event.c
===================================================================
--- linux.orig/arch/sparc/kernel/perf_event.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/perf_event.c 2013-08-22 14:37:15.946964681 -0500
@@ -1013,7 +1013,7 @@ static void update_pcrs_for_enable(struc
static void sparc_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
if (cpuc->enabled)
@@ -1031,7 +1031,7 @@ static void sparc_pmu_enable(struct pmu
static void sparc_pmu_disable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
if (!cpuc->enabled)
@@ -1065,7 +1065,7 @@ static int active_event_index(struct cpu
static void sparc_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
if (flags & PERF_EF_RELOAD) {
@@ -1080,7 +1080,7 @@ static void sparc_pmu_start(struct perf_
static void sparc_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
if (!(event->hw.state & PERF_HES_STOPPED)) {
@@ -1096,7 +1096,7 @@ static void sparc_pmu_stop(struct perf_e
static void sparc_pmu_del(struct perf_event *event, int _flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
unsigned long flags;
int i;
@@ -1133,7 +1133,7 @@ static void sparc_pmu_del(struct perf_ev
static void sparc_pmu_read(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
struct hw_perf_event *hwc = &event->hw;
@@ -1145,7 +1145,7 @@ static DEFINE_MUTEX(pmc_grab_mutex);
static void perf_stop_nmi_watchdog(void *unused)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
stop_nmi_watchdog(NULL);
@@ -1356,7 +1356,7 @@ static int collect_events(struct perf_ev
static int sparc_pmu_add(struct perf_event *event, int ef_flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int n0, ret = -EAGAIN;
unsigned long flags;
@@ -1498,7 +1498,7 @@ static int sparc_pmu_event_init(struct p
*/
static void sparc_pmu_start_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
perf_pmu_disable(pmu);
cpuhw->group_flag |= PERF_EVENT_TXN;
@@ -1511,7 +1511,7 @@ static void sparc_pmu_start_txn(struct p
*/
static void sparc_pmu_cancel_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
cpuhw->group_flag &= ~PERF_EVENT_TXN;
perf_pmu_enable(pmu);
@@ -1524,13 +1524,13 @@ static void sparc_pmu_cancel_txn(struct
*/
static int sparc_pmu_commit_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int n;
if (!sparc_pmu)
return -EINVAL;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
n = cpuc->n_events;
if (check_excludes(cpuc->event, 0, n))
return -EINVAL;
@@ -1601,7 +1601,7 @@ static int __kprobes perf_event_nmi_hand
regs = args->regs;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
/* If the PMU has the TOE IRQ enable bits, we need to do a
* dummy write to the %pcr to clear the overflow bits and thus
Index: linux/arch/sparc/kernel/sun4d_smp.c
===================================================================
--- linux.orig/arch/sparc/kernel/sun4d_smp.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/sun4d_smp.c 2013-08-22 14:37:15.946964681 -0500
@@ -204,7 +204,7 @@ static void __init smp4d_ipi_init(void)
void sun4d_ipi_interrupt(void)
{
- struct sun4d_ipi_work *work = &__get_cpu_var(sun4d_ipi_work);
+ struct sun4d_ipi_work *work = this_cpu_ptr(&sun4d_ipi_work);
if (work->single) {
work->single = 0;
Index: linux/arch/sparc/kernel/time_64.c
===================================================================
--- linux.orig/arch/sparc/kernel/time_64.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/time_64.c 2013-08-22 14:37:15.946964681 -0500
@@ -766,7 +766,7 @@ void setup_sparc64_timer(void)
: /* no outputs */
: "r" (pstate));
- sevt = &__get_cpu_var(sparc64_events);
+ sevt = this_cpu_ptr(&sparc64_events);
memcpy(sevt, &sparc64_clockevent, sizeof(*sevt));
sevt->cpumask = cpumask_of(smp_processor_id());
Index: linux/arch/sparc/mm/tlb.c
===================================================================
--- linux.orig/arch/sparc/mm/tlb.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/mm/tlb.c 2013-08-22 14:37:15.946964681 -0500
@@ -53,14 +53,14 @@ out:
void arch_enter_lazy_mmu_mode(void)
{
- struct tlb_batch *tb = &__get_cpu_var(tlb_batch);
+ struct tlb_batch *tb = this_cpu_ptr(&tlb_batch);
tb->active = 1;
}
void arch_leave_lazy_mmu_mode(void)
{
- struct tlb_batch *tb = &__get_cpu_var(tlb_batch);
+ struct tlb_batch *tb = this_cpu_ptr(&tlb_batch);
if (tb->tlb_nr)
flush_tlb_pending();
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 21/31] sparc: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, David S. Miller, sparclinux, linux-arch, Steven Rostedt,
linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/sparc/include/asm/cpudata_32.h
=================================--- linux.orig/arch/sparc/include/asm/cpudata_32.h 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/include/asm/cpudata_32.h 2013-08-22 14:37:15.942964723 -0500
@@ -26,6 +26,6 @@ typedef struct {
DECLARE_PER_CPU(cpuinfo_sparc, __cpu_data);
#define cpu_data(__cpu) per_cpu(__cpu_data, (__cpu))
-#define local_cpu_data() __get_cpu_var(__cpu_data)
+#define local_cpu_data() __this_cpu_read(__cpu_data)
#endif /* _SPARC_CPUDATA_H */
Index: linux/arch/sparc/include/asm/cpudata_64.h
=================================--- linux.orig/arch/sparc/include/asm/cpudata_64.h 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/include/asm/cpudata_64.h 2013-08-22 14:37:15.942964723 -0500
@@ -33,7 +33,7 @@ typedef struct {
DECLARE_PER_CPU(cpuinfo_sparc, __cpu_data);
#define cpu_data(__cpu) per_cpu(__cpu_data, (__cpu))
-#define local_cpu_data() __get_cpu_var(__cpu_data)
+#define local_cpu_data() __this_cpu_read(__cpu_data)
extern const struct seq_operations cpuinfo_op;
Index: linux/arch/sparc/kernel/kprobes.c
=================================--- linux.orig/arch/sparc/kernel/kprobes.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/kprobes.c 2013-08-22 14:37:15.942964723 -0500
@@ -82,7 +82,7 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
kcb->kprobe_orig_tnpc = kcb->prev_kprobe.orig_tnpc;
kcb->kprobe_orig_tstate_pil = kcb->prev_kprobe.orig_tstate_pil;
@@ -91,7 +91,7 @@ static void __kprobes restore_previous_k
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
kcb->kprobe_orig_tnpc = regs->tnpc;
kcb->kprobe_orig_tstate_pil = (regs->tstate & TSTATE_PIL);
}
@@ -154,7 +154,7 @@ static int __kprobes kprobe_handler(stru
ret = 1;
goto no_kprobe;
}
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs))
goto ss_probe;
}
Index: linux/arch/sparc/kernel/leon_smp.c
=================================--- linux.orig/arch/sparc/kernel/leon_smp.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/leon_smp.c 2013-08-22 14:37:15.942964723 -0500
@@ -354,7 +354,7 @@ static void leon_ipi_resched(int cpu)
void leonsmp_ipi_interrupt(void)
{
- struct leon_ipi_work *work = &__get_cpu_var(leon_ipi_work);
+ struct leon_ipi_work *work = this_cpu_ptr(&leon_ipi_work);
if (work->single) {
work->single = 0;
Index: linux/arch/sparc/kernel/nmi.c
=================================--- linux.orig/arch/sparc/kernel/nmi.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/nmi.c 2013-08-22 14:37:15.942964723 -0500
@@ -111,20 +111,20 @@ notrace __kprobes void perfctr_irq(int i
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
sum = local_cpu_data().irq0_irqs;
- if (__get_cpu_var(nmi_touch)) {
- __get_cpu_var(nmi_touch) = 0;
+ if (__this_cpu_read(nmi_touch)) {
+ __this_cpu_write(nmi_touch, 0);
touched = 1;
}
- if (!touched && __get_cpu_var(last_irq_sum) = sum) {
+ if (!touched && __this_cpu_read(last_irq_sum) = sum) {
__this_cpu_inc(alert_counter);
if (__this_cpu_read(alert_counter) = 30 * nmi_hz)
die_nmi("BUG: NMI Watchdog detected LOCKUP",
regs, panic_on_timeout);
} else {
- __get_cpu_var(last_irq_sum) = sum;
+ __this_cpu_write(last_irq_sum, sum);
__this_cpu_write(alert_counter, 0);
}
- if (__get_cpu_var(wd_enabled)) {
+ if (__this_cpu_read(wd_enabled)) {
pcr_ops->write_pic(0, pcr_ops->nmi_picl_value(nmi_hz));
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_enable);
}
@@ -166,7 +166,7 @@ static void report_broken_nmi(int cpu, i
void stop_nmi_watchdog(void *unused)
{
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
- __get_cpu_var(wd_enabled) = 0;
+ __this_cpu_write(wd_enabled, 0);
atomic_dec(&nmi_active);
}
@@ -219,7 +219,7 @@ error:
void start_nmi_watchdog(void *unused)
{
- __get_cpu_var(wd_enabled) = 1;
+ __this_cpu_write(wd_enabled, 1);
atomic_inc(&nmi_active);
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
@@ -230,7 +230,7 @@ void start_nmi_watchdog(void *unused)
static void nmi_adjust_hz_one(void *unused)
{
- if (!__get_cpu_var(wd_enabled))
+ if (!__this_cpu_read(wd_enabled))
return;
pcr_ops->write_pcr(0, pcr_ops->pcr_nmi_disable);
Index: linux/arch/sparc/kernel/pci_sun4v.c
=================================--- linux.orig/arch/sparc/kernel/pci_sun4v.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/pci_sun4v.c 2013-08-22 14:37:15.942964723 -0500
@@ -48,7 +48,7 @@ static int iommu_batch_initialized;
/* Interrupts must be disabled. */
static inline void iommu_batch_start(struct device *dev, unsigned long prot, unsigned long entry)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
p->dev = dev;
p->prot = prot;
@@ -94,7 +94,7 @@ static long iommu_batch_flush(struct iom
static inline void iommu_batch_new_entry(unsigned long entry)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
if (p->entry + p->npages = entry)
return;
@@ -106,7 +106,7 @@ static inline void iommu_batch_new_entry
/* Interrupts must be disabled. */
static inline long iommu_batch_add(u64 phys_page)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
BUG_ON(p->npages >= PGLIST_NENTS);
@@ -120,7 +120,7 @@ static inline long iommu_batch_add(u64 p
/* Interrupts must be disabled. */
static inline long iommu_batch_end(void)
{
- struct iommu_batch *p = &__get_cpu_var(iommu_batch);
+ struct iommu_batch *p = this_cpu_ptr(&iommu_batch);
BUG_ON(p->npages >= PGLIST_NENTS);
Index: linux/arch/sparc/kernel/perf_event.c
=================================--- linux.orig/arch/sparc/kernel/perf_event.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/perf_event.c 2013-08-22 14:37:15.946964681 -0500
@@ -1013,7 +1013,7 @@ static void update_pcrs_for_enable(struc
static void sparc_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
if (cpuc->enabled)
@@ -1031,7 +1031,7 @@ static void sparc_pmu_enable(struct pmu
static void sparc_pmu_disable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
if (!cpuc->enabled)
@@ -1065,7 +1065,7 @@ static int active_event_index(struct cpu
static void sparc_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
if (flags & PERF_EF_RELOAD) {
@@ -1080,7 +1080,7 @@ static void sparc_pmu_start(struct perf_
static void sparc_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
if (!(event->hw.state & PERF_HES_STOPPED)) {
@@ -1096,7 +1096,7 @@ static void sparc_pmu_stop(struct perf_e
static void sparc_pmu_del(struct perf_event *event, int _flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
unsigned long flags;
int i;
@@ -1133,7 +1133,7 @@ static void sparc_pmu_del(struct perf_ev
static void sparc_pmu_read(struct perf_event *event)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx = active_event_index(cpuc, event);
struct hw_perf_event *hwc = &event->hw;
@@ -1145,7 +1145,7 @@ static DEFINE_MUTEX(pmc_grab_mutex);
static void perf_stop_nmi_watchdog(void *unused)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int i;
stop_nmi_watchdog(NULL);
@@ -1356,7 +1356,7 @@ static int collect_events(struct perf_ev
static int sparc_pmu_add(struct perf_event *event, int ef_flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int n0, ret = -EAGAIN;
unsigned long flags;
@@ -1498,7 +1498,7 @@ static int sparc_pmu_event_init(struct p
*/
static void sparc_pmu_start_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
perf_pmu_disable(pmu);
cpuhw->group_flag |= PERF_EVENT_TXN;
@@ -1511,7 +1511,7 @@ static void sparc_pmu_start_txn(struct p
*/
static void sparc_pmu_cancel_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
cpuhw->group_flag &= ~PERF_EVENT_TXN;
perf_pmu_enable(pmu);
@@ -1524,13 +1524,13 @@ static void sparc_pmu_cancel_txn(struct
*/
static int sparc_pmu_commit_txn(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int n;
if (!sparc_pmu)
return -EINVAL;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
n = cpuc->n_events;
if (check_excludes(cpuc->event, 0, n))
return -EINVAL;
@@ -1601,7 +1601,7 @@ static int __kprobes perf_event_nmi_hand
regs = args->regs;
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
/* If the PMU has the TOE IRQ enable bits, we need to do a
* dummy write to the %pcr to clear the overflow bits and thus
Index: linux/arch/sparc/kernel/sun4d_smp.c
=================================--- linux.orig/arch/sparc/kernel/sun4d_smp.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/sun4d_smp.c 2013-08-22 14:37:15.946964681 -0500
@@ -204,7 +204,7 @@ static void __init smp4d_ipi_init(void)
void sun4d_ipi_interrupt(void)
{
- struct sun4d_ipi_work *work = &__get_cpu_var(sun4d_ipi_work);
+ struct sun4d_ipi_work *work = this_cpu_ptr(&sun4d_ipi_work);
if (work->single) {
work->single = 0;
Index: linux/arch/sparc/kernel/time_64.c
=================================--- linux.orig/arch/sparc/kernel/time_64.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/kernel/time_64.c 2013-08-22 14:37:15.946964681 -0500
@@ -766,7 +766,7 @@ void setup_sparc64_timer(void)
: /* no outputs */
: "r" (pstate));
- sevt = &__get_cpu_var(sparc64_events);
+ sevt = this_cpu_ptr(&sparc64_events);
memcpy(sevt, &sparc64_clockevent, sizeof(*sevt));
sevt->cpumask = cpumask_of(smp_processor_id());
Index: linux/arch/sparc/mm/tlb.c
=================================--- linux.orig/arch/sparc/mm/tlb.c 2013-08-22 14:34:54.000000000 -0500
+++ linux/arch/sparc/mm/tlb.c 2013-08-22 14:37:15.946964681 -0500
@@ -53,14 +53,14 @@ out:
void arch_enter_lazy_mmu_mode(void)
{
- struct tlb_batch *tb = &__get_cpu_var(tlb_batch);
+ struct tlb_batch *tb = this_cpu_ptr(&tlb_batch);
tb->active = 1;
}
void arch_leave_lazy_mmu_mode(void)
{
- struct tlb_batch *tb = &__get_cpu_var(tlb_batch);
+ struct tlb_batch *tb = this_cpu_ptr(&tlb_batch);
if (tb->tlb_nr)
flush_tlb_pending();
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 22/31] arm: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Russell King, Catalin Marinas, Will Deacon, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/arm/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/arm/kernel/hw_breakpoint.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm/kernel/hw_breakpoint.c 2013-08-26 13:48:40.952795024 -0500
@@ -344,13 +344,13 @@ int arch_install_hw_breakpoint(struct pe
/* Breakpoint */
ctrl_base = ARM_BASE_BCR;
val_base = ARM_BASE_BVR;
- slots = (struct perf_event **)__get_cpu_var(bp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
} else {
/* Watchpoint */
ctrl_base = ARM_BASE_WCR;
val_base = ARM_BASE_WVR;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
}
@@ -396,12 +396,12 @@ void arch_uninstall_hw_breakpoint(struct
if (info->ctrl.type == ARM_BREAKPOINT_EXECUTE) {
/* Breakpoint */
base = ARM_BASE_BCR;
- slots = (struct perf_event **)__get_cpu_var(bp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
} else {
/* Watchpoint */
base = ARM_BASE_WCR;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
}
@@ -697,7 +697,7 @@ static void watchpoint_handler(unsigned
struct arch_hw_breakpoint *info;
struct arch_hw_breakpoint_ctrl ctrl;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
for (i = 0; i < core_num_wrps; ++i) {
rcu_read_lock();
@@ -768,7 +768,7 @@ static void watchpoint_single_step_handl
struct perf_event *wp, **slots;
struct arch_hw_breakpoint *info;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
for (i = 0; i < core_num_wrps; ++i) {
rcu_read_lock();
@@ -802,7 +802,7 @@ static void breakpoint_handler(unsigned
struct arch_hw_breakpoint *info;
struct arch_hw_breakpoint_ctrl ctrl;
- slots = (struct perf_event **)__get_cpu_var(bp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(bp_on_reg);
/* The exception entry code places the amended lr in the PC. */
addr = regs->ARM_pc;
Index: linux/arch/arm/kernel/kprobes.c
===================================================================
--- linux.orig/arch/arm/kernel/kprobes.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm/kernel/kprobes.c 2013-08-26 13:48:40.952795024 -0500
@@ -171,13 +171,13 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
static void __kprobes set_current_kprobe(struct kprobe *p)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static void __kprobes
@@ -421,10 +421,10 @@ static __used __kprobes void *trampoline
continue;
if (ri->rp && ri->rp->handler) {
- __get_cpu_var(current_kprobe) = &ri->rp->kp;
+ __this_cpu_write(current_kprobe, &ri->rp->kp);
get_kprobe_ctlblk()->kprobe_status = KPROBE_HIT_ACTIVE;
ri->rp->handler(ri, regs);
- __get_cpu_var(current_kprobe) = NULL;
+ __this_cpu_write(current_kprobe, NULL);
}
orig_ret_address = (unsigned long)ri->ret_addr;
Index: linux/arch/arm/kernel/perf_event_cpu.c
===================================================================
--- linux.orig/arch/arm/kernel/perf_event_cpu.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm/kernel/perf_event_cpu.c 2013-08-26 13:48:40.952795024 -0500
@@ -68,7 +68,7 @@ EXPORT_SYMBOL_GPL(perf_num_counters);
static struct pmu_hw_events *cpu_pmu_get_cpu_events(void)
{
- return &__get_cpu_var(cpu_hw_events);
+ return this_cpu_ptr(&cpu_hw_events);
}
static void cpu_pmu_free_irq(struct arm_pmu *cpu_pmu)
Index: linux/arch/arm/kvm/arm.c
===================================================================
--- linux.orig/arch/arm/kvm/arm.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm/kvm/arm.c 2013-08-26 13:48:40.952795024 -0500
@@ -65,7 +65,7 @@ static bool vgic_present;
static void kvm_arm_set_running_vcpu(struct kvm_vcpu *vcpu)
{
BUG_ON(preemptible());
- __get_cpu_var(kvm_arm_running_vcpu) = vcpu;
+ __this_cpu_write(kvm_arm_running_vcpu, vcpu);
}
/**
@@ -75,7 +75,7 @@ static void kvm_arm_set_running_vcpu(str
struct kvm_vcpu *kvm_arm_get_running_vcpu(void)
{
BUG_ON(preemptible());
- return __get_cpu_var(kvm_arm_running_vcpu);
+ return __this_cpu_read(kvm_arm_running_vcpu);
}
/**
@@ -811,7 +811,7 @@ static void cpu_init_hyp_mode(void *dumm
boot_pgd_ptr = kvm_mmu_get_boot_httbr();
pgd_ptr = kvm_mmu_get_httbr();
- stack_page = __get_cpu_var(kvm_arm_hyp_stack_page);
+ stack_page = __this_cpu_read(kvm_arm_hyp_stack_page);
hyp_stack_ptr = stack_page + PAGE_SIZE;
vector_ptr = (unsigned long)__kvm_hyp_vector;
Index: linux/arch/arm64/kernel/debug-monitors.c
===================================================================
--- linux.orig/arch/arm64/kernel/debug-monitors.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm64/kernel/debug-monitors.c 2013-08-26 13:48:40.952795024 -0500
@@ -98,11 +98,11 @@ void enable_debug_monitors(enum debug_el
WARN_ON(preemptible());
- if (local_inc_return(&__get_cpu_var(mde_ref_count)) == 1)
+ if (this_cpu_inc_return(mde_ref_count) == 1)
enable = DBG_MDSCR_MDE;
if (el == DBG_ACTIVE_EL1 &&
- local_inc_return(&__get_cpu_var(kde_ref_count)) == 1)
+ this_cpu_inc_return(kde_ref_count) == 1)
enable |= DBG_MDSCR_KDE;
if (enable && debug_enabled) {
@@ -118,11 +118,11 @@ void disable_debug_monitors(enum debug_e
WARN_ON(preemptible());
- if (local_dec_and_test(&__get_cpu_var(mde_ref_count)))
+ if (local_dec_and_test(this_cpu_ptr(&mde_ref_count)))
disable = ~DBG_MDSCR_MDE;
if (el == DBG_ACTIVE_EL1 &&
- local_dec_and_test(&__get_cpu_var(kde_ref_count)))
+ local_dec_and_test(this_cpu_ptr(&kde_ref_count)))
disable &= ~DBG_MDSCR_KDE;
if (disable) {
Index: linux/arch/arm64/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/arm64/kernel/hw_breakpoint.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm64/kernel/hw_breakpoint.c 2013-08-26 13:48:40.952795024 -0500
@@ -184,14 +184,14 @@ int arch_install_hw_breakpoint(struct pe
/* Breakpoint */
ctrl_reg = AARCH64_DBG_REG_BCR;
val_reg = AARCH64_DBG_REG_BVR;
- slots = __get_cpu_var(bp_on_reg);
+ slots = __this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
reg_enable = !debug_info->bps_disabled;
} else {
/* Watchpoint */
ctrl_reg = AARCH64_DBG_REG_WCR;
val_reg = AARCH64_DBG_REG_WVR;
- slots = __get_cpu_var(wp_on_reg);
+ slots = __this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
reg_enable = !debug_info->wps_disabled;
}
@@ -230,12 +230,12 @@ void arch_uninstall_hw_breakpoint(struct
if (info->ctrl.type == ARM_BREAKPOINT_EXECUTE) {
/* Breakpoint */
base = AARCH64_DBG_REG_BCR;
- slots = __get_cpu_var(bp_on_reg);
+ slots = __this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
} else {
/* Watchpoint */
base = AARCH64_DBG_REG_WCR;
- slots = __get_cpu_var(wp_on_reg);
+ slots = __this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
}
@@ -505,11 +505,11 @@ static void toggle_bp_registers(int reg,
switch (reg) {
case AARCH64_DBG_REG_BCR:
- slots = __get_cpu_var(bp_on_reg);
+ slots = __this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
break;
case AARCH64_DBG_REG_WCR:
- slots = __get_cpu_var(wp_on_reg);
+ slots = __this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
break;
default:
@@ -546,7 +546,7 @@ static int breakpoint_handler(unsigned l
struct debug_info *debug_info;
struct arch_hw_breakpoint_ctrl ctrl;
- slots = (struct perf_event **)__get_cpu_var(bp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(bp_on_reg);
addr = instruction_pointer(regs);
debug_info = ¤t->thread.debug;
@@ -596,7 +596,7 @@ unlock:
user_enable_single_step(current);
} else {
toggle_bp_registers(AARCH64_DBG_REG_BCR, DBG_ACTIVE_EL1, 0);
- kernel_step = &__get_cpu_var(stepping_kernel_bp);
+ kernel_step = this_cpu_ptr(&stepping_kernel_bp);
if (*kernel_step != ARM_KERNEL_STEP_NONE)
return 0;
@@ -623,7 +623,7 @@ static int watchpoint_handler(unsigned l
struct arch_hw_breakpoint *info;
struct arch_hw_breakpoint_ctrl ctrl;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
debug_info = ¤t->thread.debug;
for (i = 0; i < core_num_wrps; ++i) {
@@ -698,7 +698,7 @@ unlock:
user_enable_single_step(current);
} else {
toggle_bp_registers(AARCH64_DBG_REG_WCR, DBG_ACTIVE_EL1, 0);
- kernel_step = &__get_cpu_var(stepping_kernel_bp);
+ kernel_step = this_cpu_ptr(&stepping_kernel_bp);
if (*kernel_step != ARM_KERNEL_STEP_NONE)
return 0;
@@ -722,7 +722,7 @@ int reinstall_suspended_bps(struct pt_re
struct debug_info *debug_info = ¤t->thread.debug;
int handled_exception = 0, *kernel_step;
- kernel_step = &__get_cpu_var(stepping_kernel_bp);
+ kernel_step = this_cpu_ptr(&stepping_kernel_bp);
/*
* Called from single-step exception handler.
Index: linux/arch/arm64/kernel/perf_event.c
===================================================================
--- linux.orig/arch/arm64/kernel/perf_event.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm64/kernel/perf_event.c 2013-08-26 13:48:40.952795024 -0500
@@ -1041,7 +1041,7 @@ static irqreturn_t armv8pmu_handle_irq(i
*/
regs = get_irq_regs();
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
for (idx = 0; idx < cpu_pmu->num_events; ++idx) {
struct perf_event *event = cpuc->events[idx];
struct hw_perf_event *hwc;
@@ -1254,7 +1254,7 @@ device_initcall(register_pmu_driver);
static struct pmu_hw_events *armpmu_get_cpu_events(void)
{
- return &__get_cpu_var(cpu_hw_events);
+ return this_cpu_ptr(&cpu_hw_events);
}
static void __init cpu_pmu_init(struct arm_pmu *armpmu)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 22/31] arm: Replace __get_cpu_var uses
@ 2013-08-26 20:44 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:44 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Russell King, Catalin Marinas, Will Deacon, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/arm/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/arm/kernel/hw_breakpoint.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm/kernel/hw_breakpoint.c 2013-08-26 13:48:40.952795024 -0500
@@ -344,13 +344,13 @@ int arch_install_hw_breakpoint(struct pe
/* Breakpoint */
ctrl_base = ARM_BASE_BCR;
val_base = ARM_BASE_BVR;
- slots = (struct perf_event **)__get_cpu_var(bp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
} else {
/* Watchpoint */
ctrl_base = ARM_BASE_WCR;
val_base = ARM_BASE_WVR;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
}
@@ -396,12 +396,12 @@ void arch_uninstall_hw_breakpoint(struct
if (info->ctrl.type == ARM_BREAKPOINT_EXECUTE) {
/* Breakpoint */
base = ARM_BASE_BCR;
- slots = (struct perf_event **)__get_cpu_var(bp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
} else {
/* Watchpoint */
base = ARM_BASE_WCR;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
}
@@ -697,7 +697,7 @@ static void watchpoint_handler(unsigned
struct arch_hw_breakpoint *info;
struct arch_hw_breakpoint_ctrl ctrl;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
for (i = 0; i < core_num_wrps; ++i) {
rcu_read_lock();
@@ -768,7 +768,7 @@ static void watchpoint_single_step_handl
struct perf_event *wp, **slots;
struct arch_hw_breakpoint *info;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
for (i = 0; i < core_num_wrps; ++i) {
rcu_read_lock();
@@ -802,7 +802,7 @@ static void breakpoint_handler(unsigned
struct arch_hw_breakpoint *info;
struct arch_hw_breakpoint_ctrl ctrl;
- slots = (struct perf_event **)__get_cpu_var(bp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(bp_on_reg);
/* The exception entry code places the amended lr in the PC. */
addr = regs->ARM_pc;
Index: linux/arch/arm/kernel/kprobes.c
===================================================================
--- linux.orig/arch/arm/kernel/kprobes.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm/kernel/kprobes.c 2013-08-26 13:48:40.952795024 -0500
@@ -171,13 +171,13 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
static void __kprobes set_current_kprobe(struct kprobe *p)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static void __kprobes
@@ -421,10 +421,10 @@ static __used __kprobes void *trampoline
continue;
if (ri->rp && ri->rp->handler) {
- __get_cpu_var(current_kprobe) = &ri->rp->kp;
+ __this_cpu_write(current_kprobe, &ri->rp->kp);
get_kprobe_ctlblk()->kprobe_status = KPROBE_HIT_ACTIVE;
ri->rp->handler(ri, regs);
- __get_cpu_var(current_kprobe) = NULL;
+ __this_cpu_write(current_kprobe, NULL);
}
orig_ret_address = (unsigned long)ri->ret_addr;
Index: linux/arch/arm/kernel/perf_event_cpu.c
===================================================================
--- linux.orig/arch/arm/kernel/perf_event_cpu.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm/kernel/perf_event_cpu.c 2013-08-26 13:48:40.952795024 -0500
@@ -68,7 +68,7 @@ EXPORT_SYMBOL_GPL(perf_num_counters);
static struct pmu_hw_events *cpu_pmu_get_cpu_events(void)
{
- return &__get_cpu_var(cpu_hw_events);
+ return this_cpu_ptr(&cpu_hw_events);
}
static void cpu_pmu_free_irq(struct arm_pmu *cpu_pmu)
Index: linux/arch/arm/kvm/arm.c
===================================================================
--- linux.orig/arch/arm/kvm/arm.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm/kvm/arm.c 2013-08-26 13:48:40.952795024 -0500
@@ -65,7 +65,7 @@ static bool vgic_present;
static void kvm_arm_set_running_vcpu(struct kvm_vcpu *vcpu)
{
BUG_ON(preemptible());
- __get_cpu_var(kvm_arm_running_vcpu) = vcpu;
+ __this_cpu_write(kvm_arm_running_vcpu, vcpu);
}
/**
@@ -75,7 +75,7 @@ static void kvm_arm_set_running_vcpu(str
struct kvm_vcpu *kvm_arm_get_running_vcpu(void)
{
BUG_ON(preemptible());
- return __get_cpu_var(kvm_arm_running_vcpu);
+ return __this_cpu_read(kvm_arm_running_vcpu);
}
/**
@@ -811,7 +811,7 @@ static void cpu_init_hyp_mode(void *dumm
boot_pgd_ptr = kvm_mmu_get_boot_httbr();
pgd_ptr = kvm_mmu_get_httbr();
- stack_page = __get_cpu_var(kvm_arm_hyp_stack_page);
+ stack_page = __this_cpu_read(kvm_arm_hyp_stack_page);
hyp_stack_ptr = stack_page + PAGE_SIZE;
vector_ptr = (unsigned long)__kvm_hyp_vector;
Index: linux/arch/arm64/kernel/debug-monitors.c
===================================================================
--- linux.orig/arch/arm64/kernel/debug-monitors.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm64/kernel/debug-monitors.c 2013-08-26 13:48:40.952795024 -0500
@@ -98,11 +98,11 @@ void enable_debug_monitors(enum debug_el
WARN_ON(preemptible());
- if (local_inc_return(&__get_cpu_var(mde_ref_count)) == 1)
+ if (this_cpu_inc_return(mde_ref_count) == 1)
enable = DBG_MDSCR_MDE;
if (el == DBG_ACTIVE_EL1 &&
- local_inc_return(&__get_cpu_var(kde_ref_count)) == 1)
+ this_cpu_inc_return(kde_ref_count) == 1)
enable |= DBG_MDSCR_KDE;
if (enable && debug_enabled) {
@@ -118,11 +118,11 @@ void disable_debug_monitors(enum debug_e
WARN_ON(preemptible());
- if (local_dec_and_test(&__get_cpu_var(mde_ref_count)))
+ if (local_dec_and_test(this_cpu_ptr(&mde_ref_count)))
disable = ~DBG_MDSCR_MDE;
if (el == DBG_ACTIVE_EL1 &&
- local_dec_and_test(&__get_cpu_var(kde_ref_count)))
+ local_dec_and_test(this_cpu_ptr(&kde_ref_count)))
disable &= ~DBG_MDSCR_KDE;
if (disable) {
Index: linux/arch/arm64/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/arm64/kernel/hw_breakpoint.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm64/kernel/hw_breakpoint.c 2013-08-26 13:48:40.952795024 -0500
@@ -184,14 +184,14 @@ int arch_install_hw_breakpoint(struct pe
/* Breakpoint */
ctrl_reg = AARCH64_DBG_REG_BCR;
val_reg = AARCH64_DBG_REG_BVR;
- slots = __get_cpu_var(bp_on_reg);
+ slots = __this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
reg_enable = !debug_info->bps_disabled;
} else {
/* Watchpoint */
ctrl_reg = AARCH64_DBG_REG_WCR;
val_reg = AARCH64_DBG_REG_WVR;
- slots = __get_cpu_var(wp_on_reg);
+ slots = __this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
reg_enable = !debug_info->wps_disabled;
}
@@ -230,12 +230,12 @@ void arch_uninstall_hw_breakpoint(struct
if (info->ctrl.type == ARM_BREAKPOINT_EXECUTE) {
/* Breakpoint */
base = AARCH64_DBG_REG_BCR;
- slots = __get_cpu_var(bp_on_reg);
+ slots = __this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
} else {
/* Watchpoint */
base = AARCH64_DBG_REG_WCR;
- slots = __get_cpu_var(wp_on_reg);
+ slots = __this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
}
@@ -505,11 +505,11 @@ static void toggle_bp_registers(int reg,
switch (reg) {
case AARCH64_DBG_REG_BCR:
- slots = __get_cpu_var(bp_on_reg);
+ slots = __this_cpu_read(bp_on_reg);
max_slots = core_num_brps;
break;
case AARCH64_DBG_REG_WCR:
- slots = __get_cpu_var(wp_on_reg);
+ slots = __this_cpu_read(wp_on_reg);
max_slots = core_num_wrps;
break;
default:
@@ -546,7 +546,7 @@ static int breakpoint_handler(unsigned l
struct debug_info *debug_info;
struct arch_hw_breakpoint_ctrl ctrl;
- slots = (struct perf_event **)__get_cpu_var(bp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(bp_on_reg);
addr = instruction_pointer(regs);
debug_info = ¤t->thread.debug;
@@ -596,7 +596,7 @@ unlock:
user_enable_single_step(current);
} else {
toggle_bp_registers(AARCH64_DBG_REG_BCR, DBG_ACTIVE_EL1, 0);
- kernel_step = &__get_cpu_var(stepping_kernel_bp);
+ kernel_step = this_cpu_ptr(&stepping_kernel_bp);
if (*kernel_step != ARM_KERNEL_STEP_NONE)
return 0;
@@ -623,7 +623,7 @@ static int watchpoint_handler(unsigned l
struct arch_hw_breakpoint *info;
struct arch_hw_breakpoint_ctrl ctrl;
- slots = (struct perf_event **)__get_cpu_var(wp_on_reg);
+ slots = (struct perf_event **)__this_cpu_read(wp_on_reg);
debug_info = ¤t->thread.debug;
for (i = 0; i < core_num_wrps; ++i) {
@@ -698,7 +698,7 @@ unlock:
user_enable_single_step(current);
} else {
toggle_bp_registers(AARCH64_DBG_REG_WCR, DBG_ACTIVE_EL1, 0);
- kernel_step = &__get_cpu_var(stepping_kernel_bp);
+ kernel_step = this_cpu_ptr(&stepping_kernel_bp);
if (*kernel_step != ARM_KERNEL_STEP_NONE)
return 0;
@@ -722,7 +722,7 @@ int reinstall_suspended_bps(struct pt_re
struct debug_info *debug_info = ¤t->thread.debug;
int handled_exception = 0, *kernel_step;
- kernel_step = &__get_cpu_var(stepping_kernel_bp);
+ kernel_step = this_cpu_ptr(&stepping_kernel_bp);
/*
* Called from single-step exception handler.
Index: linux/arch/arm64/kernel/perf_event.c
===================================================================
--- linux.orig/arch/arm64/kernel/perf_event.c 2013-08-26 13:48:40.956794980 -0500
+++ linux/arch/arm64/kernel/perf_event.c 2013-08-26 13:48:40.952795024 -0500
@@ -1041,7 +1041,7 @@ static irqreturn_t armv8pmu_handle_irq(i
*/
regs = get_irq_regs();
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
for (idx = 0; idx < cpu_pmu->num_events; ++idx) {
struct perf_event *event = cpuc->events[idx];
struct hw_perf_event *hwc;
@@ -1254,7 +1254,7 @@ device_initcall(register_pmu_driver);
static struct pmu_hw_events *armpmu_get_cpu_events(void)
{
- return &__get_cpu_var(cpu_hw_events);
+ return this_cpu_ptr(&cpu_hw_events);
}
static void __init cpu_pmu_init(struct arm_pmu *armpmu)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 25/31] avr32: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:56 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:56 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Haavard Skinnemoen, Hans-Christian Egtvedt, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/avr32/kernel/kprobes.c
===================================================================
--- linux.orig/arch/avr32/kernel/kprobes.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/avr32/kernel/kprobes.c 2013-08-26 13:27:09.758485808 -0500
@@ -104,7 +104,7 @@ static void __kprobes resume_execution(s
static void __kprobes set_current_kprobe(struct kprobe *p)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static int __kprobes kprobe_handler(struct pt_regs *regs)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 25/31] avr32: Replace __get_cpu_var uses
@ 2013-08-26 20:56 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:56 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Haavard Skinnemoen, Hans-Christian Egtvedt, linux-arch,
Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/avr32/kernel/kprobes.c
===================================================================
--- linux.orig/arch/avr32/kernel/kprobes.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/avr32/kernel/kprobes.c 2013-08-26 13:27:09.758485808 -0500
@@ -104,7 +104,7 @@ static void __kprobes resume_execution(s
static void __kprobes set_current_kprobe(struct kprobe *p)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static int __kprobes kprobe_handler(struct pt_regs *regs)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 29/31] parisc: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
2013-08-26 20:44 ` Christoph Lameter
@ 2013-08-26 20:56 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (29 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:56 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, James E.J. Bottomley, Helge Deller, linux-parisc
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/parisc/lib/memcpy.c
===================================================================
--- linux.orig/arch/parisc/lib/memcpy.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/parisc/lib/memcpy.c 2013-08-26 13:30:41.656224744 -0500
@@ -470,7 +470,7 @@ static unsigned long pa_memcpy(void *dst
return 0;
/* if a load or store fault occured we can get the faulty addr */
- d = &__get_cpu_var(exception_data);
+ d = this_cpu_ptr(&exception_data);
fault_addr = d->fault_addr;
/* error in load or store? */
Index: linux/arch/parisc/mm/fault.c
===================================================================
--- linux.orig/arch/parisc/mm/fault.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/parisc/mm/fault.c 2013-08-26 13:30:41.656224744 -0500
@@ -145,7 +145,7 @@ int fixup_exception(struct pt_regs *regs
fix = search_exception_tables(regs->iaoq[0]);
if (fix) {
struct exception_data *d;
- d = &__get_cpu_var(exception_data);
+ d = this_cpu_ptr(&exception_data);
d->fault_ip = regs->iaoq[0];
d->fault_space = regs->isr;
d->fault_addr = regs->ior;
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 23/31] tile: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 20:56 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:56 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Chris Metcalf, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/tile/include/asm/irqflags.h
===================================================================
--- linux.orig/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:57.453305409 -0500
@@ -134,7 +134,7 @@ DECLARE_PER_CPU(unsigned long long, inte
/* Re-enable all maskable interrupts. */
#define arch_local_irq_enable() \
- interrupt_mask_reset_mask(__get_cpu_var(interrupts_enabled_mask))
+ interrupt_mask_reset_mask(__this_cpu_read(interrupts_enabled_mask))
/* Disable or enable interrupts based on flag argument. */
#define arch_local_irq_restore(disabled) do { \
@@ -161,7 +161,7 @@ DECLARE_PER_CPU(unsigned long long, inte
/* Prevent the given interrupt from being enabled next time we enable irqs. */
#define arch_local_irq_mask(interrupt) \
- (__get_cpu_var(interrupts_enabled_mask) &= ~(1ULL << (interrupt)))
+ (__this_cpu_read(interrupts_enabled_mask) &= ~(1ULL << (interrupt)))
/* Prevent the given interrupt from being enabled immediately. */
#define arch_local_irq_mask_now(interrupt) do { \
@@ -171,7 +171,7 @@ DECLARE_PER_CPU(unsigned long long, inte
/* Allow the given interrupt to be enabled next time we enable irqs. */
#define arch_local_irq_unmask(interrupt) \
- (__get_cpu_var(interrupts_enabled_mask) |= (1ULL << (interrupt)))
+ (__this_cpu_read(interrupts_enabled_mask) |= (1ULL << (interrupt)))
/* Allow the given interrupt to be enabled immediately, if !irqs_disabled. */
#define arch_local_irq_unmask_now(interrupt) do { \
Index: linux/arch/tile/include/asm/mmu_context.h
===================================================================
--- linux.orig/arch/tile/include/asm/mmu_context.h 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/include/asm/mmu_context.h 2013-08-22 14:39:57.453305409 -0500
@@ -84,7 +84,7 @@ static inline void enter_lazy_tlb(struct
* clear any pending DMA interrupts.
*/
if (current->thread.tile_dma_state.enabled)
- install_page_table(mm->pgd, __get_cpu_var(current_asid));
+ install_page_table(mm->pgd, __this_cpu_read(current_asid));
#endif
}
@@ -96,12 +96,12 @@ static inline void switch_mm(struct mm_s
int cpu = smp_processor_id();
/* Pick new ASID. */
- int asid = __get_cpu_var(current_asid) + 1;
+ int asid = __this_cpu_read(current_asid) + 1;
if (asid > max_asid) {
asid = min_asid;
local_flush_tlb();
}
- __get_cpu_var(current_asid) = asid;
+ __this_cpu_read(current_asid) = asid;
/* Clear cpu from the old mm, and set it in the new one. */
cpumask_clear_cpu(cpu, mm_cpumask(prev));
Index: linux/arch/tile/kernel/irq.c
===================================================================
--- linux.orig/arch/tile/kernel/irq.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/irq.c 2013-08-22 14:39:57.453305409 -0500
@@ -77,7 +77,7 @@ static DEFINE_SPINLOCK(available_irqs_lo
*/
void tile_dev_intr(struct pt_regs *regs, int intnum)
{
- int depth = __get_cpu_var(irq_depth)++;
+ int depth = __this_cpu_inc_return(irq_depth) - 1;
unsigned long original_irqs;
unsigned long remaining_irqs;
struct pt_regs *old_regs;
@@ -124,7 +124,7 @@ void tile_dev_intr(struct pt_regs *regs,
/* Count device irqs; Linux IPIs are counted elsewhere. */
if (irq != IRQ_RESCHEDULE)
- __get_cpu_var(irq_stat).irq_dev_intr_count++;
+ __this_cpu_inc(irq_stat.irq_dev_intr_count);
generic_handle_irq(irq);
}
@@ -135,9 +135,9 @@ void tile_dev_intr(struct pt_regs *regs,
* handling.
*/
if (depth == 0)
- unmask_irqs(~__get_cpu_var(irq_disable_mask));
+ unmask_irqs(~__this_cpu_read(irq_disable_mask));
- __get_cpu_var(irq_depth)--;
+ __this_cpu_dec(irq_depth);
/*
* Track time spent against the current process again and
@@ -155,7 +155,7 @@ void tile_dev_intr(struct pt_regs *regs,
static void tile_irq_chip_enable(struct irq_data *d)
{
get_cpu_var(irq_disable_mask) &= ~(1UL << d->irq);
- if (__get_cpu_var(irq_depth) == 0)
+ if (__this_cpu_read(irq_depth) == 0)
unmask_irqs(1UL << d->irq);
put_cpu_var(irq_disable_mask);
}
@@ -201,7 +201,7 @@ static void tile_irq_chip_ack(struct irq
*/
static void tile_irq_chip_eoi(struct irq_data *d)
{
- if (!(__get_cpu_var(irq_disable_mask) & (1UL << d->irq)))
+ if (!(__this_cpu_read(irq_disable_mask) & (1UL << d->irq)))
unmask_irqs(1UL << d->irq);
}
Index: linux/arch/tile/kernel/messaging.c
===================================================================
--- linux.orig/arch/tile/kernel/messaging.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/messaging.c 2013-08-22 14:39:57.453305409 -0500
@@ -28,7 +28,7 @@ static DEFINE_PER_CPU(HV_MsgState, msg_s
void init_messaging(void)
{
/* Allocate storage for messages in kernel space */
- HV_MsgState *state = &__get_cpu_var(msg_state);
+ HV_MsgState *state = this_cpu_ptr(&msg_state);
int rc = hv_register_message_state(state);
if (rc != HV_OK)
panic("hv_register_message_state: error %d", rc);
@@ -68,7 +68,7 @@ void hv_message_intr(struct pt_regs *reg
#endif
while (1) {
- rmi = hv_receive_message(__get_cpu_var(msg_state),
+ rmi = hv_receive_message(__this_cpu_read(msg_state),
(HV_VirtAddr) message,
sizeof(message));
if (rmi.msglen == 0)
@@ -96,7 +96,7 @@ void hv_message_intr(struct pt_regs *reg
struct hv_driver_cb *cb =
(struct hv_driver_cb *)him->intarg;
cb->callback(cb, him->intdata);
- __get_cpu_var(irq_stat).irq_hv_msg_count++;
+ __this_cpu_inc(irq_stat.irq_hv_msg_count);
}
}
Index: linux/arch/tile/kernel/process.c
===================================================================
--- linux.orig/arch/tile/kernel/process.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/process.c 2013-08-22 14:39:57.453305409 -0500
@@ -63,7 +63,7 @@ early_param("idle", idle_setup);
void arch_cpu_idle(void)
{
- __get_cpu_var(irq_stat).idle_timestamp = jiffies;
+ __this_cpu_write(irq_stat.idle_timestamp, jiffies);
_cpu_idle();
}
Index: linux/arch/tile/kernel/setup.c
===================================================================
--- linux.orig/arch/tile/kernel/setup.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/setup.c 2013-08-22 14:39:57.453305409 -0500
@@ -1119,7 +1119,7 @@ static void __init validate_hv(void)
* various asid variables to their appropriate initial states.
*/
asid_range = hv_inquire_asid(0);
- __get_cpu_var(current_asid) = min_asid = asid_range.start;
+ __this_cpu_write(current_asid, min_asid = asid_range.start);
max_asid = asid_range.start + asid_range.size - 1;
if (hv_confstr(HV_CONFSTR_CHIP_MODEL, (HV_VirtAddr)chip_model,
Index: linux/arch/tile/kernel/single_step.c
===================================================================
--- linux.orig/arch/tile/kernel/single_step.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/single_step.c 2013-08-22 14:39:57.453305409 -0500
@@ -732,7 +732,7 @@ static DEFINE_PER_CPU(unsigned long, ss_
void gx_singlestep_handle(struct pt_regs *regs, int fault_num)
{
- unsigned long *ss_pc = &__get_cpu_var(ss_saved_pc);
+ unsigned long *ss_pc = this_cpu_ptr(&ss_saved_pc);
struct thread_info *info = (void *)current_thread_info();
int is_single_step = test_ti_thread_flag(info, TIF_SINGLESTEP);
unsigned long control = __insn_mfspr(SPR_SINGLE_STEP_CONTROL_K);
@@ -758,7 +758,7 @@ void gx_singlestep_handle(struct pt_regs
void single_step_once(struct pt_regs *regs)
{
- unsigned long *ss_pc = &__get_cpu_var(ss_saved_pc);
+ unsigned long *ss_pc = this_cpu_ptr(&ss_saved_pc);
unsigned long control = __insn_mfspr(SPR_SINGLE_STEP_CONTROL_K);
*ss_pc = regs->pc;
Index: linux/arch/tile/kernel/smp.c
===================================================================
--- linux.orig/arch/tile/kernel/smp.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/smp.c 2013-08-22 14:39:57.453305409 -0500
@@ -176,7 +176,7 @@ void flush_icache_range(unsigned long st
/* Called when smp_send_reschedule() triggers IRQ_RESCHEDULE. */
static irqreturn_t handle_reschedule_ipi(int irq, void *token)
{
- __get_cpu_var(irq_stat).irq_resched_count++;
+ __this_cpu_inc(irq_stat.irq_resched_count);
scheduler_ipi();
return IRQ_HANDLED;
Index: linux/arch/tile/kernel/smpboot.c
===================================================================
--- linux.orig/arch/tile/kernel/smpboot.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/smpboot.c 2013-08-22 14:39:57.453305409 -0500
@@ -41,7 +41,7 @@ void __init smp_prepare_boot_cpu(void)
int cpu = smp_processor_id();
set_cpu_online(cpu, 1);
set_cpu_present(cpu, 1);
- __get_cpu_var(cpu_state) = CPU_ONLINE;
+ __this_cpu_write(cpu_state, CPU_ONLINE);
init_messaging();
}
@@ -156,7 +156,7 @@ static void start_secondary(void)
/* printk(KERN_DEBUG "Initializing CPU#%d\n", cpuid); */
/* Initialize the current asid for our first page table. */
- __get_cpu_var(current_asid) = min_asid;
+ __this_cpu_write(current_asid, min_asid);
/* Set up this thread as another owner of the init_mm */
atomic_inc(&init_mm.mm_count);
@@ -199,7 +199,7 @@ void online_secondary(void)
notify_cpu_starting(smp_processor_id());
set_cpu_online(smp_processor_id(), 1);
- __get_cpu_var(cpu_state) = CPU_ONLINE;
+ __this_cpu_write(cpu_state, CPU_ONLINE);
/* Set up tile-specific state for this cpu. */
setup_cpu(0);
Index: linux/arch/tile/kernel/time.c
===================================================================
--- linux.orig/arch/tile/kernel/time.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/time.c 2013-08-22 14:39:57.453305409 -0500
@@ -161,7 +161,7 @@ static DEFINE_PER_CPU(struct clock_event
void setup_tile_timer(void)
{
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = this_cpu_ptr(&tile_timer);
/* Fill in fields that are speed-specific. */
clockevents_calc_mult_shift(evt, cycles_per_sec, TILE_MINSEC);
@@ -181,7 +181,7 @@ void setup_tile_timer(void)
void do_timer_interrupt(struct pt_regs *regs, int fault_num)
{
struct pt_regs *old_regs = set_irq_regs(regs);
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = this_cpu_ptr(&tile_timer);
/*
* Mask the timer interrupt here, since we are a oneshot timer
@@ -193,7 +193,7 @@ void do_timer_interrupt(struct pt_regs *
irq_enter();
/* Track interrupt count. */
- __get_cpu_var(irq_stat).irq_timer_count++;
+ __this_cpu_inc(irq_stat.irq_timer_count);
/* Call the generic timer handler */
evt->event_handler(evt);
@@ -234,6 +234,6 @@ cycles_t ns2cycles(unsigned long nsecs)
* We do not have to disable preemption here as each core has the same
* clock frequency.
*/
- struct clock_event_device *dev = &__raw_get_cpu_var(tile_timer);
+ struct clock_event_device *dev = __this_cpu_ptr(&tile_timer);
return ((u64)nsecs * dev->mult) >> dev->shift;
}
Index: linux/arch/tile/mm/highmem.c
===================================================================
--- linux.orig/arch/tile/mm/highmem.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/mm/highmem.c 2013-08-22 14:39:57.453305409 -0500
@@ -103,7 +103,7 @@ static void kmap_atomic_register(struct
spin_lock(&_lock);
/* With interrupts disabled, now fill in the per-cpu info. */
- amp = &__get_cpu_var(amps).per_type[type];
+ amp = this_cpu_ptr(&s.per_type[type]);
amp->page = page;
amp->cpu = smp_processor_id();
amp->va = va;
Index: linux/arch/tile/mm/init.c
===================================================================
--- linux.orig/arch/tile/mm/init.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/mm/init.c 2013-08-22 14:39:57.453305409 -0500
@@ -625,14 +625,14 @@ static void __init kernel_physical_mappi
interrupt_mask_set_mask(-1ULL);
rc = flush_and_install_context(__pa(pgtables),
init_pgprot((unsigned long)pgtables),
- __get_cpu_var(current_asid),
+ __this_cpu_read(current_asid),
cpumask_bits(my_cpu_mask));
interrupt_mask_restore_mask(irqmask);
BUG_ON(rc != 0);
/* Copy the page table back to the normal swapper_pg_dir. */
memcpy(pgd_base, pgtables, sizeof(pgtables));
- __install_page_table(pgd_base, __get_cpu_var(current_asid),
+ __install_page_table(pgd_base, __this_cpu_read(current_asid),
swapper_pgprot);
/*
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 29/31] parisc: Replace __get_cpu_var uses
@ 2013-08-26 20:56 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:56 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, James E.J. Bottomley, Helge Deller, linux-parisc,
linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/parisc/lib/memcpy.c
===================================================================
--- linux.orig/arch/parisc/lib/memcpy.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/parisc/lib/memcpy.c 2013-08-26 13:30:41.656224744 -0500
@@ -470,7 +470,7 @@ static unsigned long pa_memcpy(void *dst
return 0;
/* if a load or store fault occured we can get the faulty addr */
- d = &__get_cpu_var(exception_data);
+ d = this_cpu_ptr(&exception_data);
fault_addr = d->fault_addr;
/* error in load or store? */
Index: linux/arch/parisc/mm/fault.c
===================================================================
--- linux.orig/arch/parisc/mm/fault.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/parisc/mm/fault.c 2013-08-26 13:30:41.656224744 -0500
@@ -145,7 +145,7 @@ int fixup_exception(struct pt_regs *regs
fix = search_exception_tables(regs->iaoq[0]);
if (fix) {
struct exception_data *d;
- d = &__get_cpu_var(exception_data);
+ d = this_cpu_ptr(&exception_data);
d->fault_ip = regs->iaoq[0];
d->fault_space = regs->isr;
d->fault_addr = regs->ior;
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 23/31] tile: Replace __get_cpu_var uses
@ 2013-08-26 20:56 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:56 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Chris Metcalf, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/tile/include/asm/irqflags.h
===================================================================
--- linux.orig/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:57.453305409 -0500
@@ -134,7 +134,7 @@ DECLARE_PER_CPU(unsigned long long, inte
/* Re-enable all maskable interrupts. */
#define arch_local_irq_enable() \
- interrupt_mask_reset_mask(__get_cpu_var(interrupts_enabled_mask))
+ interrupt_mask_reset_mask(__this_cpu_read(interrupts_enabled_mask))
/* Disable or enable interrupts based on flag argument. */
#define arch_local_irq_restore(disabled) do { \
@@ -161,7 +161,7 @@ DECLARE_PER_CPU(unsigned long long, inte
/* Prevent the given interrupt from being enabled next time we enable irqs. */
#define arch_local_irq_mask(interrupt) \
- (__get_cpu_var(interrupts_enabled_mask) &= ~(1ULL << (interrupt)))
+ (__this_cpu_read(interrupts_enabled_mask) &= ~(1ULL << (interrupt)))
/* Prevent the given interrupt from being enabled immediately. */
#define arch_local_irq_mask_now(interrupt) do { \
@@ -171,7 +171,7 @@ DECLARE_PER_CPU(unsigned long long, inte
/* Allow the given interrupt to be enabled next time we enable irqs. */
#define arch_local_irq_unmask(interrupt) \
- (__get_cpu_var(interrupts_enabled_mask) |= (1ULL << (interrupt)))
+ (__this_cpu_read(interrupts_enabled_mask) |= (1ULL << (interrupt)))
/* Allow the given interrupt to be enabled immediately, if !irqs_disabled. */
#define arch_local_irq_unmask_now(interrupt) do { \
Index: linux/arch/tile/include/asm/mmu_context.h
===================================================================
--- linux.orig/arch/tile/include/asm/mmu_context.h 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/include/asm/mmu_context.h 2013-08-22 14:39:57.453305409 -0500
@@ -84,7 +84,7 @@ static inline void enter_lazy_tlb(struct
* clear any pending DMA interrupts.
*/
if (current->thread.tile_dma_state.enabled)
- install_page_table(mm->pgd, __get_cpu_var(current_asid));
+ install_page_table(mm->pgd, __this_cpu_read(current_asid));
#endif
}
@@ -96,12 +96,12 @@ static inline void switch_mm(struct mm_s
int cpu = smp_processor_id();
/* Pick new ASID. */
- int asid = __get_cpu_var(current_asid) + 1;
+ int asid = __this_cpu_read(current_asid) + 1;
if (asid > max_asid) {
asid = min_asid;
local_flush_tlb();
}
- __get_cpu_var(current_asid) = asid;
+ __this_cpu_read(current_asid) = asid;
/* Clear cpu from the old mm, and set it in the new one. */
cpumask_clear_cpu(cpu, mm_cpumask(prev));
Index: linux/arch/tile/kernel/irq.c
===================================================================
--- linux.orig/arch/tile/kernel/irq.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/irq.c 2013-08-22 14:39:57.453305409 -0500
@@ -77,7 +77,7 @@ static DEFINE_SPINLOCK(available_irqs_lo
*/
void tile_dev_intr(struct pt_regs *regs, int intnum)
{
- int depth = __get_cpu_var(irq_depth)++;
+ int depth = __this_cpu_inc_return(irq_depth) - 1;
unsigned long original_irqs;
unsigned long remaining_irqs;
struct pt_regs *old_regs;
@@ -124,7 +124,7 @@ void tile_dev_intr(struct pt_regs *regs,
/* Count device irqs; Linux IPIs are counted elsewhere. */
if (irq != IRQ_RESCHEDULE)
- __get_cpu_var(irq_stat).irq_dev_intr_count++;
+ __this_cpu_inc(irq_stat.irq_dev_intr_count);
generic_handle_irq(irq);
}
@@ -135,9 +135,9 @@ void tile_dev_intr(struct pt_regs *regs,
* handling.
*/
if (depth == 0)
- unmask_irqs(~__get_cpu_var(irq_disable_mask));
+ unmask_irqs(~__this_cpu_read(irq_disable_mask));
- __get_cpu_var(irq_depth)--;
+ __this_cpu_dec(irq_depth);
/*
* Track time spent against the current process again and
@@ -155,7 +155,7 @@ void tile_dev_intr(struct pt_regs *regs,
static void tile_irq_chip_enable(struct irq_data *d)
{
get_cpu_var(irq_disable_mask) &= ~(1UL << d->irq);
- if (__get_cpu_var(irq_depth) == 0)
+ if (__this_cpu_read(irq_depth) == 0)
unmask_irqs(1UL << d->irq);
put_cpu_var(irq_disable_mask);
}
@@ -201,7 +201,7 @@ static void tile_irq_chip_ack(struct irq
*/
static void tile_irq_chip_eoi(struct irq_data *d)
{
- if (!(__get_cpu_var(irq_disable_mask) & (1UL << d->irq)))
+ if (!(__this_cpu_read(irq_disable_mask) & (1UL << d->irq)))
unmask_irqs(1UL << d->irq);
}
Index: linux/arch/tile/kernel/messaging.c
===================================================================
--- linux.orig/arch/tile/kernel/messaging.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/messaging.c 2013-08-22 14:39:57.453305409 -0500
@@ -28,7 +28,7 @@ static DEFINE_PER_CPU(HV_MsgState, msg_s
void init_messaging(void)
{
/* Allocate storage for messages in kernel space */
- HV_MsgState *state = &__get_cpu_var(msg_state);
+ HV_MsgState *state = this_cpu_ptr(&msg_state);
int rc = hv_register_message_state(state);
if (rc != HV_OK)
panic("hv_register_message_state: error %d", rc);
@@ -68,7 +68,7 @@ void hv_message_intr(struct pt_regs *reg
#endif
while (1) {
- rmi = hv_receive_message(__get_cpu_var(msg_state),
+ rmi = hv_receive_message(__this_cpu_read(msg_state),
(HV_VirtAddr) message,
sizeof(message));
if (rmi.msglen == 0)
@@ -96,7 +96,7 @@ void hv_message_intr(struct pt_regs *reg
struct hv_driver_cb *cb =
(struct hv_driver_cb *)him->intarg;
cb->callback(cb, him->intdata);
- __get_cpu_var(irq_stat).irq_hv_msg_count++;
+ __this_cpu_inc(irq_stat.irq_hv_msg_count);
}
}
Index: linux/arch/tile/kernel/process.c
===================================================================
--- linux.orig/arch/tile/kernel/process.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/process.c 2013-08-22 14:39:57.453305409 -0500
@@ -63,7 +63,7 @@ early_param("idle", idle_setup);
void arch_cpu_idle(void)
{
- __get_cpu_var(irq_stat).idle_timestamp = jiffies;
+ __this_cpu_write(irq_stat.idle_timestamp, jiffies);
_cpu_idle();
}
Index: linux/arch/tile/kernel/setup.c
===================================================================
--- linux.orig/arch/tile/kernel/setup.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/setup.c 2013-08-22 14:39:57.453305409 -0500
@@ -1119,7 +1119,7 @@ static void __init validate_hv(void)
* various asid variables to their appropriate initial states.
*/
asid_range = hv_inquire_asid(0);
- __get_cpu_var(current_asid) = min_asid = asid_range.start;
+ __this_cpu_write(current_asid, min_asid = asid_range.start);
max_asid = asid_range.start + asid_range.size - 1;
if (hv_confstr(HV_CONFSTR_CHIP_MODEL, (HV_VirtAddr)chip_model,
Index: linux/arch/tile/kernel/single_step.c
===================================================================
--- linux.orig/arch/tile/kernel/single_step.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/single_step.c 2013-08-22 14:39:57.453305409 -0500
@@ -732,7 +732,7 @@ static DEFINE_PER_CPU(unsigned long, ss_
void gx_singlestep_handle(struct pt_regs *regs, int fault_num)
{
- unsigned long *ss_pc = &__get_cpu_var(ss_saved_pc);
+ unsigned long *ss_pc = this_cpu_ptr(&ss_saved_pc);
struct thread_info *info = (void *)current_thread_info();
int is_single_step = test_ti_thread_flag(info, TIF_SINGLESTEP);
unsigned long control = __insn_mfspr(SPR_SINGLE_STEP_CONTROL_K);
@@ -758,7 +758,7 @@ void gx_singlestep_handle(struct pt_regs
void single_step_once(struct pt_regs *regs)
{
- unsigned long *ss_pc = &__get_cpu_var(ss_saved_pc);
+ unsigned long *ss_pc = this_cpu_ptr(&ss_saved_pc);
unsigned long control = __insn_mfspr(SPR_SINGLE_STEP_CONTROL_K);
*ss_pc = regs->pc;
Index: linux/arch/tile/kernel/smp.c
===================================================================
--- linux.orig/arch/tile/kernel/smp.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/smp.c 2013-08-22 14:39:57.453305409 -0500
@@ -176,7 +176,7 @@ void flush_icache_range(unsigned long st
/* Called when smp_send_reschedule() triggers IRQ_RESCHEDULE. */
static irqreturn_t handle_reschedule_ipi(int irq, void *token)
{
- __get_cpu_var(irq_stat).irq_resched_count++;
+ __this_cpu_inc(irq_stat.irq_resched_count);
scheduler_ipi();
return IRQ_HANDLED;
Index: linux/arch/tile/kernel/smpboot.c
===================================================================
--- linux.orig/arch/tile/kernel/smpboot.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/smpboot.c 2013-08-22 14:39:57.453305409 -0500
@@ -41,7 +41,7 @@ void __init smp_prepare_boot_cpu(void)
int cpu = smp_processor_id();
set_cpu_online(cpu, 1);
set_cpu_present(cpu, 1);
- __get_cpu_var(cpu_state) = CPU_ONLINE;
+ __this_cpu_write(cpu_state, CPU_ONLINE);
init_messaging();
}
@@ -156,7 +156,7 @@ static void start_secondary(void)
/* printk(KERN_DEBUG "Initializing CPU#%d\n", cpuid); */
/* Initialize the current asid for our first page table. */
- __get_cpu_var(current_asid) = min_asid;
+ __this_cpu_write(current_asid, min_asid);
/* Set up this thread as another owner of the init_mm */
atomic_inc(&init_mm.mm_count);
@@ -199,7 +199,7 @@ void online_secondary(void)
notify_cpu_starting(smp_processor_id());
set_cpu_online(smp_processor_id(), 1);
- __get_cpu_var(cpu_state) = CPU_ONLINE;
+ __this_cpu_write(cpu_state, CPU_ONLINE);
/* Set up tile-specific state for this cpu. */
setup_cpu(0);
Index: linux/arch/tile/kernel/time.c
===================================================================
--- linux.orig/arch/tile/kernel/time.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/kernel/time.c 2013-08-22 14:39:57.453305409 -0500
@@ -161,7 +161,7 @@ static DEFINE_PER_CPU(struct clock_event
void setup_tile_timer(void)
{
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = this_cpu_ptr(&tile_timer);
/* Fill in fields that are speed-specific. */
clockevents_calc_mult_shift(evt, cycles_per_sec, TILE_MINSEC);
@@ -181,7 +181,7 @@ void setup_tile_timer(void)
void do_timer_interrupt(struct pt_regs *regs, int fault_num)
{
struct pt_regs *old_regs = set_irq_regs(regs);
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = this_cpu_ptr(&tile_timer);
/*
* Mask the timer interrupt here, since we are a oneshot timer
@@ -193,7 +193,7 @@ void do_timer_interrupt(struct pt_regs *
irq_enter();
/* Track interrupt count. */
- __get_cpu_var(irq_stat).irq_timer_count++;
+ __this_cpu_inc(irq_stat.irq_timer_count);
/* Call the generic timer handler */
evt->event_handler(evt);
@@ -234,6 +234,6 @@ cycles_t ns2cycles(unsigned long nsecs)
* We do not have to disable preemption here as each core has the same
* clock frequency.
*/
- struct clock_event_device *dev = &__raw_get_cpu_var(tile_timer);
+ struct clock_event_device *dev = __this_cpu_ptr(&tile_timer);
return ((u64)nsecs * dev->mult) >> dev->shift;
}
Index: linux/arch/tile/mm/highmem.c
===================================================================
--- linux.orig/arch/tile/mm/highmem.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/mm/highmem.c 2013-08-22 14:39:57.453305409 -0500
@@ -103,7 +103,7 @@ static void kmap_atomic_register(struct
spin_lock(&_lock);
/* With interrupts disabled, now fill in the per-cpu info. */
- amp = &__get_cpu_var(amps).per_type[type];
+ amp = this_cpu_ptr(&s.per_type[type]);
amp->page = page;
amp->cpu = smp_processor_id();
amp->va = va;
Index: linux/arch/tile/mm/init.c
===================================================================
--- linux.orig/arch/tile/mm/init.c 2013-08-22 14:39:14.000000000 -0500
+++ linux/arch/tile/mm/init.c 2013-08-22 14:39:57.453305409 -0500
@@ -625,14 +625,14 @@ static void __init kernel_physical_mappi
interrupt_mask_set_mask(-1ULL);
rc = flush_and_install_context(__pa(pgtables),
init_pgprot((unsigned long)pgtables),
- __get_cpu_var(current_asid),
+ __this_cpu_read(current_asid),
cpumask_bits(my_cpu_mask));
interrupt_mask_restore_mask(irqmask);
BUG_ON(rc != 0);
/* Copy the page table back to the normal swapper_pg_dir. */
memcpy(pgd_base, pgtables, sizeof(pgtables));
- __install_page_table(pgd_base, __get_cpu_var(current_asid),
+ __install_page_table(pgd_base, __this_cpu_read(current_asid),
swapper_pgprot);
/*
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 29/31] parisc: Replace __get_cpu_var uses
@ 2013-08-26 20:56 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 20:56 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, James E.J. Bottomley, Helge Deller, linux-parisc,
linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/parisc/lib/memcpy.c
===================================================================
--- linux.orig/arch/parisc/lib/memcpy.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/parisc/lib/memcpy.c 2013-08-26 13:30:41.656224744 -0500
@@ -470,7 +470,7 @@ static unsigned long pa_memcpy(void *dst
return 0;
/* if a load or store fault occured we can get the faulty addr */
- d = &__get_cpu_var(exception_data);
+ d = this_cpu_ptr(&exception_data);
fault_addr = d->fault_addr;
/* error in load or store? */
Index: linux/arch/parisc/mm/fault.c
===================================================================
--- linux.orig/arch/parisc/mm/fault.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/parisc/mm/fault.c 2013-08-26 13:30:41.656224744 -0500
@@ -145,7 +145,7 @@ int fixup_exception(struct pt_regs *regs
fix = search_exception_tables(regs->iaoq[0]);
if (fix) {
struct exception_data *d;
- d = &__get_cpu_var(exception_data);
+ d = this_cpu_ptr(&exception_data);
d->fault_ip = regs->iaoq[0];
d->fault_space = regs->isr;
d->fault_addr = regs->ior;
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 31/31] Remove __get_cpu_var and __raw_get_cpu_var macros
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 21:16 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:16 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
No user is left in the kernel source tree. Therefore we can
drop the definitions.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/asm-generic/percpu.h
===================================================================
--- linux.orig/include/asm-generic/percpu.h 2013-08-26 14:40:32.528082727 -0500
+++ linux/include/asm-generic/percpu.h 2013-08-26 14:40:32.520082812 -0500
@@ -65,9 +65,6 @@ extern unsigned long __per_cpu_offset[NR
#define this_cpu_ptr(ptr) __this_cpu_ptr(ptr)
#endif
-#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
-#define __raw_get_cpu_var(var) (*__this_cpu_ptr(&(var)))
-
#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
extern void setup_per_cpu_areas(void);
#endif
@@ -80,8 +77,6 @@ extern void setup_per_cpu_areas(void);
})
#define per_cpu(var, cpu) (*((void)(cpu), VERIFY_PERCPU_PTR(&(var))))
-#define __get_cpu_var(var) (*VERIFY_PERCPU_PTR(&(var)))
-#define __raw_get_cpu_var(var) (*VERIFY_PERCPU_PTR(&(var)))
#define this_cpu_ptr(ptr) per_cpu_ptr(ptr, 0)
#define __this_cpu_ptr(ptr) this_cpu_ptr(ptr)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 24/31] blackfin: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 21:16 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:16 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Mike Frysinger, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/blackfin/include/asm/ipipe.h
===================================================================
--- linux.orig/arch/blackfin/include/asm/ipipe.h 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/blackfin/include/asm/ipipe.h 2013-08-26 13:26:23.578979325 -0500
@@ -157,7 +157,7 @@ static inline unsigned long __ipipe_ffnz
}
#define __ipipe_do_root_xirq(ipd, irq) \
- ((ipd)->irqs[irq].handler(irq, &__raw_get_cpu_var(__ipipe_tick_regs)))
+ ((ipd)->irqs[irq].handler(irq, __this_cpu_ptr(&__ipipe_tick_regs)))
#define __ipipe_run_irqtail(irq) /* Must be a macro */ \
do { \
Index: linux/arch/blackfin/kernel/perf_event.c
===================================================================
--- linux.orig/arch/blackfin/kernel/perf_event.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/blackfin/kernel/perf_event.c 2013-08-26 13:26:23.578979325 -0500
@@ -300,7 +300,7 @@ again:
static void bfin_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -318,7 +318,7 @@ static void bfin_pmu_stop(struct perf_ev
static void bfin_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -335,7 +335,7 @@ static void bfin_pmu_start(struct perf_e
static void bfin_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
bfin_pmu_stop(event, PERF_EF_UPDATE);
__clear_bit(event->hw.idx, cpuc->used_mask);
@@ -345,7 +345,7 @@ static void bfin_pmu_del(struct perf_eve
static int bfin_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
int ret = -EAGAIN;
@@ -429,7 +429,7 @@ static int bfin_pmu_event_init(struct pe
static void bfin_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event;
struct hw_perf_event *hwc;
int i;
Index: linux/arch/blackfin/mach-common/ints-priority.c
===================================================================
--- linux.orig/arch/blackfin/mach-common/ints-priority.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/blackfin/mach-common/ints-priority.c 2013-08-26 13:26:23.578979325 -0500
@@ -1675,12 +1675,12 @@ asmlinkage int __ipipe_grab_irq(int vec,
bfin_write_TIMER_STATUS(1); /* Latch TIMIL0 */
#endif
/* This is basically what we need from the register frame. */
- __raw_get_cpu_var(__ipipe_tick_regs).ipend = regs->ipend;
- __raw_get_cpu_var(__ipipe_tick_regs).pc = regs->pc;
+ __this_cpu_write(__ipipe_tick_regs.ipend, regs->ipend);
+ __this_cpu_write(__ipipe_tick_regs.pc, regs->pc);
if (this_domain != ipipe_root_domain)
- __raw_get_cpu_var(__ipipe_tick_regs).ipend &= ~0x10;
+ __this_cpu_and(__ipipe_tick_regs.ipend, ~0x10);
else
- __raw_get_cpu_var(__ipipe_tick_regs).ipend |= 0x10;
+ __this_cpu_or(__ipipe_tick_regs.ipend, 0x10);
}
/*
Index: linux/arch/blackfin/mach-common/smp.c
===================================================================
--- linux.orig/arch/blackfin/mach-common/smp.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/blackfin/mach-common/smp.c 2013-08-26 13:26:23.578979325 -0500
@@ -146,7 +146,7 @@ static irqreturn_t ipi_handler_int1(int
platform_clear_ipi(cpu, IRQ_SUPPLE_1);
- bfin_ipi_data = &__get_cpu_var(bfin_ipi);
+ bfin_ipi_data = this_cpu_ptr(&bfin_ipi);
while ((pending = atomic_xchg(&bfin_ipi_data->bits, 0)) != 0) {
msg = 0;
do {
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 31/31] Remove __get_cpu_var and __raw_get_cpu_var macros
@ 2013-08-26 21:16 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:16 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, linux-arch, Steven Rostedt, linux-kernel
No user is left in the kernel source tree. Therefore we can
drop the definitions.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/asm-generic/percpu.h
===================================================================
--- linux.orig/include/asm-generic/percpu.h 2013-08-26 14:40:32.528082727 -0500
+++ linux/include/asm-generic/percpu.h 2013-08-26 14:40:32.520082812 -0500
@@ -65,9 +65,6 @@ extern unsigned long __per_cpu_offset[NR
#define this_cpu_ptr(ptr) __this_cpu_ptr(ptr)
#endif
-#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
-#define __raw_get_cpu_var(var) (*__this_cpu_ptr(&(var)))
-
#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
extern void setup_per_cpu_areas(void);
#endif
@@ -80,8 +77,6 @@ extern void setup_per_cpu_areas(void);
})
#define per_cpu(var, cpu) (*((void)(cpu), VERIFY_PERCPU_PTR(&(var))))
-#define __get_cpu_var(var) (*VERIFY_PERCPU_PTR(&(var)))
-#define __raw_get_cpu_var(var) (*VERIFY_PERCPU_PTR(&(var)))
#define this_cpu_ptr(ptr) per_cpu_ptr(ptr, 0)
#define __this_cpu_ptr(ptr) this_cpu_ptr(ptr)
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 24/31] blackfin: Replace __get_cpu_var uses
@ 2013-08-26 21:16 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:16 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Mike Frysinger, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/blackfin/include/asm/ipipe.h
===================================================================
--- linux.orig/arch/blackfin/include/asm/ipipe.h 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/blackfin/include/asm/ipipe.h 2013-08-26 13:26:23.578979325 -0500
@@ -157,7 +157,7 @@ static inline unsigned long __ipipe_ffnz
}
#define __ipipe_do_root_xirq(ipd, irq) \
- ((ipd)->irqs[irq].handler(irq, &__raw_get_cpu_var(__ipipe_tick_regs)))
+ ((ipd)->irqs[irq].handler(irq, __this_cpu_ptr(&__ipipe_tick_regs)))
#define __ipipe_run_irqtail(irq) /* Must be a macro */ \
do { \
Index: linux/arch/blackfin/kernel/perf_event.c
===================================================================
--- linux.orig/arch/blackfin/kernel/perf_event.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/blackfin/kernel/perf_event.c 2013-08-26 13:26:23.578979325 -0500
@@ -300,7 +300,7 @@ again:
static void bfin_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -318,7 +318,7 @@ static void bfin_pmu_stop(struct perf_ev
static void bfin_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -335,7 +335,7 @@ static void bfin_pmu_start(struct perf_e
static void bfin_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
bfin_pmu_stop(event, PERF_EF_UPDATE);
__clear_bit(event->hw.idx, cpuc->used_mask);
@@ -345,7 +345,7 @@ static void bfin_pmu_del(struct perf_eve
static int bfin_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
int ret = -EAGAIN;
@@ -429,7 +429,7 @@ static int bfin_pmu_event_init(struct pe
static void bfin_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event;
struct hw_perf_event *hwc;
int i;
Index: linux/arch/blackfin/mach-common/ints-priority.c
===================================================================
--- linux.orig/arch/blackfin/mach-common/ints-priority.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/blackfin/mach-common/ints-priority.c 2013-08-26 13:26:23.578979325 -0500
@@ -1675,12 +1675,12 @@ asmlinkage int __ipipe_grab_irq(int vec,
bfin_write_TIMER_STATUS(1); /* Latch TIMIL0 */
#endif
/* This is basically what we need from the register frame. */
- __raw_get_cpu_var(__ipipe_tick_regs).ipend = regs->ipend;
- __raw_get_cpu_var(__ipipe_tick_regs).pc = regs->pc;
+ __this_cpu_write(__ipipe_tick_regs.ipend, regs->ipend);
+ __this_cpu_write(__ipipe_tick_regs.pc, regs->pc);
if (this_domain != ipipe_root_domain)
- __raw_get_cpu_var(__ipipe_tick_regs).ipend &= ~0x10;
+ __this_cpu_and(__ipipe_tick_regs.ipend, ~0x10);
else
- __raw_get_cpu_var(__ipipe_tick_regs).ipend |= 0x10;
+ __this_cpu_or(__ipipe_tick_regs.ipend, 0x10);
}
/*
Index: linux/arch/blackfin/mach-common/smp.c
===================================================================
--- linux.orig/arch/blackfin/mach-common/smp.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/blackfin/mach-common/smp.c 2013-08-26 13:26:23.578979325 -0500
@@ -146,7 +146,7 @@ static irqreturn_t ipi_handler_int1(int
platform_clear_ipi(cpu, IRQ_SUPPLE_1);
- bfin_ipi_data = &__get_cpu_var(bfin_ipi);
+ bfin_ipi_data = this_cpu_ptr(&bfin_ipi);
while ((pending = atomic_xchg(&bfin_ipi_data->bits, 0)) != 0) {
msg = 0;
do {
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 26/31] alpha: Replace __get_cpu_var
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 21:16 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:16 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Richard Henderson, Ivan Kokshaysky, Matt Turner,
linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/alpha/kernel/perf_event.c
===================================================================
--- linux.orig/arch/alpha/kernel/perf_event.c 2013-08-26 13:55:35.904436503 -0500
+++ linux/arch/alpha/kernel/perf_event.c 2013-08-26 13:55:35.896436587 -0500
@@ -422,7 +422,7 @@ static void maybe_change_configuration(s
*/
static int alpha_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int n0;
int ret;
@@ -474,7 +474,7 @@ static int alpha_pmu_add(struct perf_eve
*/
static void alpha_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
unsigned long irq_flags;
int j;
@@ -522,7 +522,7 @@ static void alpha_pmu_read(struct perf_e
static void alpha_pmu_stop(struct perf_event *event, int flags)
{
struct hw_perf_event *hwc = &event->hw;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!(hwc->state & PERF_HES_STOPPED)) {
cpuc->idx_mask &= ~(1UL<<hwc->idx);
@@ -542,7 +542,7 @@ static void alpha_pmu_stop(struct perf_e
static void alpha_pmu_start(struct perf_event *event, int flags)
{
struct hw_perf_event *hwc = &event->hw;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
return;
@@ -713,7 +713,7 @@ static int alpha_pmu_event_init(struct p
*/
static void alpha_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->enabled)
return;
@@ -739,7 +739,7 @@ static void alpha_pmu_enable(struct pmu
static void alpha_pmu_disable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!cpuc->enabled)
return;
@@ -804,7 +804,7 @@ static void alpha_perf_event_irq_handler
int idx, j;
__this_cpu_inc(irq_pmi_count);
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
/* Completely counting through the PMC's period to trigger a new PMC
* overflow interrupt while in this interrupt routine is utterly
Index: linux/arch/alpha/kernel/time.c
===================================================================
--- linux.orig/arch/alpha/kernel/time.c 2013-08-26 13:55:35.904436503 -0500
+++ linux/arch/alpha/kernel/time.c 2013-08-26 13:55:35.896436587 -0500
@@ -86,9 +86,9 @@ unsigned long est_cycle_freq;
DEFINE_PER_CPU(u8, irq_work_pending);
-#define set_irq_work_pending_flag() __get_cpu_var(irq_work_pending) = 1
-#define test_irq_work_pending() __get_cpu_var(irq_work_pending)
-#define clear_irq_work_pending() __get_cpu_var(irq_work_pending) = 0
+#define set_irq_work_pending_flag() __this_cpu_write(irq_work_pending, 1)
+#define test_irq_work_pending() __this_cpu_read(irq_work_pending)
+#define clear_irq_work_pending() __this_cpu_writeirq_work_pending, 0)
void arch_irq_work_raise(void)
{
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 26/31] alpha: Replace __get_cpu_var
@ 2013-08-26 21:16 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:16 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Richard Henderson, Ivan Kokshaysky, Matt Turner,
linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/alpha/kernel/perf_event.c
===================================================================
--- linux.orig/arch/alpha/kernel/perf_event.c 2013-08-26 13:55:35.904436503 -0500
+++ linux/arch/alpha/kernel/perf_event.c 2013-08-26 13:55:35.896436587 -0500
@@ -422,7 +422,7 @@ static void maybe_change_configuration(s
*/
static int alpha_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int n0;
int ret;
@@ -474,7 +474,7 @@ static int alpha_pmu_add(struct perf_eve
*/
static void alpha_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
unsigned long irq_flags;
int j;
@@ -522,7 +522,7 @@ static void alpha_pmu_read(struct perf_e
static void alpha_pmu_stop(struct perf_event *event, int flags)
{
struct hw_perf_event *hwc = &event->hw;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!(hwc->state & PERF_HES_STOPPED)) {
cpuc->idx_mask &= ~(1UL<<hwc->idx);
@@ -542,7 +542,7 @@ static void alpha_pmu_stop(struct perf_e
static void alpha_pmu_start(struct perf_event *event, int flags)
{
struct hw_perf_event *hwc = &event->hw;
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
return;
@@ -713,7 +713,7 @@ static int alpha_pmu_event_init(struct p
*/
static void alpha_pmu_enable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (cpuc->enabled)
return;
@@ -739,7 +739,7 @@ static void alpha_pmu_enable(struct pmu
static void alpha_pmu_disable(struct pmu *pmu)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!cpuc->enabled)
return;
@@ -804,7 +804,7 @@ static void alpha_perf_event_irq_handler
int idx, j;
__this_cpu_inc(irq_pmi_count);
- cpuc = &__get_cpu_var(cpu_hw_events);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
/* Completely counting through the PMC's period to trigger a new PMC
* overflow interrupt while in this interrupt routine is utterly
Index: linux/arch/alpha/kernel/time.c
===================================================================
--- linux.orig/arch/alpha/kernel/time.c 2013-08-26 13:55:35.904436503 -0500
+++ linux/arch/alpha/kernel/time.c 2013-08-26 13:55:35.896436587 -0500
@@ -86,9 +86,9 @@ unsigned long est_cycle_freq;
DEFINE_PER_CPU(u8, irq_work_pending);
-#define set_irq_work_pending_flag() __get_cpu_var(irq_work_pending) = 1
-#define test_irq_work_pending() __get_cpu_var(irq_work_pending)
-#define clear_irq_work_pending() __get_cpu_var(irq_work_pending) = 0
+#define set_irq_work_pending_flag() __this_cpu_write(irq_work_pending, 1)
+#define test_irq_work_pending() __this_cpu_read(irq_work_pending)
+#define clear_irq_work_pending() __this_cpu_writeirq_work_pending, 0)
void arch_irq_work_raise(void)
{
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 28/31] arc: Replace __get_cpu_var uses throughout the kernel
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 21:56 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:56 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Vineet Gupta, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/arc/kernel/kprobes.c
===================================================================
--- linux.orig/arch/arc/kernel/kprobes.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/arc/kernel/kprobes.c 2013-08-26 13:29:19.285103021 -0500
@@ -87,13 +87,13 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
static inline void __kprobes set_current_kprobe(struct kprobe *p)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static void __kprobes resume_execution(struct kprobe *p, unsigned long addr,
@@ -237,7 +237,7 @@ int __kprobes arc_kprobe_handler(unsigne
return 1;
} else if (kprobe_running()) {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
setup_singlestep(p, regs);
kcb->kprobe_status = KPROBE_HIT_SS;
Index: linux/arch/arc/kernel/time.c
===================================================================
--- linux.orig/arch/arc/kernel/time.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/arc/kernel/time.c 2013-08-26 13:29:19.285103021 -0500
@@ -206,7 +206,7 @@ static DEFINE_PER_CPU(struct clock_event
static irqreturn_t timer_irq_handler(int irq, void *dev_id)
{
- struct clock_event_device *clk = &__get_cpu_var(arc_clockevent_device);
+ struct clock_event_device *clk = this_cpu_ptr(&arc_clockevent_device);
arc_timer_event_ack(clk->mode == CLOCK_EVT_MODE_PERIODIC);
clk->event_handler(clk);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 30/31] metag: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
@ 2013-08-26 21:56 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (30 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:56 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, James Hogan, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/metag/kernel/perf/perf_event.c
===================================================================
--- linux.orig/arch/metag/kernel/perf/perf_event.c 2013-08-26 13:30:45.220186762 -0500
+++ linux/arch/metag/kernel/perf/perf_event.c 2013-08-26 13:30:45.216186804 -0500
@@ -258,7 +258,7 @@ int metag_pmu_event_set_period(struct pe
static void metag_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -306,7 +306,7 @@ static void metag_pmu_stop(struct perf_e
static int metag_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = 0, ret = 0;
@@ -348,7 +348,7 @@ out:
static void metag_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -607,7 +607,7 @@ static int _hw_perf_event_init(struct pe
static void metag_pmu_enable_counter(struct hw_perf_event *event, int idx)
{
- struct cpu_hw_events *events = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *events = this_cpu_ptr(&cpu_hw_events);
unsigned int config = event->config;
unsigned int tmp = config & 0xf0;
unsigned long flags;
@@ -680,7 +680,7 @@ unlock:
static void metag_pmu_disable_counter(struct hw_perf_event *event, int idx)
{
- struct cpu_hw_events *events = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *events = this_cpu_ptr(&cpu_hw_events);
unsigned int tmp = 0;
unsigned long flags;
@@ -728,7 +728,7 @@ out:
static void metag_pmu_write_counter(int idx, u32 val)
{
- struct cpu_hw_events *events = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *events = this_cpu_ptr(&cpu_hw_events);
u32 tmp = 0;
unsigned long flags;
@@ -761,7 +761,7 @@ static int metag_pmu_event_map(int idx)
static irqreturn_t metag_pmu_counter_overflow(int irq, void *dev)
{
int idx = (int)dev;
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event = cpuhw->events[idx];
struct hw_perf_event *hwc = &event->hw;
struct pt_regs *regs = get_irq_regs();
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 28/31] arc: Replace __get_cpu_var uses throughout the kernel
@ 2013-08-26 21:56 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:56 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, Vineet Gupta, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/arc/kernel/kprobes.c
===================================================================
--- linux.orig/arch/arc/kernel/kprobes.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/arc/kernel/kprobes.c 2013-08-26 13:29:19.285103021 -0500
@@ -87,13 +87,13 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
static inline void __kprobes set_current_kprobe(struct kprobe *p)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static void __kprobes resume_execution(struct kprobe *p, unsigned long addr,
@@ -237,7 +237,7 @@ int __kprobes arc_kprobe_handler(unsigne
return 1;
} else if (kprobe_running()) {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
setup_singlestep(p, regs);
kcb->kprobe_status = KPROBE_HIT_SS;
Index: linux/arch/arc/kernel/time.c
===================================================================
--- linux.orig/arch/arc/kernel/time.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/arc/kernel/time.c 2013-08-26 13:29:19.285103021 -0500
@@ -206,7 +206,7 @@ static DEFINE_PER_CPU(struct clock_event
static irqreturn_t timer_irq_handler(int irq, void *dev_id)
{
- struct clock_event_device *clk = &__get_cpu_var(arc_clockevent_device);
+ struct clock_event_device *clk = this_cpu_ptr(&arc_clockevent_device);
arc_timer_event_ack(clk->mode == CLOCK_EVT_MODE_PERIODIC);
clk->event_handler(clk);
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 30/31] metag: Replace __get_cpu_var uses
@ 2013-08-26 21:56 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:56 UTC (permalink / raw)
To: Tejun Heo; +Cc: akpm, James Hogan, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/metag/kernel/perf/perf_event.c
===================================================================
--- linux.orig/arch/metag/kernel/perf/perf_event.c 2013-08-26 13:30:45.220186762 -0500
+++ linux/arch/metag/kernel/perf/perf_event.c 2013-08-26 13:30:45.216186804 -0500
@@ -258,7 +258,7 @@ int metag_pmu_event_set_period(struct pe
static void metag_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -306,7 +306,7 @@ static void metag_pmu_stop(struct perf_e
static int metag_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = 0, ret = 0;
@@ -348,7 +348,7 @@ out:
static void metag_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -607,7 +607,7 @@ static int _hw_perf_event_init(struct pe
static void metag_pmu_enable_counter(struct hw_perf_event *event, int idx)
{
- struct cpu_hw_events *events = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *events = this_cpu_ptr(&cpu_hw_events);
unsigned int config = event->config;
unsigned int tmp = config & 0xf0;
unsigned long flags;
@@ -680,7 +680,7 @@ unlock:
static void metag_pmu_disable_counter(struct hw_perf_event *event, int idx)
{
- struct cpu_hw_events *events = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *events = this_cpu_ptr(&cpu_hw_events);
unsigned int tmp = 0;
unsigned long flags;
@@ -728,7 +728,7 @@ out:
static void metag_pmu_write_counter(int idx, u32 val)
{
- struct cpu_hw_events *events = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *events = this_cpu_ptr(&cpu_hw_events);
u32 tmp = 0;
unsigned long flags;
@@ -761,7 +761,7 @@ static int metag_pmu_event_map(int idx)
static irqreturn_t metag_pmu_counter_overflow(int irq, void *dev)
{
int idx = (int)dev;
- struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuhw = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event = cpuhw->events[idx];
struct hw_perf_event *hwc = &event->hw;
struct pt_regs *regs = get_irq_regs();
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 27/31] sh: Replace __get_cpu_var uses throughout the kernel
[not found] <20130826204351.725357339@linux.com>
2013-08-26 20:44 ` Christoph Lameter
@ 2013-08-26 21:56 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
` (29 subsequent siblings)
31 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:56 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Paul Mundt, linux-sh, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/sh/kernel/hw_breakpoint.c
=================================--- linux.orig/arch/sh/kernel/hw_breakpoint.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/hw_breakpoint.c 2013-08-26 13:28:32.265604741 -0500
@@ -52,7 +52,7 @@ int arch_install_hw_breakpoint(struct pe
int i;
for (i = 0; i < sh_ubc->num_events; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (!*slot) {
*slot = bp;
@@ -84,7 +84,7 @@ void arch_uninstall_hw_breakpoint(struct
int i;
for (i = 0; i < sh_ubc->num_events; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (*slot = bp) {
*slot = NULL;
Index: linux/arch/sh/kernel/kprobes.c
=================================--- linux.orig/arch/sh/kernel/kprobes.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/kprobes.c 2013-08-26 13:28:32.265604741 -0500
@@ -102,7 +102,7 @@ int __kprobes kprobe_handle_illslot(unsi
void __kprobes arch_remove_kprobe(struct kprobe *p)
{
- struct kprobe *saved = &__get_cpu_var(saved_next_opcode);
+ struct kprobe *saved = this_cpu_ptr(&saved_next_opcode);
if (saved->addr) {
arch_disarm_kprobe(p);
@@ -111,7 +111,7 @@ void __kprobes arch_remove_kprobe(struct
saved->addr = NULL;
saved->opcode = 0;
- saved = &__get_cpu_var(saved_next_opcode2);
+ saved = this_cpu_ptr(&saved_next_opcode2);
if (saved->addr) {
arch_disarm_kprobe(saved);
@@ -129,14 +129,14 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
/*
@@ -146,15 +146,15 @@ static void __kprobes set_current_kprobe
*/
static void __kprobes prepare_singlestep(struct kprobe *p, struct pt_regs *regs)
{
- __get_cpu_var(saved_current_opcode).addr = (kprobe_opcode_t *)regs->pc;
+ __this_cpu_write(saved_current_opcode.addr, (kprobe_opcode_t *)regs->pc);
if (p != NULL) {
struct kprobe *op1, *op2;
arch_disarm_kprobe(p);
- op1 = &__get_cpu_var(saved_next_opcode);
- op2 = &__get_cpu_var(saved_next_opcode2);
+ op1 = this_cpu_ptr(&saved_next_opcode);
+ op2 = this_cpu_ptr(&saved_next_opcode2);
if (OPCODE_JSR(p->opcode) || OPCODE_JMP(p->opcode)) {
unsigned int reg_nr = ((p->opcode >> 8) & 0x000F);
@@ -249,7 +249,7 @@ static int __kprobes kprobe_handler(stru
kcb->kprobe_status = KPROBE_REENTER;
return 1;
} else {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
goto ss_probe;
}
@@ -336,9 +336,9 @@ int __kprobes trampoline_probe_handler(s
continue;
if (ri->rp && ri->rp->handler) {
- __get_cpu_var(current_kprobe) = &ri->rp->kp;
+ __this_cpu_write(current_kprobe, &ri->rp->kp);
ri->rp->handler(ri, regs);
- __get_cpu_var(current_kprobe) = NULL;
+ __this_cpu_write(current_kprobe, NULL);
}
orig_ret_address = (unsigned long)ri->ret_addr;
@@ -383,19 +383,19 @@ static int __kprobes post_kprobe_handler
cur->post_handler(cur, regs, 0);
}
- p = &__get_cpu_var(saved_next_opcode);
+ p = this_cpu_ptr(&saved_next_opcode);
if (p->addr) {
arch_disarm_kprobe(p);
p->addr = NULL;
p->opcode = 0;
- addr = __get_cpu_var(saved_current_opcode).addr;
- __get_cpu_var(saved_current_opcode).addr = NULL;
+ addr = __this_cpu_read(saved_current_opcode).addr;
+ __this_cpu_write(saved_current_opcode.addr, NULL);
p = get_kprobe(addr);
arch_arm_kprobe(p);
- p = &__get_cpu_var(saved_next_opcode2);
+ p = this_cpu_ptr(&saved_next_opcode2);
if (p->addr) {
arch_disarm_kprobe(p);
p->addr = NULL;
@@ -511,7 +511,7 @@ int __kprobes kprobe_exceptions_notify(s
if (kprobe_handler(args->regs)) {
ret = NOTIFY_STOP;
} else {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler &&
p->break_handler(p, args->regs))
ret = NOTIFY_STOP;
Index: linux/arch/sh/kernel/localtimer.c
=================================--- linux.orig/arch/sh/kernel/localtimer.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/localtimer.c 2013-08-26 13:28:32.265604741 -0500
@@ -32,7 +32,7 @@ static DEFINE_PER_CPU(struct clock_event
*/
void local_timer_interrupt(void)
{
- struct clock_event_device *clk = &__get_cpu_var(local_clockevent);
+ struct clock_event_device *clk = this_cpu_ptr(&local_clockevent);
irq_enter();
clk->event_handler(clk);
Index: linux/arch/sh/kernel/perf_event.c
=================================--- linux.orig/arch/sh/kernel/perf_event.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/perf_event.c 2013-08-26 13:28:32.265604741 -0500
@@ -227,7 +227,7 @@ again:
static void sh_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -245,7 +245,7 @@ static void sh_pmu_stop(struct perf_even
static void sh_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -262,7 +262,7 @@ static void sh_pmu_start(struct perf_eve
static void sh_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
sh_pmu_stop(event, PERF_EF_UPDATE);
__clear_bit(event->hw.idx, cpuc->used_mask);
@@ -272,7 +272,7 @@ static void sh_pmu_del(struct perf_event
static int sh_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
int ret = -EAGAIN;
Index: linux/arch/sh/kernel/smp.c
=================================--- linux.orig/arch/sh/kernel/smp.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/smp.c 2013-08-26 13:28:32.265604741 -0500
@@ -111,7 +111,7 @@ void play_dead_common(void)
irq_ctx_exit(raw_smp_processor_id());
mb();
- __get_cpu_var(cpu_state) = CPU_DEAD;
+ __this_cpu_write(cpu_state, CPU_DEAD);
local_irq_disable();
}
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 27/31] sh: Replace __get_cpu_var uses throughout the kernel
@ 2013-08-26 21:56 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:56 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Paul Mundt, linux-sh, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/sh/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/sh/kernel/hw_breakpoint.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/hw_breakpoint.c 2013-08-26 13:28:32.265604741 -0500
@@ -52,7 +52,7 @@ int arch_install_hw_breakpoint(struct pe
int i;
for (i = 0; i < sh_ubc->num_events; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (!*slot) {
*slot = bp;
@@ -84,7 +84,7 @@ void arch_uninstall_hw_breakpoint(struct
int i;
for (i = 0; i < sh_ubc->num_events; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (*slot == bp) {
*slot = NULL;
Index: linux/arch/sh/kernel/kprobes.c
===================================================================
--- linux.orig/arch/sh/kernel/kprobes.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/kprobes.c 2013-08-26 13:28:32.265604741 -0500
@@ -102,7 +102,7 @@ int __kprobes kprobe_handle_illslot(unsi
void __kprobes arch_remove_kprobe(struct kprobe *p)
{
- struct kprobe *saved = &__get_cpu_var(saved_next_opcode);
+ struct kprobe *saved = this_cpu_ptr(&saved_next_opcode);
if (saved->addr) {
arch_disarm_kprobe(p);
@@ -111,7 +111,7 @@ void __kprobes arch_remove_kprobe(struct
saved->addr = NULL;
saved->opcode = 0;
- saved = &__get_cpu_var(saved_next_opcode2);
+ saved = this_cpu_ptr(&saved_next_opcode2);
if (saved->addr) {
arch_disarm_kprobe(saved);
@@ -129,14 +129,14 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
/*
@@ -146,15 +146,15 @@ static void __kprobes set_current_kprobe
*/
static void __kprobes prepare_singlestep(struct kprobe *p, struct pt_regs *regs)
{
- __get_cpu_var(saved_current_opcode).addr = (kprobe_opcode_t *)regs->pc;
+ __this_cpu_write(saved_current_opcode.addr, (kprobe_opcode_t *)regs->pc);
if (p != NULL) {
struct kprobe *op1, *op2;
arch_disarm_kprobe(p);
- op1 = &__get_cpu_var(saved_next_opcode);
- op2 = &__get_cpu_var(saved_next_opcode2);
+ op1 = this_cpu_ptr(&saved_next_opcode);
+ op2 = this_cpu_ptr(&saved_next_opcode2);
if (OPCODE_JSR(p->opcode) || OPCODE_JMP(p->opcode)) {
unsigned int reg_nr = ((p->opcode >> 8) & 0x000F);
@@ -249,7 +249,7 @@ static int __kprobes kprobe_handler(stru
kcb->kprobe_status = KPROBE_REENTER;
return 1;
} else {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
goto ss_probe;
}
@@ -336,9 +336,9 @@ int __kprobes trampoline_probe_handler(s
continue;
if (ri->rp && ri->rp->handler) {
- __get_cpu_var(current_kprobe) = &ri->rp->kp;
+ __this_cpu_write(current_kprobe, &ri->rp->kp);
ri->rp->handler(ri, regs);
- __get_cpu_var(current_kprobe) = NULL;
+ __this_cpu_write(current_kprobe, NULL);
}
orig_ret_address = (unsigned long)ri->ret_addr;
@@ -383,19 +383,19 @@ static int __kprobes post_kprobe_handler
cur->post_handler(cur, regs, 0);
}
- p = &__get_cpu_var(saved_next_opcode);
+ p = this_cpu_ptr(&saved_next_opcode);
if (p->addr) {
arch_disarm_kprobe(p);
p->addr = NULL;
p->opcode = 0;
- addr = __get_cpu_var(saved_current_opcode).addr;
- __get_cpu_var(saved_current_opcode).addr = NULL;
+ addr = __this_cpu_read(saved_current_opcode).addr;
+ __this_cpu_write(saved_current_opcode.addr, NULL);
p = get_kprobe(addr);
arch_arm_kprobe(p);
- p = &__get_cpu_var(saved_next_opcode2);
+ p = this_cpu_ptr(&saved_next_opcode2);
if (p->addr) {
arch_disarm_kprobe(p);
p->addr = NULL;
@@ -511,7 +511,7 @@ int __kprobes kprobe_exceptions_notify(s
if (kprobe_handler(args->regs)) {
ret = NOTIFY_STOP;
} else {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler &&
p->break_handler(p, args->regs))
ret = NOTIFY_STOP;
Index: linux/arch/sh/kernel/localtimer.c
===================================================================
--- linux.orig/arch/sh/kernel/localtimer.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/localtimer.c 2013-08-26 13:28:32.265604741 -0500
@@ -32,7 +32,7 @@ static DEFINE_PER_CPU(struct clock_event
*/
void local_timer_interrupt(void)
{
- struct clock_event_device *clk = &__get_cpu_var(local_clockevent);
+ struct clock_event_device *clk = this_cpu_ptr(&local_clockevent);
irq_enter();
clk->event_handler(clk);
Index: linux/arch/sh/kernel/perf_event.c
===================================================================
--- linux.orig/arch/sh/kernel/perf_event.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/perf_event.c 2013-08-26 13:28:32.265604741 -0500
@@ -227,7 +227,7 @@ again:
static void sh_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -245,7 +245,7 @@ static void sh_pmu_stop(struct perf_even
static void sh_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -262,7 +262,7 @@ static void sh_pmu_start(struct perf_eve
static void sh_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
sh_pmu_stop(event, PERF_EF_UPDATE);
__clear_bit(event->hw.idx, cpuc->used_mask);
@@ -272,7 +272,7 @@ static void sh_pmu_del(struct perf_event
static int sh_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
int ret = -EAGAIN;
Index: linux/arch/sh/kernel/smp.c
===================================================================
--- linux.orig/arch/sh/kernel/smp.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/smp.c 2013-08-26 13:28:32.265604741 -0500
@@ -111,7 +111,7 @@ void play_dead_common(void)
irq_ctx_exit(raw_smp_processor_id());
mb();
- __get_cpu_var(cpu_state) = CPU_DEAD;
+ __this_cpu_write(cpu_state, CPU_DEAD);
local_irq_disable();
}
^ permalink raw reply [flat|nested] 96+ messages in thread
* [guv v2 27/31] sh: Replace __get_cpu_var uses throughout the kernel
@ 2013-08-26 21:56 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-26 21:56 UTC (permalink / raw)
To: Tejun Heo
Cc: akpm, Paul Mundt, linux-sh, linux-arch, Steven Rostedt, linux-kernel
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Index: linux/arch/sh/kernel/hw_breakpoint.c
===================================================================
--- linux.orig/arch/sh/kernel/hw_breakpoint.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/hw_breakpoint.c 2013-08-26 13:28:32.265604741 -0500
@@ -52,7 +52,7 @@ int arch_install_hw_breakpoint(struct pe
int i;
for (i = 0; i < sh_ubc->num_events; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (!*slot) {
*slot = bp;
@@ -84,7 +84,7 @@ void arch_uninstall_hw_breakpoint(struct
int i;
for (i = 0; i < sh_ubc->num_events; i++) {
- struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
+ struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
if (*slot == bp) {
*slot = NULL;
Index: linux/arch/sh/kernel/kprobes.c
===================================================================
--- linux.orig/arch/sh/kernel/kprobes.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/kprobes.c 2013-08-26 13:28:32.265604741 -0500
@@ -102,7 +102,7 @@ int __kprobes kprobe_handle_illslot(unsi
void __kprobes arch_remove_kprobe(struct kprobe *p)
{
- struct kprobe *saved = &__get_cpu_var(saved_next_opcode);
+ struct kprobe *saved = this_cpu_ptr(&saved_next_opcode);
if (saved->addr) {
arch_disarm_kprobe(p);
@@ -111,7 +111,7 @@ void __kprobes arch_remove_kprobe(struct
saved->addr = NULL;
saved->opcode = 0;
- saved = &__get_cpu_var(saved_next_opcode2);
+ saved = this_cpu_ptr(&saved_next_opcode2);
if (saved->addr) {
arch_disarm_kprobe(saved);
@@ -129,14 +129,14 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
/*
@@ -146,15 +146,15 @@ static void __kprobes set_current_kprobe
*/
static void __kprobes prepare_singlestep(struct kprobe *p, struct pt_regs *regs)
{
- __get_cpu_var(saved_current_opcode).addr = (kprobe_opcode_t *)regs->pc;
+ __this_cpu_write(saved_current_opcode.addr, (kprobe_opcode_t *)regs->pc);
if (p != NULL) {
struct kprobe *op1, *op2;
arch_disarm_kprobe(p);
- op1 = &__get_cpu_var(saved_next_opcode);
- op2 = &__get_cpu_var(saved_next_opcode2);
+ op1 = this_cpu_ptr(&saved_next_opcode);
+ op2 = this_cpu_ptr(&saved_next_opcode2);
if (OPCODE_JSR(p->opcode) || OPCODE_JMP(p->opcode)) {
unsigned int reg_nr = ((p->opcode >> 8) & 0x000F);
@@ -249,7 +249,7 @@ static int __kprobes kprobe_handler(stru
kcb->kprobe_status = KPROBE_REENTER;
return 1;
} else {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
goto ss_probe;
}
@@ -336,9 +336,9 @@ int __kprobes trampoline_probe_handler(s
continue;
if (ri->rp && ri->rp->handler) {
- __get_cpu_var(current_kprobe) = &ri->rp->kp;
+ __this_cpu_write(current_kprobe, &ri->rp->kp);
ri->rp->handler(ri, regs);
- __get_cpu_var(current_kprobe) = NULL;
+ __this_cpu_write(current_kprobe, NULL);
}
orig_ret_address = (unsigned long)ri->ret_addr;
@@ -383,19 +383,19 @@ static int __kprobes post_kprobe_handler
cur->post_handler(cur, regs, 0);
}
- p = &__get_cpu_var(saved_next_opcode);
+ p = this_cpu_ptr(&saved_next_opcode);
if (p->addr) {
arch_disarm_kprobe(p);
p->addr = NULL;
p->opcode = 0;
- addr = __get_cpu_var(saved_current_opcode).addr;
- __get_cpu_var(saved_current_opcode).addr = NULL;
+ addr = __this_cpu_read(saved_current_opcode).addr;
+ __this_cpu_write(saved_current_opcode.addr, NULL);
p = get_kprobe(addr);
arch_arm_kprobe(p);
- p = &__get_cpu_var(saved_next_opcode2);
+ p = this_cpu_ptr(&saved_next_opcode2);
if (p->addr) {
arch_disarm_kprobe(p);
p->addr = NULL;
@@ -511,7 +511,7 @@ int __kprobes kprobe_exceptions_notify(s
if (kprobe_handler(args->regs)) {
ret = NOTIFY_STOP;
} else {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler &&
p->break_handler(p, args->regs))
ret = NOTIFY_STOP;
Index: linux/arch/sh/kernel/localtimer.c
===================================================================
--- linux.orig/arch/sh/kernel/localtimer.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/localtimer.c 2013-08-26 13:28:32.265604741 -0500
@@ -32,7 +32,7 @@ static DEFINE_PER_CPU(struct clock_event
*/
void local_timer_interrupt(void)
{
- struct clock_event_device *clk = &__get_cpu_var(local_clockevent);
+ struct clock_event_device *clk = this_cpu_ptr(&local_clockevent);
irq_enter();
clk->event_handler(clk);
Index: linux/arch/sh/kernel/perf_event.c
===================================================================
--- linux.orig/arch/sh/kernel/perf_event.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/perf_event.c 2013-08-26 13:28:32.265604741 -0500
@@ -227,7 +227,7 @@ again:
static void sh_pmu_stop(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -245,7 +245,7 @@ static void sh_pmu_stop(struct perf_even
static void sh_pmu_start(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
@@ -262,7 +262,7 @@ static void sh_pmu_start(struct perf_eve
static void sh_pmu_del(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
sh_pmu_stop(event, PERF_EF_UPDATE);
__clear_bit(event->hw.idx, cpuc->used_mask);
@@ -272,7 +272,7 @@ static void sh_pmu_del(struct perf_event
static int sh_pmu_add(struct perf_event *event, int flags)
{
- struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
int ret = -EAGAIN;
Index: linux/arch/sh/kernel/smp.c
===================================================================
--- linux.orig/arch/sh/kernel/smp.c 2013-08-26 13:25:18.000000000 -0500
+++ linux/arch/sh/kernel/smp.c 2013-08-26 13:28:32.265604741 -0500
@@ -111,7 +111,7 @@ void play_dead_common(void)
irq_ctx_exit(raw_smp_processor_id());
mb();
- __get_cpu_var(cpu_state) = CPU_DEAD;
+ __this_cpu_write(cpu_state, CPU_DEAD);
local_irq_disable();
}
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
2013-08-26 20:56 ` Christoph Lameter
@ 2013-08-26 22:22 ` Chris Metcalf
-1 siblings, 0 replies; 96+ messages in thread
From: Chris Metcalf @ 2013-08-26 22:22 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On 8/26/2013 4:56 PM, Christoph Lameter wrote:
> Index: linux/arch/tile/include/asm/irqflags.h
> ===================================================================
> --- linux.orig/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:14.000000000 -0500
> +++ linux/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:57.453305409 -0500
Nack to the changes to this file, with all due respect :-), since I have conflicting changes in the linux-tile tree that also remove all uses of __get_cpu_var from this file. Note that there are places in this file where "this_cpu_and", etc., make sense; I don't know if it makes sense for your script to be looking for this kind of thing more generally.
> - __get_cpu_var(current_asid) = asid;
> + __this_cpu_read(current_asid) = asid;
This should be __this_cpu_write(current_asid, asid). You caught this idiom elsewhere so not sure if it was a script bug that caused it to be missed here?
> - int depth = __get_cpu_var(irq_depth)++;
> + int depth = __this_cpu_inc_return(irq_depth) - 1;
I would omit the "-1" here and instead change the one test of "depth == 0" to be "depth == 1".
> - __get_cpu_var(current_asid) = min_asid = asid_range.start;
> + __this_cpu_write(current_asid, min_asid = asid_range.start);
This was ugly before but now it's much worse :-) Please just split it into an assignment to min_asid, then use __this_cpu_write() with min_asid. Thanks.
The rest is fine. Would it make more sense for me to fold all these changes into the tile tree and just have it meet in linux-next? There are certainly a bunch of other changes staged there which likely also include more uses of __get_cpu_var...
--
Chris Metcalf, Tilera Corp.
http://www.tilera.com
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
@ 2013-08-26 22:22 ` Chris Metcalf
0 siblings, 0 replies; 96+ messages in thread
From: Chris Metcalf @ 2013-08-26 22:22 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On 8/26/2013 4:56 PM, Christoph Lameter wrote:
> Index: linux/arch/tile/include/asm/irqflags.h
> ===================================================================
> --- linux.orig/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:14.000000000 -0500
> +++ linux/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:57.453305409 -0500
Nack to the changes to this file, with all due respect :-), since I have conflicting changes in the linux-tile tree that also remove all uses of __get_cpu_var from this file. Note that there are places in this file where "this_cpu_and", etc., make sense; I don't know if it makes sense for your script to be looking for this kind of thing more generally.
> - __get_cpu_var(current_asid) = asid;
> + __this_cpu_read(current_asid) = asid;
This should be __this_cpu_write(current_asid, asid). You caught this idiom elsewhere so not sure if it was a script bug that caused it to be missed here?
> - int depth = __get_cpu_var(irq_depth)++;
> + int depth = __this_cpu_inc_return(irq_depth) - 1;
I would omit the "-1" here and instead change the one test of "depth == 0" to be "depth == 1".
> - __get_cpu_var(current_asid) = min_asid = asid_range.start;
> + __this_cpu_write(current_asid, min_asid = asid_range.start);
This was ugly before but now it's much worse :-) Please just split it into an assignment to min_asid, then use __this_cpu_write() with min_asid. Thanks.
The rest is fine. Would it make more sense for me to fold all these changes into the tile tree and just have it meet in linux-next? There are certainly a bunch of other changes staged there which likely also include more uses of __get_cpu_var...
--
Chris Metcalf, Tilera Corp.
http://www.tilera.com
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 25/31] avr32: Replace __get_cpu_var uses
2013-08-26 20:56 ` Christoph Lameter
(?)
@ 2013-08-27 6:58 ` Hans-Christian Egtvedt
-1 siblings, 0 replies; 96+ messages in thread
From: Hans-Christian Egtvedt @ 2013-08-27 6:58 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, Haavard Skinnemoen, linux-arch, Steven Rostedt,
linux-kernel
Around Mon 26 Aug 2013 20:56:16 +0000 or thereabout, Christoph Lameter wrote:
<snipp reasoning>
> Index: linux/arch/avr32/kernel/kprobes.c
> ===================================================================
> --- linux.orig/arch/avr32/kernel/kprobes.c 2013-08-26 13:25:18.000000000 -0500
> +++ linux/arch/avr32/kernel/kprobes.c 2013-08-26 13:27:09.758485808 -0500
> @@ -104,7 +104,7 @@ static void __kprobes resume_execution(s
>
> static void __kprobes set_current_kprobe(struct kprobe *p)
> {
> - __get_cpu_var(current_kprobe) = p;
> + __this_cpu_write(current_kprobe, p);
> }
>
> static int __kprobes kprobe_handler(struct pt_regs *regs)
Acked-by: Hans-Christian Egtvedt <egtvedt@samfundet.no>
--
mvh
Hans-Christian Egtvedt
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 20/31] powerpc: Replace __get_cpu_var uses
2013-08-26 20:44 ` Christoph Lameter
(?)
@ 2013-08-27 8:44 ` Geert Uytterhoeven
2013-08-27 15:37 ` Christoph Lameter
-1 siblings, 1 reply; 96+ messages in thread
From: Geert Uytterhoeven @ 2013-08-27 8:44 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, Benjamin Herrenschmidt, Paul Mackerras,
Linux-Arch, Steven Rostedt, linux-kernel
On Mon, Aug 26, 2013 at 10:44 PM, Christoph Lameter <cl@linux.com> wrote:
> __get_cpu_var() is used for multiple purposes in the kernel source. One of them is
> address calculation via the form &__get_cpu_var(x). This calculates the address for
> the instance of the percpu variable of the current processor based on an offset.
>
> Others usage cases are for storing and retrieving data from the current processors percpu area.
use cases
> __get_cpu_var() always only does a address determination. However, store and retrieve operations
an address
> This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
an explicit
> 4. Retrieve the content of a percpu struct
>
> DEFINE_PER_CPU(struct mystruct, y);
> struct mystruct x = __get_cpu_var(y);
>
> Converts to
>
> memcpy(this_cpu_ptr(&y), x, sizeof(x));
Let's hope the actual code copies in the other direction ;-)
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 22/31] arm: Replace __get_cpu_var uses
2013-08-26 20:44 ` Christoph Lameter
(?)
@ 2013-08-27 9:41 ` Catalin Marinas
-1 siblings, 0 replies; 96+ messages in thread
From: Catalin Marinas @ 2013-08-27 9:41 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, Russell King, Will Deacon, linux-arch,
Steven Rostedt, linux-kernel
On Mon, Aug 26, 2013 at 09:44:52PM +0100, Christoph Lameter wrote:
> This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
> or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
> and less registers are used when code is generated.
...
> Index: linux/arch/arm64/kernel/debug-monitors.c
> ===================================================================
> --- linux.orig/arch/arm64/kernel/debug-monitors.c 2013-08-26 13:48:40.956794980 -0500
> +++ linux/arch/arm64/kernel/debug-monitors.c 2013-08-26 13:48:40.952795024 -0500
...
> Index: linux/arch/arm64/kernel/hw_breakpoint.c
> ===================================================================
> --- linux.orig/arch/arm64/kernel/hw_breakpoint.c 2013-08-26 13:48:40.956794980 -0500
> +++ linux/arch/arm64/kernel/hw_breakpoint.c 2013-08-26 13:48:40.952795024 -0500
...
> Index: linux/arch/arm64/kernel/perf_event.c
> ===================================================================
> --- linux.orig/arch/arm64/kernel/perf_event.c 2013-08-26 13:48:40.956794980 -0500
> +++ linux/arch/arm64/kernel/perf_event.c 2013-08-26 13:48:40.952795024 -0500
For the arm64 bits:
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 30/31] metag: Replace __get_cpu_var uses
2013-08-26 21:56 ` Christoph Lameter
@ 2013-08-27 9:53 ` James Hogan
-1 siblings, 0 replies; 96+ messages in thread
From: James Hogan @ 2013-08-27 9:53 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On 26/08/13 22:56, Christoph Lameter wrote:
> Signed-off-by: Christoph Lameter <cl@linux.com>
Acked-by: James Hogan <james.hogan@imgtec.com>
Cheers
James
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 30/31] metag: Replace __get_cpu_var uses
@ 2013-08-27 9:53 ` James Hogan
0 siblings, 0 replies; 96+ messages in thread
From: James Hogan @ 2013-08-27 9:53 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On 26/08/13 22:56, Christoph Lameter wrote:
> Signed-off-by: Christoph Lameter <cl@linux.com>
Acked-by: James Hogan <james.hogan@imgtec.com>
Cheers
James
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
2013-08-26 22:22 ` Chris Metcalf
(?)
@ 2013-08-27 14:48 ` Christoph Lameter
2013-08-28 17:19 ` Chris Metcalf
-1 siblings, 1 reply; 96+ messages in thread
From: Christoph Lameter @ 2013-08-27 14:48 UTC (permalink / raw)
To: Chris Metcalf; +Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On Mon, 26 Aug 2013, Chris Metcalf wrote:
> On 8/26/2013 4:56 PM, Christoph Lameter wrote:
> > Index: linux/arch/tile/include/asm/irqflags.h
> > ===================================================================
> > --- linux.orig/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:14.000000000 -0500
> > +++ linux/arch/tile/include/asm/irqflags.h 2013-08-22 14:39:57.453305409 -0500
>
> Nack to the changes to this file, with all due respect :-), since I have conflicting changes in the linux-tile tree that also remove all uses of __get_cpu_var from this file. Note that there are places in this file where "this_cpu_and", etc., make sense; I don't know if it makes sense for your script to be looking for this kind of thing more generally.
Ok. Dropped that part.
> > - __get_cpu_var(current_asid) = asid;
> > + __this_cpu_read(current_asid) = asid;
>
> This should be __this_cpu_write(current_asid, asid). You caught this
> idiom elsewhere so not sure if it was a script bug that caused it to be
> missed here?
No these cases were rare and had to be fixed up manually. Missed that one.
> > - int depth = __get_cpu_var(irq_depth)++;
> > + int depth = __this_cpu_inc_return(irq_depth) - 1;
>
> I would omit the "-1" here and instead change the one test of "depth == 0" to be "depth == 1".
Ok.
> - __get_cpu_var(current_asid) = min_asid = asid_range.start;
> > + __this_cpu_write(current_asid, min_asid = asid_range.start);
>
> This was ugly before but now it's much worse :-) Please just split it
> into an assignment to min_asid, then use __this_cpu_write() with
> min_asid. Thanks.
Ok.
> The rest is fine. Would it make more sense for me to fold all these
> changes into the tile tree and just have it meet in linux-next? There
> are certainly a bunch of other changes staged there which likely also
> include more uses of __get_cpu_var...
Yes. Please stage this patch for the next merge window.
Fixed up patch follows:
Subject: tile: Replace __get_cpu_var uses
Cc: Chris Metcalf <cmetcalf@tilera.com>
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Others usage cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does a address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&y), x, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/tile/include/asm/mmu_context.h
===================================================================
--- linux.orig/arch/tile/include/asm/mmu_context.h 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/include/asm/mmu_context.h 2013-08-27 09:42:50.913976738 -0500
@@ -84,7 +84,7 @@ static inline void enter_lazy_tlb(struct
* clear any pending DMA interrupts.
*/
if (current->thread.tile_dma_state.enabled)
- install_page_table(mm->pgd, __get_cpu_var(current_asid));
+ install_page_table(mm->pgd, __this_cpu_read(current_asid));
#endif
}
@@ -96,12 +96,12 @@ static inline void switch_mm(struct mm_s
int cpu = smp_processor_id();
/* Pick new ASID. */
- int asid = __get_cpu_var(current_asid) + 1;
+ int asid = __this_cpu_read(current_asid) + 1;
if (asid > max_asid) {
asid = min_asid;
local_flush_tlb();
}
- __get_cpu_var(current_asid) = asid;
+ __this_cpu_write(current_asid, asid);
/* Clear cpu from the old mm, and set it in the new one. */
cpumask_clear_cpu(cpu, mm_cpumask(prev));
Index: linux/arch/tile/kernel/irq.c
===================================================================
--- linux.orig/arch/tile/kernel/irq.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/irq.c 2013-08-27 09:44:04.321219883 -0500
@@ -77,7 +77,7 @@ static DEFINE_SPINLOCK(available_irqs_lo
*/
void tile_dev_intr(struct pt_regs *regs, int intnum)
{
- int depth = __get_cpu_var(irq_depth)++;
+ int depth = __this_cpu_inc_return(irq_depth);
unsigned long original_irqs;
unsigned long remaining_irqs;
struct pt_regs *old_regs;
@@ -124,7 +124,7 @@ void tile_dev_intr(struct pt_regs *regs,
/* Count device irqs; Linux IPIs are counted elsewhere. */
if (irq != IRQ_RESCHEDULE)
- __get_cpu_var(irq_stat).irq_dev_intr_count++;
+ __this_cpu_inc(irq_stat.irq_dev_intr_count);
generic_handle_irq(irq);
}
@@ -134,10 +134,10 @@ void tile_dev_intr(struct pt_regs *regs,
* including any that were reenabled during interrupt
* handling.
*/
- if (depth == 0)
- unmask_irqs(~__get_cpu_var(irq_disable_mask));
+ if (depth == 1)
+ unmask_irqs(~__this_cpu_read(irq_disable_mask));
- __get_cpu_var(irq_depth)--;
+ __this_cpu_dec(irq_depth);
/*
* Track time spent against the current process again and
@@ -155,7 +155,7 @@ void tile_dev_intr(struct pt_regs *regs,
static void tile_irq_chip_enable(struct irq_data *d)
{
get_cpu_var(irq_disable_mask) &= ~(1UL << d->irq);
- if (__get_cpu_var(irq_depth) == 0)
+ if (__this_cpu_read(irq_depth) == 0)
unmask_irqs(1UL << d->irq);
put_cpu_var(irq_disable_mask);
}
@@ -201,7 +201,7 @@ static void tile_irq_chip_ack(struct irq
*/
static void tile_irq_chip_eoi(struct irq_data *d)
{
- if (!(__get_cpu_var(irq_disable_mask) & (1UL << d->irq)))
+ if (!(__this_cpu_read(irq_disable_mask) & (1UL << d->irq)))
unmask_irqs(1UL << d->irq);
}
Index: linux/arch/tile/kernel/messaging.c
===================================================================
--- linux.orig/arch/tile/kernel/messaging.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/messaging.c 2013-08-27 09:42:50.913976738 -0500
@@ -28,7 +28,7 @@ static DEFINE_PER_CPU(HV_MsgState, msg_s
void init_messaging(void)
{
/* Allocate storage for messages in kernel space */
- HV_MsgState *state = &__get_cpu_var(msg_state);
+ HV_MsgState *state = this_cpu_ptr(&msg_state);
int rc = hv_register_message_state(state);
if (rc != HV_OK)
panic("hv_register_message_state: error %d", rc);
@@ -68,7 +68,7 @@ void hv_message_intr(struct pt_regs *reg
#endif
while (1) {
- rmi = hv_receive_message(__get_cpu_var(msg_state),
+ rmi = hv_receive_message(__this_cpu_read(msg_state),
(HV_VirtAddr) message,
sizeof(message));
if (rmi.msglen == 0)
@@ -96,7 +96,7 @@ void hv_message_intr(struct pt_regs *reg
struct hv_driver_cb *cb =
(struct hv_driver_cb *)him->intarg;
cb->callback(cb, him->intdata);
- __get_cpu_var(irq_stat).irq_hv_msg_count++;
+ __this_cpu_inc(irq_stat.irq_hv_msg_count);
}
}
Index: linux/arch/tile/kernel/process.c
===================================================================
--- linux.orig/arch/tile/kernel/process.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/process.c 2013-08-27 09:42:50.913976738 -0500
@@ -63,7 +63,7 @@ early_param("idle", idle_setup);
void arch_cpu_idle(void)
{
- __get_cpu_var(irq_stat).idle_timestamp = jiffies;
+ __this_cpu_write(irq_stat.idle_timestamp, jiffies);
_cpu_idle();
}
Index: linux/arch/tile/kernel/setup.c
===================================================================
--- linux.orig/arch/tile/kernel/setup.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/setup.c 2013-08-27 09:44:57.600670623 -0500
@@ -1119,7 +1119,8 @@ static void __init validate_hv(void)
* various asid variables to their appropriate initial states.
*/
asid_range = hv_inquire_asid(0);
- __get_cpu_var(current_asid) = min_asid = asid_range.start;
+ min_asid = asid_range.start;
+ __this_cpu_write(current_asid, min_asid);
max_asid = asid_range.start + asid_range.size - 1;
if (hv_confstr(HV_CONFSTR_CHIP_MODEL, (HV_VirtAddr)chip_model,
Index: linux/arch/tile/kernel/single_step.c
===================================================================
--- linux.orig/arch/tile/kernel/single_step.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/single_step.c 2013-08-27 09:42:50.917976698 -0500
@@ -732,7 +732,7 @@ static DEFINE_PER_CPU(unsigned long, ss_
void gx_singlestep_handle(struct pt_regs *regs, int fault_num)
{
- unsigned long *ss_pc = &__get_cpu_var(ss_saved_pc);
+ unsigned long *ss_pc = this_cpu_ptr(&ss_saved_pc);
struct thread_info *info = (void *)current_thread_info();
int is_single_step = test_ti_thread_flag(info, TIF_SINGLESTEP);
unsigned long control = __insn_mfspr(SPR_SINGLE_STEP_CONTROL_K);
@@ -758,7 +758,7 @@ void gx_singlestep_handle(struct pt_regs
void single_step_once(struct pt_regs *regs)
{
- unsigned long *ss_pc = &__get_cpu_var(ss_saved_pc);
+ unsigned long *ss_pc = this_cpu_ptr(&ss_saved_pc);
unsigned long control = __insn_mfspr(SPR_SINGLE_STEP_CONTROL_K);
*ss_pc = regs->pc;
Index: linux/arch/tile/kernel/smp.c
===================================================================
--- linux.orig/arch/tile/kernel/smp.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/smp.c 2013-08-27 09:42:50.917976698 -0500
@@ -176,7 +176,7 @@ void flush_icache_range(unsigned long st
/* Called when smp_send_reschedule() triggers IRQ_RESCHEDULE. */
static irqreturn_t handle_reschedule_ipi(int irq, void *token)
{
- __get_cpu_var(irq_stat).irq_resched_count++;
+ __this_cpu_inc(irq_stat.irq_resched_count);
scheduler_ipi();
return IRQ_HANDLED;
Index: linux/arch/tile/kernel/smpboot.c
===================================================================
--- linux.orig/arch/tile/kernel/smpboot.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/smpboot.c 2013-08-27 09:42:50.917976698 -0500
@@ -41,7 +41,7 @@ void __init smp_prepare_boot_cpu(void)
int cpu = smp_processor_id();
set_cpu_online(cpu, 1);
set_cpu_present(cpu, 1);
- __get_cpu_var(cpu_state) = CPU_ONLINE;
+ __this_cpu_write(cpu_state, CPU_ONLINE);
init_messaging();
}
@@ -156,7 +156,7 @@ static void start_secondary(void)
/* printk(KERN_DEBUG "Initializing CPU#%d\n", cpuid); */
/* Initialize the current asid for our first page table. */
- __get_cpu_var(current_asid) = min_asid;
+ __this_cpu_write(current_asid, min_asid);
/* Set up this thread as another owner of the init_mm */
atomic_inc(&init_mm.mm_count);
@@ -199,7 +199,7 @@ void online_secondary(void)
notify_cpu_starting(smp_processor_id());
set_cpu_online(smp_processor_id(), 1);
- __get_cpu_var(cpu_state) = CPU_ONLINE;
+ __this_cpu_write(cpu_state, CPU_ONLINE);
/* Set up tile-specific state for this cpu. */
setup_cpu(0);
Index: linux/arch/tile/kernel/time.c
===================================================================
--- linux.orig/arch/tile/kernel/time.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/time.c 2013-08-27 09:42:50.917976698 -0500
@@ -161,7 +161,7 @@ static DEFINE_PER_CPU(struct clock_event
void setup_tile_timer(void)
{
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = this_cpu_ptr(&tile_timer);
/* Fill in fields that are speed-specific. */
clockevents_calc_mult_shift(evt, cycles_per_sec, TILE_MINSEC);
@@ -181,7 +181,7 @@ void setup_tile_timer(void)
void do_timer_interrupt(struct pt_regs *regs, int fault_num)
{
struct pt_regs *old_regs = set_irq_regs(regs);
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = this_cpu_ptr(&tile_timer);
/*
* Mask the timer interrupt here, since we are a oneshot timer
@@ -193,7 +193,7 @@ void do_timer_interrupt(struct pt_regs *
irq_enter();
/* Track interrupt count. */
- __get_cpu_var(irq_stat).irq_timer_count++;
+ __this_cpu_inc(irq_stat.irq_timer_count);
/* Call the generic timer handler */
evt->event_handler(evt);
@@ -234,6 +234,6 @@ cycles_t ns2cycles(unsigned long nsecs)
* We do not have to disable preemption here as each core has the same
* clock frequency.
*/
- struct clock_event_device *dev = &__raw_get_cpu_var(tile_timer);
+ struct clock_event_device *dev = __this_cpu_ptr(&tile_timer);
return ((u64)nsecs * dev->mult) >> dev->shift;
}
Index: linux/arch/tile/mm/highmem.c
===================================================================
--- linux.orig/arch/tile/mm/highmem.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/mm/highmem.c 2013-08-27 09:42:50.917976698 -0500
@@ -103,7 +103,7 @@ static void kmap_atomic_register(struct
spin_lock(&_lock);
/* With interrupts disabled, now fill in the per-cpu info. */
- amp = &__get_cpu_var(amps).per_type[type];
+ amp = this_cpu_ptr(&s.per_type[type]);
amp->page = page;
amp->cpu = smp_processor_id();
amp->va = va;
Index: linux/arch/tile/mm/init.c
===================================================================
--- linux.orig/arch/tile/mm/init.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/mm/init.c 2013-08-27 09:42:50.917976698 -0500
@@ -625,14 +625,14 @@ static void __init kernel_physical_mappi
interrupt_mask_set_mask(-1ULL);
rc = flush_and_install_context(__pa(pgtables),
init_pgprot((unsigned long)pgtables),
- __get_cpu_var(current_asid),
+ __this_cpu_read(current_asid),
cpumask_bits(my_cpu_mask));
interrupt_mask_restore_mask(irqmask);
BUG_ON(rc != 0);
/* Copy the page table back to the normal swapper_pg_dir. */
memcpy(pgd_base, pgtables, sizeof(pgtables));
- __install_page_table(pgd_base, __get_cpu_var(current_asid),
+ __install_page_table(pgd_base, __this_cpu_read(current_asid),
swapper_pgprot);
/*
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 20/31] powerpc: Replace __get_cpu_var uses
2013-08-27 8:44 ` Geert Uytterhoeven
@ 2013-08-27 15:37 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-27 15:37 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Tejun Heo, akpm, Benjamin Herrenschmidt, Paul Mackerras,
Linux-Arch, Steven Rostedt, linux-kernel
On Tue, 27 Aug 2013, Geert Uytterhoeven wrote:
> On Mon, Aug 26, 2013 at 10:44 PM, Christoph Lameter <cl@linux.com> wrote:
> > __get_cpu_var() is used for multiple purposes in the kernel source. One of them is
> > address calculation via the form &__get_cpu_var(x). This calculates the address for
> > the instance of the percpu variable of the current processor based on an offset.
> >
> > Others usage cases are for storing and retrieving data from the current processors percpu area.
>
> use cases
>
> > __get_cpu_var() always only does a address determination. However, store and retrieve operations
>
> an address
>
> > This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
>
> an explicit
>
> > 4. Retrieve the content of a percpu struct
> >
> > DEFINE_PER_CPU(struct mystruct, y);
> > struct mystruct x = __get_cpu_var(y);
> >
> > Converts to
> >
> > memcpy(this_cpu_ptr(&y), x, sizeof(x));
>
> Let's hope the actual code copies in the other direction ;-)
Ok fixed that as well.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 28/31] arc: Replace __get_cpu_var uses throughout the kernel
[not found] ` <00000140c05d28b7-f0a2bc86-fe85-40f7-b627-32664cd616a4-000000@email.amazonses.com>
@ 2013-08-28 4:24 ` Vineet Gupta
2013-08-28 13:46 ` Christoph Lameter
0 siblings, 1 reply; 96+ messages in thread
From: Vineet Gupta @ 2013-08-28 4:24 UTC (permalink / raw)
To: Christoph Lameter; +Cc: Vineet Gupta, Tejun Heo, akpm, lkml, linux-arch
On 08/27/2013 08:51 PM, Christoph Lameter wrote:
> On Tue, 27 Aug 2013, Vineet Gupta wrote:
>
>> The other thread about merging logistics doesn't seem to have concluded yet so for
>> now I'll let it sit with you - please let me know if it needs to be routed via the
>> arc tree.
> Please route via the arc tree. I will marge the last patch when all the
> bits are in.
>
Done - although it is missing you SOB line ?
-Vineet
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 18/31] s390: Replace __get_cpu_var uses
2013-08-26 20:44 ` Christoph Lameter
(?)
@ 2013-08-28 6:27 ` Heiko Carstens
-1 siblings, 0 replies; 96+ messages in thread
From: Heiko Carstens @ 2013-08-28 6:27 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, Martin Schwidefsky, linux390, linux-arch,
Steven Rostedt, linux-kernel
On Mon, Aug 26, 2013 at 08:44:40PM +0000, Christoph Lameter wrote:
> __get_cpu_var() is used for multiple purposes in the kernel source. One of them is
> address calculation via the form &__get_cpu_var(x). This calculates the address for
> the instance of the percpu variable of the current processor based on an offset.
> 4. Retrieve the content of a percpu struct
>
> DEFINE_PER_CPU(struct mystruct, y);
> struct mystruct x = __get_cpu_var(y);
>
> Converts to
>
> memcpy(this_cpu_ptr(&y), x, sizeof(x));
That's wrong (destination and source should be exchanged).
However your patch looks good.
Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com>
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 28/31] arc: Replace __get_cpu_var uses throughout the kernel
2013-08-28 4:24 ` Vineet Gupta
@ 2013-08-28 13:46 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-28 13:46 UTC (permalink / raw)
To: Vineet Gupta; +Cc: Tejun Heo, akpm, lkml, linux-arch
On Wed, 28 Aug 2013, Vineet Gupta wrote:
> On 08/27/2013 08:51 PM, Christoph Lameter wrote:
> > On Tue, 27 Aug 2013, Vineet Gupta wrote:
> >
> >> The other thread about merging logistics doesn't seem to have concluded yet so for
> >> now I'll let it sit with you - please let me know if it needs to be routed via the
> >> arc tree.
> > Please route via the arc tree. I will marge the last patch when all the
> > bits are in.
> >
>
> Done - although it is missing you SOB line ?
Sorry about that. Here is an updated version
Subject: arc: Replace __get_cpu_var uses
Cc: Vineet Gupta <vgupta@synopsys.com>
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Other use cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does an address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either an explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&x), y, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Acked-by: Vineet Gupta <vgupta@synopsys.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/arc/kernel/kprobes.c
===================================================================
--- linux.orig/arch/arc/kernel/kprobes.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/arc/kernel/kprobes.c 2013-08-26 13:29:19.285103021 -0500
@@ -87,13 +87,13 @@ static void __kprobes save_previous_kpro
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
}
static inline void __kprobes set_current_kprobe(struct kprobe *p)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
}
static void __kprobes resume_execution(struct kprobe *p, unsigned long addr,
@@ -237,7 +237,7 @@ int __kprobes arc_kprobe_handler(unsigne
return 1;
} else if (kprobe_running()) {
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs)) {
setup_singlestep(p, regs);
kcb->kprobe_status = KPROBE_HIT_SS;
Index: linux/arch/arc/kernel/time.c
===================================================================
--- linux.orig/arch/arc/kernel/time.c 2013-08-26 13:28:53.000000000 -0500
+++ linux/arch/arc/kernel/time.c 2013-08-26 13:29:19.285103021 -0500
@@ -206,7 +206,7 @@ static DEFINE_PER_CPU(struct clock_event
static irqreturn_t timer_irq_handler(int irq, void *dev_id)
{
- struct clock_event_device *clk = &__get_cpu_var(arc_clockevent_device);
+ struct clock_event_device *clk = this_cpu_ptr(&arc_clockevent_device);
arc_timer_event_ack(clk->mode == CLOCK_EVT_MODE_PERIODIC);
clk->event_handler(clk);
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
2013-08-27 14:48 ` Christoph Lameter
@ 2013-08-28 17:19 ` Chris Metcalf
0 siblings, 0 replies; 96+ messages in thread
From: Chris Metcalf @ 2013-08-28 17:19 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On 8/27/2013 10:48 AM, Christoph Lameter wrote:
> On Mon, 26 Aug 2013, Chris Metcalf wrote:
>> > The rest is fine. Would it make more sense for me to fold all these
>> > changes into the tile tree and just have it meet in linux-next? There
>> > are certainly a bunch of other changes staged there which likely also
>> > include more uses of __get_cpu_var...
> Yes. Please stage this patch for the next merge window.
Taken into the tile tree, with one extra change; in arch/tile/kernel/messaging.c, using "__this_cpu_read(msg_state)" generated warnings ("'pscr_ret__.opaque[1u]' is used uninitialized in this function", etc), since msg_state is a struct, so I changed it to be "*__this_cpu_ptr(&msg_state)". Interrupts are disabled here so preemption is not a concern.
In addition, I applied the following commit to clean up the remaining uses that are currently only being carried in the linux-tile tree.
commit db4dd3097952d2fd20e61f820dbb757ba871302d
Author: Chris Metcalf <cmetcalf@tilera.com>
Date: Wed Aug 28 12:01:44 2013 -0400
tile: remove newer uses of __get_cpu_var
See commit eb1fe08d458f for the first round.
Signed-off-by: Chris Metcalf <cmetcalf@tilera.com>
diff --git a/arch/tile/kernel/kprobes.c b/arch/tile/kernel/kprobes.c
index 1129f52..27cdcac 100644
--- a/arch/tile/kernel/kprobes.c
+++ b/arch/tile/kernel/kprobes.c
@@ -159,7 +159,7 @@ static void __kprobes save_previous_kprobe(struct kprobe_ctlblk *kcb)
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
kcb->kprobe_saved_pc = kcb->prev_kprobe.saved_pc;
}
@@ -167,7 +167,7 @@ static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
kcb->kprobe_saved_pc = regs->pc;
}
@@ -228,7 +228,7 @@ static int __kprobes kprobe_handler(struct pt_regs *regs)
ret = 1;
goto no_kprobe;
}
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs))
goto ss_probe;
}
diff --git a/arch/tile/kernel/time.c b/arch/tile/kernel/time.c
index a7a41a1..30c2c34 100644
--- a/arch/tile/kernel/time.c
+++ b/arch/tile/kernel/time.c
@@ -286,7 +286,7 @@ static int _set_clock_rate_barrier;
static int _set_clock_rate(void *arg)
{
struct _set_clock_rate_args *args = arg;
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = get_cpu_ptr(&tile_timer);
/*
* Only one CPU needs to change the timekeeping parameters and
--
Chris Metcalf, Tilera Corp.
http://www.tilera.com
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
@ 2013-08-28 17:19 ` Chris Metcalf
0 siblings, 0 replies; 96+ messages in thread
From: Chris Metcalf @ 2013-08-28 17:19 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On 8/27/2013 10:48 AM, Christoph Lameter wrote:
> On Mon, 26 Aug 2013, Chris Metcalf wrote:
>> > The rest is fine. Would it make more sense for me to fold all these
>> > changes into the tile tree and just have it meet in linux-next? There
>> > are certainly a bunch of other changes staged there which likely also
>> > include more uses of __get_cpu_var...
> Yes. Please stage this patch for the next merge window.
Taken into the tile tree, with one extra change; in arch/tile/kernel/messaging.c, using "__this_cpu_read(msg_state)" generated warnings ("'pscr_ret__.opaque[1u]' is used uninitialized in this function", etc), since msg_state is a struct, so I changed it to be "*__this_cpu_ptr(&msg_state)". Interrupts are disabled here so preemption is not a concern.
In addition, I applied the following commit to clean up the remaining uses that are currently only being carried in the linux-tile tree.
commit db4dd3097952d2fd20e61f820dbb757ba871302d
Author: Chris Metcalf <cmetcalf@tilera.com>
Date: Wed Aug 28 12:01:44 2013 -0400
tile: remove newer uses of __get_cpu_var
See commit eb1fe08d458f for the first round.
Signed-off-by: Chris Metcalf <cmetcalf@tilera.com>
diff --git a/arch/tile/kernel/kprobes.c b/arch/tile/kernel/kprobes.c
index 1129f52..27cdcac 100644
--- a/arch/tile/kernel/kprobes.c
+++ b/arch/tile/kernel/kprobes.c
@@ -159,7 +159,7 @@ static void __kprobes save_previous_kprobe(struct kprobe_ctlblk *kcb)
static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = kcb->prev_kprobe.kp;
+ __this_cpu_write(current_kprobe, kcb->prev_kprobe.kp);
kcb->kprobe_status = kcb->prev_kprobe.status;
kcb->kprobe_saved_pc = kcb->prev_kprobe.saved_pc;
}
@@ -167,7 +167,7 @@ static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb)
static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
struct kprobe_ctlblk *kcb)
{
- __get_cpu_var(current_kprobe) = p;
+ __this_cpu_write(current_kprobe, p);
kcb->kprobe_saved_pc = regs->pc;
}
@@ -228,7 +228,7 @@ static int __kprobes kprobe_handler(struct pt_regs *regs)
ret = 1;
goto no_kprobe;
}
- p = __get_cpu_var(current_kprobe);
+ p = __this_cpu_read(current_kprobe);
if (p->break_handler && p->break_handler(p, regs))
goto ss_probe;
}
diff --git a/arch/tile/kernel/time.c b/arch/tile/kernel/time.c
index a7a41a1..30c2c34 100644
--- a/arch/tile/kernel/time.c
+++ b/arch/tile/kernel/time.c
@@ -286,7 +286,7 @@ static int _set_clock_rate_barrier;
static int _set_clock_rate(void *arg)
{
struct _set_clock_rate_args *args = arg;
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = get_cpu_ptr(&tile_timer);
/*
* Only one CPU needs to change the timekeeping parameters and
--
Chris Metcalf, Tilera Corp.
http://www.tilera.com
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
2013-08-28 17:19 ` Chris Metcalf
(?)
@ 2013-08-28 17:54 ` Christoph Lameter
2013-08-28 18:13 ` Chris Metcalf
-1 siblings, 1 reply; 96+ messages in thread
From: Christoph Lameter @ 2013-08-28 17:54 UTC (permalink / raw)
To: Chris Metcalf; +Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On Wed, 28 Aug 2013, Chris Metcalf wrote:
> Taken into the tile tree, with one extra change; in
> arch/tile/kernel/messaging.c, using "__this_cpu_read(msg_state)"
> generated warnings ("'pscr_ret__.opaque[1u]' is used uninitialized in
> this function", etc), since msg_state is a struct, so I changed it to be
> "*__this_cpu_ptr(&msg_state)". Interrupts are disabled here so
> preemption is not a concern.
Well yes. __this_cpu_read can only be used to read scalars. If you want to
transfer a struct then memcpy needs to be used (or the compiler can do
that for you like above).
> In addition, I applied the following commit to clean up the remaining
> uses that are currently only being carried in the linux-tile tree.
You are going to merge these changes right? I can drop the tile patch from
my series?
One small issue below:
> diff --git a/arch/tile/kernel/time.c b/arch/tile/kernel/time.c
> index a7a41a1..30c2c34 100644
> --- a/arch/tile/kernel/time.c
> +++ b/arch/tile/kernel/time.c
> @@ -286,7 +286,7 @@ static int _set_clock_rate_barrier;
> static int _set_clock_rate(void *arg)
> {
> struct _set_clock_rate_args *args = arg;
> - struct clock_event_device *evt = &__get_cpu_var(tile_timer);
> + struct clock_event_device *evt = get_cpu_ptr(&tile_timer);
^^^^ this_cpu_ptr
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
2013-08-28 17:54 ` Christoph Lameter
@ 2013-08-28 18:13 ` Chris Metcalf
0 siblings, 0 replies; 96+ messages in thread
From: Chris Metcalf @ 2013-08-28 18:13 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On 8/28/2013 1:54 PM, Christoph Lameter wrote:
> On Wed, 28 Aug 2013, Chris Metcalf wrote:
>> In addition, I applied the following commit to clean up the remaining
>> uses that are currently only being carried in the linux-tile tree.
> You are going to merge these changes right? I can drop the tile patch from
> my series?
Yes, I am, so please feel free to drop it.
> One small issue below:
>
>> diff --git a/arch/tile/kernel/time.c b/arch/tile/kernel/time.c
>> index a7a41a1..30c2c34 100644
>> --- a/arch/tile/kernel/time.c
>> +++ b/arch/tile/kernel/time.c
>> @@ -286,7 +286,7 @@ static int _set_clock_rate_barrier;
>> static int _set_clock_rate(void *arg)
>> {
>> struct _set_clock_rate_args *args = arg;
>> - struct clock_event_device *evt = &__get_cpu_var(tile_timer);
>> + struct clock_event_device *evt = get_cpu_ptr(&tile_timer);
> ^^^^ this_cpu_ptr
Thanks, good catch. Annoying to have preemption disabled forever otherwise :-)
--
Chris Metcalf, Tilera Corp.
http://www.tilera.com
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
@ 2013-08-28 18:13 ` Chris Metcalf
0 siblings, 0 replies; 96+ messages in thread
From: Chris Metcalf @ 2013-08-28 18:13 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On 8/28/2013 1:54 PM, Christoph Lameter wrote:
> On Wed, 28 Aug 2013, Chris Metcalf wrote:
>> In addition, I applied the following commit to clean up the remaining
>> uses that are currently only being carried in the linux-tile tree.
> You are going to merge these changes right? I can drop the tile patch from
> my series?
Yes, I am, so please feel free to drop it.
> One small issue below:
>
>> diff --git a/arch/tile/kernel/time.c b/arch/tile/kernel/time.c
>> index a7a41a1..30c2c34 100644
>> --- a/arch/tile/kernel/time.c
>> +++ b/arch/tile/kernel/time.c
>> @@ -286,7 +286,7 @@ static int _set_clock_rate_barrier;
>> static int _set_clock_rate(void *arg)
>> {
>> struct _set_clock_rate_args *args = arg;
>> - struct clock_event_device *evt = &__get_cpu_var(tile_timer);
>> + struct clock_event_device *evt = get_cpu_ptr(&tile_timer);
> ^^^^ this_cpu_ptr
Thanks, good catch. Annoying to have preemption disabled forever otherwise :-)
--
Chris Metcalf, Tilera Corp.
http://www.tilera.com
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
2013-08-28 18:13 ` Chris Metcalf
(?)
@ 2013-08-28 18:22 ` Christoph Lameter
-1 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-08-28 18:22 UTC (permalink / raw)
To: Chris Metcalf; +Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel
On Wed, 28 Aug 2013, Chris Metcalf wrote:
> Yes, I am, so please feel free to drop it.
Following is the last state of the patch from my tree. Dropping it.
Subject: tile: Replace __get_cpu_var uses
Cc: Chris Metcalf <cmetcalf@tilera.com>
__get_cpu_var() is used for multiple purposes in the kernel source. One of them is
address calculation via the form &__get_cpu_var(x). This calculates the address for
the instance of the percpu variable of the current processor based on an offset.
Other use cases are for storing and retrieving data from the current processors percpu area.
__get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
#define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
__get_cpu_var() always only does an address determination. However, store and retrieve operations
could use a segment prefix (or global register on other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
optimized assembly code to read and write per cpu variables.
This patch converts __get_cpu_var into either an explicit address calculation using this_cpu_ptr()
or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
and less registers are used when code is generated.
At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
The patchset includes passes over all arches as well. Once these operations are used throughout then
specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
f.e. using a global register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu variable.
DEFINE_PER_CPU(int, u);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(this_cpu_ptr(&x), y, sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
this_cpu_inc(y)
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/arch/tile/include/asm/mmu_context.h
===================================================================
--- linux.orig/arch/tile/include/asm/mmu_context.h 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/include/asm/mmu_context.h 2013-08-27 09:42:50.913976738 -0500
@@ -84,7 +84,7 @@ static inline void enter_lazy_tlb(struct
* clear any pending DMA interrupts.
*/
if (current->thread.tile_dma_state.enabled)
- install_page_table(mm->pgd, __get_cpu_var(current_asid));
+ install_page_table(mm->pgd, __this_cpu_read(current_asid));
#endif
}
@@ -96,12 +96,12 @@ static inline void switch_mm(struct mm_s
int cpu = smp_processor_id();
/* Pick new ASID. */
- int asid = __get_cpu_var(current_asid) + 1;
+ int asid = __this_cpu_read(current_asid) + 1;
if (asid > max_asid) {
asid = min_asid;
local_flush_tlb();
}
- __get_cpu_var(current_asid) = asid;
+ __this_cpu_write(current_asid, asid);
/* Clear cpu from the old mm, and set it in the new one. */
cpumask_clear_cpu(cpu, mm_cpumask(prev));
Index: linux/arch/tile/kernel/irq.c
===================================================================
--- linux.orig/arch/tile/kernel/irq.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/irq.c 2013-08-27 09:44:04.321219883 -0500
@@ -77,7 +77,7 @@ static DEFINE_SPINLOCK(available_irqs_lo
*/
void tile_dev_intr(struct pt_regs *regs, int intnum)
{
- int depth = __get_cpu_var(irq_depth)++;
+ int depth = __this_cpu_inc_return(irq_depth);
unsigned long original_irqs;
unsigned long remaining_irqs;
struct pt_regs *old_regs;
@@ -124,7 +124,7 @@ void tile_dev_intr(struct pt_regs *regs,
/* Count device irqs; Linux IPIs are counted elsewhere. */
if (irq != IRQ_RESCHEDULE)
- __get_cpu_var(irq_stat).irq_dev_intr_count++;
+ __this_cpu_inc(irq_stat.irq_dev_intr_count);
generic_handle_irq(irq);
}
@@ -134,10 +134,10 @@ void tile_dev_intr(struct pt_regs *regs,
* including any that were reenabled during interrupt
* handling.
*/
- if (depth == 0)
- unmask_irqs(~__get_cpu_var(irq_disable_mask));
+ if (depth == 1)
+ unmask_irqs(~__this_cpu_read(irq_disable_mask));
- __get_cpu_var(irq_depth)--;
+ __this_cpu_dec(irq_depth);
/*
* Track time spent against the current process again and
@@ -155,7 +155,7 @@ void tile_dev_intr(struct pt_regs *regs,
static void tile_irq_chip_enable(struct irq_data *d)
{
get_cpu_var(irq_disable_mask) &= ~(1UL << d->irq);
- if (__get_cpu_var(irq_depth) == 0)
+ if (__this_cpu_read(irq_depth) == 0)
unmask_irqs(1UL << d->irq);
put_cpu_var(irq_disable_mask);
}
@@ -201,7 +201,7 @@ static void tile_irq_chip_ack(struct irq
*/
static void tile_irq_chip_eoi(struct irq_data *d)
{
- if (!(__get_cpu_var(irq_disable_mask) & (1UL << d->irq)))
+ if (!(__this_cpu_read(irq_disable_mask) & (1UL << d->irq)))
unmask_irqs(1UL << d->irq);
}
Index: linux/arch/tile/kernel/messaging.c
===================================================================
--- linux.orig/arch/tile/kernel/messaging.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/messaging.c 2013-08-27 09:42:50.913976738 -0500
@@ -28,7 +28,7 @@ static DEFINE_PER_CPU(HV_MsgState, msg_s
void init_messaging(void)
{
/* Allocate storage for messages in kernel space */
- HV_MsgState *state = &__get_cpu_var(msg_state);
+ HV_MsgState *state = this_cpu_ptr(&msg_state);
int rc = hv_register_message_state(state);
if (rc != HV_OK)
panic("hv_register_message_state: error %d", rc);
@@ -68,7 +68,7 @@ void hv_message_intr(struct pt_regs *reg
#endif
while (1) {
- rmi = hv_receive_message(__get_cpu_var(msg_state),
+ rmi = hv_receive_message(__this_cpu_read(msg_state),
(HV_VirtAddr) message,
sizeof(message));
if (rmi.msglen == 0)
@@ -96,7 +96,7 @@ void hv_message_intr(struct pt_regs *reg
struct hv_driver_cb *cb =
(struct hv_driver_cb *)him->intarg;
cb->callback(cb, him->intdata);
- __get_cpu_var(irq_stat).irq_hv_msg_count++;
+ __this_cpu_inc(irq_stat.irq_hv_msg_count);
}
}
Index: linux/arch/tile/kernel/process.c
===================================================================
--- linux.orig/arch/tile/kernel/process.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/process.c 2013-08-27 09:42:50.913976738 -0500
@@ -63,7 +63,7 @@ early_param("idle", idle_setup);
void arch_cpu_idle(void)
{
- __get_cpu_var(irq_stat).idle_timestamp = jiffies;
+ __this_cpu_write(irq_stat.idle_timestamp, jiffies);
_cpu_idle();
}
Index: linux/arch/tile/kernel/setup.c
===================================================================
--- linux.orig/arch/tile/kernel/setup.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/setup.c 2013-08-27 09:44:57.600670623 -0500
@@ -1119,7 +1119,8 @@ static void __init validate_hv(void)
* various asid variables to their appropriate initial states.
*/
asid_range = hv_inquire_asid(0);
- __get_cpu_var(current_asid) = min_asid = asid_range.start;
+ min_asid = asid_range.start;
+ __this_cpu_write(current_asid, min_asid);
max_asid = asid_range.start + asid_range.size - 1;
if (hv_confstr(HV_CONFSTR_CHIP_MODEL, (HV_VirtAddr)chip_model,
Index: linux/arch/tile/kernel/single_step.c
===================================================================
--- linux.orig/arch/tile/kernel/single_step.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/single_step.c 2013-08-27 09:42:50.917976698 -0500
@@ -732,7 +732,7 @@ static DEFINE_PER_CPU(unsigned long, ss_
void gx_singlestep_handle(struct pt_regs *regs, int fault_num)
{
- unsigned long *ss_pc = &__get_cpu_var(ss_saved_pc);
+ unsigned long *ss_pc = this_cpu_ptr(&ss_saved_pc);
struct thread_info *info = (void *)current_thread_info();
int is_single_step = test_ti_thread_flag(info, TIF_SINGLESTEP);
unsigned long control = __insn_mfspr(SPR_SINGLE_STEP_CONTROL_K);
@@ -758,7 +758,7 @@ void gx_singlestep_handle(struct pt_regs
void single_step_once(struct pt_regs *regs)
{
- unsigned long *ss_pc = &__get_cpu_var(ss_saved_pc);
+ unsigned long *ss_pc = this_cpu_ptr(&ss_saved_pc);
unsigned long control = __insn_mfspr(SPR_SINGLE_STEP_CONTROL_K);
*ss_pc = regs->pc;
Index: linux/arch/tile/kernel/smp.c
===================================================================
--- linux.orig/arch/tile/kernel/smp.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/smp.c 2013-08-27 09:42:50.917976698 -0500
@@ -176,7 +176,7 @@ void flush_icache_range(unsigned long st
/* Called when smp_send_reschedule() triggers IRQ_RESCHEDULE. */
static irqreturn_t handle_reschedule_ipi(int irq, void *token)
{
- __get_cpu_var(irq_stat).irq_resched_count++;
+ __this_cpu_inc(irq_stat.irq_resched_count);
scheduler_ipi();
return IRQ_HANDLED;
Index: linux/arch/tile/kernel/smpboot.c
===================================================================
--- linux.orig/arch/tile/kernel/smpboot.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/smpboot.c 2013-08-27 09:42:50.917976698 -0500
@@ -41,7 +41,7 @@ void __init smp_prepare_boot_cpu(void)
int cpu = smp_processor_id();
set_cpu_online(cpu, 1);
set_cpu_present(cpu, 1);
- __get_cpu_var(cpu_state) = CPU_ONLINE;
+ __this_cpu_write(cpu_state, CPU_ONLINE);
init_messaging();
}
@@ -156,7 +156,7 @@ static void start_secondary(void)
/* printk(KERN_DEBUG "Initializing CPU#%d\n", cpuid); */
/* Initialize the current asid for our first page table. */
- __get_cpu_var(current_asid) = min_asid;
+ __this_cpu_write(current_asid, min_asid);
/* Set up this thread as another owner of the init_mm */
atomic_inc(&init_mm.mm_count);
@@ -199,7 +199,7 @@ void online_secondary(void)
notify_cpu_starting(smp_processor_id());
set_cpu_online(smp_processor_id(), 1);
- __get_cpu_var(cpu_state) = CPU_ONLINE;
+ __this_cpu_write(cpu_state, CPU_ONLINE);
/* Set up tile-specific state for this cpu. */
setup_cpu(0);
Index: linux/arch/tile/kernel/time.c
===================================================================
--- linux.orig/arch/tile/kernel/time.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/kernel/time.c 2013-08-27 09:42:50.917976698 -0500
@@ -161,7 +161,7 @@ static DEFINE_PER_CPU(struct clock_event
void setup_tile_timer(void)
{
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = this_cpu_ptr(&tile_timer);
/* Fill in fields that are speed-specific. */
clockevents_calc_mult_shift(evt, cycles_per_sec, TILE_MINSEC);
@@ -181,7 +181,7 @@ void setup_tile_timer(void)
void do_timer_interrupt(struct pt_regs *regs, int fault_num)
{
struct pt_regs *old_regs = set_irq_regs(regs);
- struct clock_event_device *evt = &__get_cpu_var(tile_timer);
+ struct clock_event_device *evt = this_cpu_ptr(&tile_timer);
/*
* Mask the timer interrupt here, since we are a oneshot timer
@@ -193,7 +193,7 @@ void do_timer_interrupt(struct pt_regs *
irq_enter();
/* Track interrupt count. */
- __get_cpu_var(irq_stat).irq_timer_count++;
+ __this_cpu_inc(irq_stat.irq_timer_count);
/* Call the generic timer handler */
evt->event_handler(evt);
@@ -234,6 +234,6 @@ cycles_t ns2cycles(unsigned long nsecs)
* We do not have to disable preemption here as each core has the same
* clock frequency.
*/
- struct clock_event_device *dev = &__raw_get_cpu_var(tile_timer);
+ struct clock_event_device *dev = __this_cpu_ptr(&tile_timer);
return ((u64)nsecs * dev->mult) >> dev->shift;
}
Index: linux/arch/tile/mm/highmem.c
===================================================================
--- linux.orig/arch/tile/mm/highmem.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/mm/highmem.c 2013-08-27 09:42:50.917976698 -0500
@@ -103,7 +103,7 @@ static void kmap_atomic_register(struct
spin_lock(&_lock);
/* With interrupts disabled, now fill in the per-cpu info. */
- amp = &__get_cpu_var(amps).per_type[type];
+ amp = this_cpu_ptr(&s.per_type[type]);
amp->page = page;
amp->cpu = smp_processor_id();
amp->va = va;
Index: linux/arch/tile/mm/init.c
===================================================================
--- linux.orig/arch/tile/mm/init.c 2013-08-27 09:42:50.925976615 -0500
+++ linux/arch/tile/mm/init.c 2013-08-27 09:42:50.917976698 -0500
@@ -625,14 +625,14 @@ static void __init kernel_physical_mappi
interrupt_mask_set_mask(-1ULL);
rc = flush_and_install_context(__pa(pgtables),
init_pgprot((unsigned long)pgtables),
- __get_cpu_var(current_asid),
+ __this_cpu_read(current_asid),
cpumask_bits(my_cpu_mask));
interrupt_mask_restore_mask(irqmask);
BUG_ON(rc != 0);
/* Copy the page table back to the normal swapper_pg_dir. */
memcpy(pgd_base, pgtables, sizeof(pgtables));
- __install_page_table(pgd_base, __get_cpu_var(current_asid),
+ __install_page_table(pgd_base, __this_cpu_read(current_asid),
swapper_pgprot);
/*
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 08/31] tracing: Replace __get_cpu_var uses
2013-08-26 20:44 ` Christoph Lameter
(?)
@ 2013-08-28 19:48 ` Steven Rostedt
2013-08-29 3:44 ` Masami Hiramatsu
2013-08-29 16:35 ` Christoph Lameter
-1 siblings, 2 replies; 96+ messages in thread
From: Steven Rostedt @ 2013-08-28 19:48 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, Frederic Weisbecker, Ingo Molnar, linux-arch,
Steven Rostedt, linux-kernel, Masami Hiramatsu
On Mon, 26 Aug 2013 20:44:37 +0000
Christoph Lameter <cl@linux.com> wrote:
> __get_cpu_var() is used for multiple purposes in the kernel source. One of them is
> address calculation via the form &__get_cpu_var(x). This calculates the address for
> the instance of the percpu variable of the current processor based on an offset.
>
> Others usage cases are for storing and retrieving data from the current processors percpu area.
> __get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
>
> __get_cpu_var() is defined as :
>
>
> #define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
>
>
>
> __get_cpu_var() always only does a address determination. However, store and retrieve operations
> could use a segment prefix (or global register on other platforms) to avoid the address calculation.
>
> this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
> optimized assembly code to read and write per cpu variables.
>
>
> This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
> or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
> and less registers are used when code is generated.
>
> At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
>
> The patchset includes passes over all arches as well. Once these operations are used throughout then
> specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
> f.e. using a global register that may be set to the per cpu base.
You should limit the width of the change log to 76 characters.
I fixed it for now.
>
>
>
>
> Transformations done to __get_cpu_var()
>
>
> 1. Determine the address of the percpu instance of the current processor.
>
> DEFINE_PER_CPU(int, y);
> int *x = &__get_cpu_var(y);
>
> Converts to
>
> int *x = this_cpu_ptr(&y);
>
>
> 2. Same as #1 but this time an array structure is involved.
>
> DEFINE_PER_CPU(int, y[20]);
> int *x = __get_cpu_var(y);
>
> Converts to
>
> int *x = this_cpu_ptr(y);
>
>
> 3. Retrieve the content of the current processors instance of a per cpu variable.
>
> DEFINE_PER_CPU(int, u);
> int x = __get_cpu_var(y)
>
> Converts to
>
> int x = __this_cpu_read(y);
>
>
> 4. Retrieve the content of a percpu struct
>
> DEFINE_PER_CPU(struct mystruct, y);
> struct mystruct x = __get_cpu_var(y);
>
> Converts to
>
> memcpy(this_cpu_ptr(&y), x, sizeof(x));
>
>
> 5. Assignment to a per cpu variable
>
> DEFINE_PER_CPU(int, y)
> __get_cpu_var(y) = x;
>
> Converts to
>
> this_cpu_write(y, x);
>
>
> 6. Increment/Decrement etc of a per cpu variable
>
> DEFINE_PER_CPU(int, y);
> __get_cpu_var(y)++
>
> Converts to
>
> this_cpu_inc(y)
>
>
> Signed-off-by: Christoph Lameter <cl@linux.com>
>
> Index: linux/include/linux/kprobes.h
> ===================================================================
> --- linux.orig/include/linux/kprobes.h 2013-08-26 14:25:53.000000000 -0500
> +++ linux/include/linux/kprobes.h 2013-08-26 14:26:29.460993659 -0500
> @@ -329,7 +329,7 @@ static inline void reset_current_kprobe(
>
> static inline struct kprobe_ctlblk *get_kprobe_ctlblk(void)
> {
> - return (&__get_cpu_var(kprobe_ctlblk));
> + return this_cpu_ptr(&kprobe_ctlblk);
> }
Masami,
Can you give me an Acked-by: for this.
Thanks!
-- Steve
>
> int register_kprobe(struct kprobe *p);
> Index: linux/kernel/trace/ftrace.c
> ===================================================================
> --- linux.orig/kernel/trace/ftrace.c 2013-08-26 14:25:53.000000000 -0500
> +++ linux/kernel/trace/ftrace.c 2013-08-26 14:26:29.460993659 -0500
> @@ -870,7 +870,7 @@ function_profile_call(unsigned long ip,
>
> local_irq_save(flags);
>
> - stat = &__get_cpu_var(ftrace_profile_stats);
> + stat = this_cpu_ptr(&ftrace_profile_stats);
> if (!stat->hash || !ftrace_profile_enabled)
> goto out;
>
> @@ -901,7 +901,7 @@ static void profile_graph_return(struct
> unsigned long flags;
>
> local_irq_save(flags);
> - stat = &__get_cpu_var(ftrace_profile_stats);
> + stat = this_cpu_ptr(&ftrace_profile_stats);
> if (!stat->hash || !ftrace_profile_enabled)
> goto out;
>
> Index: linux/kernel/trace/trace.c
> ===================================================================
> --- linux.orig/kernel/trace/trace.c 2013-08-26 14:25:53.000000000 -0500
> +++ linux/kernel/trace/trace.c 2013-08-26 14:26:29.464993617 -0500
> @@ -1676,7 +1676,7 @@ static void __ftrace_trace_stack(struct
> */
> barrier();
> if (use_stack == 1) {
> - trace.entries = &__get_cpu_var(ftrace_stack).calls[0];
> + trace.entries = this_cpu_ptr(ftrace_stack.calls);
> trace.max_entries = FTRACE_STACK_MAX_ENTRIES;
>
> if (regs)
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 08/31] tracing: Replace __get_cpu_var uses
2013-08-28 19:48 ` Steven Rostedt
@ 2013-08-29 3:44 ` Masami Hiramatsu
2013-08-29 16:35 ` Christoph Lameter
1 sibling, 0 replies; 96+ messages in thread
From: Masami Hiramatsu @ 2013-08-29 3:44 UTC (permalink / raw)
To: Steven Rostedt
Cc: Christoph Lameter, Tejun Heo, akpm, Frederic Weisbecker,
Ingo Molnar, linux-arch, Steven Rostedt, linux-kernel
(2013/08/29 4:48), Steven Rostedt wrote:
> On Mon, 26 Aug 2013 20:44:37 +0000
> Christoph Lameter <cl@linux.com> wrote:
>
>> __get_cpu_var() is used for multiple purposes in the kernel source. One of them is
>> address calculation via the form &__get_cpu_var(x). This calculates the address for
>> the instance of the percpu variable of the current processor based on an offset.
>>
>> Others usage cases are for storing and retrieving data from the current processors percpu area.
>> __get_cpu_var() can be used as an lvalue when writing data or on the right side of an assignment.
>>
>> __get_cpu_var() is defined as :
>>
>>
>> #define __get_cpu_var(var) (*this_cpu_ptr(&(var)))
>>
>>
>>
>> __get_cpu_var() always only does a address determination. However, store and retrieve operations
>> could use a segment prefix (or global register on other platforms) to avoid the address calculation.
>>
>> this_cpu_write() and this_cpu_read() can directly take an offset into a percpu area and use
>> optimized assembly code to read and write per cpu variables.
>>
>>
>> This patch converts __get_cpu_var into either and explicit address calculation using this_cpu_ptr()
>> or into a use of this_cpu operations that use the offset. Thereby address calcualtions are avoided
>> and less registers are used when code is generated.
>>
>> At the end of the patchset all uses of __get_cpu_var have been removed so the macro is removed too.
>>
>> The patchset includes passes over all arches as well. Once these operations are used throughout then
>> specialized macros can be defined in non -x86 arches as well in order to optimize per cpu access by
>> f.e. using a global register that may be set to the per cpu base.
>
> You should limit the width of the change log to 76 characters.
>
> I fixed it for now.
>
>>
>>
>>
>>
>> Transformations done to __get_cpu_var()
>>
>>
>> 1. Determine the address of the percpu instance of the current processor.
>>
>> DEFINE_PER_CPU(int, y);
>> int *x = &__get_cpu_var(y);
>>
>> Converts to
>>
>> int *x = this_cpu_ptr(&y);
>>
>>
>> 2. Same as #1 but this time an array structure is involved.
>>
>> DEFINE_PER_CPU(int, y[20]);
>> int *x = __get_cpu_var(y);
>>
>> Converts to
>>
>> int *x = this_cpu_ptr(y);
>>
>>
>> 3. Retrieve the content of the current processors instance of a per cpu variable.
>>
>> DEFINE_PER_CPU(int, u);
>> int x = __get_cpu_var(y)
>>
>> Converts to
>>
>> int x = __this_cpu_read(y);
>>
>>
>> 4. Retrieve the content of a percpu struct
>>
>> DEFINE_PER_CPU(struct mystruct, y);
>> struct mystruct x = __get_cpu_var(y);
>>
>> Converts to
>>
>> memcpy(this_cpu_ptr(&y), x, sizeof(x));
>>
>>
>> 5. Assignment to a per cpu variable
>>
>> DEFINE_PER_CPU(int, y)
>> __get_cpu_var(y) = x;
>>
>> Converts to
>>
>> this_cpu_write(y, x);
>>
>>
>> 6. Increment/Decrement etc of a per cpu variable
>>
>> DEFINE_PER_CPU(int, y);
>> __get_cpu_var(y)++
>>
>> Converts to
>>
>> this_cpu_inc(y)
>>
>>
>> Signed-off-by: Christoph Lameter <cl@linux.com>
>>
>> Index: linux/include/linux/kprobes.h
>> ===================================================================
>> --- linux.orig/include/linux/kprobes.h 2013-08-26 14:25:53.000000000 -0500
>> +++ linux/include/linux/kprobes.h 2013-08-26 14:26:29.460993659 -0500
>> @@ -329,7 +329,7 @@ static inline void reset_current_kprobe(
>>
>> static inline struct kprobe_ctlblk *get_kprobe_ctlblk(void)
>> {
>> - return (&__get_cpu_var(kprobe_ctlblk));
>> + return this_cpu_ptr(&kprobe_ctlblk);
>> }
>
> Masami,
>
> Can you give me an Acked-by: for this.
>
Thanks for pinging me ;)
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Thank you,
--
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 08/31] tracing: Replace __get_cpu_var uses
2013-08-28 19:48 ` Steven Rostedt
2013-08-29 3:44 ` Masami Hiramatsu
@ 2013-08-29 16:35 ` Christoph Lameter
2013-08-30 7:00 ` Ingo Molnar
1 sibling, 1 reply; 96+ messages in thread
From: Christoph Lameter @ 2013-08-29 16:35 UTC (permalink / raw)
To: Steven Rostedt
Cc: Tejun Heo, akpm, Frederic Weisbecker, Ingo Molnar, linux-arch,
Steven Rostedt, linux-kernel, Masami Hiramatsu
On Wed, 28 Aug 2013, Steven Rostedt wrote:
> You should limit the width of the change log to 76 characters.
>
> I fixed it for now.
Ok. I have to do that to all patches though. Next rollup.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 08/31] tracing: Replace __get_cpu_var uses
2013-08-29 16:35 ` Christoph Lameter
@ 2013-08-30 7:00 ` Ingo Molnar
0 siblings, 0 replies; 96+ messages in thread
From: Ingo Molnar @ 2013-08-30 7:00 UTC (permalink / raw)
To: Christoph Lameter
Cc: Steven Rostedt, Tejun Heo, akpm, Frederic Weisbecker,
Ingo Molnar, linux-arch, Steven Rostedt, linux-kernel,
Masami Hiramatsu, Peter Zijlstra, Thomas Gleixner, Andrew Morton
* Christoph Lameter <cl@linux.com> wrote:
> On Wed, 28 Aug 2013, Steven Rostedt wrote:
>
> > You should limit the width of the change log to 76 characters.
> >
> > I fixed it for now.
>
> Ok. I have to do that to all patches though. Next rollup.
You'll also need to fix the preempt debug checks before spreading
this_cpu*() uses, as was pointed out to you in the original this_cpu*()
submission and which you failed to follow up to.
Consider all affected patches NAK-ed until this bug is fixed.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
2013-08-28 18:13 ` Chris Metcalf
@ 2013-08-30 16:01 ` Chris Metcalf
-1 siblings, 0 replies; 96+ messages in thread
From: Chris Metcalf @ 2013-08-30 16:01 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel,
Peter Zijlstra, Ingo Molnar, Masami Hiramatsu
On 8/28/2013 2:13 PM, Chris Metcalf wrote:
> On 8/28/2013 1:54 PM, Christoph Lameter wrote:
>> On Wed, 28 Aug 2013, Chris Metcalf wrote:
>>> In addition, I applied the following commit to clean up the remaining
>>> uses that are currently only being carried in the linux-tile tree.
>> You are going to merge these changes right? I can drop the tile patch from
>> my series?
> Yes, I am, so please feel free to drop it.
Given the nacks from Peter Ziljstra and Ingo Molnar, I'm changing my mind and dropping this patch from my tree, so please continue to include arch/tile in your tree as you figure out how you want to restructure the overall series.
I reviewed the one commit in the linux-tile tree that added new __get_cpu_var uses (kprobes support), and concluded the code can safely use __this_cpu_write etc (and in fact the generic reset_current_kprobe method already does so), so I modified the code not to use __get_cpu_var. So there shouldn't be any new uses from the tile tree after the merge window opens.
Their concerns feel particularly relevant to arch/tile since one of the commits queued for the merge window is support for preemption :-)
Thanks!
--
Chris Metcalf, Tilera Corp.
http://www.tilera.com
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 23/31] tile: Replace __get_cpu_var uses
@ 2013-08-30 16:01 ` Chris Metcalf
0 siblings, 0 replies; 96+ messages in thread
From: Chris Metcalf @ 2013-08-30 16:01 UTC (permalink / raw)
To: Christoph Lameter
Cc: Tejun Heo, akpm, linux-arch, Steven Rostedt, linux-kernel,
Peter Zijlstra, Ingo Molnar, Masami Hiramatsu
On 8/28/2013 2:13 PM, Chris Metcalf wrote:
> On 8/28/2013 1:54 PM, Christoph Lameter wrote:
>> On Wed, 28 Aug 2013, Chris Metcalf wrote:
>>> In addition, I applied the following commit to clean up the remaining
>>> uses that are currently only being carried in the linux-tile tree.
>> You are going to merge these changes right? I can drop the tile patch from
>> my series?
> Yes, I am, so please feel free to drop it.
Given the nacks from Peter Ziljstra and Ingo Molnar, I'm changing my mind and dropping this patch from my tree, so please continue to include arch/tile in your tree as you figure out how you want to restructure the overall series.
I reviewed the one commit in the linux-tile tree that added new __get_cpu_var uses (kprobes support), and concluded the code can safely use __this_cpu_write etc (and in fact the generic reset_current_kprobe method already does so), so I modified the code not to use __get_cpu_var. So there shouldn't be any new uses from the tile tree after the merge window opens.
Their concerns feel particularly relevant to arch/tile since one of the commits queued for the merge window is support for preemption :-)
Thanks!
--
Chris Metcalf, Tilera Corp.
http://www.tilera.com
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 04/31] net: Replace __get_cpu_var uses
[not found] <20130826204351.725357339@linux.com>
` (30 preceding siblings ...)
2013-08-26 21:56 ` Christoph Lameter
@ 2013-09-02 21:35 ` David Howells
2013-09-03 3:33 ` David Miller
` (2 more replies)
31 siblings, 3 replies; 96+ messages in thread
From: David Howells @ 2013-09-02 21:35 UTC (permalink / raw)
To: Christoph Lameter
Cc: dhowells, Tejun Heo, akpm, David S. Miller, netdev, linux-arch,
Steven Rostedt, linux-kernel
Would it be possible to use __thread annotations for per-CPU variables, I
wonder?
David
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 04/31] net: Replace __get_cpu_var uses
2013-09-02 21:35 ` [guv v2 04/31] net: Replace __get_cpu_var uses David Howells
@ 2013-09-03 3:33 ` David Miller
2013-09-03 14:21 ` Christoph Lameter
2013-09-03 14:20 ` Christoph Lameter
2013-09-03 20:39 ` David Howells
2 siblings, 1 reply; 96+ messages in thread
From: David Miller @ 2013-09-03 3:33 UTC (permalink / raw)
To: dhowells; +Cc: cl, tj, akpm, netdev, linux-arch, srostedt, linux-kernel
From: David Howells <dhowells@redhat.com>
Date: Mon, 02 Sep 2013 22:35:06 +0100
> Would it be possible to use __thread annotations for per-CPU
> variables, I wonder?
Paul Mackerras tried it on powerpc and you can't do it.
The problem is that there is no way to tell the compiler that sched()
and similar (potentially) change the thread pointer base.
It really will cache pre-computed __thread pointer calculations across
sched().
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 04/31] net: Replace __get_cpu_var uses
2013-09-02 21:35 ` [guv v2 04/31] net: Replace __get_cpu_var uses David Howells
2013-09-03 3:33 ` David Miller
@ 2013-09-03 14:20 ` Christoph Lameter
2013-09-03 20:39 ` David Howells
2 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-09-03 14:20 UTC (permalink / raw)
To: David Howells
Cc: Tejun Heo, akpm, David S. Miller, netdev, linux-arch,
Steven Rostedt, linux-kernel
On Mon, 2 Sep 2013, David Howells wrote:
> Would it be possible to use __thread annotations for per-CPU variables, I
> wonder?
We already have a __percpu annotation. Looked at __thread a couple of
years ago but found that support for the kernel segment register was not
available. If we had __thread then most of the operations could simply be
provided by the compiler.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 04/31] net: Replace __get_cpu_var uses
2013-09-03 3:33 ` David Miller
@ 2013-09-03 14:21 ` Christoph Lameter
0 siblings, 0 replies; 96+ messages in thread
From: Christoph Lameter @ 2013-09-03 14:21 UTC (permalink / raw)
To: David Miller
Cc: dhowells, tj, akpm, netdev, linux-arch, srostedt, linux-kernel
On Mon, 2 Sep 2013, David Miller wrote:
> It really will cache pre-computed __thread pointer calculations across
> sched().
On x86 the compiler could use the segment prefix and then it may work.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [guv v2 04/31] net: Replace __get_cpu_var uses
2013-09-02 21:35 ` [guv v2 04/31] net: Replace __get_cpu_var uses David Howells
2013-09-03 3:33 ` David Miller
2013-09-03 14:20 ` Christoph Lameter
@ 2013-09-03 20:39 ` David Howells
2 siblings, 0 replies; 96+ messages in thread
From: David Howells @ 2013-09-03 20:39 UTC (permalink / raw)
To: David Miller
Cc: dhowells, cl, tj, akpm, netdev, linux-arch, srostedt, linux-kernel
David Miller <davem@davemloft.net> wrote:
> > Would it be possible to use __thread annotations for per-CPU
> > variables, I wonder?
>
> Paul Mackerras tried it on powerpc and you can't do it.
>
> The problem is that there is no way to tell the compiler that sched()
> and similar (potentially) change the thread pointer base.
>
> It really will cache pre-computed __thread pointer calculations across
> sched().
Yeah... I thought there might be something like that - but I could see x86_64
using %fs as a base.
David
^ permalink raw reply [flat|nested] 96+ messages in thread
end of thread, other threads:[~2013-09-03 20:39 UTC | newest]
Thread overview: 96+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20130826204351.725357339@linux.com>
2013-08-26 20:44 ` [guv v2 02/31] percpu: Make __verify_pcu_ptr handle per cpu pointers to arrays Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 03/31] Coccinelle script for __get_cpu_var conversion Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 05/31] time: Replace __get_cpu_var uses Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 04/31] net: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 06/31] scheduler: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 08/31] tracing: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-28 19:48 ` Steven Rostedt
2013-08-29 3:44 ` Masami Hiramatsu
2013-08-29 16:35 ` Christoph Lameter
2013-08-30 7:00 ` Ingo Molnar
2013-08-26 20:44 ` [guv v2 09/31] block: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 11/31] percpu: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 07/31] mm: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 10/31] rcu: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 14/31] drivers: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 13/31] kernel misc: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 12/31] watchdog: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 18/31] s390: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-28 6:27 ` Heiko Carstens
2013-08-26 20:44 ` [guv v2 17/31] mips: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 15/31] zcache/zsmalloc: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 16/31] x86: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 01/31] x86: Use this_cpu_inc/dec for debug registers Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 19/31] ia64: Replace __get_cpu_var uses Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 20/31] powerpc: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-27 8:44 ` Geert Uytterhoeven
2013-08-27 15:37 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 21/31] sparc: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-26 20:44 ` [guv v2 22/31] arm: " Christoph Lameter
2013-08-26 20:44 ` Christoph Lameter
2013-08-27 9:41 ` Catalin Marinas
2013-08-26 20:56 ` [guv v2 25/31] avr32: " Christoph Lameter
2013-08-26 20:56 ` Christoph Lameter
2013-08-27 6:58 ` Hans-Christian Egtvedt
2013-08-26 20:56 ` [guv v2 23/31] tile: " Christoph Lameter
2013-08-26 20:56 ` Christoph Lameter
2013-08-26 22:22 ` Chris Metcalf
2013-08-26 22:22 ` Chris Metcalf
2013-08-27 14:48 ` Christoph Lameter
2013-08-28 17:19 ` Chris Metcalf
2013-08-28 17:19 ` Chris Metcalf
2013-08-28 17:54 ` Christoph Lameter
2013-08-28 18:13 ` Chris Metcalf
2013-08-28 18:13 ` Chris Metcalf
2013-08-28 18:22 ` Christoph Lameter
2013-08-30 16:01 ` Chris Metcalf
2013-08-30 16:01 ` Chris Metcalf
2013-08-26 20:56 ` [guv v2 29/31] parisc: " Christoph Lameter
2013-08-26 20:56 ` Christoph Lameter
2013-08-26 20:56 ` Christoph Lameter
2013-08-26 21:16 ` [guv v2 31/31] Remove __get_cpu_var and __raw_get_cpu_var macros Christoph Lameter
2013-08-26 21:16 ` Christoph Lameter
2013-08-26 21:16 ` [guv v2 24/31] blackfin: Replace __get_cpu_var uses Christoph Lameter
2013-08-26 21:16 ` Christoph Lameter
2013-08-26 21:16 ` [guv v2 26/31] alpha: Replace __get_cpu_var Christoph Lameter
2013-08-26 21:16 ` Christoph Lameter
2013-08-26 21:56 ` [guv v2 30/31] metag: Replace __get_cpu_var uses Christoph Lameter
2013-08-26 21:56 ` Christoph Lameter
2013-08-27 9:53 ` James Hogan
2013-08-27 9:53 ` James Hogan
2013-08-26 21:56 ` [guv v2 28/31] arc: Replace __get_cpu_var uses throughout the kernel Christoph Lameter
2013-08-26 21:56 ` Christoph Lameter
[not found] ` <521C2A9F.3070400@synopsys.com>
[not found] ` <00000140c05d28b7-f0a2bc86-fe85-40f7-b627-32664cd616a4-000000@email.amazonses.com>
2013-08-28 4:24 ` Vineet Gupta
2013-08-28 13:46 ` Christoph Lameter
2013-08-26 21:56 ` [guv v2 27/31] sh: " Christoph Lameter
2013-08-26 21:56 ` Christoph Lameter
2013-08-26 21:56 ` Christoph Lameter
2013-09-02 21:35 ` [guv v2 04/31] net: Replace __get_cpu_var uses David Howells
2013-09-03 3:33 ` David Miller
2013-09-03 14:21 ` Christoph Lameter
2013-09-03 14:20 ` Christoph Lameter
2013-09-03 20:39 ` David Howells
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.