* [PATCH net-next-2.6] rps: Various optimizations
@ 2010-05-06 8:58 Eric Dumazet
2010-05-07 5:07 ` David Miller
0 siblings, 1 reply; 7+ messages in thread
From: Eric Dumazet @ 2010-05-06 8:58 UTC (permalink / raw)
To: David Miller; +Cc: netdev, Tom Herbert
Introduce ____napi_schedule() helper for callers in irq disabled
contexts. rps_trigger_softirq() becomes a leaf function.
Use container_of() in process_backlog() instead of accessing per_cpu
address.
Use a custom inlined version of __napi_complete() in process_backlog()
to avoid one locked instruction :
only current cpu owns and manipulates this napi,
and NAPI_STATE_SCHED is the only possible flag set on backlog.
we can use a plain write instead of clear_bit(),
and we dont need an smp_mb() memory barrier, since RPS is on,
backlog is protected by a spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
net/core/dev.c | 28 ++++++++++++++++++++++------
1 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/net/core/dev.c b/net/core/dev.c
index 36d53be..c6861e4 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2205,6 +2205,14 @@ int netdev_max_backlog __read_mostly = 1000;
int netdev_budget __read_mostly = 300;
int weight_p __read_mostly = 64; /* old backlog weight */
+/* Called with irq disabled */
+static inline void ____napi_schedule(struct softnet_data *sd,
+ struct napi_struct *napi)
+{
+ list_add_tail(&napi->poll_list, &sd->poll_list);
+ __raise_softirq_irqoff(NET_RX_SOFTIRQ);
+}
+
#ifdef CONFIG_RPS
/* One global table that all flow-based protocols share. */
@@ -2363,7 +2371,7 @@ static void rps_trigger_softirq(void *data)
{
struct softnet_data *sd = data;
- __napi_schedule(&sd->backlog);
+ ____napi_schedule(sd, &sd->backlog);
sd->received_rps++;
}
@@ -2421,7 +2429,7 @@ enqueue:
/* Schedule NAPI for backlog device */
if (napi_schedule_prep(&sd->backlog)) {
if (!rps_ipi_queued(sd))
- __napi_schedule(&sd->backlog);
+ ____napi_schedule(sd, &sd->backlog);
}
goto enqueue;
}
@@ -3280,7 +3288,7 @@ static void net_rps_action_and_irq_enable(struct softnet_data *sd)
static int process_backlog(struct napi_struct *napi, int quota)
{
int work = 0;
- struct softnet_data *sd = &__get_cpu_var(softnet_data);
+ struct softnet_data *sd = container_of(napi, struct softnet_data, backlog);
#ifdef CONFIG_RPS
/* Check if we have pending ipi, its better to send them now,
@@ -3313,7 +3321,16 @@ static int process_backlog(struct napi_struct *napi, int quota)
&sd->process_queue);
}
if (qlen < quota - work) {
- __napi_complete(napi);
+ /*
+ * Inline a custom version of __napi_complete().
+ * only current cpu owns and manipulates this napi,
+ * and NAPI_STATE_SCHED is the only possible flag set on backlog.
+ * we can use a plain write instead of clear_bit(),
+ * and we dont need an smp_mb() memory barrier.
+ */
+ list_del(&napi->poll_list);
+ napi->state = 0;
+
quota = work + qlen;
}
rps_unlock(sd);
@@ -3334,8 +3351,7 @@ void __napi_schedule(struct napi_struct *n)
unsigned long flags;
local_irq_save(flags);
- list_add_tail(&n->poll_list, &__get_cpu_var(softnet_data).poll_list);
- __raise_softirq_irqoff(NET_RX_SOFTIRQ);
+ ____napi_schedule(&__get_cpu_var(softnet_data), n);
local_irq_restore(flags);
}
EXPORT_SYMBOL(__napi_schedule);
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH net-next-2.6] rps: Various optimizations
2010-05-06 8:58 [PATCH net-next-2.6] rps: Various optimizations Eric Dumazet
@ 2010-05-07 5:07 ` David Miller
2010-05-07 5:16 ` Eric Dumazet
0 siblings, 1 reply; 7+ messages in thread
From: David Miller @ 2010-05-07 5:07 UTC (permalink / raw)
To: eric.dumazet; +Cc: netdev, therbert
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 06 May 2010 10:58:42 +0200
> Introduce ____napi_schedule() helper for callers in irq disabled
> contexts. rps_trigger_softirq() becomes a leaf function.
>
> Use container_of() in process_backlog() instead of accessing per_cpu
> address.
>
> Use a custom inlined version of __napi_complete() in process_backlog()
> to avoid one locked instruction :
>
> only current cpu owns and manipulates this napi,
> and NAPI_STATE_SCHED is the only possible flag set on backlog.
> we can use a plain write instead of clear_bit(),
> and we dont need an smp_mb() memory barrier, since RPS is on,
> backlog is protected by a spinlock.
>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Looks great, applied, thanks Eric.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH net-next-2.6] rps: Various optimizations
2010-05-07 5:07 ` David Miller
@ 2010-05-07 5:16 ` Eric Dumazet
2010-05-07 9:51 ` [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog Eric Dumazet
0 siblings, 1 reply; 7+ messages in thread
From: Eric Dumazet @ 2010-05-07 5:16 UTC (permalink / raw)
To: David Miller; +Cc: netdev, therbert
Le jeudi 06 mai 2010 à 22:07 -0700, David Miller a écrit :
> Looks great, applied, thanks Eric.
Thanks, I have a followup to avoid one atomic in enqueue phase too ;)
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog
2010-05-07 5:16 ` Eric Dumazet
@ 2010-05-07 9:51 ` Eric Dumazet
2010-05-07 10:01 ` Changli Gao
2010-05-18 0:22 ` David Miller
0 siblings, 2 replies; 7+ messages in thread
From: Eric Dumazet @ 2010-05-07 9:51 UTC (permalink / raw)
To: David Miller; +Cc: netdev, therbert
Le vendredi 07 mai 2010 à 07:16 +0200, Eric Dumazet a écrit :
> Le jeudi 06 mai 2010 à 22:07 -0700, David Miller a écrit :
>
> > Looks great, applied, thanks Eric.
>
> Thanks, I have a followup to avoid one atomic in enqueue phase too ;)
>
[PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog
If CONFIG_SMP=y, then we own a queue spinlock, we can avoid the atomic
test_and_set_bit() from napi_schedule_prep().
We now have same number of atomic ops per netif_rx() calls than with
pre-RPS kernel.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
diff --git a/net/core/dev.c b/net/core/dev.c
index 32611c8..49fa5a6 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2426,8 +2426,10 @@ enqueue:
return NET_RX_SUCCESS;
}
- /* Schedule NAPI for backlog device */
- if (napi_schedule_prep(&sd->backlog)) {
+ /* Schedule NAPI for backlog device
+ * We can use non atomic operation since we own the queue lock
+ */
+ if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) {
if (!rps_ipi_queued(sd))
____napi_schedule(sd, &sd->backlog);
}
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog
2010-05-07 9:51 ` [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog Eric Dumazet
@ 2010-05-07 10:01 ` Changli Gao
2010-05-07 10:05 ` Eric Dumazet
2010-05-18 0:22 ` David Miller
1 sibling, 1 reply; 7+ messages in thread
From: Changli Gao @ 2010-05-07 10:01 UTC (permalink / raw)
To: Eric Dumazet; +Cc: David Miller, netdev, therbert
On Fri, May 7, 2010 at 5:51 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le vendredi 07 mai 2010 à 07:16 +0200, Eric Dumazet a écrit :
>> Le jeudi 06 mai 2010 à 22:07 -0700, David Miller a écrit :
>>
>> > Looks great, applied, thanks Eric.
>>
>> Thanks, I have a followup to avoid one atomic in enqueue phase too ;)
>>
>
> [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog
>
> If CONFIG_SMP=y, then we own a queue spinlock, we can avoid the atomic
> test_and_set_bit() from napi_schedule_prep().
>
> We now have same number of atomic ops per netif_rx() calls than with
> pre-RPS kernel.
>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> ---
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 32611c8..49fa5a6 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -2426,8 +2426,10 @@ enqueue:
> return NET_RX_SUCCESS;
> }
>
> - /* Schedule NAPI for backlog device */
> - if (napi_schedule_prep(&sd->backlog)) {
> + /* Schedule NAPI for backlog device
> + * We can use non atomic operation since we own the queue lock
> + */
> + if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) {
> if (!rps_ipi_queued(sd))
> ____napi_schedule(sd, &sd->backlog);
> }
>
Why not use a wrapper function?
sth. like:
static inline int __napi_schedule_prep(struct napi_struct *n)
{
return (!__test_and_set_bit(NAPI_STATE_SCHED, &n->state)
}
--
Regards,
Changli Gao(xiaosuo@gmail.com)
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog
2010-05-07 10:01 ` Changli Gao
@ 2010-05-07 10:05 ` Eric Dumazet
0 siblings, 0 replies; 7+ messages in thread
From: Eric Dumazet @ 2010-05-07 10:05 UTC (permalink / raw)
To: Changli Gao; +Cc: David Miller, netdev, therbert
Le vendredi 07 mai 2010 à 18:01 +0800, Changli Gao a écrit :
> On Fri, May 7, 2010 at 5:51 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > Le vendredi 07 mai 2010 à 07:16 +0200, Eric Dumazet a écrit :
> >> Le jeudi 06 mai 2010 à 22:07 -0700, David Miller a écrit :
> >>
> >> > Looks great, applied, thanks Eric.
> >>
> >> Thanks, I have a followup to avoid one atomic in enqueue phase too ;)
> >>
> >
> > [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog
> >
> > If CONFIG_SMP=y, then we own a queue spinlock, we can avoid the atomic
> > test_and_set_bit() from napi_schedule_prep().
> >
> > We now have same number of atomic ops per netif_rx() calls than with
> > pre-RPS kernel.
> >
> > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> > ---
> > diff --git a/net/core/dev.c b/net/core/dev.c
> > index 32611c8..49fa5a6 100644
> > --- a/net/core/dev.c
> > +++ b/net/core/dev.c
> > @@ -2426,8 +2426,10 @@ enqueue:
> > return NET_RX_SUCCESS;
> > }
> >
> > - /* Schedule NAPI for backlog device */
> > - if (napi_schedule_prep(&sd->backlog)) {
> > + /* Schedule NAPI for backlog device
> > + * We can use non atomic operation since we own the queue lock
> > + */
> > + if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) {
> > if (!rps_ipi_queued(sd))
> > ____napi_schedule(sd, &sd->backlog);
> > }
> >
>
> Why not use a wrapper function?
>
> sth. like:
>
> static inline int __napi_schedule_prep(struct napi_struct *n)
> {
> return (!__test_and_set_bit(NAPI_STATE_SCHED, &n->state)
> }
>
For one user ?
Not sure it helps code readability.
Right now we all have our minds knowing every bit of this code, but next
year ?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog
2010-05-07 9:51 ` [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog Eric Dumazet
2010-05-07 10:01 ` Changli Gao
@ 2010-05-18 0:22 ` David Miller
1 sibling, 0 replies; 7+ messages in thread
From: David Miller @ 2010-05-18 0:22 UTC (permalink / raw)
To: eric.dumazet; +Cc: netdev, therbert
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Fri, 07 May 2010 11:51:21 +0200
> Le vendredi 07 mai 2010 à 07:16 +0200, Eric Dumazet a écrit :
>> Le jeudi 06 mai 2010 à 22:07 -0700, David Miller a écrit :
>>
>> > Looks great, applied, thanks Eric.
>>
>> Thanks, I have a followup to avoid one atomic in enqueue phase too ;)
>>
>
> [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog
>
> If CONFIG_SMP=y, then we own a queue spinlock, we can avoid the atomic
> test_and_set_bit() from napi_schedule_prep().
>
> We now have same number of atomic ops per netif_rx() calls than with
> pre-RPS kernel.
>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Also applied, thanks Eric.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2010-05-18 0:22 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-05-06 8:58 [PATCH net-next-2.6] rps: Various optimizations Eric Dumazet
2010-05-07 5:07 ` David Miller
2010-05-07 5:16 ` Eric Dumazet
2010-05-07 9:51 ` [PATCH net-next-2.6] rps: avoid one atomic in enqueue_to_backlog Eric Dumazet
2010-05-07 10:01 ` Changli Gao
2010-05-07 10:05 ` Eric Dumazet
2010-05-18 0:22 ` David Miller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).