* [PATCH net-next] ipv4: speedup ip_idents_reserve()
@ 2015-05-01 17:37 Eric Dumazet
2015-05-01 23:44 ` David Miller
0 siblings, 1 reply; 2+ messages in thread
From: Eric Dumazet @ 2015-05-01 17:37 UTC (permalink / raw)
To: David Miller; +Cc: netdev
From: Eric Dumazet <edumazet@google.com>
Under stress, ip_idents_reserve() is accessing a contended
cache line twice, with non optimal MESI transactions.
If we place timestamps in separate location, we reduce this
pressure by ~50% and allow atomic_add_return() to issue
a Request for Ownership.
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
net/ipv4/route.c | 20 +++++++++++---------
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index a78540f28276..f7559cba4343 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -457,12 +457,9 @@ static struct neighbour *ipv4_neigh_lookup(const struct dst_entry *dst,
}
#define IP_IDENTS_SZ 2048u
-struct ip_ident_bucket {
- atomic_t id;
- u32 stamp32;
-};
-static struct ip_ident_bucket *ip_idents __read_mostly;
+static atomic_t *ip_idents __read_mostly;
+static u32 *ip_tstamps __read_mostly;
/* In order to protect privacy, we add a perturbation to identifiers
* if one generator is seldom used. This makes hard for an attacker
@@ -470,15 +467,16 @@ static struct ip_ident_bucket *ip_idents __read_mostly;
*/
u32 ip_idents_reserve(u32 hash, int segs)
{
- struct ip_ident_bucket *bucket = ip_idents + hash % IP_IDENTS_SZ;
- u32 old = ACCESS_ONCE(bucket->stamp32);
+ u32 *p_tstamp = ip_tstamps + hash % IP_IDENTS_SZ;
+ atomic_t *p_id = ip_idents + hash % IP_IDENTS_SZ;
+ u32 old = ACCESS_ONCE(*p_tstamp);
u32 now = (u32)jiffies;
u32 delta = 0;
- if (old != now && cmpxchg(&bucket->stamp32, old, now) == old)
+ if (old != now && cmpxchg(p_tstamp, old, now) == old)
delta = prandom_u32_max(now - old);
- return atomic_add_return(segs + delta, &bucket->id) - segs;
+ return atomic_add_return(segs + delta, p_id) - segs;
}
EXPORT_SYMBOL(ip_idents_reserve);
@@ -2741,6 +2739,10 @@ int __init ip_rt_init(void)
prandom_bytes(ip_idents, IP_IDENTS_SZ * sizeof(*ip_idents));
+ ip_tstamps = kcalloc(IP_IDENTS_SZ, sizeof(*ip_tstamps), GFP_KERNEL);
+ if (!ip_tstamps)
+ panic("IP: failed to allocate ip_tstamps\n");
+
for_each_possible_cpu(cpu) {
struct uncached_list *ul = &per_cpu(rt_uncached_list, cpu);
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH net-next] ipv4: speedup ip_idents_reserve()
2015-05-01 17:37 [PATCH net-next] ipv4: speedup ip_idents_reserve() Eric Dumazet
@ 2015-05-01 23:44 ` David Miller
0 siblings, 0 replies; 2+ messages in thread
From: David Miller @ 2015-05-01 23:44 UTC (permalink / raw)
To: eric.dumazet; +Cc: netdev
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Fri, 01 May 2015 10:37:49 -0700
> From: Eric Dumazet <edumazet@google.com>
>
> Under stress, ip_idents_reserve() is accessing a contended
> cache line twice, with non optimal MESI transactions.
>
> If we place timestamps in separate location, we reduce this
> pressure by ~50% and allow atomic_add_return() to issue
> a Request for Ownership.
>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
Looks good, applied, thanks Eric.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2015-05-01 23:44 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-05-01 17:37 [PATCH net-next] ipv4: speedup ip_idents_reserve() Eric Dumazet
2015-05-01 23:44 ` David Miller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).