From: Heiko Carstens <heiko.carstens@de.ibm.com>
To: "David S. Miller" <davem@davemloft.net>
Cc: shemminger@osdl.org, jgarzik@pobox.com, akpm@osdl.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
fpavlic@de.ibm.com, davem@sunset.davemloft.net
Subject: Re: [patch] ipv4: initialize arp_tbl rw lock
Date: Mon, 24 Apr 2006 12:18:52 +0200 [thread overview]
Message-ID: <20060424101852.GA16007@osiris.boeblingen.de.ibm.com> (raw)
In-Reply-To: <20060415.003457.103031290.davem@davemloft.net>
> > Tried to figure out what is causing the delays I experienced when I replaced
> > module_init() in af_inet.c with fs_initcall(). After all it turned out that
> > synchronize_net() which is basicically nothing else than synchronize_rcu()
> > sometimes takes several seconds to complete?! No idea why that is...
> >
> > callchain: inet_init() -> inet_register_protosw() -> synchronize_net()
>
> The problem can't be rcu_init(), that gets done very early
> in init/main.c
>
> Maybe it's some timer or something else specific to s390?
>
> It could also be that there's perhaps nothing to context
> switch to, thus the RCU takes forever to "happen".
Yes, it's more or less s390 specific.
What happens is the following: synchronize_rcu() enqueues an RCU callback on
cpu 0. Later on cpu 0 handles a bunch of RCU batches, but without handling
this specific request (it's in rdp->curlist). Since this cpu has nothing else
to do it enters cpu_idle() (it's a nohz idle, therefore it might be quite a
long time in idle state).
While cpu 0 is in idle state cpu 2 calls cpu_quiet() which in turn will call
rcu_start_batch(). If cpu 0 would run now, it would notice rdp->curlist moved
to rdp->donelist and that there is something to do. Unfortunately it doesn't
get notified from rcu_start_batch(). That's why I ended up waiting several
seconds until finally some interrupt arrived at cpu 0 which made things go
on finally.
Avoiding this could be done if we look at rdp->curlist before going into
a nohz idle wait, or we could send an interprocessor interrupt to idle
cpus. Sending an interrupt while looking only on nohz_cpu_mask seems to
be a bit racy, since other cpus might have entered cpu idle after
nohz_cpu_mask has been read...
At least the initcall change for inet_init() can go in, since it just
revealed a problem that we have anyway.
next prev parent reply other threads:[~2006-04-24 10:18 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2006-04-07 8:15 [patch] ipv4: initialize arp_tbl rw lock Heiko Carstens
2006-04-07 14:46 ` Stephen Hemminger
2006-04-08 10:02 ` Heiko Carstens
2006-04-08 10:12 ` Andrew Morton
2006-04-19 10:45 ` Christian Borntraeger
2006-04-19 20:12 ` David S. Miller
2006-04-20 13:11 ` Heiko Carstens
2006-04-08 10:14 ` David S. Miller
2006-04-08 10:42 ` Heiko Carstens
2006-04-08 12:14 ` Sam Ravnborg
2006-04-15 7:27 ` Heiko Carstens
2006-04-15 7:34 ` David S. Miller
2006-04-15 23:00 ` Heiko Carstens
2006-04-24 10:18 ` Heiko Carstens [this message]
2006-04-24 10:22 ` [patch] ipv4: inet_init() -> fs_initcall Heiko Carstens
2006-04-07 20:14 ` [patch] ipv4: initialize arp_tbl rw lock David S. Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20060424101852.GA16007@osiris.boeblingen.de.ibm.com \
--to=heiko.carstens@de.ibm.com \
--cc=akpm@osdl.org \
--cc=davem@davemloft.net \
--cc=davem@sunset.davemloft.net \
--cc=fpavlic@de.ibm.com \
--cc=jgarzik@pobox.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=shemminger@osdl.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).