From: Willem de Bruijn <willemdebruijn.kernel@gmail.com>
To: baoyou.xie@gmail.com
Cc: David Miller <davem@davemloft.net>,
Willem de Bruijn <willemb@google.com>,
Al Viro <viro@zeniv.linux.org.uk>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
pombredanne@nexb.com, Tobias Klauser <tklauser@distanz.ch>,
Matthew Dawson <matthew@mjdsystems.ca>,
Network Development <netdev@vger.kernel.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2] datagram: return from __skb_recv_datagram() as soon as possible
Date: Mon, 16 Jul 2018 15:17:54 -0700 [thread overview]
Message-ID: <CAF=yD-+N6BBcbKFo3B=FmVcp9VD+NUr2mfk-qkvtkVZYDhsRHQ@mail.gmail.com> (raw)
In-Reply-To: <1531653191-6456-1-git-send-email-baoyou.xie@gmail.com>
On Sun, Jul 15, 2018 at 4:17 AM Baoyou Xie <baoyou.xie@gmail.com> wrote:
>
> We got a soft lockup in a heavy busy cloud server where RIP is
> at _raw_spin_unlock_irqrestore+0x1b/0x40:
> [] finish_wait+0x56/0x70
> [] __skb_recv_datagram+0x3fb/0x5a0
> [] ? datagram_poll+0x100/0x100
> [] skb_recv_datagram+0x41/0x60
> [] netlink_recvmsg+0x62/0x450
> [] sock_recvmsg+0xbf/0x100
> [] ? futex_wait+0x193/0x280
> [] ? finish_task_switch+0x108/0x170
> [] SYSC_recvfrom+0xe8/0x160
> [] ? __schedule+0x3c8/0x990
> [] SyS_recvfrom+0xe/0x10
> [] system_call_fastpath+0x16/0x1b
>
> In fact, a mistake exists in __skb_recv_datagram(). For example,
> if a datagram come in persistently after go through the socket
> queue, then __skb_wait_for_more_packets() will find out that the
> last peeked skb is not the real last one, so it return 0. this
> results in long time outer loop, and can trigger soft lockup.
Is this with MSG_PEEK?
If the above occurs, that implies that the queue is not empty so the
next iteration of the loop in __skb_recv_datagram should return
the oldest packet on the queue.
This is a netlink socket. Those do not support SO_PEEK_OFF,
simplifying the problem somewhat.
I do not yet see how this can loop until timeout if data is queued.
> So this patch changes the loop condition to prevent soft lockup.
Bounding waiting time in this manner should not be needed, as
__skb_wait_for_more_packets reduces remaining timeo on wake.
next prev parent reply other threads:[~2018-07-16 22:18 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-07-15 11:13 [PATCH v2] datagram: return from __skb_recv_datagram() as soon as possible Baoyou Xie
2018-07-16 22:17 ` Willem de Bruijn [this message]
2018-07-18 5:15 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAF=yD-+N6BBcbKFo3B=FmVcp9VD+NUr2mfk-qkvtkVZYDhsRHQ@mail.gmail.com' \
--to=willemdebruijn.kernel@gmail.com \
--cc=baoyou.xie@gmail.com \
--cc=davem@davemloft.net \
--cc=gregkh@linuxfoundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=matthew@mjdsystems.ca \
--cc=netdev@vger.kernel.org \
--cc=pombredanne@nexb.com \
--cc=tklauser@distanz.ch \
--cc=viro@zeniv.linux.org.uk \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).