* [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon @ 2014-11-19 10:50 Dexuan Cui 2014-11-19 10:49 ` Vitaly Kuznetsov 0 siblings, 1 reply; 8+ messages in thread From: Dexuan Cui @ 2014-11-19 10:50 UTC (permalink / raw) To: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang; +Cc: kys, haiyangz Under high memory pressure and very high KVP R/W test pressure, the netlink recvfrom() may transiently return ENOBUFS to the daemon -- we found this during a 2-week stress test. We'd better not terminate the daemon on this failure, because a typical KVP user can re-try the R/W and hopefully it will succeed next time. Cc: K. Y. Srinivasan <kys@microsoft.com> Signed-off-by: Dexuan Cui <decui@microsoft.com> --- tools/hv/hv_kvp_daemon.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c index 22b0764..9f4b303 100644 --- a/tools/hv/hv_kvp_daemon.c +++ b/tools/hv/hv_kvp_daemon.c @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[]) addr_p, &addr_l); if (len < 0) { + int saved_errno = errno; syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s", addr.nl_pid, errno, strerror(errno)); + + if (saved_errno == ENOBUFS) { + syslog(LOG_ERR, "error = ENOBUFS: ignored"); + continue; + } + close(fd); return -1; } -- 1.9.1 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon 2014-11-19 10:50 [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon Dexuan Cui @ 2014-11-19 10:49 ` Vitaly Kuznetsov 2014-11-19 12:15 ` Dexuan Cui 0 siblings, 1 reply; 8+ messages in thread From: Vitaly Kuznetsov @ 2014-11-19 10:49 UTC (permalink / raw) To: Dexuan Cui Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang, haiyangz Dexuan Cui <decui@microsoft.com> writes: > Under high memory pressure and very high KVP R/W test pressure, the netlink > recvfrom() may transiently return ENOBUFS to the daemon -- we found this > during a 2-week stress test. > > We'd better not terminate the daemon on this failure, because a typical KVP > user can re-try the R/W and hopefully it will succeed next time. > > Cc: K. Y. Srinivasan <kys@microsoft.com> > Signed-off-by: Dexuan Cui <decui@microsoft.com> > --- > tools/hv/hv_kvp_daemon.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c > index 22b0764..9f4b303 100644 > --- a/tools/hv/hv_kvp_daemon.c > +++ b/tools/hv/hv_kvp_daemon.c > @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[]) > addr_p, &addr_l); > > if (len < 0) { > + int saved_errno = errno; > syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s", > addr.nl_pid, errno, strerror(errno)); > + > + if (saved_errno == ENOBUFS) { is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd suggest we ignore these as well in such case. Ignoring ENOMEM here is doubtful, I think. But possible. > + syslog(LOG_ERR, "error = ENOBUFS: ignored"); > + continue; > + } > + > close(fd); > return -1; > } -- Vitaly ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon 2014-11-19 10:49 ` Vitaly Kuznetsov @ 2014-11-19 12:15 ` Dexuan Cui 2014-11-19 12:40 ` Vitaly Kuznetsov 0 siblings, 1 reply; 8+ messages in thread From: Dexuan Cui @ 2014-11-19 12:15 UTC (permalink / raw) To: Vitaly Kuznetsov Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang, Haiyang Zhang > -----Original Message----- > From: Vitaly Kuznetsov > Sent: Wednesday, November 19, 2014 18:50 PM > To: Dexuan Cui > Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; driverdev- > devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com; > jasowang@redhat.com; Haiyang Zhang > Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon > > Dexuan Cui writes: > > > Under high memory pressure and very high KVP R/W test pressure, the > netlink > > recvfrom() may transiently return ENOBUFS to the daemon -- we found > this > > during a 2-week stress test. > > > > We'd better not terminate the daemon on this failure, because a typical > KVP > > user can re-try the R/W and hopefully it will succeed next time. > > > > diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c > > index 22b0764..9f4b303 100644 > > --- a/tools/hv/hv_kvp_daemon.c > > +++ b/tools/hv/hv_kvp_daemon.c > > @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[]) > > addr_p, &addr_l); > > > > if (len < 0) { > > + int saved_errno = errno; > > syslog(LOG_ERR, "recvfrom failed; pid:%u > error:%d %s", > > addr.nl_pid, errno, strerror(errno)); > > + > > + if (saved_errno == ENOBUFS) { > > is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd suggest > we ignore these as well in such case. Ignoring ENOMEM here is doubtful, > I think. But possible. > > Vitaly I don't think EAGAIN is possible because "man recvfrom" says "If no messages are available at the socket, the receive calls wait for a message to arrive, unless the socket is nonblocking (see fcntl(2)), in which case the value -1 is returned and the external variable errno is set to EAGAIN or EWOULDBLOCK". The same man page mention ENOMEM for recvmsg(), but not recvfrom(). -- Dexuan ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon 2014-11-19 12:15 ` Dexuan Cui @ 2014-11-19 12:40 ` Vitaly Kuznetsov 2014-11-19 12:57 ` Dexuan Cui 0 siblings, 1 reply; 8+ messages in thread From: Vitaly Kuznetsov @ 2014-11-19 12:40 UTC (permalink / raw) To: Dexuan Cui Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang, Haiyang Zhang Dexuan Cui <decui@microsoft.com> writes: >> -----Original Message----- >> From: Vitaly Kuznetsov >> Sent: Wednesday, November 19, 2014 18:50 PM >> To: Dexuan Cui >> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; driverdev- >> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com; >> jasowang@redhat.com; Haiyang Zhang >> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon >> >> Dexuan Cui writes: >> >> > Under high memory pressure and very high KVP R/W test pressure, the netlink >> > recvfrom() may transiently return ENOBUFS to the daemon -- we found this >> > during a 2-week stress test. >> > >> > We'd better not terminate the daemon on this failure, because a typical KVP >> > user can re-try the R/W and hopefully it will succeed next time. >> > >> > diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c >> > index 22b0764..9f4b303 100644 >> > --- a/tools/hv/hv_kvp_daemon.c >> > +++ b/tools/hv/hv_kvp_daemon.c >> > @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[]) >> > addr_p, &addr_l); >> > >> > if (len < 0) { >> > + int saved_errno = errno; >> > syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s", >> > addr.nl_pid, errno, strerror(errno)); >> > + >> > + if (saved_errno == ENOBUFS) { >> >> is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd suggest >> we ignore these as well in such case. Ignoring ENOMEM here is doubtful, >> I think. But possible. >> >> Vitaly > > I don't think EAGAIN is possible because "man recvfrom" says > "If no messages are available at the socket, the receive calls wait for a > message to arrive, unless the socket is nonblocking (see fcntl(2)), in which > case the value -1 is returned and the external variable errno is set to > EAGAIN or EWOULDBLOCK". > > The same man page mention ENOMEM for recvmsg(), but not recvfrom(). Ah, sorry, I though your patch patches the other place: call to netlink_send() which does sendmsg() (and my EAGAIN/EWOULDBLOCK/ENOMEM comment was about it). It could also make sense to patch them both as I think it is possible to hit these as well. > > -- Dexuan -- Vitaly ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon 2014-11-19 12:40 ` Vitaly Kuznetsov @ 2014-11-19 12:57 ` Dexuan Cui 2014-11-19 13:05 ` Vitaly Kuznetsov 0 siblings, 1 reply; 8+ messages in thread From: Dexuan Cui @ 2014-11-19 12:57 UTC (permalink / raw) To: Vitaly Kuznetsov Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang, Haiyang Zhang > -----Original Message----- > From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com] > Sent: Wednesday, November 19, 2014 20:41 PM > To: Dexuan Cui > Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; driverdev- > devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com; > jasowang@redhat.com; Haiyang Zhang > Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon > > Dexuan Cui <decui@microsoft.com> writes: > > >> -----Original Message----- > >> From: Vitaly Kuznetsov > >> Sent: Wednesday, November 19, 2014 18:50 PM > >> To: Dexuan Cui > >> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; > driverdev- > >> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com; > >> jasowang@redhat.com; Haiyang Zhang > >> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon > >> > >> Dexuan Cui writes: > >> > >> > Under high memory pressure and very high KVP R/W test pressure, > the netlink > >> > recvfrom() may transiently return ENOBUFS to the daemon -- we found > this > >> > during a 2-week stress test. > >> > > >> > We'd better not terminate the daemon on this failure, because a > typical KVP > >> > user can re-try the R/W and hopefully it will succeed next time. > >> > > >> > diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c > >> > index 22b0764..9f4b303 100644 > >> > --- a/tools/hv/hv_kvp_daemon.c > >> > +++ b/tools/hv/hv_kvp_daemon.c > >> > @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[]) > >> > addr_p, &addr_l); > >> > > >> > if (len < 0) { > >> > + int saved_errno = errno; > >> > syslog(LOG_ERR, "recvfrom failed; pid:%u > error:%d %s", > >> > addr.nl_pid, errno, strerror(errno)); > >> > + > >> > + if (saved_errno == ENOBUFS) { > >> > >> is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd > suggest > >> we ignore these as well in such case. Ignoring ENOMEM here is doubtful, > >> I think. But possible. > >> > >> Vitaly > > > > I don't think EAGAIN is possible because "man recvfrom" says > > "If no messages are available at the socket, the receive calls wait for a > > message to arrive, unless the socket is nonblocking (see fcntl(2)), in > which > > case the value -1 is returned and the external variable errno is set to > > EAGAIN or EWOULDBLOCK". > > > > The same man page mention ENOMEM for recvmsg(), but not recvfrom(). > > Ah, sorry, I though your patch patches the other place: call to > netlink_send() which does sendmsg() (and my > EAGAIN/EWOULDBLOCK/ENOMEM > comment was about it). It could also make sense to patch them both as I > think it is possible to hit these as well. > > > -- Dexuan > -- > Vitaly OK, I can add this new check: (I'll send out the v2 tomorrow in case people have new comments) --- a/tools/hv/hv_kvp_daemon.c +++ b/tools/hv/hv_kvp_daemon.c @@ -1770,8 +1770,15 @@ kvp_done: len = netlink_send(fd, incoming_cn_msg); if (len < 0) { + int saved_errno = errno; syslog(LOG_ERR, "net_link send failed; error: %d %s", errno, strerror(errno)); + + if (saved_errno == ENOMEM || saved_errno == EAGAIN) { + syslog(LOG_ERR, "send error: ignored"); + continue; + } + exit(EXIT_FAILURE); } } Thanks, -- Dexuan ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon 2014-11-19 12:57 ` Dexuan Cui @ 2014-11-19 13:05 ` Vitaly Kuznetsov 2014-11-19 13:13 ` Dexuan Cui 0 siblings, 1 reply; 8+ messages in thread From: Vitaly Kuznetsov @ 2014-11-19 13:05 UTC (permalink / raw) To: Dexuan Cui Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang, Haiyang Zhang Dexuan Cui <decui@microsoft.com> writes: >> -----Original Message----- >> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com] >> Sent: Wednesday, November 19, 2014 20:41 PM >> To: Dexuan Cui >> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; driverdev- >> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com; >> jasowang@redhat.com; Haiyang Zhang >> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon >> >> Dexuan Cui <decui@microsoft.com> writes: >> >> >> -----Original Message----- >> >> From: Vitaly Kuznetsov >> >> Sent: Wednesday, November 19, 2014 18:50 PM >> >> To: Dexuan Cui >> >> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; >> driverdev- >> >> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com; >> >> jasowang@redhat.com; Haiyang Zhang >> >> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon >> >> >> >> Dexuan Cui writes: >> >> >> >> > Under high memory pressure and very high KVP R/W test pressure, >> the netlink >> >> > recvfrom() may transiently return ENOBUFS to the daemon -- we found >> this >> >> > during a 2-week stress test. >> >> > >> >> > We'd better not terminate the daemon on this failure, because a >> typical KVP >> >> > user can re-try the R/W and hopefully it will succeed next time. >> >> > >> >> > diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c >> >> > index 22b0764..9f4b303 100644 >> >> > --- a/tools/hv/hv_kvp_daemon.c >> >> > +++ b/tools/hv/hv_kvp_daemon.c >> >> > @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[]) >> >> > addr_p, &addr_l); >> >> > >> >> > if (len < 0) { >> >> > + int saved_errno = errno; >> >> > syslog(LOG_ERR, "recvfrom failed; pid:%u >> error:%d %s", >> >> > addr.nl_pid, errno, strerror(errno)); >> >> > + >> >> > + if (saved_errno == ENOBUFS) { >> >> >> >> is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd >> suggest >> >> we ignore these as well in such case. Ignoring ENOMEM here is doubtful, >> >> I think. But possible. >> >> >> >> Vitaly >> > >> > I don't think EAGAIN is possible because "man recvfrom" says >> > "If no messages are available at the socket, the receive calls wait for a >> > message to arrive, unless the socket is nonblocking (see fcntl(2)), in >> which >> > case the value -1 is returned and the external variable errno is set to >> > EAGAIN or EWOULDBLOCK". >> > >> > The same man page mention ENOMEM for recvmsg(), but not recvfrom(). >> >> Ah, sorry, I though your patch patches the other place: call to >> netlink_send() which does sendmsg() (and my >> EAGAIN/EWOULDBLOCK/ENOMEM >> comment was about it). It could also make sense to patch them both as I >> think it is possible to hit these as well. >> >> > -- Dexuan >> -- >> Vitaly > > OK, I can add this new check: > (I'll send out the v2 tomorrow in case people have new comments) > Thanks! > --- a/tools/hv/hv_kvp_daemon.c > +++ b/tools/hv/hv_kvp_daemon.c > @@ -1770,8 +1770,15 @@ kvp_done: > > len = netlink_send(fd, incoming_cn_msg); > if (len < 0) { > + int saved_errno = errno; > syslog(LOG_ERR, "net_link send failed; error: %d %s", errno, > strerror(errno)); > + > + if (saved_errno == ENOMEM || saved_errno == EAGAIN) { Sorry for being pushy, but it seems ENOBUFS is also possible here (at least man sendmsg mentions it). > + syslog(LOG_ERR, "send error: ignored"); > + continue; > + } > + > exit(EXIT_FAILURE); > } > } > > Thanks, > -- Dexuan -- Vitaly ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon 2014-11-19 13:05 ` Vitaly Kuznetsov @ 2014-11-19 13:13 ` Dexuan Cui 2014-11-19 14:39 ` Vitaly Kuznetsov 0 siblings, 1 reply; 8+ messages in thread From: Dexuan Cui @ 2014-11-19 13:13 UTC (permalink / raw) To: Vitaly Kuznetsov Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang, Haiyang Zhang > -----Original Message----- > From: Vitaly Kuznetsov > >> -- > >> Vitaly > > > > OK, I can add this new check: > > (I'll send out the v2 tomorrow in case people have new comments) > > > > Thanks! > > > --- a/tools/hv/hv_kvp_daemon.c > > +++ b/tools/hv/hv_kvp_daemon.c > > @@ -1770,8 +1770,15 @@ kvp_done: > > > > len = netlink_send(fd, incoming_cn_msg); > > if (len < 0) { > > + int saved_errno = errno; > > syslog(LOG_ERR, "net_link send failed; error: %d %s", errno, > > strerror(errno)); > > + > > + if (saved_errno == ENOMEM || saved_errno == EAGAIN) { > > Sorry for being pushy, but it seems ENOBUFS is also possible here (at > least man sendmsg mentions it). OK, I'll add this too. :-) BTW, I realized sendmsg() can't return EAGAIN here as that's for non-blocking socket. Here I simply ignore the error, hoping the other end will re-try. > > > + syslog(LOG_ERR, "send error: ignored"); > > + continue; > > + } > > + > > exit(EXIT_FAILURE); > > } > > } > > > > Thanks, > > -- Dexuan > > Vitaly -- Dexuan ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon 2014-11-19 13:13 ` Dexuan Cui @ 2014-11-19 14:39 ` Vitaly Kuznetsov 0 siblings, 0 replies; 8+ messages in thread From: Vitaly Kuznetsov @ 2014-11-19 14:39 UTC (permalink / raw) To: Dexuan Cui Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang, Haiyang Zhang Dexuan Cui <decui@microsoft.com> writes: >> -----Original Message----- >> From: Vitaly Kuznetsov >> >> -- >> >> Vitaly >> > >> > OK, I can add this new check: >> > (I'll send out the v2 tomorrow in case people have new comments) >> > >> >> Thanks! >> >> > --- a/tools/hv/hv_kvp_daemon.c >> > +++ b/tools/hv/hv_kvp_daemon.c >> > @@ -1770,8 +1770,15 @@ kvp_done: >> > >> > len = netlink_send(fd, incoming_cn_msg); >> > if (len < 0) { >> > + int saved_errno = errno; >> > syslog(LOG_ERR, "net_link send failed; error: %d %s", errno, >> > strerror(errno)); >> > + >> > + if (saved_errno == ENOMEM || saved_errno == EAGAIN) { >> >> Sorry for being pushy, but it seems ENOBUFS is also possible here (at >> least man sendmsg mentions it). > OK, I'll add this too. :-) > > BTW, I realized sendmsg() can't return EAGAIN here as that's for non-blocking > socket. > > Here I simply ignore the error, hoping the other end will re-try. > I agree, it's sufficient to ignore ENOBUFS on recieve path and both ENOMEM/ENOBUFS on send. Thanks! >> >> > + syslog(LOG_ERR, "send error: ignored"); >> > + continue; >> > + } >> > + >> > exit(EXIT_FAILURE); >> > } >> > } >> > >> > Thanks, >> > -- Dexuan >> >> Vitaly > > -- Dexuan -- Vitaly ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-11-19 14:40 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-11-19 10:50 [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon Dexuan Cui 2014-11-19 10:49 ` Vitaly Kuznetsov 2014-11-19 12:15 ` Dexuan Cui 2014-11-19 12:40 ` Vitaly Kuznetsov 2014-11-19 12:57 ` Dexuan Cui 2014-11-19 13:05 ` Vitaly Kuznetsov 2014-11-19 13:13 ` Dexuan Cui 2014-11-19 14:39 ` Vitaly Kuznetsov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).