linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
  2014-11-19 10:50 [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon Dexuan Cui
@ 2014-11-19 10:49 ` Vitaly Kuznetsov
  2014-11-19 12:15   ` Dexuan Cui
  0 siblings, 1 reply; 8+ messages in thread
From: Vitaly Kuznetsov @ 2014-11-19 10:49 UTC (permalink / raw)
  To: Dexuan Cui
  Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang, haiyangz

Dexuan Cui <decui@microsoft.com> writes:

> Under high memory pressure and very high KVP R/W test pressure, the netlink
> recvfrom() may transiently return ENOBUFS to the daemon -- we found this
> during a 2-week stress test.
>
> We'd better not terminate the daemon on this failure, because a typical KVP
> user can re-try the R/W and hopefully it will succeed next time.
>
> Cc: K. Y. Srinivasan <kys@microsoft.com>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> ---
>  tools/hv/hv_kvp_daemon.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
> index 22b0764..9f4b303 100644
> --- a/tools/hv/hv_kvp_daemon.c
> +++ b/tools/hv/hv_kvp_daemon.c
> @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
>  				addr_p, &addr_l);
>
>  		if (len < 0) {
> +			int saved_errno = errno;
>  			syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s",
>  					addr.nl_pid, errno, strerror(errno));
> +
> +			if (saved_errno == ENOBUFS) {

is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd suggest
we ignore these as well in such case. Ignoring ENOMEM here is doubtful,
I think. But possible.

> +				syslog(LOG_ERR, "error = ENOBUFS: ignored");
> +				continue;
> +			}
> +
>  			close(fd);
>  			return -1;
>  		}

-- 
  Vitaly

^ permalink raw reply	[flat|nested] 8+ messages in thread
* [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
@ 2014-11-19 10:50 Dexuan Cui
  2014-11-19 10:49 ` Vitaly Kuznetsov
  0 siblings, 1 reply; 8+ messages in thread
From: Dexuan Cui @ 2014-11-19 10:50 UTC (permalink / raw)
  To: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang; +Cc: kys, haiyangz

Under high memory pressure and very high KVP R/W test pressure, the netlink
recvfrom() may transiently return ENOBUFS to the daemon -- we found this
during a 2-week stress test.

We'd better not terminate the daemon on this failure, because a typical KVP
user can re-try the R/W and hopefully it will succeed next time.

Cc: K. Y. Srinivasan <kys@microsoft.com>
Signed-off-by: Dexuan Cui <decui@microsoft.com>
---
 tools/hv/hv_kvp_daemon.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
index 22b0764..9f4b303 100644
--- a/tools/hv/hv_kvp_daemon.c
+++ b/tools/hv/hv_kvp_daemon.c
@@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
 				addr_p, &addr_l);
 
 		if (len < 0) {
+			int saved_errno = errno;
 			syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s",
 					addr.nl_pid, errno, strerror(errno));
+
+			if (saved_errno == ENOBUFS) {
+				syslog(LOG_ERR, "error = ENOBUFS: ignored");
+				continue;
+			}
+
 			close(fd);
 			return -1;
 		}
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread
* RE: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
  2014-11-19 10:49 ` Vitaly Kuznetsov
@ 2014-11-19 12:15   ` Dexuan Cui
  2014-11-19 12:40     ` Vitaly Kuznetsov
  0 siblings, 1 reply; 8+ messages in thread
From: Dexuan Cui @ 2014-11-19 12:15 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang,
	Haiyang Zhang

> -----Original Message-----
> From: Vitaly Kuznetsov
> Sent: Wednesday, November 19, 2014 18:50 PM
> To: Dexuan Cui
> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; driverdev-
> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> jasowang@redhat.com; Haiyang Zhang
> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
> 
> Dexuan Cui  writes:
> 
> > Under high memory pressure and very high KVP R/W test pressure, the
> netlink
> > recvfrom() may transiently return ENOBUFS to the daemon -- we found
> this
> > during a 2-week stress test.
> >
> > We'd better not terminate the daemon on this failure, because a typical
> KVP
> > user can re-try the R/W and hopefully it will succeed next time.
> >
> > diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
> > index 22b0764..9f4b303 100644
> > --- a/tools/hv/hv_kvp_daemon.c
> > +++ b/tools/hv/hv_kvp_daemon.c
> > @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
> >  				addr_p, &addr_l);
> >
> >  		if (len < 0) {
> > +			int saved_errno = errno;
> >  			syslog(LOG_ERR, "recvfrom failed; pid:%u
> error:%d %s",
> >  					addr.nl_pid, errno, strerror(errno));
> > +
> > +			if (saved_errno == ENOBUFS) {
> 
> is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd suggest
> we ignore these as well in such case. Ignoring ENOMEM here is doubtful,
> I think. But possible.
> 
>   Vitaly

I don't think EAGAIN is possible  because "man recvfrom" says
   "If  no messages are available at the socket, the receive calls wait for a
     message to arrive, unless the socket is nonblocking (see fcntl(2)), in which
     case the value -1 is returned and  the  external variable  errno is set to
    EAGAIN or EWOULDBLOCK".

The same man page mention ENOMEM for recvmsg(), but not recvfrom().

-- Dexuan

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
  2014-11-19 12:15   ` Dexuan Cui
@ 2014-11-19 12:40     ` Vitaly Kuznetsov
  2014-11-19 12:57       ` Dexuan Cui
  0 siblings, 1 reply; 8+ messages in thread
From: Vitaly Kuznetsov @ 2014-11-19 12:40 UTC (permalink / raw)
  To: Dexuan Cui
  Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang,
	Haiyang Zhang

Dexuan Cui <decui@microsoft.com> writes:

>> -----Original Message-----
>> From: Vitaly Kuznetsov
>> Sent: Wednesday, November 19, 2014 18:50 PM
>> To: Dexuan Cui
>> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; driverdev-
>> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
>> jasowang@redhat.com; Haiyang Zhang
>> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
>> 
>> Dexuan Cui  writes:
>> 
>> > Under high memory pressure and very high KVP R/W test pressure, the netlink
>> > recvfrom() may transiently return ENOBUFS to the daemon -- we found this
>> > during a 2-week stress test.
>> >
>> > We'd better not terminate the daemon on this failure, because a typical KVP
>> > user can re-try the R/W and hopefully it will succeed next time.
>> >
>> > diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
>> > index 22b0764..9f4b303 100644
>> > --- a/tools/hv/hv_kvp_daemon.c
>> > +++ b/tools/hv/hv_kvp_daemon.c
>> > @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
>> >  				addr_p, &addr_l);
>> >
>> >  		if (len < 0) {
>> > +			int saved_errno = errno;
>> >  			syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s",
>> >  					addr.nl_pid, errno, strerror(errno));
>> > +
>> > +			if (saved_errno == ENOBUFS) {
>> 
>> is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd suggest
>> we ignore these as well in such case. Ignoring ENOMEM here is doubtful,
>> I think. But possible.
>> 
>>   Vitaly
>
> I don't think EAGAIN is possible  because "man recvfrom" says
>    "If  no messages are available at the socket, the receive calls wait for a
>      message to arrive, unless the socket is nonblocking (see fcntl(2)), in which
>      case the value -1 is returned and  the  external variable  errno is set to
>     EAGAIN or EWOULDBLOCK".
>
> The same man page mention ENOMEM for recvmsg(), but not recvfrom().

Ah, sorry, I though your patch patches the other place: call to
netlink_send() which does sendmsg() (and my EAGAIN/EWOULDBLOCK/ENOMEM
comment was about it). It could also make sense to patch them both as I
think it is possible to hit these as well.

>
> -- Dexuan

-- 
  Vitaly

^ permalink raw reply	[flat|nested] 8+ messages in thread
* RE: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
  2014-11-19 12:40     ` Vitaly Kuznetsov
@ 2014-11-19 12:57       ` Dexuan Cui
  2014-11-19 13:05         ` Vitaly Kuznetsov
  0 siblings, 1 reply; 8+ messages in thread
From: Dexuan Cui @ 2014-11-19 12:57 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang,
	Haiyang Zhang

> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Wednesday, November 19, 2014 20:41 PM
> To: Dexuan Cui
> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; driverdev-
> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> jasowang@redhat.com; Haiyang Zhang
> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
> 
> Dexuan Cui <decui@microsoft.com> writes:
> 
> >> -----Original Message-----
> >> From: Vitaly Kuznetsov
> >> Sent: Wednesday, November 19, 2014 18:50 PM
> >> To: Dexuan Cui
> >> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> driverdev-
> >> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> >> jasowang@redhat.com; Haiyang Zhang
> >> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
> >>
> >> Dexuan Cui  writes:
> >>
> >> > Under high memory pressure and very high KVP R/W test pressure,
> the netlink
> >> > recvfrom() may transiently return ENOBUFS to the daemon -- we found
> this
> >> > during a 2-week stress test.
> >> >
> >> > We'd better not terminate the daemon on this failure, because a
> typical KVP
> >> > user can re-try the R/W and hopefully it will succeed next time.
> >> >
> >> > diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
> >> > index 22b0764..9f4b303 100644
> >> > --- a/tools/hv/hv_kvp_daemon.c
> >> > +++ b/tools/hv/hv_kvp_daemon.c
> >> > @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
> >> >  				addr_p, &addr_l);
> >> >
> >> >  		if (len < 0) {
> >> > +			int saved_errno = errno;
> >> >  			syslog(LOG_ERR, "recvfrom failed; pid:%u
> error:%d %s",
> >> >  					addr.nl_pid, errno, strerror(errno));
> >> > +
> >> > +			if (saved_errno == ENOBUFS) {
> >>
> >> is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd
> suggest
> >> we ignore these as well in such case. Ignoring ENOMEM here is doubtful,
> >> I think. But possible.
> >>
> >>   Vitaly
> >
> > I don't think EAGAIN is possible  because "man recvfrom" says
> >    "If  no messages are available at the socket, the receive calls wait for a
> >      message to arrive, unless the socket is nonblocking (see fcntl(2)), in
> which
> >      case the value -1 is returned and  the  external variable  errno is set to
> >     EAGAIN or EWOULDBLOCK".
> >
> > The same man page mention ENOMEM for recvmsg(), but not recvfrom().
> 
> Ah, sorry, I though your patch patches the other place: call to
> netlink_send() which does sendmsg() (and my
> EAGAIN/EWOULDBLOCK/ENOMEM
> comment was about it). It could also make sense to patch them both as I
> think it is possible to hit these as well.
> 
> > -- Dexuan
> --
>   Vitaly

OK, I can add this new check:
(I'll send out the v2 tomorrow in case  people have new comments)

--- a/tools/hv/hv_kvp_daemon.c
+++ b/tools/hv/hv_kvp_daemon.c
@@ -1770,8 +1770,15 @@ kvp_done:

                len = netlink_send(fd, incoming_cn_msg);
                if (len < 0) {
+                       int saved_errno = errno;
                        syslog(LOG_ERR, "net_link send failed; error: %d %s", errno,
                                        strerror(errno));
+
+                       if (saved_errno == ENOMEM || saved_errno == EAGAIN) {
+                               syslog(LOG_ERR, "send error: ignored");
+                               continue;
+                       }
+
                        exit(EXIT_FAILURE);
                }
        }

Thanks,
-- Dexuan

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
  2014-11-19 12:57       ` Dexuan Cui
@ 2014-11-19 13:05         ` Vitaly Kuznetsov
  2014-11-19 13:13           ` Dexuan Cui
  0 siblings, 1 reply; 8+ messages in thread
From: Vitaly Kuznetsov @ 2014-11-19 13:05 UTC (permalink / raw)
  To: Dexuan Cui
  Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang,
	Haiyang Zhang

Dexuan Cui <decui@microsoft.com> writes:

>> -----Original Message-----
>> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
>> Sent: Wednesday, November 19, 2014 20:41 PM
>> To: Dexuan Cui
>> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org; driverdev-
>> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
>> jasowang@redhat.com; Haiyang Zhang
>> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
>> 
>> Dexuan Cui <decui@microsoft.com> writes:
>> 
>> >> -----Original Message-----
>> >> From: Vitaly Kuznetsov
>> >> Sent: Wednesday, November 19, 2014 18:50 PM
>> >> To: Dexuan Cui
>> >> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
>> driverdev-
>> >> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
>> >> jasowang@redhat.com; Haiyang Zhang
>> >> Subject: Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
>> >>
>> >> Dexuan Cui  writes:
>> >>
>> >> > Under high memory pressure and very high KVP R/W test pressure,
>> the netlink
>> >> > recvfrom() may transiently return ENOBUFS to the daemon -- we found
>> this
>> >> > during a 2-week stress test.
>> >> >
>> >> > We'd better not terminate the daemon on this failure, because a
>> typical KVP
>> >> > user can re-try the R/W and hopefully it will succeed next time.
>> >> >
>> >> > diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
>> >> > index 22b0764..9f4b303 100644
>> >> > --- a/tools/hv/hv_kvp_daemon.c
>> >> > +++ b/tools/hv/hv_kvp_daemon.c
>> >> > @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
>> >> >  				addr_p, &addr_l);
>> >> >
>> >> >  		if (len < 0) {
>> >> > +			int saved_errno = errno;
>> >> >  			syslog(LOG_ERR, "recvfrom failed; pid:%u
>> error:%d %s",
>> >> >  					addr.nl_pid, errno, strerror(errno));
>> >> > +
>> >> > +			if (saved_errno == ENOBUFS) {
>> >>
>> >> is it possible to meet EAGAIN (or EWOULDBLOCK) here as well? I'd
>> suggest
>> >> we ignore these as well in such case. Ignoring ENOMEM here is doubtful,
>> >> I think. But possible.
>> >>
>> >>   Vitaly
>> >
>> > I don't think EAGAIN is possible  because "man recvfrom" says
>> >    "If  no messages are available at the socket, the receive calls wait for a
>> >      message to arrive, unless the socket is nonblocking (see fcntl(2)), in
>> which
>> >      case the value -1 is returned and  the  external variable  errno is set to
>> >     EAGAIN or EWOULDBLOCK".
>> >
>> > The same man page mention ENOMEM for recvmsg(), but not recvfrom().
>> 
>> Ah, sorry, I though your patch patches the other place: call to
>> netlink_send() which does sendmsg() (and my
>> EAGAIN/EWOULDBLOCK/ENOMEM
>> comment was about it). It could also make sense to patch them both as I
>> think it is possible to hit these as well.
>> 
>> > -- Dexuan
>> --
>>   Vitaly
>
> OK, I can add this new check:
> (I'll send out the v2 tomorrow in case  people have new comments)
>

Thanks!

> --- a/tools/hv/hv_kvp_daemon.c
> +++ b/tools/hv/hv_kvp_daemon.c
> @@ -1770,8 +1770,15 @@ kvp_done:
>
>                 len = netlink_send(fd, incoming_cn_msg);
>                 if (len < 0) {
> +                       int saved_errno = errno;
>                         syslog(LOG_ERR, "net_link send failed; error: %d %s", errno,
>                                         strerror(errno));
> +
> +                       if (saved_errno == ENOMEM || saved_errno ==  EAGAIN) {

Sorry for being pushy, but it seems ENOBUFS is also possible here (at
least man sendmsg mentions it).

> +                               syslog(LOG_ERR, "send error: ignored");
> +                               continue;
> +                       }
> +
>                         exit(EXIT_FAILURE);
>                 }
>         }
>
> Thanks,
> -- Dexuan

-- 
  Vitaly

^ permalink raw reply	[flat|nested] 8+ messages in thread
* RE: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
  2014-11-19 13:05         ` Vitaly Kuznetsov
@ 2014-11-19 13:13           ` Dexuan Cui
  2014-11-19 14:39             ` Vitaly Kuznetsov
  0 siblings, 1 reply; 8+ messages in thread
From: Dexuan Cui @ 2014-11-19 13:13 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang,
	Haiyang Zhang

> -----Original Message-----
> From: Vitaly Kuznetsov 
> >> --
> >>   Vitaly
> >
> > OK, I can add this new check:
> > (I'll send out the v2 tomorrow in case  people have new comments)
> >
> 
> Thanks!
> 
> > --- a/tools/hv/hv_kvp_daemon.c
> > +++ b/tools/hv/hv_kvp_daemon.c
> > @@ -1770,8 +1770,15 @@ kvp_done:
> >
> >                 len = netlink_send(fd, incoming_cn_msg);
> >                 if (len < 0) {
> > +                       int saved_errno = errno;
> >                         syslog(LOG_ERR, "net_link send failed; error: %d %s", errno,
> >                                         strerror(errno));
> > +
> > +                       if (saved_errno == ENOMEM || saved_errno ==  EAGAIN) {
> 
> Sorry for being pushy, but it seems ENOBUFS is also possible here (at
> least man sendmsg mentions it).
OK, I'll add this too. :-)

BTW, I realized sendmsg() can't return EAGAIN here as that's for non-blocking
socket.

Here I simply ignore the error, hoping the other end will re-try.

> 
> > +                               syslog(LOG_ERR, "send error: ignored");
> > +                               continue;
> > +                       }
> > +
> >                         exit(EXIT_FAILURE);
> >                 }
> >         }
> >
> > Thanks,
> > -- Dexuan
> 
>   Vitaly

-- Dexuan

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon
  2014-11-19 13:13           ` Dexuan Cui
@ 2014-11-19 14:39             ` Vitaly Kuznetsov
  0 siblings, 0 replies; 8+ messages in thread
From: Vitaly Kuznetsov @ 2014-11-19 14:39 UTC (permalink / raw)
  To: Dexuan Cui
  Cc: gregkh, linux-kernel, driverdev-devel, olaf, apw, jasowang,
	Haiyang Zhang

Dexuan Cui <decui@microsoft.com> writes:

>> -----Original Message-----
>> From: Vitaly Kuznetsov 
>> >> --
>> >>   Vitaly
>> >
>> > OK, I can add this new check:
>> > (I'll send out the v2 tomorrow in case  people have new comments)
>> >
>> 
>> Thanks!
>> 
>> > --- a/tools/hv/hv_kvp_daemon.c
>> > +++ b/tools/hv/hv_kvp_daemon.c
>> > @@ -1770,8 +1770,15 @@ kvp_done:
>> >
>> >                 len = netlink_send(fd, incoming_cn_msg);
>> >                 if (len < 0) {
>> > +                       int saved_errno = errno;
>> >                         syslog(LOG_ERR, "net_link send failed; error: %d %s", errno,
>> >                                         strerror(errno));
>> > +
>> > +                       if (saved_errno == ENOMEM || saved_errno ==  EAGAIN) {
>> 
>> Sorry for being pushy, but it seems ENOBUFS is also possible here (at
>> least man sendmsg mentions it).
> OK, I'll add this too. :-)
>
> BTW, I realized sendmsg() can't return EAGAIN here as that's for non-blocking
> socket.
>
> Here I simply ignore the error, hoping the other end will re-try.
>

I agree, it's sufficient to ignore ENOBUFS on recieve path and both
ENOMEM/ENOBUFS on send.

Thanks!

>> 
>> > +                               syslog(LOG_ERR, "send error: ignored");
>> > +                               continue;
>> > +                       }
>> > +
>> >                         exit(EXIT_FAILURE);
>> >                 }
>> >         }
>> >
>> > Thanks,
>> > -- Dexuan
>> 
>>   Vitaly
>
> -- Dexuan

-- 
  Vitaly

^ permalink raw reply	[flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-11-19 14:40 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-11-19 10:50 [PATCH] tools: hv: ignore ENOBUFS in the KVP daemon Dexuan Cui
2014-11-19 10:49 ` Vitaly Kuznetsov
2014-11-19 12:15   ` Dexuan Cui
2014-11-19 12:40     ` Vitaly Kuznetsov
2014-11-19 12:57       ` Dexuan Cui
2014-11-19 13:05         ` Vitaly Kuznetsov
2014-11-19 13:13           ` Dexuan Cui
2014-11-19 14:39             ` Vitaly Kuznetsov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).