* [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06
@ 2017-11-06 15:33 Daniel P. Berrange
2017-11-06 15:33 ` [Qemu-devel] [PULL v1 1/2] sockets: avoid leak of listen file descriptor Daniel P. Berrange
` (2 more replies)
0 siblings, 3 replies; 12+ messages in thread
From: Daniel P. Berrange @ 2017-11-06 15:33 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell, Daniel P. Berrange
The following changes since commit ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c:
Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2017-11-06 10:04:16 +0000)
are available in the git repository at:
git://github.com/berrange/qemu tags/pull-qio-2017-11-06-1
for you to fetch changes up to 74c0035408d5b327eac5985d976bca3009e0c5d6:
tests: Add test-listen - a stress test for QEMU socket listen (2017-11-06 11:49:15 +0000)
----------------------------------------------------------------
Merge qio 2017-11-06 v1
----------------------------------------------------------------
Daniel P. Berrange (1):
sockets: avoid leak of listen file descriptor
Knut Omang (1):
tests: Add test-listen - a stress test for QEMU socket listen
tests/Makefile.include | 2 +
tests/test-listen.c | 253 +++++++++++++++++++++++++++++++++++++++++++++++++
util/qemu-sockets.c | 52 +++++-----
3 files changed, 284 insertions(+), 23 deletions(-)
create mode 100644 tests/test-listen.c
--
2.13.6
^ permalink raw reply [flat|nested] 12+ messages in thread* [Qemu-devel] [PULL v1 1/2] sockets: avoid leak of listen file descriptor 2017-11-06 15:33 [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 Daniel P. Berrange @ 2017-11-06 15:33 ` Daniel P. Berrange 2017-11-06 15:33 ` [Qemu-devel] [PULL v1 2/2] tests: Add test-listen - a stress test for QEMU socket listen Daniel P. Berrange 2017-11-06 16:11 ` [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 Peter Maydell 2 siblings, 0 replies; 12+ messages in thread From: Daniel P. Berrange @ 2017-11-06 15:33 UTC (permalink / raw) To: qemu-devel; +Cc: Peter Maydell, Daniel P. Berrange If we iterate over the full port range without successfully binding+listening on the socket, we'll try the next address, whereupon we overwrite the slisten file descriptor variable without closing it. Rather than having two places where we open + close socket FDs on different iterations of nested for loops, re-arrange the code to always open+close within the same loop iteration. Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Daniel P. Berrange <berrange@redhat.com> --- util/qemu-sockets.c | 52 +++++++++++++++++++++++++++++----------------------- 1 file changed, 29 insertions(+), 23 deletions(-) diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c index b47fb45885..8b75541ce4 100644 --- a/util/qemu-sockets.c +++ b/util/qemu-sockets.c @@ -207,7 +207,7 @@ static int inet_listen_saddr(InetSocketAddress *saddr, char uaddr[INET6_ADDRSTRLEN+1]; char uport[33]; int rc, port_min, port_max, p; - int slisten = 0; + int slisten = -1; int saved_errno = 0; bool socket_created = false; Error *err = NULL; @@ -267,31 +267,42 @@ static int inet_listen_saddr(InetSocketAddress *saddr, uaddr,INET6_ADDRSTRLEN,uport,32, NI_NUMERICHOST | NI_NUMERICSERV); - slisten = create_fast_reuse_socket(e); - if (slisten < 0) { - continue; - } - - socket_created = true; port_min = inet_getport(e); port_max = saddr->has_to ? saddr->to + port_offset : port_min; for (p = port_min; p <= port_max; p++) { inet_setport(e, p); - rc = try_bind(slisten, saddr, e); - if (rc) { - if (errno == EADDRINUSE) { + + slisten = create_fast_reuse_socket(e); + if (slisten < 0) { + /* First time we expect we might fail to create the socket + * eg if 'e' has AF_INET6 but ipv6 kmod is not loaded. + * Later iterations should always succeed if first iteration + * worked though, so treat that as fatal. + */ + if (p == port_min) { continue; } else { - error_setg_errno(errp, errno, "Failed to bind socket"); + error_setg_errno(errp, errno, + "Failed to recreate failed listening socket"); goto listen_failed; } } - if (!listen(slisten, 1)) { - goto listen_ok; - } - if (errno != EADDRINUSE) { - error_setg_errno(errp, errno, "Failed to listen on socket"); - goto listen_failed; + socket_created = true; + + rc = try_bind(slisten, saddr, e); + if (rc < 0) { + if (errno != EADDRINUSE) { + error_setg_errno(errp, errno, "Failed to bind socket"); + goto listen_failed; + } + } else { + if (!listen(slisten, 1)) { + goto listen_ok; + } + if (errno != EADDRINUSE) { + error_setg_errno(errp, errno, "Failed to listen on socket"); + goto listen_failed; + } } /* Someone else managed to bind to the same port and beat us * to listen on it! Socket semantics does not allow us to @@ -299,12 +310,7 @@ static int inet_listen_saddr(InetSocketAddress *saddr, * socket to allow bind attempts for subsequent ports: */ closesocket(slisten); - slisten = create_fast_reuse_socket(e); - if (slisten < 0) { - error_setg_errno(errp, errno, - "Failed to recreate failed listening socket"); - goto listen_failed; - } + slisten = -1; } } error_setg_errno(errp, errno, -- 2.13.6 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Qemu-devel] [PULL v1 2/2] tests: Add test-listen - a stress test for QEMU socket listen 2017-11-06 15:33 [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 Daniel P. Berrange 2017-11-06 15:33 ` [Qemu-devel] [PULL v1 1/2] sockets: avoid leak of listen file descriptor Daniel P. Berrange @ 2017-11-06 15:33 ` Daniel P. Berrange 2018-05-01 14:00 ` Knut Omang 2017-11-06 16:11 ` [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 Peter Maydell 2 siblings, 1 reply; 12+ messages in thread From: Daniel P. Berrange @ 2017-11-06 15:33 UTC (permalink / raw) To: qemu-devel; +Cc: Peter Maydell, Knut Omang, Daniel P . Berrange From: Knut Omang <knut.omang@oracle.com> There's a potential race condition between multiple bind()'s attempting to bind to the same port, which occasionally allows more than one bind to succeed against the same port. When a subsequent listen() call is made with the same socket only one will succeed. The current QEMU code does however not take this situation into account and the listen will cause the code to break out and fail even when there are actually available ports to use. This test exposes two subtests: /socket/listen-serial /socket/listen-compete The "compete" subtest creates a number of threads and have them all trying to bind to the same port with a large enough offset input to allow all threads to get it's own port. The "serial" subtest just does the same, except in series in a single thread. The serial version passes, probably in most versions of QEMU. The parallel version exposes the problem in a relatively reliable way, eg. it fails a majority of times, but not with a 100% rate, occasional passes can be seen. Nevertheless this is quite good given that the bug was tricky to reproduce and has been left undetected for a while. The problem seems to be present in all versions of QEMU. The original failure scenario occurred with VNC port allocation in a traditional Xen based build, in different code but with similar functionality. Reported-by: Bhavesh Davda <bhavesh.davda@oracle.com> Signed-off-by: Knut Omang <knut.omang@oracle.com> Reviewed-by: Yuval Shaia <yuval.shaia@oracle.com> Reviewed-by: Bhavesh Davda <bhavesh.davda@oracle.com> Reviewed-by: Girish Moodalbail <girish.moodalbail@oracle.com> Signed-off-by: Daniel P. Berrange <berrange@redhat.com> --- tests/Makefile.include | 2 + tests/test-listen.c | 253 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 255 insertions(+) create mode 100644 tests/test-listen.c diff --git a/tests/Makefile.include b/tests/Makefile.include index 434a2ce868..e4bb88bd3d 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -154,6 +154,7 @@ gcov-files-check-bufferiszero-y = util/bufferiszero.c check-unit-y += tests/test-uuid$(EXESUF) check-unit-y += tests/ptimer-test$(EXESUF) gcov-files-ptimer-test-y = hw/core/ptimer.c +check-unit-y += tests/test-listen$(EXESUF) check-unit-y += tests/test-qapi-util$(EXESUF) gcov-files-test-qapi-util-y = qapi/qapi-util.c @@ -804,6 +805,7 @@ tests/test-arm-mptimer$(EXESUF): tests/test-arm-mptimer.o tests/test-qapi-util$(EXESUF): tests/test-qapi-util.o $(test-util-obj-y) tests/numa-test$(EXESUF): tests/numa-test.o tests/vmgenid-test$(EXESUF): tests/vmgenid-test.o tests/boot-sector.o tests/acpi-utils.o +tests/test-listen$(EXESUF): tests/test-listen.o $(test-util-obj-y) tests/migration/stress$(EXESUF): tests/migration/stress.o $(call quiet-command, $(LINKPROG) -static -O3 $(PTHREAD_LIB) -o $@ $< ,"LINK","$(TARGET_DIR)$@") diff --git a/tests/test-listen.c b/tests/test-listen.c new file mode 100644 index 0000000000..03c4c8f03b --- /dev/null +++ b/tests/test-listen.c @@ -0,0 +1,253 @@ +/* + * Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. + * Author: Knut Omang <knut.omang@oracle.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 or later + * as published by the Free Software Foundation. + * + * Test parallel port listen configuration with + * dynamic port allocation + */ + +#include "qemu/osdep.h" +#include "libqtest.h" +#include "qemu-common.h" +#include "qemu/thread.h" +#include "qemu/sockets.h" +#include "qapi/error.h" + +#define NAME_LEN 1024 +#define PORT_LEN 16 + +struct thr_info { + QemuThread thread; + int to_port; + bool ipv4; + bool ipv6; + int got_port; + int eno; + int fd; + const char *errstr; + char hostname[NAME_LEN + 1]; + char port[PORT_LEN + 1]; +}; + + +/* These two functions taken from test-io-channel-socket.c */ +static int check_bind(const char *hostname, bool *has_proto) +{ + int fd = -1; + struct addrinfo ai, *res = NULL; + int rc; + int ret = -1; + + memset(&ai, 0, sizeof(ai)); + ai.ai_flags = AI_CANONNAME | AI_ADDRCONFIG; + ai.ai_family = AF_UNSPEC; + ai.ai_socktype = SOCK_STREAM; + + /* lookup */ + rc = getaddrinfo(hostname, NULL, &ai, &res); + if (rc != 0) { + if (rc == EAI_ADDRFAMILY || + rc == EAI_FAMILY) { + *has_proto = false; + goto done; + } + goto cleanup; + } + + fd = qemu_socket(res->ai_family, res->ai_socktype, res->ai_protocol); + if (fd < 0) { + goto cleanup; + } + + if (bind(fd, res->ai_addr, res->ai_addrlen) < 0) { + if (errno == EADDRNOTAVAIL) { + *has_proto = false; + goto done; + } + goto cleanup; + } + + *has_proto = true; + done: + ret = 0; + + cleanup: + if (fd != -1) { + close(fd); + } + if (res) { + freeaddrinfo(res); + } + return ret; +} + +static int check_protocol_support(bool *has_ipv4, bool *has_ipv6) +{ + if (check_bind("127.0.0.1", has_ipv4) < 0) { + return -1; + } + if (check_bind("::1", has_ipv6) < 0) { + return -1; + } + + return 0; +} + +static void *listener_thread(void *arg) +{ + struct thr_info *thr = (struct thr_info *)arg; + SocketAddress addr = { + .type = SOCKET_ADDRESS_TYPE_INET, + .u = { + .inet = { + .host = thr->hostname, + .port = thr->port, + .has_ipv4 = thr->ipv4, + .ipv4 = thr->ipv4, + .has_ipv6 = thr->ipv6, + .ipv6 = thr->ipv6, + .has_to = true, + .to = thr->to_port, + }, + }, + }; + Error *err = NULL; + int fd; + + fd = socket_listen(&addr, &err); + if (fd < 0) { + thr->eno = errno; + thr->errstr = error_get_pretty(err); + } else { + struct sockaddr_in a; + socklen_t a_len = sizeof(a); + g_assert_cmpint(getsockname(fd, (struct sockaddr *)&a, &a_len), ==, 0); + thr->got_port = ntohs(a.sin_port); + thr->fd = fd; + } + return arg; +} + + +static void listen_compete_nthr(bool threaded, int nthreads, + int start_port, int max_offset, + bool ipv4, bool ipv6) +{ + int i; + int failed_listens = 0; + struct thr_info *thr = g_new0(struct thr_info, nthreads); + int used[max_offset + 1]; + + memset(used, 0, sizeof(used)); + for (i = 0; i < nthreads; i++) { + snprintf(thr[i].port, PORT_LEN, "%d", start_port); + strcpy(thr[i].hostname, "localhost"); + thr[i].to_port = start_port + max_offset; + thr[i].ipv4 = ipv4; + thr[i].ipv6 = ipv6; + } + + for (i = 0; i < nthreads; i++) { + if (threaded) { + qemu_thread_create(&thr[i].thread, "listener", + listener_thread, &thr[i], + QEMU_THREAD_JOINABLE); + } else { + listener_thread(&thr[i]); + } + } + + if (threaded) { + for (i = 0; i < nthreads; i++) { + qemu_thread_join(&thr[i].thread); + } + } + for (i = 0; i < nthreads; i++) { + if (thr[i].got_port) { + closesocket(thr[i].fd); + } + } + + for (i = 0; i < nthreads; i++) { + if (thr[i].eno != 0) { + const char *m; + g_printerr("** Failed to assign a port to thread %d (errno = %d)\n", + i, thr[i].eno); + /* This is what we are interested in capturing - + * catch and report details if something unexpected happens: + */ + m = strstr(thr[i].errstr, "Failed to listen on socket"); + if (m != NULL) { + g_assert_cmpstr(thr[i].errstr, ==, + "Failed to listen on socket: Address already in use"); + } + failed_listens++; + } else { + int assigned_port = thr[i].got_port; + g_assert_cmpint(assigned_port, <= , thr[i].to_port); + g_assert_cmpint(used[assigned_port - start_port], == , 0); + } + } + g_assert_cmpint(failed_listens, ==, 0); + g_free(thr); +} + + +static void listen_compete_ipv4(void) +{ + listen_compete_nthr(true, 200, 5920, 300, true, false); +} + +static void listen_serial_ipv4(void) +{ + listen_compete_nthr(false, 200, 6300, 300, true, false); +} + +static void listen_compete_ipv6(void) +{ + listen_compete_nthr(true, 200, 5920, 300, true, false); +} + +static void listen_serial_ipv6(void) +{ + listen_compete_nthr(false, 200, 6300, 300, false, true); +} + +static void listen_compete_gen(void) +{ + listen_compete_nthr(true, 200, 5920, 300, true, true); +} + +static void listen_serial_gen(void) +{ + listen_compete_nthr(false, 200, 6300, 300, true, true); +} + + +int main(int argc, char **argv) +{ + bool has_ipv4, has_ipv6; + g_test_init(&argc, &argv, NULL); + + if (check_protocol_support(&has_ipv4, &has_ipv6) < 0) { + return 1; + } + + if (has_ipv4) { + g_test_add_func("/socket/listen-serial/ipv4", listen_serial_ipv4); + g_test_add_func("/socket/listen-compete/ipv4", listen_compete_ipv4); + } + if (has_ipv6) { + g_test_add_func("/socket/listen-serial/ipv6", listen_serial_ipv6); + g_test_add_func("/socket/listen-compete/ipv6", listen_compete_ipv6); + } + if (has_ipv4 && has_ipv6) { + g_test_add_func("/socket/listen-serial/generic", listen_serial_gen); + g_test_add_func("/socket/listen-compete/generic", listen_compete_gen); + } + return g_test_run(); +} -- 2.13.6 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PULL v1 2/2] tests: Add test-listen - a stress test for QEMU socket listen 2017-11-06 15:33 ` [Qemu-devel] [PULL v1 2/2] tests: Add test-listen - a stress test for QEMU socket listen Daniel P. Berrange @ 2018-05-01 14:00 ` Knut Omang 2018-05-01 14:07 ` Daniel P. Berrangé 0 siblings, 1 reply; 12+ messages in thread From: Knut Omang @ 2018-05-01 14:00 UTC (permalink / raw) To: Daniel P. Berrange, qemu-devel; +Cc: Peter Maydell Hi Peter, Seems this test was lost along the way? I thought it got merged with Daniel's fix to the potential leak that the test exposed in your build, but it seems not? Thanks, Knut On Mon, 2017-11-06 at 15:33 +0000, Daniel P. Berrange wrote: > From: Knut Omang <knut.omang@oracle.com> > > There's a potential race condition between multiple bind()'s > attempting to bind to the same port, which occasionally > allows more than one bind to succeed against the same port. > > When a subsequent listen() call is made with the same socket > only one will succeed. > > The current QEMU code does however not take this situation into account > and the listen will cause the code to break out and fail even > when there are actually available ports to use. > > This test exposes two subtests: > > /socket/listen-serial > /socket/listen-compete > > The "compete" subtest creates a number of threads and have them all trying to bind > to the same port with a large enough offset input to > allow all threads to get it's own port. > The "serial" subtest just does the same, except in series in a > single thread. > > The serial version passes, probably in most versions of QEMU. > > The parallel version exposes the problem in a relatively reliable way, > eg. it fails a majority of times, but not with a 100% rate, occasional > passes can be seen. Nevertheless this is quite good given that > the bug was tricky to reproduce and has been left undetected for > a while. > > The problem seems to be present in all versions of QEMU. > > The original failure scenario occurred with VNC port allocation > in a traditional Xen based build, in different code > but with similar functionality. > > Reported-by: Bhavesh Davda <bhavesh.davda@oracle.com> > Signed-off-by: Knut Omang <knut.omang@oracle.com> > Reviewed-by: Yuval Shaia <yuval.shaia@oracle.com> > Reviewed-by: Bhavesh Davda <bhavesh.davda@oracle.com> > Reviewed-by: Girish Moodalbail <girish.moodalbail@oracle.com> > Signed-off-by: Daniel P. Berrange <berrange@redhat.com> > --- > tests/Makefile.include | 2 + > tests/test-listen.c | 253 +++++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 255 insertions(+) > create mode 100644 tests/test-listen.c > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 434a2ce868..e4bb88bd3d 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -154,6 +154,7 @@ gcov-files-check-bufferiszero-y = util/bufferiszero.c > check-unit-y += tests/test-uuid$(EXESUF) > check-unit-y += tests/ptimer-test$(EXESUF) > gcov-files-ptimer-test-y = hw/core/ptimer.c > +check-unit-y += tests/test-listen$(EXESUF) > check-unit-y += tests/test-qapi-util$(EXESUF) > gcov-files-test-qapi-util-y = qapi/qapi-util.c > > @@ -804,6 +805,7 @@ tests/test-arm-mptimer$(EXESUF): tests/test-arm-mptimer.o > tests/test-qapi-util$(EXESUF): tests/test-qapi-util.o $(test-util-obj-y) > tests/numa-test$(EXESUF): tests/numa-test.o > tests/vmgenid-test$(EXESUF): tests/vmgenid-test.o tests/boot-sector.o tests/acpi- > utils.o > +tests/test-listen$(EXESUF): tests/test-listen.o $(test-util-obj-y) > > tests/migration/stress$(EXESUF): tests/migration/stress.o > $(call quiet-command, $(LINKPROG) -static -O3 $(PTHREAD_LIB) -o $@ $< > ,"LINK","$(TARGET_DIR)$@") > diff --git a/tests/test-listen.c b/tests/test-listen.c > new file mode 100644 > index 0000000000..03c4c8f03b > --- /dev/null > +++ b/tests/test-listen.c > @@ -0,0 +1,253 @@ > +/* > + * Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. > + * Author: Knut Omang <knut.omang@oracle.com> > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 or later > + * as published by the Free Software Foundation. > + * > + * Test parallel port listen configuration with > + * dynamic port allocation > + */ > + > +#include "qemu/osdep.h" > +#include "libqtest.h" > +#include "qemu-common.h" > +#include "qemu/thread.h" > +#include "qemu/sockets.h" > +#include "qapi/error.h" > + > +#define NAME_LEN 1024 > +#define PORT_LEN 16 > + > +struct thr_info { > + QemuThread thread; > + int to_port; > + bool ipv4; > + bool ipv6; > + int got_port; > + int eno; > + int fd; > + const char *errstr; > + char hostname[NAME_LEN + 1]; > + char port[PORT_LEN + 1]; > +}; > + > + > +/* These two functions taken from test-io-channel-socket.c */ > +static int check_bind(const char *hostname, bool *has_proto) > +{ > + int fd = -1; > + struct addrinfo ai, *res = NULL; > + int rc; > + int ret = -1; > + > + memset(&ai, 0, sizeof(ai)); > + ai.ai_flags = AI_CANONNAME | AI_ADDRCONFIG; > + ai.ai_family = AF_UNSPEC; > + ai.ai_socktype = SOCK_STREAM; > + > + /* lookup */ > + rc = getaddrinfo(hostname, NULL, &ai, &res); > + if (rc != 0) { > + if (rc == EAI_ADDRFAMILY || > + rc == EAI_FAMILY) { > + *has_proto = false; > + goto done; > + } > + goto cleanup; > + } > + > + fd = qemu_socket(res->ai_family, res->ai_socktype, res->ai_protocol); > + if (fd < 0) { > + goto cleanup; > + } > + > + if (bind(fd, res->ai_addr, res->ai_addrlen) < 0) { > + if (errno == EADDRNOTAVAIL) { > + *has_proto = false; > + goto done; > + } > + goto cleanup; > + } > + > + *has_proto = true; > + done: > + ret = 0; > + > + cleanup: > + if (fd != -1) { > + close(fd); > + } > + if (res) { > + freeaddrinfo(res); > + } > + return ret; > +} > + > +static int check_protocol_support(bool *has_ipv4, bool *has_ipv6) > +{ > + if (check_bind("127.0.0.1", has_ipv4) < 0) { > + return -1; > + } > + if (check_bind("::1", has_ipv6) < 0) { > + return -1; > + } > + > + return 0; > +} > + > +static void *listener_thread(void *arg) > +{ > + struct thr_info *thr = (struct thr_info *)arg; > + SocketAddress addr = { > + .type = SOCKET_ADDRESS_TYPE_INET, > + .u = { > + .inet = { > + .host = thr->hostname, > + .port = thr->port, > + .has_ipv4 = thr->ipv4, > + .ipv4 = thr->ipv4, > + .has_ipv6 = thr->ipv6, > + .ipv6 = thr->ipv6, > + .has_to = true, > + .to = thr->to_port, > + }, > + }, > + }; > + Error *err = NULL; > + int fd; > + > + fd = socket_listen(&addr, &err); > + if (fd < 0) { > + thr->eno = errno; > + thr->errstr = error_get_pretty(err); > + } else { > + struct sockaddr_in a; > + socklen_t a_len = sizeof(a); > + g_assert_cmpint(getsockname(fd, (struct sockaddr *)&a, &a_len), ==, 0); > + thr->got_port = ntohs(a.sin_port); > + thr->fd = fd; > + } > + return arg; > +} > + > + > +static void listen_compete_nthr(bool threaded, int nthreads, > + int start_port, int max_offset, > + bool ipv4, bool ipv6) > +{ > + int i; > + int failed_listens = 0; > + struct thr_info *thr = g_new0(struct thr_info, nthreads); > + int used[max_offset + 1]; > + > + memset(used, 0, sizeof(used)); > + for (i = 0; i < nthreads; i++) { > + snprintf(thr[i].port, PORT_LEN, "%d", start_port); > + strcpy(thr[i].hostname, "localhost"); > + thr[i].to_port = start_port + max_offset; > + thr[i].ipv4 = ipv4; > + thr[i].ipv6 = ipv6; > + } > + > + for (i = 0; i < nthreads; i++) { > + if (threaded) { > + qemu_thread_create(&thr[i].thread, "listener", > + listener_thread, &thr[i], > + QEMU_THREAD_JOINABLE); > + } else { > + listener_thread(&thr[i]); > + } > + } > + > + if (threaded) { > + for (i = 0; i < nthreads; i++) { > + qemu_thread_join(&thr[i].thread); > + } > + } > + for (i = 0; i < nthreads; i++) { > + if (thr[i].got_port) { > + closesocket(thr[i].fd); > + } > + } > + > + for (i = 0; i < nthreads; i++) { > + if (thr[i].eno != 0) { > + const char *m; > + g_printerr("** Failed to assign a port to thread %d (errno = %d)\n", > + i, thr[i].eno); > + /* This is what we are interested in capturing - > + * catch and report details if something unexpected happens: > + */ > + m = strstr(thr[i].errstr, "Failed to listen on socket"); > + if (m != NULL) { > + g_assert_cmpstr(thr[i].errstr, ==, > + "Failed to listen on socket: Address already in use"); > + } > + failed_listens++; > + } else { > + int assigned_port = thr[i].got_port; > + g_assert_cmpint(assigned_port, <= , thr[i].to_port); > + g_assert_cmpint(used[assigned_port - start_port], == , 0); > + } > + } > + g_assert_cmpint(failed_listens, ==, 0); > + g_free(thr); > +} > + > + > +static void listen_compete_ipv4(void) > +{ > + listen_compete_nthr(true, 200, 5920, 300, true, false); > +} > + > +static void listen_serial_ipv4(void) > +{ > + listen_compete_nthr(false, 200, 6300, 300, true, false); > +} > + > +static void listen_compete_ipv6(void) > +{ > + listen_compete_nthr(true, 200, 5920, 300, true, false); > +} > + > +static void listen_serial_ipv6(void) > +{ > + listen_compete_nthr(false, 200, 6300, 300, false, true); > +} > + > +static void listen_compete_gen(void) > +{ > + listen_compete_nthr(true, 200, 5920, 300, true, true); > +} > + > +static void listen_serial_gen(void) > +{ > + listen_compete_nthr(false, 200, 6300, 300, true, true); > +} > + > + > +int main(int argc, char **argv) > +{ > + bool has_ipv4, has_ipv6; > + g_test_init(&argc, &argv, NULL); > + > + if (check_protocol_support(&has_ipv4, &has_ipv6) < 0) { > + return 1; > + } > + > + if (has_ipv4) { > + g_test_add_func("/socket/listen-serial/ipv4", listen_serial_ipv4); > + g_test_add_func("/socket/listen-compete/ipv4", listen_compete_ipv4); > + } > + if (has_ipv6) { > + g_test_add_func("/socket/listen-serial/ipv6", listen_serial_ipv6); > + g_test_add_func("/socket/listen-compete/ipv6", listen_compete_ipv6); > + } > + if (has_ipv4 && has_ipv6) { > + g_test_add_func("/socket/listen-serial/generic", listen_serial_gen); > + g_test_add_func("/socket/listen-compete/generic", listen_compete_gen); > + } > + return g_test_run(); > +} ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PULL v1 2/2] tests: Add test-listen - a stress test for QEMU socket listen 2018-05-01 14:00 ` Knut Omang @ 2018-05-01 14:07 ` Daniel P. Berrangé 2018-05-01 14:21 ` Knut Omang 0 siblings, 1 reply; 12+ messages in thread From: Daniel P. Berrangé @ 2018-05-01 14:07 UTC (permalink / raw) To: Knut Omang; +Cc: qemu-devel, Peter Maydell On Tue, May 01, 2018 at 04:00:35PM +0200, Knut Omang wrote: > Hi Peter, > > Seems this test was lost along the way? > > I thought it got merged with Daniel's fix to the potential leak > that the test exposed in your build, but it seems not? I've never been able to get this test, nor another one of my own to succesfully pass automated testing in all the various test envs Peter and our many CI systems use. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PULL v1 2/2] tests: Add test-listen - a stress test for QEMU socket listen 2018-05-01 14:07 ` Daniel P. Berrangé @ 2018-05-01 14:21 ` Knut Omang 2018-05-01 14:36 ` Daniel P. Berrangé 0 siblings, 1 reply; 12+ messages in thread From: Knut Omang @ 2018-05-01 14:21 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: qemu-devel, Peter Maydell On Tue, 2018-05-01 at 15:07 +0100, Daniel P. Berrangé wrote: > On Tue, May 01, 2018 at 04:00:35PM +0200, Knut Omang wrote: > > Hi Peter, > > > > Seems this test was lost along the way? > > > > I thought it got merged with Daniel's fix to the potential leak > > that the test exposed in your build, but it seems not? > > I've never been able to get this test, nor another one of my own > to succesfully pass automated testing in all the various test > envs Peter and our many CI systems use. Ok, that explains it - no big deal from my side, I just thought it was "value lost" against future errors. Would it make sense to let the test check for environment and just really run in cases where it reliably passes? That will also serve as documentation as to where there might be additional issues hidden? Knut ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PULL v1 2/2] tests: Add test-listen - a stress test for QEMU socket listen 2018-05-01 14:21 ` Knut Omang @ 2018-05-01 14:36 ` Daniel P. Berrangé 0 siblings, 0 replies; 12+ messages in thread From: Daniel P. Berrangé @ 2018-05-01 14:36 UTC (permalink / raw) To: Knut Omang; +Cc: qemu-devel, Peter Maydell On Tue, May 01, 2018 at 04:21:01PM +0200, Knut Omang wrote: > On Tue, 2018-05-01 at 15:07 +0100, Daniel P. Berrangé wrote: > > On Tue, May 01, 2018 at 04:00:35PM +0200, Knut Omang wrote: > > > Hi Peter, > > > > > > Seems this test was lost along the way? > > > > > > I thought it got merged with Daniel's fix to the potential leak > > > that the test exposed in your build, but it seems not? > > > > I've never been able to get this test, nor another one of my own > > to succesfully pass automated testing in all the various test > > envs Peter and our many CI systems use. > > Ok, that explains it - no big deal from my side, I just thought > it was "value lost" against future errors. > > Would it make sense to let the test check for environment and just > really run in cases where it reliably passes? That's why I've actually been trying to do - I hacked some checks into the start of the test suite that were supposed to detect if it can run reliably but it still got failures. Because of the large number of socketss it opens it was hitting FD resource limits at times IIRC FWIW, I still have the patch on my local queue and hope one day I'll have an insight to fix it enough to merge... Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 2017-11-06 15:33 [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 Daniel P. Berrange 2017-11-06 15:33 ` [Qemu-devel] [PULL v1 1/2] sockets: avoid leak of listen file descriptor Daniel P. Berrange 2017-11-06 15:33 ` [Qemu-devel] [PULL v1 2/2] tests: Add test-listen - a stress test for QEMU socket listen Daniel P. Berrange @ 2017-11-06 16:11 ` Peter Maydell 2017-11-06 17:43 ` Daniel P. Berrange 2 siblings, 1 reply; 12+ messages in thread From: Peter Maydell @ 2017-11-06 16:11 UTC (permalink / raw) To: Daniel P. Berrange; +Cc: QEMU Developers On 6 November 2017 at 15:33, Daniel P. Berrange <berrange@redhat.com> wrote: > The following changes since commit ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c: > > Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2017-11-06 10:04:16 +0000) > > are available in the git repository at: > > git://github.com/berrange/qemu tags/pull-qio-2017-11-06-1 > > for you to fetch changes up to 74c0035408d5b327eac5985d976bca3009e0c5d6: > > tests: Add test-listen - a stress test for QEMU socket listen (2017-11-06 11:49:15 +0000) > > ---------------------------------------------------------------- > Merge qio 2017-11-06 v1 > > ---------------------------------------------------------------- Hi. I'm afraid this fails on various hosts. arm 32 bit (in a chroot on a 64 bit box): TEST: tests/test-listen... (pid=319) /socket/listen-serial/ipv4: OK /socket/listen-serial/ipv6: ** Failed to assign a port to thread 0 (errno = 98) ** Failed to assign a port to thread 1 (errno = 98) ** Failed to assign a port to thread 2 (errno = 98) ** Failed to assign a port to thread 3 (errno = 98) ** Failed to assign a port to thread 4 (errno = 98) [...repeats all the way up to...] ** Failed to assign a port to thread 197 (errno = 98) ** Failed to assign a port to thread 198 (errno = 98) ** Failed to assign a port to thread 199 (errno = 98) ** ERROR:/home/peter.maydell/qemu/tests/test-listen.c:195:listen_compete_nthr: assertion failed (failed_listens == 0): (200 == 0) FAIL GTester: last random seed: R02Sbad2a742dbd146e85cb412ce8f8f3750 (pid=321) /socket/listen-serial/generic: OK /socket/listen-compete/ipv4: OK /socket/listen-compete/ipv6: OK /socket/listen-compete/generic: OK FAIL: tests/test-listen Compile failure for the test on FreeBSD: /root/qemu/tests/test-listen.c:53:19: error: use of undeclared identifier 'EAI_ADDRFAMILY' if (rc == EAI_ADDRFAMILY || ^ Test failure on OpenBSD: TEST: tests/test-listen... (pid=9208) /socket/listen-serial/ipv4: ** Failed to assign a port to thread 124 (errno = 48) ** Failed to assign a port to thread 125 (errno = 48) ** Failed to assign a port to thread 126 (errno = 48) ** Failed to assign a port to thread 127 (errno = 48) [repeats up to] ** Failed to assign a port to thread 197 (errno = 48) ** Failed to assign a port to thread 198 (errno = 48) ** Failed to assign a port to thread 199 (errno = 48) ** ERROR:/home/qemu/tests/test-listen.c:195:listen_compete_nthr: assertion failed (failed_listens == 0): (76 == 0) FAIL GTester: last random seed: R02S17d50ab00329e514ffbe630191e76e83 (pid=80652) & similarly for the other subtests. Linux x86-64: make: Entering directory '/home/petmay01/linaro/qemu-for-merges/build/all' GTESTER tests/test-listen ** Failed to assign a port to thread 0 (errno = 98) ** Failed to assign a port to thread 1 (errno = 98) ** Failed to assign a port to thread 2 (errno = 98) ** Failed to assign a port to thread 3 (errno = 98) ** Failed to assign a port to thread 4 (errno = 98) ** Failed to assign a port to thread 5 (errno = 98) ** Failed to assign a port to thread 6 (errno = 98) [up to] ** Failed to assign a port to thread 199 (errno = 98) ** ERROR:/home/petmay01/linaro/qemu-for-merges/tests/test-listen.c:195:listen_compete_nthr: assertion failed (failed_listens == 0): (200 == 0) GTester: last random seed: R02S041a056d256c9ae190cf026f5adfd979 /home/petmay01/linaro/qemu-for-merges/tests/Makefile.include:863: recipe for target 'check-tests/test-listen' failed thanks -- PMM ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 2017-11-06 16:11 ` [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 Peter Maydell @ 2017-11-06 17:43 ` Daniel P. Berrange 2017-11-06 17:58 ` Peter Maydell 0 siblings, 1 reply; 12+ messages in thread From: Daniel P. Berrange @ 2017-11-06 17:43 UTC (permalink / raw) To: Peter Maydell; +Cc: QEMU Developers On Mon, Nov 06, 2017 at 04:11:56PM +0000, Peter Maydell wrote: > On 6 November 2017 at 15:33, Daniel P. Berrange <berrange@redhat.com> wrote: > > The following changes since commit ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c: > > > > Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2017-11-06 10:04:16 +0000) > > > > are available in the git repository at: > > > > git://github.com/berrange/qemu tags/pull-qio-2017-11-06-1 > > > > for you to fetch changes up to 74c0035408d5b327eac5985d976bca3009e0c5d6: > > > > tests: Add test-listen - a stress test for QEMU socket listen (2017-11-06 11:49:15 +0000) > > > > ---------------------------------------------------------------- > > Merge qio 2017-11-06 v1 > > > > ---------------------------------------------------------------- > > Hi. I'm afraid this fails on various hosts. Sigh, anything network related in unit tests is soo hard to get right :-( I was hoping the previous fail you saw with this test was all due to the file descriptor leak the first patch fixed, but I guess not.... > > arm 32 bit (in a chroot on a 64 bit box): > TEST: tests/test-listen... (pid=319) > /socket/listen-serial/ipv4: OK > /socket/listen-serial/ipv6: > ** Failed to assign a port to > thread 0 (errno = 98) > ** Failed to assign a port to thread 1 (errno = 98) > ** Failed to assign a port to thread 2 (errno = 98) > ** Failed to assign a port to thread 3 (errno = 98) > ** Failed to assign a port to thread 4 (errno = 98) > [...repeats all the way up to...] > ** Failed to assign a port to thread 197 (errno = 98) > ** Failed to assign a port to thread 198 (errno = 98) > ** Failed to assign a port to thread 199 (errno = 98) > ** > ERROR:/home/peter.maydell/qemu/tests/test-listen.c:195:listen_compete_nthr: > assertion failed (failed_listens == 0): (200 == 0) Hmm, every single thread failed, which suggests some fundamental config in this environment that prevents any use of networking at all from tests. Anything interesting about this build env related to network availability that would prevent use of even localhost ? > FAIL > GTester: last random seed: R02Sbad2a742dbd146e85cb412ce8f8f3750 > (pid=321) > /socket/listen-serial/generic: OK > /socket/listen-compete/ipv4: OK > /socket/listen-compete/ipv6: OK > /socket/listen-compete/generic: OK > FAIL: tests/test-listen > > Compile failure for the test on FreeBSD: > > /root/qemu/tests/test-listen.c:53:19: error: use of undeclared > identifier 'EAI_ADDRFAMILY' > if (rc == EAI_ADDRFAMILY || > ^ Ok, that's at least an easy one to fix. > > > Test failure on OpenBSD: > > TEST: tests/test-listen... (pid=9208) > /socket/listen-serial/ipv4: > ** Failed to assign a port to > thread 124 (errno = 48) > ** Failed to assign a port to thread 125 (errno = 48) > ** Failed to assign a port to thread 126 (errno = 48) > ** Failed to assign a port to thread 127 (errno = 48) > [repeats up to] > ** Failed to assign a port to thread 197 (errno = 48) > ** Failed to assign a port to thread 198 (errno = 48) > ** Failed to assign a port to thread 199 (errno = 48) > ** > ERROR:/home/qemu/tests/test-listen.c:195:listen_compete_nthr: > assertion failed (failed_listens == 0): (76 == 0) > FAIL > GTester: last random seed: R02S17d50ab00329e514ffbe630191e76e83 > (pid=80652) Only a subset failed, which is more interesting - this is definitely more like the problem seen when FD leaks. Could you mention what the ulimits are for your OpenBSD host - I wondering if it hits max FDs due to having a parallel builds / test run. > & similarly for the other subtests. > > Linux x86-64: > > make: Entering directory '/home/petmay01/linaro/qemu-for-merges/build/all' > GTESTER tests/test-listen > ** Failed to assign a port to thread 0 (errno = 98) > ** Failed to assign a port to thread 1 (errno = 98) > ** Failed to assign a port to thread 2 (errno = 98) > ** Failed to assign a port to thread 3 (errno = 98) > ** Failed to assign a port to thread 4 (errno = 98) > ** Failed to assign a port to thread 5 (errno = 98) > ** Failed to assign a port to thread 6 (errno = 98) > [up to] > ** Failed to assign a port to thread 199 (errno = 98) > ** > ERROR:/home/petmay01/linaro/qemu-for-merges/tests/test-listen.c:195:listen_compete_nthr: > assertion failed (failed_listens == 0): (200 == 0) > GTester: last random seed: R02S041a056d256c9ae190cf026f5adfd979 > /home/petmay01/linaro/qemu-for-merges/tests/Makefile.include:863: > recipe for target 'check-tests/test-listen' failed Looks same as your arm env, which suggests no network access at all. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 2017-11-06 17:43 ` Daniel P. Berrange @ 2017-11-06 17:58 ` Peter Maydell 2017-11-07 9:19 ` Daniel P. Berrange 0 siblings, 1 reply; 12+ messages in thread From: Peter Maydell @ 2017-11-06 17:58 UTC (permalink / raw) To: Daniel P. Berrange; +Cc: QEMU Developers On 6 November 2017 at 17:43, Daniel P. Berrange <berrange@redhat.com> wrote: > On Mon, Nov 06, 2017 at 04:11:56PM +0000, Peter Maydell wrote: >> On 6 November 2017 at 15:33, Daniel P. Berrange <berrange@redhat.com> wrote: >> > The following changes since commit ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c: >> > >> > Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2017-11-06 10:04:16 +0000) >> > >> > are available in the git repository at: >> > >> > git://github.com/berrange/qemu tags/pull-qio-2017-11-06-1 >> > >> > for you to fetch changes up to 74c0035408d5b327eac5985d976bca3009e0c5d6: >> > >> > tests: Add test-listen - a stress test for QEMU socket listen (2017-11-06 11:49:15 +0000) >> > >> > ---------------------------------------------------------------- >> > Merge qio 2017-11-06 v1 >> > >> > ---------------------------------------------------------------- >> >> Hi. I'm afraid this fails on various hosts. > > Sigh, anything network related in unit tests is soo hard to get right :-( > I was hoping the previous fail you saw with this test was all due to the > file descriptor leak the first patch fixed, but I guess not.... > >> >> arm 32 bit (in a chroot on a 64 bit box): >> TEST: tests/test-listen... (pid=319) >> /socket/listen-serial/ipv4: OK >> /socket/listen-serial/ipv6: >> ** Failed to assign a port to >> thread 0 (errno = 98) >> ** Failed to assign a port to thread 1 (errno = 98) >> ** Failed to assign a port to thread 2 (errno = 98) >> ** Failed to assign a port to thread 3 (errno = 98) >> ** Failed to assign a port to thread 4 (errno = 98) >> [...repeats all the way up to...] >> ** Failed to assign a port to thread 197 (errno = 98) >> ** Failed to assign a port to thread 198 (errno = 98) >> ** Failed to assign a port to thread 199 (errno = 98) >> ** >> ERROR:/home/peter.maydell/qemu/tests/test-listen.c:195:listen_compete_nthr: >> assertion failed (failed_listens == 0): (200 == 0) > > Hmm, every single thread failed, which suggests some fundamental config in > this environment that prevents any use of networking at all from tests. > > Anything interesting about this build env related to network availability > that would prevent use of even localhost ? You'll notice that the IPV4 test worked fine. I assume this is an IPV6 specific problem. >> Test failure on OpenBSD: >> >> TEST: tests/test-listen... (pid=9208) >> /socket/listen-serial/ipv4: >> ** Failed to assign a port to >> thread 124 (errno = 48) >> ** Failed to assign a port to thread 125 (errno = 48) >> ** Failed to assign a port to thread 126 (errno = 48) >> ** Failed to assign a port to thread 127 (errno = 48) >> [repeats up to] >> ** Failed to assign a port to thread 197 (errno = 48) >> ** Failed to assign a port to thread 198 (errno = 48) >> ** Failed to assign a port to thread 199 (errno = 48) >> ** >> ERROR:/home/qemu/tests/test-listen.c:195:listen_compete_nthr: >> assertion failed (failed_listens == 0): (76 == 0) >> FAIL >> GTester: last random seed: R02S17d50ab00329e514ffbe630191e76e83 >> (pid=80652) > > Only a subset failed, which is more interesting - this is definitely > more like the problem seen when FD leaks. Could you mention what the > ulimits are for your OpenBSD host - I wondering if it hits max FDs > due to having a parallel builds / test run. # ulimit -a time(cpu-seconds) unlimited file(blocks) unlimited coredump(blocks) unlimited data(kbytes) 33554432 stack(kbytes) 8192 lockedmem(kbytes) 670784 memory(kbytes) 2010588 nofiles(descriptors) 128 processes 1310 That nofiles limit looks quite close to the number where we start failing... >> Linux x86-64: >> >> make: Entering directory '/home/petmay01/linaro/qemu-for-merges/build/all' >> GTESTER tests/test-listen >> ** Failed to assign a port to thread 0 (errno = 98) >> ** Failed to assign a port to thread 1 (errno = 98) >> ** Failed to assign a port to thread 2 (errno = 98) >> ** Failed to assign a port to thread 3 (errno = 98) >> ** Failed to assign a port to thread 4 (errno = 98) >> ** Failed to assign a port to thread 5 (errno = 98) >> ** Failed to assign a port to thread 6 (errno = 98) >> [up to] >> ** Failed to assign a port to thread 199 (errno = 98) >> ** >> ERROR:/home/petmay01/linaro/qemu-for-merges/tests/test-listen.c:195:listen_compete_nthr: >> assertion failed (failed_listens == 0): (200 == 0) >> GTester: last random seed: R02S041a056d256c9ae190cf026f5adfd979 >> /home/petmay01/linaro/qemu-for-merges/tests/Makefile.include:863: >> recipe for target 'check-tests/test-listen' failed > > Looks same as your arm env, which suggests no network access at all. This is running native on my primary development machine, so its network access is fine :-) thanks -- PMM ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 2017-11-06 17:58 ` Peter Maydell @ 2017-11-07 9:19 ` Daniel P. Berrange 2017-11-07 14:40 ` Daniel P. Berrange 0 siblings, 1 reply; 12+ messages in thread From: Daniel P. Berrange @ 2017-11-07 9:19 UTC (permalink / raw) To: Peter Maydell; +Cc: QEMU Developers On Mon, Nov 06, 2017 at 05:58:41PM +0000, Peter Maydell wrote: > On 6 November 2017 at 17:43, Daniel P. Berrange <berrange@redhat.com> wrote: > > On Mon, Nov 06, 2017 at 04:11:56PM +0000, Peter Maydell wrote: > >> On 6 November 2017 at 15:33, Daniel P. Berrange <berrange@redhat.com> wrote: > >> > The following changes since commit ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c: > >> > > >> > Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2017-11-06 10:04:16 +0000) > >> > > >> > are available in the git repository at: > >> > > >> > git://github.com/berrange/qemu tags/pull-qio-2017-11-06-1 > >> > > >> > for you to fetch changes up to 74c0035408d5b327eac5985d976bca3009e0c5d6: > >> > > >> > tests: Add test-listen - a stress test for QEMU socket listen (2017-11-06 11:49:15 +0000) > >> > > >> > ---------------------------------------------------------------- > >> > Merge qio 2017-11-06 v1 > >> > > >> > ---------------------------------------------------------------- > >> > >> Hi. I'm afraid this fails on various hosts. > > > > Sigh, anything network related in unit tests is soo hard to get right :-( > > I was hoping the previous fail you saw with this test was all due to the > > file descriptor leak the first patch fixed, but I guess not.... > > > >> > >> arm 32 bit (in a chroot on a 64 bit box): > >> TEST: tests/test-listen... (pid=319) > >> /socket/listen-serial/ipv4: OK > >> /socket/listen-serial/ipv6: > >> ** Failed to assign a port to > >> thread 0 (errno = 98) > >> ** Failed to assign a port to thread 1 (errno = 98) > >> ** Failed to assign a port to thread 2 (errno = 98) > >> ** Failed to assign a port to thread 3 (errno = 98) > >> ** Failed to assign a port to thread 4 (errno = 98) > >> [...repeats all the way up to...] > >> ** Failed to assign a port to thread 197 (errno = 98) > >> ** Failed to assign a port to thread 198 (errno = 98) > >> ** Failed to assign a port to thread 199 (errno = 98) > >> ** > >> ERROR:/home/peter.maydell/qemu/tests/test-listen.c:195:listen_compete_nthr: > >> assertion failed (failed_listens == 0): (200 == 0) > > > > Hmm, every single thread failed, which suggests some fundamental config in > > this environment that prevents any use of networking at all from tests. > > > > Anything interesting about this build env related to network availability > > that would prevent use of even localhost ? > > You'll notice that the IPV4 test worked fine. I assume this is an > IPV6 specific problem. Oh yes, so it is. The strange thing is that the test does check if we have a protocol available by doing a test bind() at start. > >> Test failure on OpenBSD: > >> > >> TEST: tests/test-listen... (pid=9208) > >> /socket/listen-serial/ipv4: > >> ** Failed to assign a port to > >> thread 124 (errno = 48) > >> ** Failed to assign a port to thread 125 (errno = 48) > >> ** Failed to assign a port to thread 126 (errno = 48) > >> ** Failed to assign a port to thread 127 (errno = 48) > >> [repeats up to] > >> ** Failed to assign a port to thread 197 (errno = 48) > >> ** Failed to assign a port to thread 198 (errno = 48) > >> ** Failed to assign a port to thread 199 (errno = 48) > >> ** > >> ERROR:/home/qemu/tests/test-listen.c:195:listen_compete_nthr: > >> assertion failed (failed_listens == 0): (76 == 0) > >> FAIL > >> GTester: last random seed: R02S17d50ab00329e514ffbe630191e76e83 > >> (pid=80652) > > > > Only a subset failed, which is more interesting - this is definitely > > more like the problem seen when FD leaks. Could you mention what the > > ulimits are for your OpenBSD host - I wondering if it hits max FDs > > due to having a parallel builds / test run. > > # ulimit -a > time(cpu-seconds) unlimited > file(blocks) unlimited > coredump(blocks) unlimited > data(kbytes) 33554432 > stack(kbytes) 8192 > lockedmem(kbytes) 670784 > memory(kbytes) 2010588 > nofiles(descriptors) 128 > processes 1310 > > That nofiles limit looks quite close to the number where we > start failing... Ok, I think I'll just make the test skip unless nofiles >= 1024 so we have a good margin of safety when things run very parallel Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 2017-11-07 9:19 ` Daniel P. Berrange @ 2017-11-07 14:40 ` Daniel P. Berrange 0 siblings, 0 replies; 12+ messages in thread From: Daniel P. Berrange @ 2017-11-07 14:40 UTC (permalink / raw) To: Peter Maydell; +Cc: QEMU Developers On Tue, Nov 07, 2017 at 09:19:45AM +0000, Daniel P. Berrange wrote: > On Mon, Nov 06, 2017 at 05:58:41PM +0000, Peter Maydell wrote: > > On 6 November 2017 at 17:43, Daniel P. Berrange <berrange@redhat.com> wrote: > > > On Mon, Nov 06, 2017 at 04:11:56PM +0000, Peter Maydell wrote: > > >> On 6 November 2017 at 15:33, Daniel P. Berrange <berrange@redhat.com> wrote: > > >> > The following changes since commit ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c: > > >> > > > >> > Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2017-11-06 10:04:16 +0000) > > >> > > > >> > are available in the git repository at: > > >> > > > >> > git://github.com/berrange/qemu tags/pull-qio-2017-11-06-1 > > >> > > > >> > for you to fetch changes up to 74c0035408d5b327eac5985d976bca3009e0c5d6: > > >> > > > >> > tests: Add test-listen - a stress test for QEMU socket listen (2017-11-06 11:49:15 +0000) > > >> > > > >> > ---------------------------------------------------------------- > > >> > Merge qio 2017-11-06 v1 > > >> > > > >> > ---------------------------------------------------------------- > > >> > > >> Hi. I'm afraid this fails on various hosts. > > > > > > Sigh, anything network related in unit tests is soo hard to get right :-( > > > I was hoping the previous fail you saw with this test was all due to the > > > file descriptor leak the first patch fixed, but I guess not.... > > > > > >> > > >> arm 32 bit (in a chroot on a 64 bit box): > > >> TEST: tests/test-listen... (pid=319) > > >> /socket/listen-serial/ipv4: OK > > >> /socket/listen-serial/ipv6: > > >> ** Failed to assign a port to > > >> thread 0 (errno = 98) > > >> ** Failed to assign a port to thread 1 (errno = 98) > > >> ** Failed to assign a port to thread 2 (errno = 98) > > >> ** Failed to assign a port to thread 3 (errno = 98) > > >> ** Failed to assign a port to thread 4 (errno = 98) > > >> [...repeats all the way up to...] > > >> ** Failed to assign a port to thread 197 (errno = 98) > > >> ** Failed to assign a port to thread 198 (errno = 98) > > >> ** Failed to assign a port to thread 199 (errno = 98) > > >> ** > > >> ERROR:/home/peter.maydell/qemu/tests/test-listen.c:195:listen_compete_nthr: > > >> assertion failed (failed_listens == 0): (200 == 0) > > > > > > Hmm, every single thread failed, which suggests some fundamental config in > > > this environment that prevents any use of networking at all from tests. > > > > > > Anything interesting about this build env related to network availability > > > that would prevent use of even localhost ? > > > > You'll notice that the IPV4 test worked fine. I assume this is an > > IPV6 specific problem. > > Oh yes, so it is. The strange thing is that the test does check if we have > a protocol available by doing a test bind() at start. > > > >> Test failure on OpenBSD: > > >> > > >> TEST: tests/test-listen... (pid=9208) > > >> /socket/listen-serial/ipv4: > > >> ** Failed to assign a port to > > >> thread 124 (errno = 48) > > >> ** Failed to assign a port to thread 125 (errno = 48) > > >> ** Failed to assign a port to thread 126 (errno = 48) > > >> ** Failed to assign a port to thread 127 (errno = 48) > > >> [repeats up to] > > >> ** Failed to assign a port to thread 197 (errno = 48) > > >> ** Failed to assign a port to thread 198 (errno = 48) > > >> ** Failed to assign a port to thread 199 (errno = 48) > > >> ** > > >> ERROR:/home/qemu/tests/test-listen.c:195:listen_compete_nthr: > > >> assertion failed (failed_listens == 0): (76 == 0) > > >> FAIL > > >> GTester: last random seed: R02S17d50ab00329e514ffbe630191e76e83 > > >> (pid=80652) > > > > > > Only a subset failed, which is more interesting - this is definitely > > > more like the problem seen when FD leaks. Could you mention what the > > > ulimits are for your OpenBSD host - I wondering if it hits max FDs > > > due to having a parallel builds / test run. > > > > # ulimit -a > > time(cpu-seconds) unlimited > > file(blocks) unlimited > > coredump(blocks) unlimited > > data(kbytes) 33554432 > > stack(kbytes) 8192 > > lockedmem(kbytes) 670784 > > memory(kbytes) 2010588 > > nofiles(descriptors) 128 > > processes 1310 > > > > That nofiles limit looks quite close to the number where we > > start failing... > > Ok, I think I'll just make the test skip unless nofiles >= 1024 > so we have a good margin of safety when things run very parallel FYI, I've dropped the test suite addition in my 2nd PULL request, as I'm not confident I can solve all the portability problems quickly, and don't want to waste your time going back + forth testing new versions during freeze. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2018-05-01 14:36 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-11-06 15:33 [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 Daniel P. Berrange 2017-11-06 15:33 ` [Qemu-devel] [PULL v1 1/2] sockets: avoid leak of listen file descriptor Daniel P. Berrange 2017-11-06 15:33 ` [Qemu-devel] [PULL v1 2/2] tests: Add test-listen - a stress test for QEMU socket listen Daniel P. Berrange 2018-05-01 14:00 ` Knut Omang 2018-05-01 14:07 ` Daniel P. Berrangé 2018-05-01 14:21 ` Knut Omang 2018-05-01 14:36 ` Daniel P. Berrangé 2017-11-06 16:11 ` [Qemu-devel] [PULL v1 0/2] Merge IO 2017/11/06 Peter Maydell 2017-11-06 17:43 ` Daniel P. Berrange 2017-11-06 17:58 ` Peter Maydell 2017-11-07 9:19 ` Daniel P. Berrange 2017-11-07 14:40 ` Daniel P. Berrange
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.