From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:55780)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <borntraeger@de.ibm.com>) id 1aO9UI-0004sA-KX
	for qemu-devel@nongnu.org; Tue, 26 Jan 2016 14:41:18 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <borntraeger@de.ibm.com>) id 1aO9UE-00038K-G4
	for qemu-devel@nongnu.org; Tue, 26 Jan 2016 14:41:14 -0500
Received: from e06smtp16.uk.ibm.com ([195.75.94.112]:36471)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <borntraeger@de.ibm.com>) id 1aO9UE-00036D-6X
	for qemu-devel@nongnu.org; Tue, 26 Jan 2016 14:41:10 -0500
Received: from localhost
	by e06smtp16.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use
	Only! Violators will be prosecuted
	for <qemu-devel@nongnu.org> from <borntraeger@de.ibm.com>;
	Tue, 26 Jan 2016 19:41:07 -0000
Received: from b06cxnps3074.portsmouth.uk.ibm.com
	(d06relay09.portsmouth.uk.ibm.com [9.149.109.194])
	by d06dlp03.portsmouth.uk.ibm.com (Postfix) with ESMTP id 406E91B08061
	for <qemu-devel@nongnu.org>; Tue, 26 Jan 2016 19:41:12 +0000 (GMT)
Received: from d06av09.portsmouth.uk.ibm.com (d06av09.portsmouth.uk.ibm.com
	[9.149.37.250])
	by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with
	ESMTP id u0QJf46i3801530
	for <qemu-devel@nongnu.org>; Tue, 26 Jan 2016 19:41:04 GMT
Received: from d06av09.portsmouth.uk.ibm.com (localhost [127.0.0.1])
	by d06av09.portsmouth.uk.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with
	ESMTP id u0QJf4Qd013029
	for <qemu-devel@nongnu.org>; Tue, 26 Jan 2016 12:41:04 -0700
References: <1453716498-27238-1-git-send-email-dgilbert@redhat.com>
From: Christian Borntraeger <borntraeger@de.ibm.com>
Message-ID: <56A7CBCF.2070004@de.ibm.com>
Date: Tue, 26 Jan 2016 20:41:03 +0100
MIME-Version: 1.0
In-Reply-To: <1453716498-27238-1-git-send-email-dgilbert@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
Subject: Re: [Qemu-devel] [PATCH] use broadcast on qemu_pause_cond
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com>, qemu-devel@nongnu.org, pbonzini@redhat.com
Cc: jdenemar@redhat.com, "Jason J. Herne" <jjherne@linux.vnet.ibm.com>

On 01/25/2016 11:08 AM, Dr. David Alan Gilbert (git) wrote:
> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> 
> Jiri saw a hang on pause_all_vcpus called from postcopy_start,
> where the cpus are all apparently stopped ('stopped' flag set)
> but pause_all_vcpus is still stuck on a cond_wait on qemu_paused_cond.
> We suspect this is happening if a qmp_stop is called at about the
> same time as the postcopy code calls that pause_all_vcpus;
> although they both should have the main lock held, Paolo spotted
> the cond_wait unlocks the global lock so perhaps they both
> could end up waiting at the same time?

We have been chasing a similar problem, with many guests with lots of cpus, that
sometimes thread 1 waits like
Thread 1 (Thread 0x3fffa670c00 (LWP 15652)):
#0  0x000003fffcdf21b2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000008023f8f2 in qemu_cond_wait ()
#2  0x0000000080060332 in pause_all_vcpus ()
#3  0x00000000800603e8 in vm_stop ()
#4  0x00000000800f9b04 in qmp_marshal_input_stop ()
#5  0x0000000080063154 in handle_qmp_command ()
#6  0x000000008023b77e in json_message_process_token ()
#7  0x000000008024ef98 in json_lexer_feed_char ()
---Type <return> to continue, or q <return> to quit---
#8  0x000000008024f056 in json_lexer_feed ()
#9  0x0000000080061756 in monitor_qmp_read ()
#10 0x00000000800e4966 in tcp_chr_read ()
#11 0x000003fffcce3fb6 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#12 0x00000000801bd18e in main_loop_wait ()
#13 0x000000008002e244 in main ()
(gdb) 

One thread was still running inside KVM, not being kicked out into userspace.
Now: This might actually be the same problem. I was chasing the still running
CPU (why it does not exit, and I was able to make progress with killall -SIGUSR1 
qemu), but in fact, the problem might have been that thread 1 did not get
notified by the LAST CPUs (notify getting lost), therefore, never kicked this
CPU out.

The problem was never reproducable with qemu 2.3, so maybe the BQL avoided the
issue?
We will test if this fixes our problem as well.

> 
> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Reported-by: Jiri Denemark <jdenemar@redhat.com>
> ---
>  cpus.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index 3efff6b..1e97cc4 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -986,7 +986,7 @@ static void qemu_wait_io_event_common(CPUState *cpu)
>      if (cpu->stop) {
>          cpu->stop = false;
>          cpu->stopped = true;
> -        qemu_cond_signal(&qemu_pause_cond);
> +        qemu_cond_broadcast(&qemu_pause_cond);
>      }
>      flush_queued_work(cpu);
>      cpu->thread_kicked = false;
> @@ -1396,7 +1396,7 @@ void cpu_stop_current(void)
>          current_cpu->stop = false;
>          current_cpu->stopped = true;
>          cpu_exit(current_cpu);
> -        qemu_cond_signal(&qemu_pause_cond);
> +        qemu_cond_broadcast(&qemu_pause_cond);
>      }
>  }
>