Re: Question on memory commit during MR finalize()

From: Peter Xu <peterx@redhat.com>
To: Thanos Makatos <thanos.makatos@nutanix.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>,
	John Levon <john.levon@nutanix.com>,
	John G Johnson <john.g.johnson@oracle.com>,
	Markus Armbruster <armbru@redhat.com>,
	QEMU Devel Mailing List <qemu-devel@nongnu.org>
Subject: Re: Question on memory commit during MR finalize()
Date: Thu, 15 Jul 2021 14:35:29 -0400	[thread overview]
Message-ID: <YPB/8f0pv7mDR310@t490s> (raw)
In-Reply-To: <CH0PR02MB78989C2F77880AA5355037748B129@CH0PR02MB7898.namprd02.prod.outlook.com>

On Thu, Jul 15, 2021 at 02:27:48PM +0000, Thanos Makatos wrote:
> Hi Peter,

Hi, Thanos,

> We're hitting this issue using a QEMU branch where JJ is using vfio-user as the transport for multiprocess-qemu (https://github.com/oracle/qemu/issues/9). We can reproduce it fairly reliably by migrating a virtual SPDK NVMe controller (the NVMf/vfio-user target with experimental migration support, https://review.spdk.io/gerrit/c/spdk/spdk/+/7617/14). I can provide detailed repro instructions but first I want to make sure we're not missing any patches.

I don't think you missed any bug fix patches, as the issue I mentioned can only
be trigger with my own branch at that time, and that's fixed when my patchset
got merged.

However if you encountered the same issue, it's possible that there's an
incorrect use of qemu memory/cpu API too somewhere there so similar issue is
triggered.  For example, in my case it was run_on_cpu() called incorrectly
within memory layout changing so BQL is released without being noticed.

I've got a series that tries to expose these hard to debug issues:

https://lore.kernel.org/qemu-devel/20200421162108.594796-1-peterx@redhat.com/

Obviously the series didn't track enough interest so it didn't get merged.
However maybe that's also something useful to what you're debugging, so you can
apply those patches onto your branch and see the stack when it reproduces
again. Logically with these sanity patches it could fail earlier than what
you've hit right now (which I believe should be within the RCU thread; btw it
would be interesting to share your stack too when it's hit) and it could
provide more useful information.

I saw that the old series won't apply onto master any more, so I rebased it and
pushed it here (with one patch dropped since someone wrote a similar patch and
got merged, so there're only 7 patches in the new tree):

https://github.com/xzpeter/qemu/tree/memory-sanity

No guarantee it'll help, but IMHO worth trying.

Thanks,

-- 
Peter Xu