RE: Question on memory commit during MR finalize()

[Thanos Makatos <thanos.makatos@nutanix.com>]

> -----Original Message-----
> From: Peter Xu <peterx@redhat.com>
> Sent: 16 July 2021 15:19
> To: Thanos Makatos <thanos.makatos@nutanix.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>; Markus Armbruster
> <armbru@redhat.com>; QEMU Devel Mailing List <qemu-
> devel@nongnu.org>; John Levon <john.levon@nutanix.com>; John G
> Johnson <john.g.johnson@oracle.com>
> Subject: Re: Question on memory commit during MR finalize()
> 
> On Fri, Jul 16, 2021 at 11:42:02AM +0000, Thanos Makatos wrote:
> > > -----Original Message-----
> > > From: Peter Xu <peterx@redhat.com>
> > > Sent: 15 July 2021 19:35
> > > To: Thanos Makatos <thanos.makatos@nutanix.com>
> > > Cc: Paolo Bonzini <pbonzini@redhat.com>; Markus Armbruster
> > > <armbru@redhat.com>; QEMU Devel Mailing List <qemu-
> > > devel@nongnu.org>; John Levon <john.levon@nutanix.com>; John G
> > > Johnson <john.g.johnson@oracle.com>
> > > Subject: Re: Question on memory commit during MR finalize()
> > >
> > > On Thu, Jul 15, 2021 at 02:27:48PM +0000, Thanos Makatos wrote:
> > > > Hi Peter,
> > >
> > > Hi, Thanos,
> > >
> > > > We're hitting this issue using a QEMU branch where JJ is using
> > > > vfio-user as
> > > the transport for multiprocess-qemu
> > > (https://urldefense.proofpoint.com/v2/url?u=https-
> > >
> 3A__github.com_oracle_qemu_issues_9&d=DwIBaQ&c=s883GpUCOChKOHi
> > >
> ocYtGcg&r=XTpYsh5Ps2zJvtw6ogtti46atk736SI4vgsJiUKIyDE&m=9nFuGF9kg5l
> > >
> ZsKPi03BNzo9pckG8DlodVG0LuEofnKw&s=dcp70CIgJljcWFwSRZm5zZRJj80jX
> > > XERLwpbH6ZcgzQ&e= ). We can reproduce it fairly reliably by
> > > migrating a virtual SPDK NVMe controller (the NVMf/vfio-user target
> > > with experimental migration support,
> > > https://urldefense.proofpoint.com/v2/url?u=https-
> > > 3A__review.spdk.io_gerrit_c_spdk_spdk_-
> > >
> 2B_7617_14&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw
> > >
> 6ogtti46atk736SI4vgsJiUKIyDE&m=9nFuGF9kg5lZsKPi03BNzo9pckG8DlodVG0
> > > LuEofnKw&s=iXolOQM5sYj4IB-cf__Ta8jgKXZqisYE-uuwq6qnbLo&e= ). I
> can
> > > provide detailed repro instructions but first I want to make sure
> > > we're not missing any patches.
> > >
> > > I don't think you missed any bug fix patches, as the issue I
> > > mentioned can only be trigger with my own branch at that time, and
> > > that's fixed when my patchset got merged.
> > >
> > > However if you encountered the same issue, it's possible that
> > > there's an incorrect use of qemu memory/cpu API too somewhere there
> > > so similar issue is triggered.  For example, in my case it was
> > > run_on_cpu() called incorrectly within memory layout changing so BQL
> > > is released without being noticed.
> > >
> > > I've got a series that tries to expose these hard to debug issues:
> > >
> > > https://urldefense.proofpoint.com/v2/url?u=https-
> > > 3A__lore.kernel.org_qemu-2Ddevel_20200421162108.594796-2D1-
> 2Dpeterx-
> > >
> 40redhat.com_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJ
> > >
> vtw6ogtti46atk736SI4vgsJiUKIyDE&m=9nFuGF9kg5lZsKPi03BNzo9pckG8Dlod
> > > VG0LuEofnKw&s=kQRJEb4CQmxEirS-III15QJz_phzhCYLIgjOF-SB9Pk&e=
> > >
> > > Obviously the series didn't track enough interest so it didn't get merged.
> > > However maybe that's also something useful to what you're debugging,
> > > so you can apply those patches onto your branch and see the stack
> > > when it reproduces again. Logically with these sanity patches it
> > > could fail earlier than what you've hit right now (which I believe
> > > should be within the RCU thread; btw it would be interesting to
> > > share your stack too when it's hit) and it could provide more useful
> information.
> > >
> > > I saw that the old series won't apply onto master any more, so I
> > > rebased it and pushed it here (with one patch dropped since someone
> > > wrote a similar patch and got merged, so there're only 7 patches in the
> new tree):
> > >
> > > https://urldefense.proofpoint.com/v2/url?u=https-
> > > 3A__github.com_xzpeter_qemu_tree_memory-
> > >
> 2Dsanity&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6og
> > >
> tti46atk736SI4vgsJiUKIyDE&m=9nFuGF9kg5lZsKPi03BNzo9pckG8DlodVG0LuE
> > > ofnKw&s=G-8FV-H-VcZTgCVRfTEVKo1GALIk2PqBvTdAcAXFoZ0&e=
> > >
> > > No guarantee it'll help, but IMHO worth trying.
> >
> > The memory-sanity branch fails to build:
> >
> > ./configure --prefix=/opt/qemu-xzpeter --target-list=x86_64-linux-user
> > --enable-debug make -j 8 ...
> > [697/973] Linking target qemu-x86_64
> > FAILED: qemu-x86_64
> > c++  -o qemu-x86_64 libcommon.fa.p/cpus-common.c.o
> > c++ libcommon.fa.p/page-vary-common.c.o libcommon.fa.p/disas_i386.c.o
> > c++ libcommon.fa.p/disas_capstone.c.o
> > c++ libcommon.fa.p/hw_core_cpu-common.c.o
> > c++ libcommon.fa.p/ebpf_ebpf_rss-stub.c.o
> > c++ libcommon.fa.p/accel_accel-user.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_user_excp_helper.c.
> > c++ o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_user_seg_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_x86_64_signal.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_x86_64_cpu_loop.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_cpu.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_gdbstub.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_xsave_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_cpu-dump.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_sev-stub.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_kvm_kvm-stub.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_bpt_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_cc_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_excp_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_fpu_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_int_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_mem_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_misc_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_mpx_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_seg_helper.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_tcg-cpu.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/target_i386_tcg_translate.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/trace_control-target.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/cpu.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/disas.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/gdbstub.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/page-vary.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/tcg_optimize.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/tcg_region.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/tcg_tcg.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/tcg_tcg-common.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/tcg_tcg-op.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/tcg_tcg-op-gvec.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/tcg_tcg-op-vec.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/fpu_softfloat.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_accel-common.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_tcg-all.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_cpu-exec-common.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_cpu-exec.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_tcg-runtime-gvec.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_tcg-runtime.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_translate-all.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_translator.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_user-exec.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_user-exec-stub.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_tcg_plugin-gen.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_stubs_hax-stub.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_stubs_xen-stub.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/accel_stubs_kvm-stub.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/plugins_loader.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/plugins_core.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/plugins_api.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_elfload.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_exit.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_fd-trans.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_linuxload.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_main.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_mmap.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_safe-syscall.S.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_signal.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_strace.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_syscall.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_uaccess.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/linux-user_uname.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/thunk.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/meson-generated_.._x86_64-linux-
> use
> > c++ r-gdbstub-xml.c.o
> > c++ libqemu-x86_64-linux-user.fa.p/meson-
> generated_.._trace_generated-
> > c++ helpers.c.o -Wl,--as-needed -Wl,--no-undefined -pie
> > c++ -Wl,--whole-archive libhwcore.fa libqom.fa -Wl,--no-whole-archive
> > c++ -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -m64
> > c++ -fstack-protector-strong -Wl,--start-group libcapstone.a
> > c++ libqemuutil.a libhwcore.fa libqom.fa -ldl
> > c++ -Wl,--dynamic-list=/root/src/qemu/build/qemu-plugins-ld.symbols
> > c++ -lrt -lutil -lm -pthread -Wl,--export-dynamic -lgmodule-2.0
> > c++ -lglib-2.0 -lstdc++ -Wl,--end-group
> > /usr/bin/ld: libcommon.fa.p/cpus-common.c.o: in function
> `do_run_on_cpu':
> > /root/src/qemu/build/../cpus-common.c:153: undefined reference to
> `qemu_cond_wait_iothread'
> > collect2: error: ld returned 1 exit status [698/973] Compiling C
> > object
> > tests/fp/libsoftfloat.a.p/berkeley-softfloat-3_source_f32_to_ui64_r_mi
> > nMag.c.o [699/973] Compiling C object
> > tests/fp/libsoftfloat.a.p/berkeley-softfloat-3_source_f32_to_i32_r_min
> > Mag.c.o [700/973] Compiling C object
> > tests/fp/libsoftfloat.a.p/berkeley-softfloat-3_source_f32_to_f16.c.o
> > [701/973] Compiling C object
> > tests/fp/libsoftfloat.a.p/berkeley-softfloat-3_source_f32_to_f64.c.o
> > [702/973] Compiling C object
> > tests/fp/libsoftfloat.a.p/berkeley-softfloat-3_source_f32_to_i64_r_min
> > Mag.c.o [703/973] Compiling C object
> > tests/fp/libsoftfloat.a.p/berkeley-softfloat-3_source_f32_to_extF80M.c
> > .o [704/973] Compiling C object
> > tests/fp/libsoftfloat.a.p/berkeley-softfloat-3_source_f32_to_extF80.c.
> > o
> > ninja: build stopped: subcommand failed.
> > make[1]: *** [Makefile:154: run-ninja] Error 1
> > make[1]: Leaving directory '/root/src/qemu/build'
> > make: *** [GNUmakefile:11: all] Error 2
> 
> So it fails linux-user...  I can fix the compilation, but it should pass x86_64-
> softmmu. More importantly - are you using linux-user binaries?  The thing is
> my branch will only be helpful to debug BQL related issues, so if that's the
> case then please ignore the branch as linux-user shouldn't be using bql, then
> my branch won't help.

Doesn't matter, I can use x86_64-softmmu.

> 
> >
> > Regarding the stack trace, I can very easily reproduce it on our branch, I
> know exactly where to set the breakpoint:
> >
> > (gdb) r
> > Starting prThread 0x7fffeffff7 In: __pthread_cond_waitu host -enable-kvm
> -smp 4 -nographic -m 2G -object memory-backend-
> file,id=mem0,size=2G,mem-path=/dev/hugepages,share=on,prealloc=yes, -
> numa node,memdev=mem0 -L88   PC: 0x7ffff772700cuThread 8 "qemu-
> system-x86" received signal SIGUSR1, User defined signal 1.
> >                         f58c1        GI_raise
> 50               58f7bb
> > #0  0x00007ffff758f7bb in __GI_raise (sig=sig@entry=6) at
> > ../sysdeps/unix/sysv/linux/raise.c:50
> > #1  0x00007ffff757a535 in __GI_abort () at abort.c:79
> > #2  0x0000555555c9301e in kvm_set_phys_mem (kml=0x5555568ee830,
> > section=0x7ffff58c05e0, add=true) at ../accel/kvm/kvm-all.c:1194
> > #3  0x0000555555c930cd in kvm_region_add (listener=0x5555568ee830,
> > section=0x7ffff58c05e0) at ../accel/kvm/kvm-all.c:1211
> > #4  0x0000555555bd6c9e in address_space_update_topology_pass
> > (as=0x555556648420 <address_space_memory>,
> old_view=0x555556f21730,
> > new_view=0x7ffff0001cb0, adding=true) at ../softmmu/memory.c:971
> > #5  0x0000555555bd6f98 in address_space_set_flatview
> > (as=0x555556648420 <address_space_memory>) at
> ../softmmu/memory.c:1047
> > #6  0x0000555555bd713f in memory_region_transaction_commit () at
> > ../softmmu/memory.c:1099
> > #7  0x0000555555bd89a5 in memory_region_finalize (obj=0x555556e21800)
> > at ../softmmu/memory.c:1751
> > #8  0x0000555555cca132 in object_deinit (obj=0x555556e21800,
> > type=0x5555566a8f80) at ../qom/object.c:673
> > #9  0x0000555555cca1a4 in object_finalize (data=0x555556e21800) at
> > ../qom/object.c:687
> > #10 0x0000555555ccb196 in object_unref (objptr=0x555556e21800) at
> > ../qom/object.c:1186
> > #11 0x0000555555bb11f0 in phys_section_destroy (mr=0x555556e21800) at
> > ../softmmu/physmem.c:1171
> > #12 0x0000555555bb124a in phys_sections_free (map=0x5555572cf9a0) at
> > ../softmmu/physmem.c:1180
> > #13 0x0000555555bb4632 in address_space_dispatch_free
> > (d=0x5555572cf990) at ../softmmu/physmem.c:2562
> > #14 0x0000555555bd4485 in flatview_destroy (view=0x5555572cf950) at
> > ../softmmu/memory.c:291
> > #15 0x0000555555e367e8 in call_rcu_thread (opaque=0x0) at
> > ../util/rcu.c:281
> > #16 0x0000555555e68e57 in qemu_thread_start (args=0x555556665e30) at
> > ../util/qemu-thread-posix.c:521
> > #17 0x00007ffff7720fa3 in start_thread (arg=<optimized out>) at
> > pthread_create.c:486lot=10, start=0xfebd1000, size=0x1000: File exists
> > #18 0x00007ffff76514cf in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
> 
> Yes indeed it looks alike.
> 
> --
> Peter Xu

I can trivially trigger an assertion with a build where I merged the recent vfio-user patches (https://patchew.org/QEMU/cover.1626675354.git.elena.ufimtseva@oracle.com/) to master and then merging the result into your xzpeter/memory-sanity branch, I've pushed the branch here: https://github.com/tmakatos/qemu/tree/memory-sanity. I explain the repro steps below in case you want to take a look:

Build as follows:

./configure --prefix=/opt/qemu-xzpeter --target-list=x86_64-softmmu --enable-kvm  --enable-debug --enable-multiprocess && make -j `nproc` && make install

Then build and run the GPIO sample from libvfio-user (https://github.com/nutanix/libvfio-user):

libvfio-user/build/dbg/samples/gpio-pci-idio-16 -v /var/run/vfio-user.sock

And then run QEMU as follows:

gdb --args /opt/qemu-xzpeter/bin/qemu-system-x86_64 -cpu host -enable-kvm -smp 4 -m 2G -object memory-backend-file,id=mem0,size=2G,mem-path=/dev/hugepages,share=on,prealloc=yes -numa node,memdev=mem0 -kernel bionic-server-cloudimg-amd64-vmlinuz-generic -initrd bionic-server-cloudimg-amd64-initrd-generic -append 'console=ttyS0 root=/dev/sda1 single' -hda bionic-server-cloudimg-amd64-0.raw -nic user,model=virtio-net-pci -machine pc-q35-3.1 -device vfio-user-pci,socket=/var/run/vfio-user.sock -nographic

I immediately get the following stack trace:

Thread 5 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7fffe6e82700 (LWP 151973)]
__lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
103     ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) bt
#0  0x00007ffff655d29c in __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1  0x00007ffff6558642 in __pthread_mutex_cond_lock (mutex=mutex@entry=0x5555568bb280 <qemu_global_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007ffff6559ef8 in __pthread_cond_wait_common (abstime=0x0, mutex=0x5555568bb280 <qemu_global_mutex>, cond=0x555556cecc30) at pthread_cond_wait.c:645
#3  0x00007ffff6559ef8 in __pthread_cond_wait (cond=0x555556cecc30, mutex=0x5555568bb280 <qemu_global_mutex>) at pthread_cond_wait.c:655
#4  0x000055555604f717 in qemu_cond_wait_impl (cond=0x555556cecc30, mutex=0x5555568bb280 <qemu_global_mutex>, file=0x5555561ca869 "../softmmu/cpus.c", line=514) at ../util/qemu-thread-posix.c:194
#5  0x0000555555d28a4a in qemu_cond_wait_iothread (cond=0x555556cecc30) at ../softmmu/cpus.c:514
#6  0x0000555555d28781 in qemu_wait_io_event (cpu=0x555556ce02c0) at ../softmmu/cpus.c:425
#7  0x0000555555e5da75 in kvm_vcpu_thread_fn (arg=0x555556ce02c0) at ../accel/kvm/kvm-accel-ops.c:54
#8  0x000055555604feed in qemu_thread_start (args=0x555556cecc70) at ../util/qemu-thread-posix.c:541
#9  0x00007ffff6553fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007ffff64824cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95