[Qemu-devel] Question about wrong ram-node0 reference

* [Qemu-devel] Question about wrong ram-node0 reference
@ 2019-05-25  3:35 liujunjie (A)
  2019-05-27  7:57 ` Igor Mammedov
  0 siblings, 1 reply; 3+ messages in thread
From: liujunjie (A) @ 2019-05-25  3:35 UTC (permalink / raw)
  To: pbonzini, afaerber, ehabkost, imammedo, qemu-devel
  Cc: fangying, wangxin (U), Huangweidong (C), Zhoujian (jay)

Hi, I have met a problem:

The QEMU version is 2.8.1, the virtual machine is configured with 1G huge pages, two NUMA nodes and four pass-through NVME SSDs.

After we started the VM, in addition to some QMP queries nothing more has been done, the QEMU aborted after some months later.
After that, the VM is restarted, and the problem does not reproduce yet.
And The backtrace of the RCU thread is as follows:
(gdb) bt
#0  0x00007fd2695f0197 in raise () from /usr/lib64/libc.so.6
#1  0x00007fd2695f1888 in abort () from /usr/lib64/libc.so.6
#2  0x00007fd2695e9206 in __assert_fail_base () from /usr/lib64/libc.so.6
#3  0x00007fd2695e92b2 in __assert_fail () from /usr/lib64/libc.so.6
#4  0x0000000000476a84 in memory_region_finalize (obj=<optimized out>)
    at /home/abuild/rpmbuild/BUILD/qemu-kvm-2.8.1/memory.c:1512
#5  0x0000000000763105 in object_deinit (obj=obj@entry=0x1dc1fd0,
    type=type@entry=0x1d065b0) at qom/object.c:448
#6  0x0000000000763153 in object_finalize (data=0x1dc1fd0) at qom/object.c:462
#7  0x00000000007627cc in object_property_del_all (obj=obj@entry=0x1dc1f70)
    at qom/object.c:399
#8  0x0000000000763148 in object_finalize (data=0x1dc1f70) at qom/object.c:461
#9  0x0000000000764426 in object_unref (obj=<optimized out>) at qom/object.c:897
#10 0x0000000000473b6b in memory_region_unref (mr=<optimized out>)
    at /home/abuild/rpmbuild/BUILD/qemu-kvm-2.8.1/memory.c:1560
#11 0x0000000000473bc7 in flatview_destroy (view=0x7fc188b9cb90)
    at /home/abuild/rpmbuild/BUILD/qemu-kvm-2.8.1/memory.c:289
#12 0x0000000000843be0 in call_rcu_thread (opaque=<optimized out>)
    at util/rcu.c:279
#13 0x00000000008325c2 in qemu_thread_start (args=args@entry=0x1d00810)
    at util/qemu_thread_posix.c:496
#14 0x00007fd269983dc5 in start_thread () from /usr/lib64/libpthread.so.0
#15 0x00007fd2696b27bd in clone () from /usr/lib64/libc.so.6

In this core, I found that the reference of "/objects/ram-node0"( the type of ram-node0 is struct "HostMemoryBackendFile") equals to 0 , while the reference of "/objects/ram-node1" equals to 129, more details can be seen at the end of this email.

I searched through the community, and found a case that had the same error report: https://mail.coreboot.org/pipermail/seabios/2017-September/011799.html
However, I did not configure pcie_pci_bridge. Besides, qemu aborted in device initialization phase in this case.

Also, I try to find out which can reference "/objects/ram-node0" so as to look for the one that may un reference improperly, most of them lie in the function of "render_memory_region" or "phys_section_add" when memory topology changes.
Later, the temporary flatviews are destroyed by RCU thread, so un reference happened and the backtrace is similar to the one shown above.
But I am not familiar with the detail of these process, it is hard to keep trace of these memory topology changes.

My question is:
How can ram-node0's reference comes down to 0 when the virtual machine is still running?

Maybe someone who is familiar with memory_region_ref or memory-backend-file can help me figure out.
Any idea is appreciated.

---
(gdb) p *((HostMemoryBackendFile *) 0x1dc1f70)
$24 = {parent_obj = {parent = {class = 0x1d70880, free = 0x7fd26a812580 <g_free>, properties = 0x1db7920, ref = 0, parent = 0x1da9710}, size = 68719476736, merge = true, dump = false, prealloc = true, force_prealloc = false, is_mapped = true, host_nodes = {1, 0, 0}, policy = HOST_MEM_POLICY_BIND, mr = {parent_obj = {class = 0x1d6d790, free = 0x0, properties = 0x1db79e0, ref = 0, parent = 0x0}, romd_mode = true, ram = true, subpage = false, readonly = false, rom_device = false, flush_coalesced_mmio = false, global_locking = true, dirty_log_mask = 0 '\000', ram_block = 0x1dc2960, owner = 0x1dc1f70, iommu_ops = 0x0, ops = 0xcb0fe0 <unassigned_mem_ops>, opaque = 0x0, container = 0x200d4c0, size = 0x00000000000000000000001000000000, addr = 0, destructor = 0x470800 <memory_region_destructor_ram>, align = 1073741824, terminates = true, ram_device = false, enabled = true, warning_printed = false, vga_logging_count = 0 '\000', alias = 0x0, alias_offset = 0, priority = 0, subregions = {tqh_first = 0x0, tqh_last = 0x1dc2078}, subregions_link = {tqe_next = 0x0, tqe_prev = 0x1dc2c68}, coalesced = {tqh_first = 0x0, tqh_last = 0x1dc2098}, name = 0x1dc27a0 "/objects/ram-node0", ioeventfd_nb = 0, ioeventfds = 0x0, iommu_notify = {lh_first = 0x0}, iommu_notify_flags = IOMMU_NOTIFIER_NONE}}, share = true, mem_path = 0x1dc2350 "/dev/hugepages/libvirt/qemu/118-instance-00025bf8"}

(gdb) p *((HostMemoryBackendFile *) 0x1dc2b50)
$205 = {parent_obj = {parent = {class = 0x1d70880, free = 0x7fd26a812580 <g_free>, properties = 0x1db7a40, ref = 129, parent = 0x1da9710}, size = 68719476736, merge = true, dump = false, prealloc = true, force_prealloc = false, is_mapped = true, host_nodes = {2, 0, 0}, policy = HOST_MEM_POLICY_BIND, mr = {parent_obj = {class = 0x1d6d790, free = 0x0, properties = 0x1db7aa0, ref = 1, parent = 0x1dc2b50}, romd_mode = true, ram = true, subpage = false, readonly = false, rom_device = false, flush_coalesced_mmio = false, global_locking = true, dirty_log_mask = 0 '\000', ram_block = 0x1dc3470, owner = 0x1dc2b50, iommu_ops = 0x0, ops = 0xcb0fe0 <unassigned_mem_ops>, opaque = 0x0, container = 0x200d4c0, size = 0x00000000000000000000001000000000, addr = 68719476736, destructor = 0x470800 <memory_region_destructor_ram>, align = 1073741824, terminates = true, ram_device = false, enabled = true, warning_printed = false, vga_logging_count = 0 '\000', alias = 0x0, alias_offset = 0, priority = 0, subregions = {tqh_first = 0x0, tqh_last = 0x1dc2c58}, subregions_link = {tqe_next = 0x1dc1fd0, tqe_prev = 0x200d568}, coalesced = {tqh_first = 0x0, tqh_last = 0x1dc2c78}, name = 0x1dc32b0 "/objects/ram-node1", ioeventfd_nb = 0, ioeventfds = 0x0, iommu_notify = {lh_first = 0x0}, iommu_notify_flags = IOMMU_NOTIFIER_NONE}}, share = true, mem_path = 0x1da8c40 "/dev/hugepages/libvirt/qemu/118-instance-00025bf8"}

Thanks,
Junjie Liu

^ permalink raw reply	[flat|nested] 3+ messages in thread