* null domains after xl destroy
@ 2017-04-11 5:25 Glenn Enright
2017-04-11 5:59 ` Juergen Gross
0 siblings, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-04-11 5:25 UTC (permalink / raw)
To: xen-devel
Hi all
We are seeing an odd issue with domu domains from xl destroy, under
recent 4.9 kernels a (null) domain is left behind.
This has occurred on a variety of hardware, with no obvious commonality.
4.4.55 does not show this behavior.
On my test machine I have the following packages installed under
centos6, from https://xen.crc.id.au/
~]# rpm -qa | grep xen
xen47-licenses-4.7.2-4.el6.x86_64
xen47-4.7.2-4.el6.x86_64
kernel-xen-4.9.21-1.el6xen.x86_64
xen47-ocaml-4.7.2-4.el6.x86_64
xen47-libs-4.7.2-4.el6.x86_64
xen47-libcacard-4.7.2-4.el6.x86_64
xen47-hypervisor-4.7.2-4.el6.x86_64
xen47-runtime-4.7.2-4.el6.x86_64
kernel-xen-firmware-4.9.21-1.el6xen.x86_64
I've also replicated the issue with 4.9.17 and 4.9.20
To replicate, on a cleanly booted dom0 with one pv VM, I run the
following on the VM
{
while true; do
dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
done
}
Then on the dom0 I do this sequence to reliably get a null domain. This
occurs with oxenstored and xenstored both.
{
xl sync 1
xl destroy 1
}
xl list then renders something like ...
(null) 1 4 4 --p--d
9.8 0
From what I can see it appears to be disk related. Affected VMs all use
lvm storage for their boot disk. lvdisplay of the affected lv shows that
the lv has is being help open by something.
~]# lvdisplay test/test.img | grep open
# open 1
I've not been able to determine what that thing is as yet. I tried lsof,
dmsetup, various lv tools. Waiting for the disk to be released does not
work.
~]# xl list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 1512 2 r-----
29.0
(null) 1 4 4 --p--d
9.8
xenstore-ls reports nothing for the null domain id that I can see.
I can later start and restart the affected domain, but direct operations
on the lv such as removing it don't work since it is 'busy'.
Does anyone have thoughts on this? Happy to provide any output that
might be useful, not subscribed to this list so please CC.
Regards, Glenn
http://rimuhosting.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: null domains after xl destroy 2017-04-11 5:25 null domains after xl destroy Glenn Enright @ 2017-04-11 5:59 ` Juergen Gross 2017-04-11 8:03 ` Glenn Enright 0 siblings, 1 reply; 28+ messages in thread From: Juergen Gross @ 2017-04-11 5:59 UTC (permalink / raw) To: glenn, xen-devel On 11/04/17 07:25, Glenn Enright wrote: > Hi all > > We are seeing an odd issue with domu domains from xl destroy, under > recent 4.9 kernels a (null) domain is left behind. I guess this is the dom0 kernel version? > This has occurred on a variety of hardware, with no obvious commonality. > > 4.4.55 does not show this behavior. > > On my test machine I have the following packages installed under > centos6, from https://xen.crc.id.au/ > > ~]# rpm -qa | grep xen > xen47-licenses-4.7.2-4.el6.x86_64 > xen47-4.7.2-4.el6.x86_64 > kernel-xen-4.9.21-1.el6xen.x86_64 > xen47-ocaml-4.7.2-4.el6.x86_64 > xen47-libs-4.7.2-4.el6.x86_64 > xen47-libcacard-4.7.2-4.el6.x86_64 > xen47-hypervisor-4.7.2-4.el6.x86_64 > xen47-runtime-4.7.2-4.el6.x86_64 > kernel-xen-firmware-4.9.21-1.el6xen.x86_64 > > I've also replicated the issue with 4.9.17 and 4.9.20 > > To replicate, on a cleanly booted dom0 with one pv VM, I run the > following on the VM > > { > while true; do > dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync > done > } > > Then on the dom0 I do this sequence to reliably get a null domain. This > occurs with oxenstored and xenstored both. > > { > xl sync 1 > xl destroy 1 > } > > xl list then renders something like ... > > (null) 1 4 4 --p--d > 9.8 0 Something is referencing the domain, e.g. some of its memory pages are still mapped by dom0. > From what I can see it appears to be disk related. Affected VMs all use > lvm storage for their boot disk. lvdisplay of the affected lv shows that > the lv has is being help open by something. How are the disks configured? Especially the backend type is important. > > ~]# lvdisplay test/test.img | grep open > # open 1 > > I've not been able to determine what that thing is as yet. I tried lsof, > dmsetup, various lv tools. Waiting for the disk to be released does not > work. > > ~]# xl list > Name ID Mem VCPUs State > Time(s) > Domain-0 0 1512 2 r----- > 29.0 > (null) 1 4 4 --p--d > 9.8 > > xenstore-ls reports nothing for the null domain id that I can see. Any qemu process related to the domain still running? Any dom0 kernel messages related to Xen? Juergen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-11 5:59 ` Juergen Gross @ 2017-04-11 8:03 ` Glenn Enright 2017-04-11 9:49 ` Dietmar Hahn 0 siblings, 1 reply; 28+ messages in thread From: Glenn Enright @ 2017-04-11 8:03 UTC (permalink / raw) To: Juergen Gross, xen-devel On 11/04/17 17:59, Juergen Gross wrote: > On 11/04/17 07:25, Glenn Enright wrote: >> Hi all >> >> We are seeing an odd issue with domu domains from xl destroy, under >> recent 4.9 kernels a (null) domain is left behind. > > I guess this is the dom0 kernel version? > >> This has occurred on a variety of hardware, with no obvious commonality. >> >> 4.4.55 does not show this behavior. >> >> On my test machine I have the following packages installed under >> centos6, from https://xen.crc.id.au/ >> >> ~]# rpm -qa | grep xen >> xen47-licenses-4.7.2-4.el6.x86_64 >> xen47-4.7.2-4.el6.x86_64 >> kernel-xen-4.9.21-1.el6xen.x86_64 >> xen47-ocaml-4.7.2-4.el6.x86_64 >> xen47-libs-4.7.2-4.el6.x86_64 >> xen47-libcacard-4.7.2-4.el6.x86_64 >> xen47-hypervisor-4.7.2-4.el6.x86_64 >> xen47-runtime-4.7.2-4.el6.x86_64 >> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 >> >> I've also replicated the issue with 4.9.17 and 4.9.20 >> >> To replicate, on a cleanly booted dom0 with one pv VM, I run the >> following on the VM >> >> { >> while true; do >> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync >> done >> } >> >> Then on the dom0 I do this sequence to reliably get a null domain. This >> occurs with oxenstored and xenstored both. >> >> { >> xl sync 1 >> xl destroy 1 >> } >> >> xl list then renders something like ... >> >> (null) 1 4 4 --p--d >> 9.8 0 > > Something is referencing the domain, e.g. some of its memory pages are > still mapped by dom0. > >> From what I can see it appears to be disk related. Affected VMs all use >> lvm storage for their boot disk. lvdisplay of the affected lv shows that >> the lv has is being help open by something. > > How are the disks configured? Especially the backend type is important. > >> >> ~]# lvdisplay test/test.img | grep open >> # open 1 >> >> I've not been able to determine what that thing is as yet. I tried lsof, >> dmsetup, various lv tools. Waiting for the disk to be released does not >> work. >> >> ~]# xl list >> Name ID Mem VCPUs State >> Time(s) >> Domain-0 0 1512 2 r----- >> 29.0 >> (null) 1 4 4 --p--d >> 9.8 >> >> xenstore-ls reports nothing for the null domain id that I can see. > > Any qemu process related to the domain still running? > > Any dom0 kernel messages related to Xen? > > > Juergen > Yep, 4.9 dom0 kernel Typically we see an xl process running, but that has already gone away in this case. The domU is a PV guest using phy definition, the basic startup is like this... xl -v create -f paramfile extra="console=hvc0 elevator=noop xen-blkfront.max=64" There are no qemu processes or threads anywhere I can see. I dont see any meaningful messages in the linux kernel log, and nothing at all in the hypervisor log. Here is output from the dom0 starting and then stopping a domU using the above mechanism br0: port 2(vif3.0) entered disabled state br0: port 2(vif4.0) entered blocking state br0: port 2(vif4.0) entered disabled state device vif4.0 entered promiscuous mode IPv6: ADDRCONF(NETDEV_UP): vif4.0: link is not ready xen-blkback: backend/vbd/4/51713: using 2 queues, protocol 1 (x86_64-abi) persistent grants xen-blkback: backend/vbd/4/51721: using 2 queues, protocol 1 (x86_64-abi) persistent grants vif vif-4-0 vif4.0: Guest Rx ready IPv6: ADDRCONF(NETDEV_CHANGE): vif4.0: link becomes ready br0: port 2(vif4.0) entered blocking state br0: port 2(vif4.0) entered forwarding state br0: port 2(vif4.0) entered disabled state br0: port 2(vif4.0) entered disabled state device vif4.0 left promiscuous mode br0: port 2(vif4.0) entered disabled state ... here is xl info ... host : xxxxxxxxxxxx release : 4.9.21-1.el6xen.x86_64 version : #1 SMP Sat Apr 8 18:03:45 AEST 2017 machine : x86_64 nr_cpus : 4 max_cpu_id : 3 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 2394 hw_caps : b7ebfbff:0000e3bd:20100800:00000001:00000000:00000000:00000000:00000000 virt_caps : total_memory : 8190 free_memory : 6577 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 7 xen_extra : .2 xen_version : 4.7.2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : xen_commandline : dom0_mem=1512M cpufreq=xen dom0_max_vcpus=2 dom0_vcpus_pin log_lvl=all guest_loglvl=all vcpu_migration_delay=1000 cc_compiler : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17) cc_compile_by : mockbuild cc_compile_domain : (none) cc_compile_date : Mon Apr 3 12:17:20 AEST 2017 build_id : 0ec32d14d7c34e5d9deaaf6e3b7ea0c8006d68fa xend_config_format : 4 # cat /proc/cmdline ro root=UUID=xxxxxxxxxx rd_MD_UUID=xxxxxxxxxxxx rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_MD_UUID=xxxxxxxxxxxxx SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet pcie_aspm=off panic=30 max_loop=64 dm_mod.use_blk_mq=y xen-blkfront.max=64 The domu is using an lvm on top of a md raid1 array, on direct connected HDDs. Nothing special hardware wise. The disk line for that domU looks functionally like... disk = [ 'phy:/dev/testlv/test.img,xvda1,w' ] I would appreciate any suggestions on how to increase the debug level in a relevant way or where to look to get more useful information on what is happening. To clarify the actual shutdown sequence that causes problems... # xl sysrq $id s # xl destroy $id Regards, Glenn _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-11 8:03 ` Glenn Enright @ 2017-04-11 9:49 ` Dietmar Hahn 2017-04-11 22:13 ` Glenn Enright 0 siblings, 1 reply; 28+ messages in thread From: Dietmar Hahn @ 2017-04-11 9:49 UTC (permalink / raw) To: xen-devel, glenn; +Cc: Juergen Gross Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright: > On 11/04/17 17:59, Juergen Gross wrote: > > On 11/04/17 07:25, Glenn Enright wrote: > >> Hi all > >> > >> We are seeing an odd issue with domu domains from xl destroy, under > >> recent 4.9 kernels a (null) domain is left behind. > > > > I guess this is the dom0 kernel version? > > > >> This has occurred on a variety of hardware, with no obvious commonality. > >> > >> 4.4.55 does not show this behavior. > >> > >> On my test machine I have the following packages installed under > >> centos6, from https://xen.crc.id.au/ > >> > >> ~]# rpm -qa | grep xen > >> xen47-licenses-4.7.2-4.el6.x86_64 > >> xen47-4.7.2-4.el6.x86_64 > >> kernel-xen-4.9.21-1.el6xen.x86_64 > >> xen47-ocaml-4.7.2-4.el6.x86_64 > >> xen47-libs-4.7.2-4.el6.x86_64 > >> xen47-libcacard-4.7.2-4.el6.x86_64 > >> xen47-hypervisor-4.7.2-4.el6.x86_64 > >> xen47-runtime-4.7.2-4.el6.x86_64 > >> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 > >> > >> I've also replicated the issue with 4.9.17 and 4.9.20 > >> > >> To replicate, on a cleanly booted dom0 with one pv VM, I run the > >> following on the VM > >> > >> { > >> while true; do > >> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync > >> done > >> } > >> > >> Then on the dom0 I do this sequence to reliably get a null domain. This > >> occurs with oxenstored and xenstored both. > >> > >> { > >> xl sync 1 > >> xl destroy 1 > >> } > >> > >> xl list then renders something like ... > >> > >> (null) 1 4 4 --p--d > >> 9.8 0 > > > > Something is referencing the domain, e.g. some of its memory pages are > > still mapped by dom0. You can try # xl debug-keys q and further # xl dmesg to see the output of the previous command. The 'q' dumps domain (and guest debug) info. # xl debug-keys h prints all possible parameters for more informations. Dietmar. > > > >> From what I can see it appears to be disk related. Affected VMs all use > >> lvm storage for their boot disk. lvdisplay of the affected lv shows that > >> the lv has is being help open by something. > > > > How are the disks configured? Especially the backend type is important. > > > >> > >> ~]# lvdisplay test/test.img | grep open > >> # open 1 > >> > >> I've not been able to determine what that thing is as yet. I tried lsof, > >> dmsetup, various lv tools. Waiting for the disk to be released does not > >> work. > >> > >> ~]# xl list > >> Name ID Mem VCPUs State > >> Time(s) > >> Domain-0 0 1512 2 r----- > >> 29.0 > >> (null) 1 4 4 --p--d > >> 9.8 > >> > >> xenstore-ls reports nothing for the null domain id that I can see. > > > > Any qemu process related to the domain still running? > > > > Any dom0 kernel messages related to Xen? > > > > > > Juergen > > > > Yep, 4.9 dom0 kernel > > Typically we see an xl process running, but that has already gone away > in this case. The domU is a PV guest using phy definition, the basic > startup is like this... > > xl -v create -f paramfile extra="console=hvc0 elevator=noop > xen-blkfront.max=64" > > There are no qemu processes or threads anywhere I can see. > > I dont see any meaningful messages in the linux kernel log, and nothing > at all in the hypervisor log. Here is output from the dom0 starting and > then stopping a domU using the above mechanism > > br0: port 2(vif3.0) entered disabled state > br0: port 2(vif4.0) entered blocking state > br0: port 2(vif4.0) entered disabled state > device vif4.0 entered promiscuous mode > IPv6: ADDRCONF(NETDEV_UP): vif4.0: link is not ready > xen-blkback: backend/vbd/4/51713: using 2 queues, protocol 1 > (x86_64-abi) persistent grants > xen-blkback: backend/vbd/4/51721: using 2 queues, protocol 1 > (x86_64-abi) persistent grants > vif vif-4-0 vif4.0: Guest Rx ready > IPv6: ADDRCONF(NETDEV_CHANGE): vif4.0: link becomes ready > br0: port 2(vif4.0) entered blocking state > br0: port 2(vif4.0) entered forwarding state > br0: port 2(vif4.0) entered disabled state > br0: port 2(vif4.0) entered disabled state > device vif4.0 left promiscuous mode > br0: port 2(vif4.0) entered disabled state > > ... here is xl info ... > > host : xxxxxxxxxxxx > release : 4.9.21-1.el6xen.x86_64 > version : #1 SMP Sat Apr 8 18:03:45 AEST 2017 > machine : x86_64 > nr_cpus : 4 > max_cpu_id : 3 > nr_nodes : 1 > cores_per_socket : 4 > threads_per_core : 1 > cpu_mhz : 2394 > hw_caps : > b7ebfbff:0000e3bd:20100800:00000001:00000000:00000000:00000000:00000000 > virt_caps : > total_memory : 8190 > free_memory : 6577 > sharing_freed_memory : 0 > sharing_used_memory : 0 > outstanding_claims : 0 > free_cpus : 0 > xen_major : 4 > xen_minor : 7 > xen_extra : .2 > xen_version : 4.7.2 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : > xen_commandline : dom0_mem=1512M cpufreq=xen dom0_max_vcpus=2 > dom0_vcpus_pin log_lvl=all guest_loglvl=all vcpu_migration_delay=1000 > cc_compiler : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17) > cc_compile_by : mockbuild > cc_compile_domain : (none) > cc_compile_date : Mon Apr 3 12:17:20 AEST 2017 > build_id : 0ec32d14d7c34e5d9deaaf6e3b7ea0c8006d68fa > xend_config_format : 4 > > > # cat /proc/cmdline > ro root=UUID=xxxxxxxxxx rd_MD_UUID=xxxxxxxxxxxx rd_NO_LUKS > KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_MD_UUID=xxxxxxxxxxxxx > SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet > pcie_aspm=off panic=30 max_loop=64 dm_mod.use_blk_mq=y xen-blkfront.max=64 > > The domu is using an lvm on top of a md raid1 array, on direct connected > HDDs. Nothing special hardware wise. The disk line for that domU looks > functionally like... > > disk = [ 'phy:/dev/testlv/test.img,xvda1,w' ] > > I would appreciate any suggestions on how to increase the debug level in > a relevant way or where to look to get more useful information on what > is happening. > > To clarify the actual shutdown sequence that causes problems... > > # xl sysrq $id s > # xl destroy $id > > > Regards, Glenn > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > https://lists.xen.org/xen-devel -- Company details: http://ts.fujitsu.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-11 9:49 ` Dietmar Hahn @ 2017-04-11 22:13 ` Glenn Enright 2017-04-11 22:23 ` Andrew Cooper 0 siblings, 1 reply; 28+ messages in thread From: Glenn Enright @ 2017-04-11 22:13 UTC (permalink / raw) To: Dietmar Hahn, xen-devel; +Cc: Juergen Gross On 11/04/17 21:49, Dietmar Hahn wrote: > Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright: >> On 11/04/17 17:59, Juergen Gross wrote: >>> On 11/04/17 07:25, Glenn Enright wrote: >>>> Hi all >>>> >>>> We are seeing an odd issue with domu domains from xl destroy, under >>>> recent 4.9 kernels a (null) domain is left behind. >>> >>> I guess this is the dom0 kernel version? >>> >>>> This has occurred on a variety of hardware, with no obvious commonality. >>>> >>>> 4.4.55 does not show this behavior. >>>> >>>> On my test machine I have the following packages installed under >>>> centos6, from https://xen.crc.id.au/ >>>> >>>> ~]# rpm -qa | grep xen >>>> xen47-licenses-4.7.2-4.el6.x86_64 >>>> xen47-4.7.2-4.el6.x86_64 >>>> kernel-xen-4.9.21-1.el6xen.x86_64 >>>> xen47-ocaml-4.7.2-4.el6.x86_64 >>>> xen47-libs-4.7.2-4.el6.x86_64 >>>> xen47-libcacard-4.7.2-4.el6.x86_64 >>>> xen47-hypervisor-4.7.2-4.el6.x86_64 >>>> xen47-runtime-4.7.2-4.el6.x86_64 >>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 >>>> >>>> I've also replicated the issue with 4.9.17 and 4.9.20 >>>> >>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the >>>> following on the VM >>>> >>>> { >>>> while true; do >>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync >>>> done >>>> } >>>> >>>> Then on the dom0 I do this sequence to reliably get a null domain. This >>>> occurs with oxenstored and xenstored both. >>>> >>>> { >>>> xl sync 1 >>>> xl destroy 1 >>>> } >>>> >>>> xl list then renders something like ... >>>> >>>> (null) 1 4 4 --p--d >>>> 9.8 0 >>> >>> Something is referencing the domain, e.g. some of its memory pages are >>> still mapped by dom0. > > You can try > # xl debug-keys q > and further > # xl dmesg > to see the output of the previous command. The 'q' dumps domain > (and guest debug) info. > # xl debug-keys h > prints all possible parameters for more informations. > > Dietmar. > I've done this as requested, below is the output. (XEN) 'q' pressed -> dumping domain info (now=0x92:D6C271CE) (XEN) General information for domain 0: (XEN) refcnt=3 dying=0 pause_count=0 (XEN) nr_pages=387072 xenheap_pages=5 shared_pages=0 paged_pages=0 dirty_cpus={0-1} max_pages=4294967295 (XEN) handle=00000000-0000-0000-0000-000000000000 vm_assist=0000000d (XEN) Rangesets belonging to domain 0: (XEN) I/O Ports { 0-1f, 22-3f, 44-60, 62-9f, a2-cfb, d00-1007, 100c-ffff } (XEN) log-dirty { } (XEN) Interrupts { 1-30 } (XEN) I/O Memory { 0-fedff, fef00-ffffff } (XEN) Memory pages belonging to domain 0: (XEN) DomPage list too long to display (XEN) XenPage 000000000020e9c5: caf=c000000000000002, taf=7400000000000002 (XEN) XenPage 000000000020e9c4: caf=c000000000000001, taf=7400000000000001 (XEN) XenPage 000000000020e9c3: caf=c000000000000001, taf=7400000000000001 (XEN) XenPage 000000000020e9c2: caf=c000000000000001, taf=7400000000000001 (XEN) XenPage 00000000000e7d2e: caf=c000000000000002, taf=7400000000000002 (XEN) NODE affinity for domain 0: [0] (XEN) VCPU information and callbacks for domain 0: (XEN) VCPU0: CPU0 [has=T] poll=0 upcall_pend=01 upcall_mask=00 dirty_cpus={0} (XEN) cpu_hard_affinity={0} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) VCPU1: CPU1 [has=T] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={1} (XEN) cpu_hard_affinity={1} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) General information for domain 1: (XEN) refcnt=1 dying=2 pause_count=2 (XEN) nr_pages=2114 xenheap_pages=0 shared_pages=0 paged_pages=0 dirty_cpus={} max_pages=1280256 (XEN) handle=a481c2eb-31e3-4ae6-9809-290e746c8eec vm_assist=0000000d (XEN) Rangesets belonging to domain 1: (XEN) I/O Ports { } (XEN) log-dirty { } (XEN) Interrupts { } (XEN) I/O Memory { } (XEN) Memory pages belonging to domain 1: (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001 (XEN) NODE affinity for domain 1: [0] (XEN) VCPU information and callbacks for domain 1: (XEN) VCPU0: CPU0 [has=F] poll=0 upcall_pend=00 upcall_mask=01 dirty_cpus={} (XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) VCPU1: CPU1 [has=F] poll=0 upcall_pend=00 upcall_mask=01 dirty_cpus={} (XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) VCPU2: CPU2 [has=F] poll=0 upcall_pend=00 upcall_mask=01 dirty_cpus={} (XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) VCPU3: CPU3 [has=F] poll=0 upcall_pend=00 upcall_mask=01 dirty_cpus={} (XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) Notifying guest 0:0 (virq 1, port 4) (XEN) Notifying guest 0:1 (virq 1, port 10) (XEN) Notifying guest 1:0 (virq 1, port 0) (XEN) Notifying guest 1:1 (virq 1, port 0) (XEN) Notifying guest 1:2 (virq 1, port 0) (XEN) Notifying guest 1:3 (virq 1, port 0) (XEN) Shared frames 0 -- Saved frames 0 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-11 22:13 ` Glenn Enright @ 2017-04-11 22:23 ` Andrew Cooper 2017-04-11 22:45 ` Glenn Enright 0 siblings, 1 reply; 28+ messages in thread From: Andrew Cooper @ 2017-04-11 22:23 UTC (permalink / raw) To: glenn, Dietmar Hahn, xen-devel; +Cc: Juergen Gross On 11/04/2017 23:13, Glenn Enright wrote: > On 11/04/17 21:49, Dietmar Hahn wrote: >> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright: >>> On 11/04/17 17:59, Juergen Gross wrote: >>>> On 11/04/17 07:25, Glenn Enright wrote: >>>>> Hi all >>>>> >>>>> We are seeing an odd issue with domu domains from xl destroy, under >>>>> recent 4.9 kernels a (null) domain is left behind. >>>> >>>> I guess this is the dom0 kernel version? >>>> >>>>> This has occurred on a variety of hardware, with no obvious >>>>> commonality. >>>>> >>>>> 4.4.55 does not show this behavior. >>>>> >>>>> On my test machine I have the following packages installed under >>>>> centos6, from https://xen.crc.id.au/ >>>>> >>>>> ~]# rpm -qa | grep xen >>>>> xen47-licenses-4.7.2-4.el6.x86_64 >>>>> xen47-4.7.2-4.el6.x86_64 >>>>> kernel-xen-4.9.21-1.el6xen.x86_64 >>>>> xen47-ocaml-4.7.2-4.el6.x86_64 >>>>> xen47-libs-4.7.2-4.el6.x86_64 >>>>> xen47-libcacard-4.7.2-4.el6.x86_64 >>>>> xen47-hypervisor-4.7.2-4.el6.x86_64 >>>>> xen47-runtime-4.7.2-4.el6.x86_64 >>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 >>>>> >>>>> I've also replicated the issue with 4.9.17 and 4.9.20 >>>>> >>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the >>>>> following on the VM >>>>> >>>>> { >>>>> while true; do >>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync >>>>> done >>>>> } >>>>> >>>>> Then on the dom0 I do this sequence to reliably get a null domain. >>>>> This >>>>> occurs with oxenstored and xenstored both. >>>>> >>>>> { >>>>> xl sync 1 >>>>> xl destroy 1 >>>>> } >>>>> >>>>> xl list then renders something like ... >>>>> >>>>> (null) 1 4 4 --p--d >>>>> 9.8 0 >>>> >>>> Something is referencing the domain, e.g. some of its memory pages are >>>> still mapped by dom0. >> >> You can try >> # xl debug-keys q >> and further >> # xl dmesg >> to see the output of the previous command. The 'q' dumps domain >> (and guest debug) info. >> # xl debug-keys h >> prints all possible parameters for more informations. >> >> Dietmar. >> > > I've done this as requested, below is the output. > > <snip> > (XEN) Memory pages belonging to domain 1: > (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001 > (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001 There are 16 pages still referenced from somewhere. Can you grab all of `xenstore-ls -f` in dom0? There is probably a backend still attached. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-11 22:23 ` Andrew Cooper @ 2017-04-11 22:45 ` Glenn Enright 2017-04-18 8:36 ` Juergen Gross 0 siblings, 1 reply; 28+ messages in thread From: Glenn Enright @ 2017-04-11 22:45 UTC (permalink / raw) To: Andrew Cooper, Dietmar Hahn, xen-devel; +Cc: Juergen Gross On 12/04/17 10:23, Andrew Cooper wrote: > On 11/04/2017 23:13, Glenn Enright wrote: >> On 11/04/17 21:49, Dietmar Hahn wrote: >>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright: >>>> On 11/04/17 17:59, Juergen Gross wrote: >>>>> On 11/04/17 07:25, Glenn Enright wrote: >>>>>> Hi all >>>>>> >>>>>> We are seeing an odd issue with domu domains from xl destroy, under >>>>>> recent 4.9 kernels a (null) domain is left behind. >>>>> >>>>> I guess this is the dom0 kernel version? >>>>> >>>>>> This has occurred on a variety of hardware, with no obvious >>>>>> commonality. >>>>>> >>>>>> 4.4.55 does not show this behavior. >>>>>> >>>>>> On my test machine I have the following packages installed under >>>>>> centos6, from https://xen.crc.id.au/ >>>>>> >>>>>> ~]# rpm -qa | grep xen >>>>>> xen47-licenses-4.7.2-4.el6.x86_64 >>>>>> xen47-4.7.2-4.el6.x86_64 >>>>>> kernel-xen-4.9.21-1.el6xen.x86_64 >>>>>> xen47-ocaml-4.7.2-4.el6.x86_64 >>>>>> xen47-libs-4.7.2-4.el6.x86_64 >>>>>> xen47-libcacard-4.7.2-4.el6.x86_64 >>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64 >>>>>> xen47-runtime-4.7.2-4.el6.x86_64 >>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 >>>>>> >>>>>> I've also replicated the issue with 4.9.17 and 4.9.20 >>>>>> >>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the >>>>>> following on the VM >>>>>> >>>>>> { >>>>>> while true; do >>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync >>>>>> done >>>>>> } >>>>>> >>>>>> Then on the dom0 I do this sequence to reliably get a null domain. >>>>>> This >>>>>> occurs with oxenstored and xenstored both. >>>>>> >>>>>> { >>>>>> xl sync 1 >>>>>> xl destroy 1 >>>>>> } >>>>>> >>>>>> xl list then renders something like ... >>>>>> >>>>>> (null) 1 4 4 --p--d >>>>>> 9.8 0 >>>>> >>>>> Something is referencing the domain, e.g. some of its memory pages are >>>>> still mapped by dom0. >>> >>> You can try >>> # xl debug-keys q >>> and further >>> # xl dmesg >>> to see the output of the previous command. The 'q' dumps domain >>> (and guest debug) info. >>> # xl debug-keys h >>> prints all possible parameters for more informations. >>> >>> Dietmar. >>> >> >> I've done this as requested, below is the output. >> >> <snip> >> (XEN) Memory pages belonging to domain 1: >> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001 >> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001 > > There are 16 pages still referenced from somewhere. > > Can you grab all of `xenstore-ls -f` in dom0? There is probably a > backend still attached. > > ~Andrew > Note this is under oxenstored presently. # xenstore-ls -f /local = "" /local/domain = "" /local/domain/0 = "" /local/domain/0/control = "" /local/domain/0/control/feature-poweroff = "1" /local/domain/0/control/feature-reboot = "1" /local/domain/0/control/feature-suspend = "1" /local/domain/0/domid = "0" /local/domain/0/name = "Domain-0" /vm = "" /libxl = "" # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 1512 2 r----- 35.0 (null) 1 8 4 --p--d 8.5 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-11 22:45 ` Glenn Enright @ 2017-04-18 8:36 ` Juergen Gross 2017-04-19 1:02 ` Glenn Enright 0 siblings, 1 reply; 28+ messages in thread From: Juergen Gross @ 2017-04-18 8:36 UTC (permalink / raw) To: glenn, Andrew Cooper, Dietmar Hahn, xen-devel [-- Attachment #1: Type: text/plain, Size: 3823 bytes --] On 12/04/17 00:45, Glenn Enright wrote: > On 12/04/17 10:23, Andrew Cooper wrote: >> On 11/04/2017 23:13, Glenn Enright wrote: >>> On 11/04/17 21:49, Dietmar Hahn wrote: >>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright: >>>>> On 11/04/17 17:59, Juergen Gross wrote: >>>>>> On 11/04/17 07:25, Glenn Enright wrote: >>>>>>> Hi all >>>>>>> >>>>>>> We are seeing an odd issue with domu domains from xl destroy, under >>>>>>> recent 4.9 kernels a (null) domain is left behind. >>>>>> >>>>>> I guess this is the dom0 kernel version? >>>>>> >>>>>>> This has occurred on a variety of hardware, with no obvious >>>>>>> commonality. >>>>>>> >>>>>>> 4.4.55 does not show this behavior. >>>>>>> >>>>>>> On my test machine I have the following packages installed under >>>>>>> centos6, from https://xen.crc.id.au/ >>>>>>> >>>>>>> ~]# rpm -qa | grep xen >>>>>>> xen47-licenses-4.7.2-4.el6.x86_64 >>>>>>> xen47-4.7.2-4.el6.x86_64 >>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64 >>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64 >>>>>>> xen47-libs-4.7.2-4.el6.x86_64 >>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64 >>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64 >>>>>>> xen47-runtime-4.7.2-4.el6.x86_64 >>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 >>>>>>> >>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20 >>>>>>> >>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the >>>>>>> following on the VM >>>>>>> >>>>>>> { >>>>>>> while true; do >>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync >>>>>>> done >>>>>>> } >>>>>>> >>>>>>> Then on the dom0 I do this sequence to reliably get a null domain. >>>>>>> This >>>>>>> occurs with oxenstored and xenstored both. >>>>>>> >>>>>>> { >>>>>>> xl sync 1 >>>>>>> xl destroy 1 >>>>>>> } >>>>>>> >>>>>>> xl list then renders something like ... >>>>>>> >>>>>>> (null) 1 4 4 >>>>>>> --p--d >>>>>>> 9.8 0 >>>>>> >>>>>> Something is referencing the domain, e.g. some of its memory pages >>>>>> are >>>>>> still mapped by dom0. >>>> >>>> You can try >>>> # xl debug-keys q >>>> and further >>>> # xl dmesg >>>> to see the output of the previous command. The 'q' dumps domain >>>> (and guest debug) info. >>>> # xl debug-keys h >>>> prints all possible parameters for more informations. >>>> >>>> Dietmar. >>>> >>> >>> I've done this as requested, below is the output. >>> >>> <snip> >>> (XEN) Memory pages belonging to domain 1: >>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001 >>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001 >> >> There are 16 pages still referenced from somewhere. Just a wild guess: could you please try the attached kernel patch? This might give us some more diagnostic data... Juergen [-- Attachment #2: debug.patch --] [-- Type: text/x-patch, Size: 1426 bytes --] diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index 8fe61b5dc5a6..304d5d130e0c 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -313,7 +313,7 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif) static void xen_blkif_free(struct xen_blkif *blkif) { - xen_blkif_disconnect(blkif); + WARN_ON(xen_blkif_disconnect(blkif)); xen_vbd_free(&blkif->vbd); /* Make sure everything is drained before shutting down */ @@ -505,7 +505,7 @@ static int xen_blkbk_remove(struct xenbus_device *dev) dev_set_drvdata(&dev->dev, NULL); if (be->blkif) - xen_blkif_disconnect(be->blkif); + WARN_ON(xen_blkif_disconnect(be->blkif)); /* Put the reference we set in xen_blkif_alloc(). */ xen_blkif_put(be->blkif); @@ -792,7 +792,7 @@ static void frontend_changed(struct xenbus_device *dev, * Clean up so that memory resources can be used by * other devices. connect_ring reported already error. */ - xen_blkif_disconnect(be->blkif); + WARN_ON(xen_blkif_disconnect(be->blkif)); break; } xen_update_blkif_status(be->blkif); @@ -803,7 +803,7 @@ static void frontend_changed(struct xenbus_device *dev, break; case XenbusStateClosed: - xen_blkif_disconnect(be->blkif); + WARN_ON(xen_blkif_disconnect(be->blkif)); xenbus_switch_state(dev, XenbusStateClosed); if (xenbus_dev_is_online(dev)) break; [-- Attachment #3: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-18 8:36 ` Juergen Gross @ 2017-04-19 1:02 ` Glenn Enright 2017-04-19 4:39 ` Juergen Gross 0 siblings, 1 reply; 28+ messages in thread From: Glenn Enright @ 2017-04-19 1:02 UTC (permalink / raw) To: Juergen Gross, Andrew Cooper, Dietmar Hahn, xen-devel On 18/04/17 20:36, Juergen Gross wrote: > On 12/04/17 00:45, Glenn Enright wrote: >> On 12/04/17 10:23, Andrew Cooper wrote: >>> On 11/04/2017 23:13, Glenn Enright wrote: >>>> On 11/04/17 21:49, Dietmar Hahn wrote: >>>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright: >>>>>> On 11/04/17 17:59, Juergen Gross wrote: >>>>>>> On 11/04/17 07:25, Glenn Enright wrote: >>>>>>>> Hi all >>>>>>>> >>>>>>>> We are seeing an odd issue with domu domains from xl destroy, under >>>>>>>> recent 4.9 kernels a (null) domain is left behind. >>>>>>> >>>>>>> I guess this is the dom0 kernel version? >>>>>>> >>>>>>>> This has occurred on a variety of hardware, with no obvious >>>>>>>> commonality. >>>>>>>> >>>>>>>> 4.4.55 does not show this behavior. >>>>>>>> >>>>>>>> On my test machine I have the following packages installed under >>>>>>>> centos6, from https://xen.crc.id.au/ >>>>>>>> >>>>>>>> ~]# rpm -qa | grep xen >>>>>>>> xen47-licenses-4.7.2-4.el6.x86_64 >>>>>>>> xen47-4.7.2-4.el6.x86_64 >>>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64 >>>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64 >>>>>>>> xen47-libs-4.7.2-4.el6.x86_64 >>>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64 >>>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64 >>>>>>>> xen47-runtime-4.7.2-4.el6.x86_64 >>>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 >>>>>>>> >>>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20 >>>>>>>> >>>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the >>>>>>>> following on the VM >>>>>>>> >>>>>>>> { >>>>>>>> while true; do >>>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync >>>>>>>> done >>>>>>>> } >>>>>>>> >>>>>>>> Then on the dom0 I do this sequence to reliably get a null domain. >>>>>>>> This >>>>>>>> occurs with oxenstored and xenstored both. >>>>>>>> >>>>>>>> { >>>>>>>> xl sync 1 >>>>>>>> xl destroy 1 >>>>>>>> } >>>>>>>> >>>>>>>> xl list then renders something like ... >>>>>>>> >>>>>>>> (null) 1 4 4 >>>>>>>> --p--d >>>>>>>> 9.8 0 >>>>>>> >>>>>>> Something is referencing the domain, e.g. some of its memory pages >>>>>>> are >>>>>>> still mapped by dom0. >>>>> >>>>> You can try >>>>> # xl debug-keys q >>>>> and further >>>>> # xl dmesg >>>>> to see the output of the previous command. The 'q' dumps domain >>>>> (and guest debug) info. >>>>> # xl debug-keys h >>>>> prints all possible parameters for more informations. >>>>> >>>>> Dietmar. >>>>> >>>> >>>> I've done this as requested, below is the output. >>>> >>>> <snip> >>>> (XEN) Memory pages belonging to domain 1: >>>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001 >>>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001 >>> >>> There are 16 pages still referenced from somewhere. > > Just a wild guess: could you please try the attached kernel patch? This > might give us some more diagnostic data... > > > Juergen > Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still shows the issue. When replicating the leak I now see this trace (via dmesg). Hopefully that is useful. Please note, I'm going to be offline next week, but am keen to keep on with this, it may just be a while before I followup is all. Regards, Glenn http://rimuhosting.com ------------[ cut here ]------------ WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 xen_blkbk_remove+0x138/0x140 Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 Call Trace: [<ffffffff8136b61f>] dump_stack+0x67/0x98 [<ffffffff8108007d>] __warn+0xfd/0x120 [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 [<ffffffff814b93a2>] device_del+0x112/0x210 [<ffffffff81448113>] ? xenbus_read+0x53/0x70 [<ffffffff814b94c2>] device_unregister+0x22/0x60 [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 [<ffffffff81449fe0>] frontend_changed+0x10/0x20 [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 [<ffffffff810c0c5d>] ? complete+0x4d/0x60 [<ffffffff81447760>] ? split+0xf0/0xf0 [<ffffffff810a051d>] kthread+0xcd/0xf0 [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 ---[ end trace ee097287c9865a62 ]--- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-19 1:02 ` Glenn Enright @ 2017-04-19 4:39 ` Juergen Gross 2017-04-19 7:16 ` Roger Pau Monné 0 siblings, 1 reply; 28+ messages in thread From: Juergen Gross @ 2017-04-19 4:39 UTC (permalink / raw) To: glenn, xen-devel, Konrad Rzeszutek Wilk, Roger Pau Monné Cc: Andrew Cooper, Dietmar Hahn On 19/04/17 03:02, Glenn Enright wrote: > On 18/04/17 20:36, Juergen Gross wrote: >> On 12/04/17 00:45, Glenn Enright wrote: >>> On 12/04/17 10:23, Andrew Cooper wrote: >>>> On 11/04/2017 23:13, Glenn Enright wrote: >>>>> On 11/04/17 21:49, Dietmar Hahn wrote: >>>>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright: >>>>>>> On 11/04/17 17:59, Juergen Gross wrote: >>>>>>>> On 11/04/17 07:25, Glenn Enright wrote: >>>>>>>>> Hi all >>>>>>>>> >>>>>>>>> We are seeing an odd issue with domu domains from xl destroy, >>>>>>>>> under >>>>>>>>> recent 4.9 kernels a (null) domain is left behind. >>>>>>>> >>>>>>>> I guess this is the dom0 kernel version? >>>>>>>> >>>>>>>>> This has occurred on a variety of hardware, with no obvious >>>>>>>>> commonality. >>>>>>>>> >>>>>>>>> 4.4.55 does not show this behavior. >>>>>>>>> >>>>>>>>> On my test machine I have the following packages installed under >>>>>>>>> centos6, from https://xen.crc.id.au/ >>>>>>>>> >>>>>>>>> ~]# rpm -qa | grep xen >>>>>>>>> xen47-licenses-4.7.2-4.el6.x86_64 >>>>>>>>> xen47-4.7.2-4.el6.x86_64 >>>>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64 >>>>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64 >>>>>>>>> xen47-libs-4.7.2-4.el6.x86_64 >>>>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64 >>>>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64 >>>>>>>>> xen47-runtime-4.7.2-4.el6.x86_64 >>>>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 >>>>>>>>> >>>>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20 >>>>>>>>> >>>>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the >>>>>>>>> following on the VM >>>>>>>>> >>>>>>>>> { >>>>>>>>> while true; do >>>>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync >>>>>>>>> done >>>>>>>>> } >>>>>>>>> >>>>>>>>> Then on the dom0 I do this sequence to reliably get a null domain. >>>>>>>>> This >>>>>>>>> occurs with oxenstored and xenstored both. >>>>>>>>> >>>>>>>>> { >>>>>>>>> xl sync 1 >>>>>>>>> xl destroy 1 >>>>>>>>> } >>>>>>>>> >>>>>>>>> xl list then renders something like ... >>>>>>>>> >>>>>>>>> (null) 1 4 4 >>>>>>>>> --p--d >>>>>>>>> 9.8 0 >>>>>>>> >>>>>>>> Something is referencing the domain, e.g. some of its memory pages >>>>>>>> are >>>>>>>> still mapped by dom0. >>>>>> >>>>>> You can try >>>>>> # xl debug-keys q >>>>>> and further >>>>>> # xl dmesg >>>>>> to see the output of the previous command. The 'q' dumps domain >>>>>> (and guest debug) info. >>>>>> # xl debug-keys h >>>>>> prints all possible parameters for more informations. >>>>>> >>>>>> Dietmar. >>>>>> >>>>> >>>>> I've done this as requested, below is the output. >>>>> >>>>> <snip> >>>>> (XEN) Memory pages belonging to domain 1: >>>>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001 >>>>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001 >>>> >>>> There are 16 pages still referenced from somewhere. >> >> Just a wild guess: could you please try the attached kernel patch? This >> might give us some more diagnostic data... >> >> >> Juergen >> > > Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still > shows the issue. When replicating the leak I now see this trace (via > dmesg). Hopefully that is useful. > > Please note, I'm going to be offline next week, but am keen to keep on > with this, it may just be a while before I followup is all. > > Regards, Glenn > http://rimuhosting.com > > > ------------[ cut here ]------------ > WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 > xen_blkbk_remove+0x138/0x140 > Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev > xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 > ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security > iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 > nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc > ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus > i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy > dm_mirror dm_region_hash dm_log dm_mod > CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 > Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 > ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 > 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d > ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 > Call Trace: > [<ffffffff8136b61f>] dump_stack+0x67/0x98 > [<ffffffff8108007d>] __warn+0xfd/0x120 > [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 > [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 > [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 > [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 > [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 > [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 > [<ffffffff814b93a2>] device_del+0x112/0x210 > [<ffffffff81448113>] ? xenbus_read+0x53/0x70 > [<ffffffff814b94c2>] device_unregister+0x22/0x60 > [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 > [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 > [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 > [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 > [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 > [<ffffffff81449fe0>] frontend_changed+0x10/0x20 > [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 > [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 > [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 > [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 > [<ffffffff810c0c5d>] ? complete+0x4d/0x60 > [<ffffffff81447760>] ? split+0xf0/0xf0 > [<ffffffff810a051d>] kthread+0xcd/0xf0 > [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 > [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 > [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 > [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 > ---[ end trace ee097287c9865a62 ]--- Konrad, Roger, this was triggered by a debug patch in xen_blkbk_remove(): if (be->blkif) - xen_blkif_disconnect(be->blkif); + WARN_ON(xen_blkif_disconnect(be->blkif)); So I guess we need something like xen_blk_drain_io() in case of calls to xen_blkif_disconnect() which are not allowed to fail (either at the call sites of xen_blkif_disconnect() or in this function depending on a new boolean parameter indicating it should wait for outstanding I/Os). I can try a patch, but I'd appreciate if you could confirm this wouldn't add further problems... Juergen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-19 4:39 ` Juergen Gross @ 2017-04-19 7:16 ` Roger Pau Monné 2017-04-19 7:35 ` Juergen Gross 2017-04-19 10:09 ` Juergen Gross 0 siblings, 2 replies; 28+ messages in thread From: Roger Pau Monné @ 2017-04-19 7:16 UTC (permalink / raw) To: Juergen Gross; +Cc: Andrew Cooper, xen-devel, Dietmar Hahn, glenn On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: > On 19/04/17 03:02, Glenn Enright wrote: > > On 18/04/17 20:36, Juergen Gross wrote: > >> On 12/04/17 00:45, Glenn Enright wrote: > >>> On 12/04/17 10:23, Andrew Cooper wrote: > >>>> On 11/04/2017 23:13, Glenn Enright wrote: > >>>>> On 11/04/17 21:49, Dietmar Hahn wrote: > >>>>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright: > >>>>>>> On 11/04/17 17:59, Juergen Gross wrote: > >>>>>>>> On 11/04/17 07:25, Glenn Enright wrote: > >>>>>>>>> Hi all > >>>>>>>>> > >>>>>>>>> We are seeing an odd issue with domu domains from xl destroy, > >>>>>>>>> under > >>>>>>>>> recent 4.9 kernels a (null) domain is left behind. > >>>>>>>> > >>>>>>>> I guess this is the dom0 kernel version? > >>>>>>>> > >>>>>>>>> This has occurred on a variety of hardware, with no obvious > >>>>>>>>> commonality. > >>>>>>>>> > >>>>>>>>> 4.4.55 does not show this behavior. > >>>>>>>>> > >>>>>>>>> On my test machine I have the following packages installed under > >>>>>>>>> centos6, from https://xen.crc.id.au/ > >>>>>>>>> > >>>>>>>>> ~]# rpm -qa | grep xen > >>>>>>>>> xen47-licenses-4.7.2-4.el6.x86_64 > >>>>>>>>> xen47-4.7.2-4.el6.x86_64 > >>>>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64 > >>>>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64 > >>>>>>>>> xen47-libs-4.7.2-4.el6.x86_64 > >>>>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64 > >>>>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64 > >>>>>>>>> xen47-runtime-4.7.2-4.el6.x86_64 > >>>>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 > >>>>>>>>> > >>>>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20 > >>>>>>>>> > >>>>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the > >>>>>>>>> following on the VM > >>>>>>>>> > >>>>>>>>> { > >>>>>>>>> while true; do > >>>>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync > >>>>>>>>> done > >>>>>>>>> } > >>>>>>>>> > >>>>>>>>> Then on the dom0 I do this sequence to reliably get a null domain. > >>>>>>>>> This > >>>>>>>>> occurs with oxenstored and xenstored both. > >>>>>>>>> > >>>>>>>>> { > >>>>>>>>> xl sync 1 > >>>>>>>>> xl destroy 1 > >>>>>>>>> } > >>>>>>>>> > >>>>>>>>> xl list then renders something like ... > >>>>>>>>> > >>>>>>>>> (null) 1 4 4 > >>>>>>>>> --p--d > >>>>>>>>> 9.8 0 > >>>>>>>> > >>>>>>>> Something is referencing the domain, e.g. some of its memory pages > >>>>>>>> are > >>>>>>>> still mapped by dom0. > >>>>>> > >>>>>> You can try > >>>>>> # xl debug-keys q > >>>>>> and further > >>>>>> # xl dmesg > >>>>>> to see the output of the previous command. The 'q' dumps domain > >>>>>> (and guest debug) info. > >>>>>> # xl debug-keys h > >>>>>> prints all possible parameters for more informations. > >>>>>> > >>>>>> Dietmar. > >>>>>> > >>>>> > >>>>> I've done this as requested, below is the output. > >>>>> > >>>>> <snip> > >>>>> (XEN) Memory pages belonging to domain 1: > >>>>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001 > >>>>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001 > >>>> > >>>> There are 16 pages still referenced from somewhere. > >> > >> Just a wild guess: could you please try the attached kernel patch? This > >> might give us some more diagnostic data... > >> > >> > >> Juergen > >> > > > > Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still > > shows the issue. When replicating the leak I now see this trace (via > > dmesg). Hopefully that is useful. > > > > Please note, I'm going to be offline next week, but am keen to keep on > > with this, it may just be a while before I followup is all. > > > > Regards, Glenn > > http://rimuhosting.com > > > > > > ------------[ cut here ]------------ > > WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 > > xen_blkbk_remove+0x138/0x140 > > Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev > > xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 > > ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security > > iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 > > nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc > > ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus > > i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy > > dm_mirror dm_region_hash dm_log dm_mod > > CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 > > Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 > > ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 > > 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d > > ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 > > Call Trace: > > [<ffffffff8136b61f>] dump_stack+0x67/0x98 > > [<ffffffff8108007d>] __warn+0xfd/0x120 > > [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 > > [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 > > [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 > > [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 > > [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 > > [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 > > [<ffffffff814b93a2>] device_del+0x112/0x210 > > [<ffffffff81448113>] ? xenbus_read+0x53/0x70 > > [<ffffffff814b94c2>] device_unregister+0x22/0x60 > > [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 > > [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 > > [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 > > [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 > > [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 > > [<ffffffff81449fe0>] frontend_changed+0x10/0x20 > > [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 > > [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 > > [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 > > [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 > > [<ffffffff810c0c5d>] ? complete+0x4d/0x60 > > [<ffffffff81447760>] ? split+0xf0/0xf0 > > [<ffffffff810a051d>] kthread+0xcd/0xf0 > > [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 > > [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 > > [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 > > [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 > > ---[ end trace ee097287c9865a62 ]--- > > Konrad, Roger, > > this was triggered by a debug patch in xen_blkbk_remove(): > > if (be->blkif) > - xen_blkif_disconnect(be->blkif); > + WARN_ON(xen_blkif_disconnect(be->blkif)); > > So I guess we need something like xen_blk_drain_io() in case of calls to > xen_blkif_disconnect() which are not allowed to fail (either at the call > sites of xen_blkif_disconnect() or in this function depending on a new > boolean parameter indicating it should wait for outstanding I/Os). > > I can try a patch, but I'd appreciate if you could confirm this wouldn't > add further problems... Hello, Thanks for debugging this, the easiest solution seems to be to replace the ring->inflight atomic_read check in xen_blkif_disconnect with a call to xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to prevent further issues like this one). Roger. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-19 7:16 ` Roger Pau Monné @ 2017-04-19 7:35 ` Juergen Gross 2017-04-19 10:09 ` Juergen Gross 1 sibling, 0 replies; 28+ messages in thread From: Juergen Gross @ 2017-04-19 7:35 UTC (permalink / raw) To: Roger Pau Monné; +Cc: Andrew Cooper, xen-devel, Dietmar Hahn, glenn On 19/04/17 09:16, Roger Pau Monné wrote: > On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: >> On 19/04/17 03:02, Glenn Enright wrote: >>> On 18/04/17 20:36, Juergen Gross wrote: >>>> On 12/04/17 00:45, Glenn Enright wrote: >>>>> On 12/04/17 10:23, Andrew Cooper wrote: >>>>>> On 11/04/2017 23:13, Glenn Enright wrote: >>>>>>> On 11/04/17 21:49, Dietmar Hahn wrote: >>>>>>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright: >>>>>>>>> On 11/04/17 17:59, Juergen Gross wrote: >>>>>>>>>> On 11/04/17 07:25, Glenn Enright wrote: >>>>>>>>>>> Hi all >>>>>>>>>>> >>>>>>>>>>> We are seeing an odd issue with domu domains from xl destroy, >>>>>>>>>>> under >>>>>>>>>>> recent 4.9 kernels a (null) domain is left behind. >>>>>>>>>> >>>>>>>>>> I guess this is the dom0 kernel version? >>>>>>>>>> >>>>>>>>>>> This has occurred on a variety of hardware, with no obvious >>>>>>>>>>> commonality. >>>>>>>>>>> >>>>>>>>>>> 4.4.55 does not show this behavior. >>>>>>>>>>> >>>>>>>>>>> On my test machine I have the following packages installed under >>>>>>>>>>> centos6, from https://xen.crc.id.au/ >>>>>>>>>>> >>>>>>>>>>> ~]# rpm -qa | grep xen >>>>>>>>>>> xen47-licenses-4.7.2-4.el6.x86_64 >>>>>>>>>>> xen47-4.7.2-4.el6.x86_64 >>>>>>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64 >>>>>>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64 >>>>>>>>>>> xen47-libs-4.7.2-4.el6.x86_64 >>>>>>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64 >>>>>>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64 >>>>>>>>>>> xen47-runtime-4.7.2-4.el6.x86_64 >>>>>>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64 >>>>>>>>>>> >>>>>>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20 >>>>>>>>>>> >>>>>>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the >>>>>>>>>>> following on the VM >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> while true; do >>>>>>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync >>>>>>>>>>> done >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> Then on the dom0 I do this sequence to reliably get a null domain. >>>>>>>>>>> This >>>>>>>>>>> occurs with oxenstored and xenstored both. >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> xl sync 1 >>>>>>>>>>> xl destroy 1 >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> xl list then renders something like ... >>>>>>>>>>> >>>>>>>>>>> (null) 1 4 4 >>>>>>>>>>> --p--d >>>>>>>>>>> 9.8 0 >>>>>>>>>> >>>>>>>>>> Something is referencing the domain, e.g. some of its memory pages >>>>>>>>>> are >>>>>>>>>> still mapped by dom0. >>>>>>>> >>>>>>>> You can try >>>>>>>> # xl debug-keys q >>>>>>>> and further >>>>>>>> # xl dmesg >>>>>>>> to see the output of the previous command. The 'q' dumps domain >>>>>>>> (and guest debug) info. >>>>>>>> # xl debug-keys h >>>>>>>> prints all possible parameters for more informations. >>>>>>>> >>>>>>>> Dietmar. >>>>>>>> >>>>>>> >>>>>>> I've done this as requested, below is the output. >>>>>>> >>>>>>> <snip> >>>>>>> (XEN) Memory pages belonging to domain 1: >>>>>>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001 >>>>>>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001 >>>>>> >>>>>> There are 16 pages still referenced from somewhere. >>>> >>>> Just a wild guess: could you please try the attached kernel patch? This >>>> might give us some more diagnostic data... >>>> >>>> >>>> Juergen >>>> >>> >>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still >>> shows the issue. When replicating the leak I now see this trace (via >>> dmesg). Hopefully that is useful. >>> >>> Please note, I'm going to be offline next week, but am keen to keep on >>> with this, it may just be a while before I followup is all. >>> >>> Regards, Glenn >>> http://rimuhosting.com >>> >>> >>> ------------[ cut here ]------------ >>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 >>> xen_blkbk_remove+0x138/0x140 >>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev >>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security >>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc >>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus >>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy >>> dm_mirror dm_region_hash dm_log dm_mod >>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 >>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 >>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d >>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 >>> Call Trace: >>> [<ffffffff8136b61f>] dump_stack+0x67/0x98 >>> [<ffffffff8108007d>] __warn+0xfd/0x120 >>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 >>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 >>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 >>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 >>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 >>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 >>> [<ffffffff814b93a2>] device_del+0x112/0x210 >>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70 >>> [<ffffffff814b94c2>] device_unregister+0x22/0x60 >>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 >>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 >>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20 >>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 >>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 >>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 >>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60 >>> [<ffffffff81447760>] ? split+0xf0/0xf0 >>> [<ffffffff810a051d>] kthread+0xcd/0xf0 >>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 >>> ---[ end trace ee097287c9865a62 ]--- >> >> Konrad, Roger, >> >> this was triggered by a debug patch in xen_blkbk_remove(): >> >> if (be->blkif) >> - xen_blkif_disconnect(be->blkif); >> + WARN_ON(xen_blkif_disconnect(be->blkif)); >> >> So I guess we need something like xen_blk_drain_io() in case of calls to >> xen_blkif_disconnect() which are not allowed to fail (either at the call >> sites of xen_blkif_disconnect() or in this function depending on a new >> boolean parameter indicating it should wait for outstanding I/Os). >> >> I can try a patch, but I'd appreciate if you could confirm this wouldn't >> add further problems... > > Hello, > > Thanks for debugging this, the easiest solution seems to be to replace the > ring->inflight atomic_read check in xen_blkif_disconnect with a call to > xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to > prevent further issues like this one). Nah, this isn't going to work. Or at least it won't work as it was designed to. :-) The main problem seems to be that xen_blkif_get/put() are used for multiple purposes: they shouldn't be used by xen_blkif_alloc_rings() and xen_blkif_disconnect() as they will prohibit xen_blkif_deferred_free() being called when an I/O is terminated and xen_blkif_put() is meant to free all remaining resources. I'll write a patch to correct this. Juergen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-19 7:16 ` Roger Pau Monné 2017-04-19 7:35 ` Juergen Gross @ 2017-04-19 10:09 ` Juergen Gross 2017-04-19 16:22 ` Steven Haigh 2017-05-01 0:55 ` Glenn Enright 1 sibling, 2 replies; 28+ messages in thread From: Juergen Gross @ 2017-04-19 10:09 UTC (permalink / raw) To: glenn; +Cc: Andrew Cooper, xen-devel, Dietmar Hahn, Roger Pau Monné [-- Attachment #1: Type: text/plain, Size: 4099 bytes --] On 19/04/17 09:16, Roger Pau Monné wrote: > On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: >> On 19/04/17 03:02, Glenn Enright wrote: >>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still >>> shows the issue. When replicating the leak I now see this trace (via >>> dmesg). Hopefully that is useful. >>> >>> Please note, I'm going to be offline next week, but am keen to keep on >>> with this, it may just be a while before I followup is all. >>> >>> Regards, Glenn >>> http://rimuhosting.com >>> >>> >>> ------------[ cut here ]------------ >>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 >>> xen_blkbk_remove+0x138/0x140 >>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev >>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security >>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc >>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus >>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy >>> dm_mirror dm_region_hash dm_log dm_mod >>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 >>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 >>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d >>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 >>> Call Trace: >>> [<ffffffff8136b61f>] dump_stack+0x67/0x98 >>> [<ffffffff8108007d>] __warn+0xfd/0x120 >>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 >>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 >>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 >>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 >>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 >>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 >>> [<ffffffff814b93a2>] device_del+0x112/0x210 >>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70 >>> [<ffffffff814b94c2>] device_unregister+0x22/0x60 >>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 >>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 >>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20 >>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 >>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 >>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 >>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60 >>> [<ffffffff81447760>] ? split+0xf0/0xf0 >>> [<ffffffff810a051d>] kthread+0xcd/0xf0 >>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 >>> ---[ end trace ee097287c9865a62 ]--- >> >> Konrad, Roger, >> >> this was triggered by a debug patch in xen_blkbk_remove(): >> >> if (be->blkif) >> - xen_blkif_disconnect(be->blkif); >> + WARN_ON(xen_blkif_disconnect(be->blkif)); >> >> So I guess we need something like xen_blk_drain_io() in case of calls to >> xen_blkif_disconnect() which are not allowed to fail (either at the call >> sites of xen_blkif_disconnect() or in this function depending on a new >> boolean parameter indicating it should wait for outstanding I/Os). >> >> I can try a patch, but I'd appreciate if you could confirm this wouldn't >> add further problems... > > Hello, > > Thanks for debugging this, the easiest solution seems to be to replace the > ring->inflight atomic_read check in xen_blkif_disconnect with a call to > xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to > prevent further issues like this one). Glenn, can you please try the attached patch (in dom0)? Juergen [-- Attachment #2: blkback.patch --] [-- Type: text/x-patch, Size: 2698 bytes --] commit fd4252549f5f3e4de6d887d9ae4c4d7f35d7de52 Author: Juergen Gross <jgross@suse.com> Date: Wed Apr 19 11:19:51 2017 +0200 xen/blkback: fix disconnect while I/Os in flight Today disconnecting xen-blkback is broken in case there are still I/Os in flight: xen_blkif_disconnect() will bail out early without releasing all resources in the hope it will be called again when the last request has terminated. This, however, won't happen as xen_blkif_free() won't be called on termination of the last running request: xen_blkif_put() won't decrement the blkif refcnt to 0 as xen_blkif_disconnect() didn't finish before thus some xen_blkif_put() calls in xen_blkif_disconnect() didn't happen. To solve this deadlock xen_blkif_disconnect() and xen_blkif_alloc_rings() shouldn't use xen_blkif_put() and xen_blkif_get() but use some other way to do their accounting of resources. This at once fixes another error in xen_blkif_disconnect(): when it returned early with -EBUSY for another ring than 0 it would call xen_blkif_put() again for already handled rings on a subsequent call. This will lead to inconsistencies in the refcnt handling. Cc: stable@vger.kernel.org diff --git a/drivers/block/xen-blkback/common.h b/drivers/block/xen-blkback/common.h index dea61f6ab8cb..953f38802333 100644 --- a/drivers/block/xen-blkback/common.h +++ b/drivers/block/xen-blkback/common.h @@ -281,6 +281,7 @@ struct xen_blkif_ring { wait_queue_head_t wq; atomic_t inflight; + int active; /* One thread per blkif ring. */ struct task_struct *xenblkd; unsigned int waiting_reqs; diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index 8fe61b5dc5a6..411d2ded2456 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -159,7 +159,7 @@ static int xen_blkif_alloc_rings(struct xen_blkif *blkif) init_waitqueue_head(&ring->shutdown_wq); ring->blkif = blkif; ring->st_print = jiffies; - xen_blkif_get(blkif); + ring->active = 1; } return 0; @@ -249,6 +249,9 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif) struct xen_blkif_ring *ring = &blkif->rings[r]; unsigned int i = 0; + if (!ring->active) + continue; + if (ring->xenblkd) { kthread_stop(ring->xenblkd); wake_up(&ring->shutdown_wq); @@ -296,7 +299,7 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif) BUG_ON(ring->free_pages_num != 0); BUG_ON(ring->persistent_gnt_c != 0); WARN_ON(i != (XEN_BLKIF_REQS_PER_PAGE * blkif->nr_ring_pages)); - xen_blkif_put(blkif); + ring->active = 0; } blkif->nr_ring_pages = 0; /* [-- Attachment #3: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-19 10:09 ` Juergen Gross @ 2017-04-19 16:22 ` Steven Haigh 2017-04-21 8:42 ` Steven Haigh 2017-05-01 0:55 ` Glenn Enright 1 sibling, 1 reply; 28+ messages in thread From: Steven Haigh @ 2017-04-19 16:22 UTC (permalink / raw) To: Juergen Gross, glenn Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel [-- Attachment #1.1.1: Type: text/plain, Size: 4754 bytes --] On 19/04/17 20:09, Juergen Gross wrote: > On 19/04/17 09:16, Roger Pau Monné wrote: >> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: >>> On 19/04/17 03:02, Glenn Enright wrote: >>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still >>>> shows the issue. When replicating the leak I now see this trace (via >>>> dmesg). Hopefully that is useful. >>>> >>>> Please note, I'm going to be offline next week, but am keen to keep on >>>> with this, it may just be a while before I followup is all. >>>> >>>> Regards, Glenn >>>> http://rimuhosting.com >>>> >>>> >>>> ------------[ cut here ]------------ >>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 >>>> xen_blkbk_remove+0x138/0x140 >>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev >>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security >>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc >>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus >>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy >>>> dm_mirror dm_region_hash dm_log dm_mod >>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 >>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 >>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d >>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 >>>> Call Trace: >>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98 >>>> [<ffffffff8108007d>] __warn+0xfd/0x120 >>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 >>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 >>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 >>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 >>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 >>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 >>>> [<ffffffff814b93a2>] device_del+0x112/0x210 >>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70 >>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60 >>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 >>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 >>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20 >>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 >>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 >>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 >>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60 >>>> [<ffffffff81447760>] ? split+0xf0/0xf0 >>>> [<ffffffff810a051d>] kthread+0xcd/0xf0 >>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 >>>> ---[ end trace ee097287c9865a62 ]--- >>> >>> Konrad, Roger, >>> >>> this was triggered by a debug patch in xen_blkbk_remove(): >>> >>> if (be->blkif) >>> - xen_blkif_disconnect(be->blkif); >>> + WARN_ON(xen_blkif_disconnect(be->blkif)); >>> >>> So I guess we need something like xen_blk_drain_io() in case of calls to >>> xen_blkif_disconnect() which are not allowed to fail (either at the call >>> sites of xen_blkif_disconnect() or in this function depending on a new >>> boolean parameter indicating it should wait for outstanding I/Os). >>> >>> I can try a patch, but I'd appreciate if you could confirm this wouldn't >>> add further problems... >> >> Hello, >> >> Thanks for debugging this, the easiest solution seems to be to replace the >> ring->inflight atomic_read check in xen_blkif_disconnect with a call to >> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to >> prevent further issues like this one). > > Glenn, > > can you please try the attached patch (in dom0)? For what its worth, I have applied this in kernel package 4.9.23-2 as follows: * Wed Apr 19 2017 Steven Haigh <netwiz@crc.id.au> - 4.9.23-2 - xen/blkback: fix disconnect while I/Os in flight Its available from any 'in sync' mirror: https://xen.crc.id.au/downloads/ Feedback welcome for both mine and Juergen's sake. -- Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 [-- Attachment #1.2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] [-- Attachment #2: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-19 16:22 ` Steven Haigh @ 2017-04-21 8:42 ` Steven Haigh 2017-04-21 8:44 ` Juergen Gross 0 siblings, 1 reply; 28+ messages in thread From: Steven Haigh @ 2017-04-21 8:42 UTC (permalink / raw) To: Juergen Gross, glenn Cc: Andrew Cooper, xen-devel, Dietmar Hahn, Roger Pau Monné [-- Attachment #1.1.1: Type: text/plain, Size: 4853 bytes --] On 20/04/17 02:22, Steven Haigh wrote: > On 19/04/17 20:09, Juergen Gross wrote: >> On 19/04/17 09:16, Roger Pau Monné wrote: >>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: >>>> On 19/04/17 03:02, Glenn Enright wrote: >>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still >>>>> shows the issue. When replicating the leak I now see this trace (via >>>>> dmesg). Hopefully that is useful. >>>>> >>>>> Please note, I'm going to be offline next week, but am keen to keep on >>>>> with this, it may just be a while before I followup is all. >>>>> >>>>> Regards, Glenn >>>>> http://rimuhosting.com >>>>> >>>>> >>>>> ------------[ cut here ]------------ >>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 >>>>> xen_blkbk_remove+0x138/0x140 >>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev >>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security >>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc >>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus >>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy >>>>> dm_mirror dm_region_hash dm_log dm_mod >>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 >>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 >>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d >>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 >>>>> Call Trace: >>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98 >>>>> [<ffffffff8108007d>] __warn+0xfd/0x120 >>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 >>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 >>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 >>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 >>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 >>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 >>>>> [<ffffffff814b93a2>] device_del+0x112/0x210 >>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70 >>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60 >>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 >>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 >>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20 >>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 >>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 >>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 >>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60 >>>>> [<ffffffff81447760>] ? split+0xf0/0xf0 >>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0 >>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 >>>>> ---[ end trace ee097287c9865a62 ]--- >>>> >>>> Konrad, Roger, >>>> >>>> this was triggered by a debug patch in xen_blkbk_remove(): >>>> >>>> if (be->blkif) >>>> - xen_blkif_disconnect(be->blkif); >>>> + WARN_ON(xen_blkif_disconnect(be->blkif)); >>>> >>>> So I guess we need something like xen_blk_drain_io() in case of calls to >>>> xen_blkif_disconnect() which are not allowed to fail (either at the call >>>> sites of xen_blkif_disconnect() or in this function depending on a new >>>> boolean parameter indicating it should wait for outstanding I/Os). >>>> >>>> I can try a patch, but I'd appreciate if you could confirm this wouldn't >>>> add further problems... >>> >>> Hello, >>> >>> Thanks for debugging this, the easiest solution seems to be to replace the >>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to >>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to >>> prevent further issues like this one). >> >> Glenn, >> >> can you please try the attached patch (in dom0)? Tested-by: Steven Haigh <netwiz@crc.id.au> I've tried specifically with 4.9.23 and can no long make this occur in my scenario. Also built with 4.9.24 and expecting similar results. I'm aware Glenn has a much wider test schedule and number of systems than me, however my testing is successful. -- Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 [-- Attachment #1.2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] [-- Attachment #2: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-21 8:42 ` Steven Haigh @ 2017-04-21 8:44 ` Juergen Gross 0 siblings, 0 replies; 28+ messages in thread From: Juergen Gross @ 2017-04-21 8:44 UTC (permalink / raw) To: Steven Haigh, glenn Cc: Andrew Cooper, xen-devel, Dietmar Hahn, Roger Pau Monné On 21/04/17 10:42, Steven Haigh wrote: > On 20/04/17 02:22, Steven Haigh wrote: >> On 19/04/17 20:09, Juergen Gross wrote: >>> On 19/04/17 09:16, Roger Pau Monné wrote: >>>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: >>>>> On 19/04/17 03:02, Glenn Enright wrote: >>>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still >>>>>> shows the issue. When replicating the leak I now see this trace (via >>>>>> dmesg). Hopefully that is useful. >>>>>> >>>>>> Please note, I'm going to be offline next week, but am keen to keep on >>>>>> with this, it may just be a while before I followup is all. >>>>>> >>>>>> Regards, Glenn >>>>>> http://rimuhosting.com >>>>>> >>>>>> >>>>>> ------------[ cut here ]------------ >>>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 >>>>>> xen_blkbk_remove+0x138/0x140 >>>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev >>>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >>>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security >>>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >>>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc >>>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus >>>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy >>>>>> dm_mirror dm_region_hash dm_log dm_mod >>>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 >>>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >>>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 >>>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d >>>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 >>>>>> Call Trace: >>>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98 >>>>>> [<ffffffff8108007d>] __warn+0xfd/0x120 >>>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 >>>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 >>>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 >>>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 >>>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 >>>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 >>>>>> [<ffffffff814b93a2>] device_del+0x112/0x210 >>>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70 >>>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60 >>>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 >>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 >>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20 >>>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 >>>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 >>>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 >>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60 >>>>>> [<ffffffff81447760>] ? split+0xf0/0xf0 >>>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0 >>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 >>>>>> ---[ end trace ee097287c9865a62 ]--- >>>>> >>>>> Konrad, Roger, >>>>> >>>>> this was triggered by a debug patch in xen_blkbk_remove(): >>>>> >>>>> if (be->blkif) >>>>> - xen_blkif_disconnect(be->blkif); >>>>> + WARN_ON(xen_blkif_disconnect(be->blkif)); >>>>> >>>>> So I guess we need something like xen_blk_drain_io() in case of calls to >>>>> xen_blkif_disconnect() which are not allowed to fail (either at the call >>>>> sites of xen_blkif_disconnect() or in this function depending on a new >>>>> boolean parameter indicating it should wait for outstanding I/Os). >>>>> >>>>> I can try a patch, but I'd appreciate if you could confirm this wouldn't >>>>> add further problems... >>>> >>>> Hello, >>>> >>>> Thanks for debugging this, the easiest solution seems to be to replace the >>>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to >>>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to >>>> prevent further issues like this one). >>> >>> Glenn, >>> >>> can you please try the attached patch (in dom0)? > > Tested-by: Steven Haigh <netwiz@crc.id.au> > > I've tried specifically with 4.9.23 and can no long make this occur in > my scenario. Also built with 4.9.24 and expecting similar results. Thanks for testing! Will post the patch officially now. Juergen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-04-19 10:09 ` Juergen Gross 2017-04-19 16:22 ` Steven Haigh @ 2017-05-01 0:55 ` Glenn Enright 2017-05-03 10:45 ` Steven Haigh 1 sibling, 1 reply; 28+ messages in thread From: Glenn Enright @ 2017-05-01 0:55 UTC (permalink / raw) To: Juergen Gross Cc: Andrew Cooper, xen-devel, Dietmar Hahn, Roger Pau Monné On 19/04/17 22:09, Juergen Gross wrote: > On 19/04/17 09:16, Roger Pau Monné wrote: >> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: >>> On 19/04/17 03:02, Glenn Enright wrote: >>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still >>>> shows the issue. When replicating the leak I now see this trace (via >>>> dmesg). Hopefully that is useful. >>>> >>>> Please note, I'm going to be offline next week, but am keen to keep on >>>> with this, it may just be a while before I followup is all. >>>> >>>> Regards, Glenn >>>> http://rimuhosting.com >>>> >>>> >>>> ------------[ cut here ]------------ >>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 >>>> xen_blkbk_remove+0x138/0x140 >>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev >>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security >>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc >>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus >>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy >>>> dm_mirror dm_region_hash dm_log dm_mod >>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 >>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 >>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d >>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 >>>> Call Trace: >>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98 >>>> [<ffffffff8108007d>] __warn+0xfd/0x120 >>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 >>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 >>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 >>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 >>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 >>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 >>>> [<ffffffff814b93a2>] device_del+0x112/0x210 >>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70 >>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60 >>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 >>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 >>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20 >>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 >>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 >>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 >>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60 >>>> [<ffffffff81447760>] ? split+0xf0/0xf0 >>>> [<ffffffff810a051d>] kthread+0xcd/0xf0 >>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 >>>> ---[ end trace ee097287c9865a62 ]--- >>> >>> Konrad, Roger, >>> >>> this was triggered by a debug patch in xen_blkbk_remove(): >>> >>> if (be->blkif) >>> - xen_blkif_disconnect(be->blkif); >>> + WARN_ON(xen_blkif_disconnect(be->blkif)); >>> >>> So I guess we need something like xen_blk_drain_io() in case of calls to >>> xen_blkif_disconnect() which are not allowed to fail (either at the call >>> sites of xen_blkif_disconnect() or in this function depending on a new >>> boolean parameter indicating it should wait for outstanding I/Os). >>> >>> I can try a patch, but I'd appreciate if you could confirm this wouldn't >>> add further problems... >> >> Hello, >> >> Thanks for debugging this, the easiest solution seems to be to replace the >> ring->inflight atomic_read check in xen_blkif_disconnect with a call to >> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to >> prevent further issues like this one). > > Glenn, > > can you please try the attached patch (in dom0)? > > > Juergen > (resending with full CC list) I'm back. After testing unfortunately I'm still seeing the leak. The below trace is with the debug patch applied as well under 4.9.25. It looks very similar to me. I am still able to replicate this reliably. Regards, Glenn http://rimuhosting.com ------------[ cut here ]------------ WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:511 xen_blkbk_remove+0x138/0x140 Modules linked in: ebt_ip xen_pciback xen_netback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc ipv6 crc_ccitt ppdev parport_pc parport serio_raw i2c_i801 i2c_smbus i2c_core sg e1000e ptp pps_core i3000_edac edac_core raid1 sd_mod ahci libahci floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.25-1.el6xen.x86_64 #1 Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 ffffc90040cfbb98 ffffffff8136b76f 0000000000000013 0000000000000000 0000000000000000 0000000000000000 ffffc90040cfbbe8 ffffffff8108007d ffffea0000141720 000001ff41334434 ffff880000000001 ffff88004d3aedc0 Call Trace: [<ffffffff8136b76f>] dump_stack+0x67/0x98 [<ffffffff8108007d>] __warn+0xfd/0x120 [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 [<ffffffff814ec0a8>] xen_blkbk_remove+0x138/0x140 [<ffffffff81449b07>] xenbus_dev_remove+0x47/0xa0 [<ffffffff814bd2b4>] __device_release_driver+0xb4/0x160 [<ffffffff814bd38d>] device_release_driver+0x2d/0x40 [<ffffffff814bc2b4>] bus_remove_device+0x124/0x190 [<ffffffff814b9682>] device_del+0x112/0x210 [<ffffffff81448423>] ? xenbus_read+0x53/0x70 [<ffffffff814b97a2>] device_unregister+0x22/0x60 [<ffffffff814eda9d>] frontend_changed+0xad/0x4c0 [<ffffffff81449e67>] xenbus_otherend_changed+0xc7/0x140 [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20 [<ffffffff8144a2f0>] frontend_changed+0x10/0x20 [<ffffffff81447b0c>] xenwatch_thread+0x9c/0x140 [<ffffffff810bffb0>] ? woken_wake_function+0x20/0x20 [<ffffffff816ed98a>] ? schedule+0x3a/0xa0 [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20 [<ffffffff810c0c6d>] ? complete+0x4d/0x60 [<ffffffff81447a70>] ? split+0xf0/0xf0 [<ffffffff810a0535>] kthread+0xe5/0x100 [<ffffffff810a051d>] ? kthread+0xcd/0x100 [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 [<ffffffff816f1bc5>] ret_from_fork+0x25/0x30 ---[ end trace ea3a48c80e4ad79d ]--- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-01 0:55 ` Glenn Enright @ 2017-05-03 10:45 ` Steven Haigh 2017-05-03 13:38 ` Juergen Gross 2017-05-03 15:53 ` Juergen Gross 0 siblings, 2 replies; 28+ messages in thread From: Steven Haigh @ 2017-05-03 10:45 UTC (permalink / raw) To: glenn, Juergen Gross Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel [-- Attachment #1.1.1: Type: text/plain, Size: 7725 bytes --] Just wanted to give this a little nudge now people seem to be back on deck... On 01/05/17 10:55, Glenn Enright wrote: > On 19/04/17 22:09, Juergen Gross wrote: >> On 19/04/17 09:16, Roger Pau Monné wrote: >>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: >>>> On 19/04/17 03:02, Glenn Enright wrote: >>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still >>>>> shows the issue. When replicating the leak I now see this trace (via >>>>> dmesg). Hopefully that is useful. >>>>> >>>>> Please note, I'm going to be offline next week, but am keen to keep on >>>>> with this, it may just be a while before I followup is all. >>>>> >>>>> Regards, Glenn >>>>> http://rimuhosting.com >>>>> >>>>> >>>>> ------------[ cut here ]------------ >>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 >>>>> xen_blkbk_remove+0x138/0x140 >>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev >>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state >>>>> iptable_security >>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp >>>>> llc >>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 >>>>> i2c_smbus >>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci >>>>> floppy >>>>> dm_mirror dm_region_hash dm_log dm_mod >>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 >>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 >>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d >>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 >>>>> Call Trace: >>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98 >>>>> [<ffffffff8108007d>] __warn+0xfd/0x120 >>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 >>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 >>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 >>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 >>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 >>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 >>>>> [<ffffffff814b93a2>] device_del+0x112/0x210 >>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70 >>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60 >>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 >>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 >>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20 >>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 >>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 >>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 >>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60 >>>>> [<ffffffff81447760>] ? split+0xf0/0xf0 >>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0 >>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 >>>>> ---[ end trace ee097287c9865a62 ]--- >>>> >>>> Konrad, Roger, >>>> >>>> this was triggered by a debug patch in xen_blkbk_remove(): >>>> >>>> if (be->blkif) >>>> - xen_blkif_disconnect(be->blkif); >>>> + WARN_ON(xen_blkif_disconnect(be->blkif)); >>>> >>>> So I guess we need something like xen_blk_drain_io() in case of >>>> calls to >>>> xen_blkif_disconnect() which are not allowed to fail (either at the >>>> call >>>> sites of xen_blkif_disconnect() or in this function depending on a new >>>> boolean parameter indicating it should wait for outstanding I/Os). >>>> >>>> I can try a patch, but I'd appreciate if you could confirm this >>>> wouldn't >>>> add further problems... >>> >>> Hello, >>> >>> Thanks for debugging this, the easiest solution seems to be to >>> replace the >>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to >>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void >>> (to >>> prevent further issues like this one). >> >> Glenn, >> >> can you please try the attached patch (in dom0)? >> >> >> Juergen >> > > (resending with full CC list) > > I'm back. After testing unfortunately I'm still seeing the leak. The > below trace is with the debug patch applied as well under 4.9.25. It > looks very similar to me. I am still able to replicate this reliably. > > Regards, Glenn > http://rimuhosting.com > > ------------[ cut here ]------------ > WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:511 > xen_blkbk_remove+0x138/0x140 > Modules linked in: ebt_ip xen_pciback xen_netback xen_gntalloc > xen_gntdev xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 > ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security > iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 > nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc > ipv6 crc_ccitt ppdev parport_pc parport serio_raw i2c_i801 i2c_smbus > i2c_core sg e1000e ptp pps_core i3000_edac edac_core raid1 sd_mod ahci > libahci floppy dm_mirror dm_region_hash dm_log dm_mod > CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.25-1.el6xen.x86_64 #1 > Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 > ffffc90040cfbb98 ffffffff8136b76f 0000000000000013 0000000000000000 > 0000000000000000 0000000000000000 ffffc90040cfbbe8 ffffffff8108007d > ffffea0000141720 000001ff41334434 ffff880000000001 ffff88004d3aedc0 > Call Trace: > [<ffffffff8136b76f>] dump_stack+0x67/0x98 > [<ffffffff8108007d>] __warn+0xfd/0x120 > [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 > [<ffffffff814ec0a8>] xen_blkbk_remove+0x138/0x140 > [<ffffffff81449b07>] xenbus_dev_remove+0x47/0xa0 > [<ffffffff814bd2b4>] __device_release_driver+0xb4/0x160 > [<ffffffff814bd38d>] device_release_driver+0x2d/0x40 > [<ffffffff814bc2b4>] bus_remove_device+0x124/0x190 > [<ffffffff814b9682>] device_del+0x112/0x210 > [<ffffffff81448423>] ? xenbus_read+0x53/0x70 > [<ffffffff814b97a2>] device_unregister+0x22/0x60 > [<ffffffff814eda9d>] frontend_changed+0xad/0x4c0 > [<ffffffff81449e67>] xenbus_otherend_changed+0xc7/0x140 > [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20 > [<ffffffff8144a2f0>] frontend_changed+0x10/0x20 > [<ffffffff81447b0c>] xenwatch_thread+0x9c/0x140 > [<ffffffff810bffb0>] ? woken_wake_function+0x20/0x20 > [<ffffffff816ed98a>] ? schedule+0x3a/0xa0 > [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20 > [<ffffffff810c0c6d>] ? complete+0x4d/0x60 > [<ffffffff81447a70>] ? split+0xf0/0xf0 > [<ffffffff810a0535>] kthread+0xe5/0x100 > [<ffffffff810a051d>] ? kthread+0xcd/0x100 > [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 > [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 > [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 > [<ffffffff816f1bc5>] ret_from_fork+0x25/0x30 > ---[ end trace ea3a48c80e4ad79d ]--- > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > https://lists.xen.org/xen-devel -- Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 [-- Attachment #1.2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] [-- Attachment #2: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-03 10:45 ` Steven Haigh @ 2017-05-03 13:38 ` Juergen Gross 2017-05-03 15:53 ` Juergen Gross 1 sibling, 0 replies; 28+ messages in thread From: Juergen Gross @ 2017-05-03 13:38 UTC (permalink / raw) To: Steven Haigh, glenn Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel On 03/05/17 12:45, Steven Haigh wrote: > Just wanted to give this a little nudge now people seem to be back on > deck... Things seem to be more complicated than I thought. There are clearly paths leading to use-after-free scenarios, e.g. the one of the backtrace below: xen_blkbk_remove() will free be regardless of the return value of xen_blkif_disconnect(). If -EBUSY is returned xen_blkif_disconnect() will be called again at the end of an I/O still in progress, leading to a call of xenbus_unmap_ring_vfree(blkif->be->dev, ...) with be already freed... As Roger already said: this is a complete mess. Working on a patch now... BTW: Glenn, the debug patch isn't important any longer. It was just meant to locate the problem which is now known. Juergen > > On 01/05/17 10:55, Glenn Enright wrote: >> On 19/04/17 22:09, Juergen Gross wrote: >>> On 19/04/17 09:16, Roger Pau Monné wrote: >>>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: >>>>> On 19/04/17 03:02, Glenn Enright wrote: >>>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still >>>>>> shows the issue. When replicating the leak I now see this trace (via >>>>>> dmesg). Hopefully that is useful. >>>>>> >>>>>> Please note, I'm going to be offline next week, but am keen to keep on >>>>>> with this, it may just be a while before I followup is all. >>>>>> >>>>>> Regards, Glenn >>>>>> http://rimuhosting.com >>>>>> >>>>>> >>>>>> ------------[ cut here ]------------ >>>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 >>>>>> xen_blkbk_remove+0x138/0x140 >>>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev >>>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >>>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state >>>>>> iptable_security >>>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >>>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp >>>>>> llc >>>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 >>>>>> i2c_smbus >>>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci >>>>>> floppy >>>>>> dm_mirror dm_region_hash dm_log dm_mod >>>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1 >>>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >>>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000 >>>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d >>>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0 >>>>>> Call Trace: >>>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98 >>>>>> [<ffffffff8108007d>] __warn+0xfd/0x120 >>>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 >>>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140 >>>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0 >>>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160 >>>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40 >>>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190 >>>>>> [<ffffffff814b93a2>] device_del+0x112/0x210 >>>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70 >>>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60 >>>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0 >>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140 >>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20 >>>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140 >>>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20 >>>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0 >>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60 >>>>>> [<ffffffff81447760>] ? split+0xf0/0xf0 >>>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0 >>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0 >>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >>>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30 >>>>>> ---[ end trace ee097287c9865a62 ]--- >>>>> >>>>> Konrad, Roger, >>>>> >>>>> this was triggered by a debug patch in xen_blkbk_remove(): >>>>> >>>>> if (be->blkif) >>>>> - xen_blkif_disconnect(be->blkif); >>>>> + WARN_ON(xen_blkif_disconnect(be->blkif)); >>>>> >>>>> So I guess we need something like xen_blk_drain_io() in case of >>>>> calls to >>>>> xen_blkif_disconnect() which are not allowed to fail (either at the >>>>> call >>>>> sites of xen_blkif_disconnect() or in this function depending on a new >>>>> boolean parameter indicating it should wait for outstanding I/Os). >>>>> >>>>> I can try a patch, but I'd appreciate if you could confirm this >>>>> wouldn't >>>>> add further problems... >>>> >>>> Hello, >>>> >>>> Thanks for debugging this, the easiest solution seems to be to >>>> replace the >>>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to >>>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void >>>> (to >>>> prevent further issues like this one). >>> >>> Glenn, >>> >>> can you please try the attached patch (in dom0)? >>> >>> >>> Juergen >>> >> >> (resending with full CC list) >> >> I'm back. After testing unfortunately I'm still seeing the leak. The >> below trace is with the debug patch applied as well under 4.9.25. It >> looks very similar to me. I am still able to replicate this reliably. >> >> Regards, Glenn >> http://rimuhosting.com >> >> ------------[ cut here ]------------ >> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:511 >> xen_blkbk_remove+0x138/0x140 >> Modules linked in: ebt_ip xen_pciback xen_netback xen_gntalloc >> xen_gntdev xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security >> iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc >> ipv6 crc_ccitt ppdev parport_pc parport serio_raw i2c_i801 i2c_smbus >> i2c_core sg e1000e ptp pps_core i3000_edac edac_core raid1 sd_mod ahci >> libahci floppy dm_mirror dm_region_hash dm_log dm_mod >> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.25-1.el6xen.x86_64 #1 >> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >> ffffc90040cfbb98 ffffffff8136b76f 0000000000000013 0000000000000000 >> 0000000000000000 0000000000000000 ffffc90040cfbbe8 ffffffff8108007d >> ffffea0000141720 000001ff41334434 ffff880000000001 ffff88004d3aedc0 >> Call Trace: >> [<ffffffff8136b76f>] dump_stack+0x67/0x98 >> [<ffffffff8108007d>] __warn+0xfd/0x120 >> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20 >> [<ffffffff814ec0a8>] xen_blkbk_remove+0x138/0x140 >> [<ffffffff81449b07>] xenbus_dev_remove+0x47/0xa0 >> [<ffffffff814bd2b4>] __device_release_driver+0xb4/0x160 >> [<ffffffff814bd38d>] device_release_driver+0x2d/0x40 >> [<ffffffff814bc2b4>] bus_remove_device+0x124/0x190 >> [<ffffffff814b9682>] device_del+0x112/0x210 >> [<ffffffff81448423>] ? xenbus_read+0x53/0x70 >> [<ffffffff814b97a2>] device_unregister+0x22/0x60 >> [<ffffffff814eda9d>] frontend_changed+0xad/0x4c0 >> [<ffffffff81449e67>] xenbus_otherend_changed+0xc7/0x140 >> [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >> [<ffffffff8144a2f0>] frontend_changed+0x10/0x20 >> [<ffffffff81447b0c>] xenwatch_thread+0x9c/0x140 >> [<ffffffff810bffb0>] ? woken_wake_function+0x20/0x20 >> [<ffffffff816ed98a>] ? schedule+0x3a/0xa0 >> [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20 >> [<ffffffff810c0c6d>] ? complete+0x4d/0x60 >> [<ffffffff81447a70>] ? split+0xf0/0xf0 >> [<ffffffff810a0535>] kthread+0xe5/0x100 >> [<ffffffff810a051d>] ? kthread+0xcd/0x100 >> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40 >> [<ffffffff816f1bc5>] ret_from_fork+0x25/0x30 >> ---[ end trace ea3a48c80e4ad79d ]--- >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> https://lists.xen.org/xen-devel > _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-03 10:45 ` Steven Haigh 2017-05-03 13:38 ` Juergen Gross @ 2017-05-03 15:53 ` Juergen Gross 2017-05-03 16:58 ` Steven Haigh 1 sibling, 1 reply; 28+ messages in thread From: Juergen Gross @ 2017-05-03 15:53 UTC (permalink / raw) To: Steven Haigh, glenn Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel [-- Attachment #1: Type: text/plain, Size: 356 bytes --] On 03/05/17 12:45, Steven Haigh wrote: > Just wanted to give this a little nudge now people seem to be back on > deck... Glenn, could you please give the attached patch a try? It should be applied on top of the other correction, the old debug patch should not be applied. I have added some debug output to make sure we see what is happening. Juergen [-- Attachment #2: blk.patch --] [-- Type: text/x-patch, Size: 1969 bytes --] commit 246aaf60bd934b7571944b98a31078d519d637c6 Author: Juergen Gross <jgross@suse.com> Date: Wed May 3 15:57:18 2017 +0200 xen/blkback: don't free be structure too early The be structure must nor be freed when freeing the blkif structure isn't done. Otherwise a use-after-free of be when unmapping the ring used for communicating with the frontend will occur in case of a late call of xenblk_disconnect() (e.g. due to an I/O still active when trying to disconnect). diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index 411d2ded2456..0614fb294e2b 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -33,6 +33,7 @@ struct backend_info { unsigned major; unsigned minor; char *mode; + int delayed; }; static struct kmem_cache *xen_blkif_cachep; @@ -262,8 +263,11 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif) * don't have any discard_io or other_io requests. So, checking * for inflight IO is enough. */ - if (atomic_read(&ring->inflight) > 0) + if (atomic_read(&ring->inflight) > 0) { + pr_warn("xen_blkif_disconnect: busy\n"); + blkif->be->delayed = 1; return -EBUSY; + } if (ring->irq) { unbind_from_irqhandler(ring->irq, ring); @@ -315,9 +319,11 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif) static void xen_blkif_free(struct xen_blkif *blkif) { - - xen_blkif_disconnect(blkif); + pr_warn("xen_blkif_free: delayed = %d\n", blkif->be->delayed); + WARN_ON(xen_blkif_disconnect(blkif)); xen_vbd_free(&blkif->vbd); + kfree(blkif->be->mode); + kfree(blkif->be); /* Make sure everything is drained before shutting down */ kmem_cache_free(xen_blkif_cachep, blkif); @@ -512,8 +518,7 @@ static int xen_blkbk_remove(struct xenbus_device *dev) /* Put the reference we set in xen_blkif_alloc(). */ xen_blkif_put(be->blkif); - kfree(be->mode); - kfree(be); + return 0; } [-- Attachment #3: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-03 15:53 ` Juergen Gross @ 2017-05-03 16:58 ` Steven Haigh 2017-05-03 22:17 ` Glenn Enright 0 siblings, 1 reply; 28+ messages in thread From: Steven Haigh @ 2017-05-03 16:58 UTC (permalink / raw) To: Juergen Gross, glenn Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel [-- Attachment #1.1.1: Type: text/plain, Size: 613 bytes --] On 04/05/17 01:53, Juergen Gross wrote: > On 03/05/17 12:45, Steven Haigh wrote: >> Just wanted to give this a little nudge now people seem to be back on >> deck... > > Glenn, could you please give the attached patch a try? > > It should be applied on top of the other correction, the old debug > patch should not be applied. > > I have added some debug output to make sure we see what is happening. This patch is included in kernel-xen-4.9.26-1 It should be in the repos now. -- Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 [-- Attachment #1.2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] [-- Attachment #2: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-03 16:58 ` Steven Haigh @ 2017-05-03 22:17 ` Glenn Enright 2017-05-08 9:10 ` Juergen Gross 0 siblings, 1 reply; 28+ messages in thread From: Glenn Enright @ 2017-05-03 22:17 UTC (permalink / raw) To: Steven Haigh, Juergen Gross Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel On 04/05/17 04:58, Steven Haigh wrote: > On 04/05/17 01:53, Juergen Gross wrote: >> On 03/05/17 12:45, Steven Haigh wrote: >>> Just wanted to give this a little nudge now people seem to be back on >>> deck... >> >> Glenn, could you please give the attached patch a try? >> >> It should be applied on top of the other correction, the old debug >> patch should not be applied. >> >> I have added some debug output to make sure we see what is happening. > > This patch is included in kernel-xen-4.9.26-1 > > It should be in the repos now. > Still seeing the same issue. Without the extra debug patch all I see in the logs after destroy is this... xen-blkback: xen_blkif_disconnect: busy xen-blkback: xen_blkif_free: delayed = 0 br0: port 2(vif1.0) entered disabled state br0: port 2(vif1.0) entered disabled state device vif1.0 left promiscuous mode br0: port 2(vif1.0) entered disabled state Without the dd running in the domU, the domU exits cleanly on destroy and I don't see the busy message from above, eg xen-blkback: xen_blkif_free: delayed = 0 xen-blkback: xen_blkif_free: delayed = 0 br0: port 2(vif5.0) entered disabled state br0: port 2(vif5.0) entered disabled state device vif5.0 left promiscuous mode br0: port 2(vif5.0) entered disabled state Regards, Glenn http://rimuhosting.com { domid=1; xl sysrq $domid s; sleep 2; xl destroy $domid; } # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 1512 2 r----- 37.9 (null) 1 8 4 --p--d 9.5 # xl inf host : host480.rimuhosting.com release : 4.9.26-1.el6xen.x86_64 version : #1 SMP Thu May 4 02:07:34 AEST 2017 machine : x86_64 nr_cpus : 4 max_cpu_id : 3 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 2394 hw_caps : b7ebfbff:0000e3bd:20100800:00000001:00000000:00000000:00000000:00000000 virt_caps : total_memory : 8190 free_memory : 6573 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 7 xen_extra : .2 xen_version : 4.7.2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : xen_commandline : dom0_mem=1512M cpufreq=xen dom0_max_vcpus=2 dom0_vcpus_pin loglvl=debug vcpu_migration_delay=1000 cc_compiler : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18) cc_compile_by : mockbuild cc_compile_domain : xen.crc.id.au cc_compile_date : Wed Apr 19 18:17:37 AEST 2017 build_id : d5f616d8c9d8b5decfdbdca9d19eacb60c666418 xend_config_format : 4 (XEN) 'q' pressed -> dumping domain info (now=0x22E:795D0BA5) (XEN) General information for domain 0: (XEN) refcnt=3 dying=0 pause_count=0 (XEN) nr_pages=387072 xenheap_pages=5 shared_pages=0 paged_pages=0 dirty_cpus={0-1} max_pages=4294967295 (XEN) handle=00000000-0000-0000-0000-000000000000 vm_assist=0000000d (XEN) Rangesets belonging to domain 0: (XEN) I/O Ports { 0-1f, 22-3f, 44-60, 62-9f, a2-cfb, d00-1007, 100c-ffff } (XEN) log-dirty { } (XEN) Interrupts { 1-30 } (XEN) I/O Memory { 0-fedff, fef00-ffffff } (XEN) Memory pages belonging to domain 0: (XEN) DomPage list too long to display (XEN) XenPage 000000000020e9c4: caf=c000000000000002, taf=7400000000000002 (XEN) XenPage 000000000020e9c3: caf=c000000000000001, taf=7400000000000001 (XEN) XenPage 000000000020e9c2: caf=c000000000000001, taf=7400000000000001 (XEN) XenPage 000000000020e9c1: caf=c000000000000001, taf=7400000000000001 (XEN) XenPage 00000000000e7d2e: caf=c000000000000002, taf=7400000000000002 (XEN) NODE affinity for domain 0: [0] (XEN) VCPU information and callbacks for domain 0: (XEN) VCPU0: CPU0 [has=T] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={0} (XEN) cpu_hard_affinity={0} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) VCPU1: CPU1 [has=T] poll=0 upcall_pend=00 upcall_mask=01 dirty_cpus={1} (XEN) cpu_hard_affinity={1} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) General information for domain 1: (XEN) refcnt=1 dying=2 pause_count=2 (XEN) nr_pages=2113 xenheap_pages=0 shared_pages=0 paged_pages=0 dirty_cpus={} max_pages=1280256 (XEN) handle=7a5642fc-2372-4174-9508-aae2ad1f6908 vm_assist=0000000d (XEN) Rangesets belonging to domain 1: (XEN) I/O Ports { } (XEN) log-dirty { } (XEN) Interrupts { } (XEN) I/O Memory { } (XEN) Memory pages belonging to domain 1: (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001 (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001 (XEN) NODE affinity for domain 1: [0] (XEN) VCPU information and callbacks for domain 1: (XEN) VCPU0: CPU0 [has=F] poll=0 upcall_pend=00 upcall_mask=01 dirty_cpus={} (XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) VCPU1: CPU1 [has=F] poll=0 upcall_pend=00 upcall_mask=01 dirty_cpus={} (XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) VCPU2: CPU2 [has=F] poll=0 upcall_pend=00 upcall_mask=01 dirty_cpus={} (XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) VCPU3: CPU3 [has=F] poll=0 upcall_pend=00 upcall_mask=01 dirty_cpus={} (XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3} (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) Notifying guest 0:0 (virq 1, port 4) (XEN) Notifying guest 0:1 (virq 1, port 10) (XEN) Notifying guest 1:0 (virq 1, port 0) (XEN) Notifying guest 1:1 (virq 1, port 0) (XEN) Notifying guest 1:2 (virq 1, port 0) (XEN) Notifying guest 1:3 (virq 1, port 0) (XEN) Shared frames 0 -- Saved frames 0 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-03 22:17 ` Glenn Enright @ 2017-05-08 9:10 ` Juergen Gross 2017-05-09 9:24 ` Roger Pau Monné 0 siblings, 1 reply; 28+ messages in thread From: Juergen Gross @ 2017-05-08 9:10 UTC (permalink / raw) To: glenn, Steven Haigh, Jennifer Herbert Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel On 04/05/17 00:17, Glenn Enright wrote: > On 04/05/17 04:58, Steven Haigh wrote: >> On 04/05/17 01:53, Juergen Gross wrote: >>> On 03/05/17 12:45, Steven Haigh wrote: >>>> Just wanted to give this a little nudge now people seem to be back on >>>> deck... >>> >>> Glenn, could you please give the attached patch a try? >>> >>> It should be applied on top of the other correction, the old debug >>> patch should not be applied. >>> >>> I have added some debug output to make sure we see what is happening. >> >> This patch is included in kernel-xen-4.9.26-1 >> >> It should be in the repos now. >> > > Still seeing the same issue. Without the extra debug patch all I see in > the logs after destroy is this... > > xen-blkback: xen_blkif_disconnect: busy > xen-blkback: xen_blkif_free: delayed = 0 Hmm, to me it seems as if some grant isn't being unmapped. Looking at gnttab_unmap_refs_async() I wonder how this is supposed to work: I don't see how a grant would ever be unmapped in case of page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it does is deferring the call to the unmap operation again and again. Or am I missing something here? Juergen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-08 9:10 ` Juergen Gross @ 2017-05-09 9:24 ` Roger Pau Monné 2017-05-13 4:02 ` Glenn Enright 0 siblings, 1 reply; 28+ messages in thread From: Roger Pau Monné @ 2017-05-09 9:24 UTC (permalink / raw) To: Juergen Gross Cc: Steven Haigh, Andrew Cooper, Jennifer Herbert, Dietmar Hahn, xen-devel, glenn On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote: > On 04/05/17 00:17, Glenn Enright wrote: > > On 04/05/17 04:58, Steven Haigh wrote: > >> On 04/05/17 01:53, Juergen Gross wrote: > >>> On 03/05/17 12:45, Steven Haigh wrote: > >>>> Just wanted to give this a little nudge now people seem to be back on > >>>> deck... > >>> > >>> Glenn, could you please give the attached patch a try? > >>> > >>> It should be applied on top of the other correction, the old debug > >>> patch should not be applied. > >>> > >>> I have added some debug output to make sure we see what is happening. > >> > >> This patch is included in kernel-xen-4.9.26-1 > >> > >> It should be in the repos now. > >> > > > > Still seeing the same issue. Without the extra debug patch all I see in > > the logs after destroy is this... > > > > xen-blkback: xen_blkif_disconnect: busy > > xen-blkback: xen_blkif_free: delayed = 0 > > Hmm, to me it seems as if some grant isn't being unmapped. > > Looking at gnttab_unmap_refs_async() I wonder how this is supposed to > work: > > I don't see how a grant would ever be unmapped in case of > page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it > does is deferring the call to the unmap operation again and again. Or > am I missing something here? No, I don't think you are missing anything, but I cannot see how this can be solved in a better way, unmapping a page that's still referenced is certainly not the best option, or else we risk triggering a page-fault elsewhere. IMHO, gnttab_unmap_refs_async should have a timeout, and return an error at some point. Also, I'm wondering whether there's a way to keep track of who has references on a specific page, but so far I haven't been able to figure out how to get this information from Linux. Also, I've noticed that __gnttab_unmap_refs_async uses page_count, shouldn't it use page_ref_count instead? Roger. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-09 9:24 ` Roger Pau Monné @ 2017-05-13 4:02 ` Glenn Enright 2017-05-15 9:57 ` Juergen Gross 0 siblings, 1 reply; 28+ messages in thread From: Glenn Enright @ 2017-05-13 4:02 UTC (permalink / raw) To: Roger Pau Monné, Juergen Gross Cc: Jennifer Herbert, xen-devel, Steven Haigh, Dietmar Hahn, Andrew Cooper On 09/05/17 21:24, Roger Pau Monné wrote: > On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote: >> On 04/05/17 00:17, Glenn Enright wrote: >>> On 04/05/17 04:58, Steven Haigh wrote: >>>> On 04/05/17 01:53, Juergen Gross wrote: >>>>> On 03/05/17 12:45, Steven Haigh wrote: >>>>>> Just wanted to give this a little nudge now people seem to be back on >>>>>> deck... >>>>> >>>>> Glenn, could you please give the attached patch a try? >>>>> >>>>> It should be applied on top of the other correction, the old debug >>>>> patch should not be applied. >>>>> >>>>> I have added some debug output to make sure we see what is happening. >>>> >>>> This patch is included in kernel-xen-4.9.26-1 >>>> >>>> It should be in the repos now. >>>> >>> >>> Still seeing the same issue. Without the extra debug patch all I see in >>> the logs after destroy is this... >>> >>> xen-blkback: xen_blkif_disconnect: busy >>> xen-blkback: xen_blkif_free: delayed = 0 >> >> Hmm, to me it seems as if some grant isn't being unmapped. >> >> Looking at gnttab_unmap_refs_async() I wonder how this is supposed to >> work: >> >> I don't see how a grant would ever be unmapped in case of >> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it >> does is deferring the call to the unmap operation again and again. Or >> am I missing something here? > > No, I don't think you are missing anything, but I cannot see how this can be > solved in a better way, unmapping a page that's still referenced is certainly > not the best option, or else we risk triggering a page-fault elsewhere. > > IMHO, gnttab_unmap_refs_async should have a timeout, and return an error at > some point. Also, I'm wondering whether there's a way to keep track of who has > references on a specific page, but so far I haven't been able to figure out how > to get this information from Linux. > > Also, I've noticed that __gnttab_unmap_refs_async uses page_count, shouldn't it > use page_ref_count instead? > > Roger. > In case it helps, I have continued to work on this. I notices processed left behind (under 4.9.27). The same issue is ongoing. # ps auxf | grep [x]vda root 2983 0.0 0.0 0 0 ? S 01:44 0:00 \_ [1.xvda1-1] root 5457 0.0 0.0 0 0 ? S 02:06 0:00 \_ [3.xvda1-1] root 7382 0.0 0.0 0 0 ? S 02:36 0:00 \_ [4.xvda1-1] root 9668 0.0 0.0 0 0 ? S 02:51 0:00 \_ [6.xvda1-1] root 11080 0.0 0.0 0 0 ? S 02:57 0:00 \_ [7.xvda1-1] # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 1512 2 r----- 118.5 (null) 1 8 4 --p--d 43.8 (null) 3 8 4 --p--d 6.3 (null) 4 8 4 --p--d 73.4 (null) 6 8 4 --p--d 14.7 (null) 7 8 4 --p--d 30 Those all have... [root 11080]# cat wchan xen_blkif_schedule [root 11080]# cat stack [<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40 [<ffffffff810a0555>] kthread+0xe5/0x100 [<ffffffff816f1c45>] ret_from_fork+0x25/0x30 [<ffffffffffffffff>] 0xffffffffffffffff So I can give anything that is in the /proc/pid space. 'maps' and 'pagemap' are empty fwiw. I also have perf tools installed on this box in case anyone has anything I can run with that which might help. A reminder I can replicate by doing xl destroy $domU while running this inside the domU. Has anyone else been able to repeat this? { while true; do dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync done } This does not work in 4.4 kernels, so there is possibly a regression of some sort? Regards, Glenn _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-13 4:02 ` Glenn Enright @ 2017-05-15 9:57 ` Juergen Gross 2017-05-16 0:49 ` Glenn Enright 0 siblings, 1 reply; 28+ messages in thread From: Juergen Gross @ 2017-05-15 9:57 UTC (permalink / raw) To: glenn, Roger Pau Monné Cc: Jennifer Herbert, xen-devel, Steven Haigh, Dietmar Hahn, Andrew Cooper [-- Attachment #1: Type: text/plain, Size: 3641 bytes --] On 13/05/17 06:02, Glenn Enright wrote: > On 09/05/17 21:24, Roger Pau Monné wrote: >> On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote: >>> On 04/05/17 00:17, Glenn Enright wrote: >>>> On 04/05/17 04:58, Steven Haigh wrote: >>>>> On 04/05/17 01:53, Juergen Gross wrote: >>>>>> On 03/05/17 12:45, Steven Haigh wrote: >>>>>>> Just wanted to give this a little nudge now people seem to be >>>>>>> back on >>>>>>> deck... >>>>>> >>>>>> Glenn, could you please give the attached patch a try? >>>>>> >>>>>> It should be applied on top of the other correction, the old debug >>>>>> patch should not be applied. >>>>>> >>>>>> I have added some debug output to make sure we see what is happening. >>>>> >>>>> This patch is included in kernel-xen-4.9.26-1 >>>>> >>>>> It should be in the repos now. >>>>> >>>> >>>> Still seeing the same issue. Without the extra debug patch all I see in >>>> the logs after destroy is this... >>>> >>>> xen-blkback: xen_blkif_disconnect: busy >>>> xen-blkback: xen_blkif_free: delayed = 0 >>> >>> Hmm, to me it seems as if some grant isn't being unmapped. >>> >>> Looking at gnttab_unmap_refs_async() I wonder how this is supposed to >>> work: >>> >>> I don't see how a grant would ever be unmapped in case of >>> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it >>> does is deferring the call to the unmap operation again and again. Or >>> am I missing something here? >> >> No, I don't think you are missing anything, but I cannot see how this >> can be >> solved in a better way, unmapping a page that's still referenced is >> certainly >> not the best option, or else we risk triggering a page-fault elsewhere. >> >> IMHO, gnttab_unmap_refs_async should have a timeout, and return an >> error at >> some point. Also, I'm wondering whether there's a way to keep track of >> who has >> references on a specific page, but so far I haven't been able to >> figure out how >> to get this information from Linux. >> >> Also, I've noticed that __gnttab_unmap_refs_async uses page_count, >> shouldn't it >> use page_ref_count instead? >> >> Roger. >> > > In case it helps, I have continued to work on this. I notices processed > left behind (under 4.9.27). The same issue is ongoing. > > # ps auxf | grep [x]vda > root 2983 0.0 0.0 0 0 ? S 01:44 0:00 \_ > [1.xvda1-1] > root 5457 0.0 0.0 0 0 ? S 02:06 0:00 \_ > [3.xvda1-1] > root 7382 0.0 0.0 0 0 ? S 02:36 0:00 \_ > [4.xvda1-1] > root 9668 0.0 0.0 0 0 ? S 02:51 0:00 \_ > [6.xvda1-1] > root 11080 0.0 0.0 0 0 ? S 02:57 0:00 \_ > [7.xvda1-1] > > # xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 1512 2 r----- 118.5 > (null) 1 8 4 --p--d 43.8 > (null) 3 8 4 --p--d 6.3 > (null) 4 8 4 --p--d 73.4 > (null) 6 8 4 --p--d 14.7 > (null) 7 8 4 --p--d 30 > > Those all have... > > [root 11080]# cat wchan > xen_blkif_schedule > > [root 11080]# cat stack > [<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40 > [<ffffffff810a0555>] kthread+0xe5/0x100 > [<ffffffff816f1c45>] ret_from_fork+0x25/0x30 > [<ffffffffffffffff>] 0xffffffffffffffff And found another reference count bug. Would you like to give the attached patch (to be applied additionally to the previous ones) a try? Juergen [-- Attachment #2: 0003-xen-blkback-don-t-use-xen_blkif_get-in-xen-blkback-k.patch --] [-- Type: text/x-patch, Size: 1850 bytes --] >From ef37fe77b7b1b1d733011df5615634e543e21c4f Mon Sep 17 00:00:00 2001 From: Juergen Gross <jgross@suse.com> Date: Mon, 15 May 2017 11:41:18 +0200 Subject: [PATCH 3/3] xen/blkback: don't use xen_blkif_get() in xen-blkback kthread There is no need to use xen_blkif_get()/xen_blkif_put() in the kthread of xen-blkback. Thread stopping is synchronous and using the blkif reference counting in the kthread will avoid to ever let the reference count drop to zero at the end of an I/O running concurrent to disconnecting and multiple rings. Setting ring->xenblkd to NULL after stopping the kthread isn't needed as the kthread does this already. Signed-off-by: Juergen Gross <jgross@suse.com> --- drivers/block/xen-blkback/blkback.c | 3 --- drivers/block/xen-blkback/xenbus.c | 1 - 2 files changed, 4 deletions(-) diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c index 726c32e35db9..6b14c509f3c7 100644 --- a/drivers/block/xen-blkback/blkback.c +++ b/drivers/block/xen-blkback/blkback.c @@ -609,8 +609,6 @@ int xen_blkif_schedule(void *arg) unsigned long timeout; int ret; - xen_blkif_get(blkif); - set_freezable(); while (!kthread_should_stop()) { if (try_to_freeze()) @@ -665,7 +663,6 @@ int xen_blkif_schedule(void *arg) print_stats(ring); ring->xenblkd = NULL; - xen_blkif_put(blkif); return 0; } diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index ec7eaeee3765..9f8d136d7636 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -255,7 +255,6 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif) if (ring->xenblkd) { kthread_stop(ring->xenblkd); wake_up(&ring->shutdown_wq); - ring->xenblkd = NULL; } /* The above kthread_stop() guarantees that at this point we -- 2.12.0 [-- Attachment #3: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-15 9:57 ` Juergen Gross @ 2017-05-16 0:49 ` Glenn Enright 2017-05-16 1:18 ` Steven Haigh 0 siblings, 1 reply; 28+ messages in thread From: Glenn Enright @ 2017-05-16 0:49 UTC (permalink / raw) To: Juergen Gross, Roger Pau Monné Cc: Jennifer Herbert, xen-devel, Steven Haigh, Dietmar Hahn, Andrew Cooper On 15/05/17 21:57, Juergen Gross wrote: > On 13/05/17 06:02, Glenn Enright wrote: >> On 09/05/17 21:24, Roger Pau Monné wrote: >>> On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote: >>>> On 04/05/17 00:17, Glenn Enright wrote: >>>>> On 04/05/17 04:58, Steven Haigh wrote: >>>>>> On 04/05/17 01:53, Juergen Gross wrote: >>>>>>> On 03/05/17 12:45, Steven Haigh wrote: >>>>>>>> Just wanted to give this a little nudge now people seem to be >>>>>>>> back on >>>>>>>> deck... >>>>>>> >>>>>>> Glenn, could you please give the attached patch a try? >>>>>>> >>>>>>> It should be applied on top of the other correction, the old debug >>>>>>> patch should not be applied. >>>>>>> >>>>>>> I have added some debug output to make sure we see what is happening. >>>>>> >>>>>> This patch is included in kernel-xen-4.9.26-1 >>>>>> >>>>>> It should be in the repos now. >>>>>> >>>>> >>>>> Still seeing the same issue. Without the extra debug patch all I see in >>>>> the logs after destroy is this... >>>>> >>>>> xen-blkback: xen_blkif_disconnect: busy >>>>> xen-blkback: xen_blkif_free: delayed = 0 >>>> >>>> Hmm, to me it seems as if some grant isn't being unmapped. >>>> >>>> Looking at gnttab_unmap_refs_async() I wonder how this is supposed to >>>> work: >>>> >>>> I don't see how a grant would ever be unmapped in case of >>>> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it >>>> does is deferring the call to the unmap operation again and again. Or >>>> am I missing something here? >>> >>> No, I don't think you are missing anything, but I cannot see how this >>> can be >>> solved in a better way, unmapping a page that's still referenced is >>> certainly >>> not the best option, or else we risk triggering a page-fault elsewhere. >>> >>> IMHO, gnttab_unmap_refs_async should have a timeout, and return an >>> error at >>> some point. Also, I'm wondering whether there's a way to keep track of >>> who has >>> references on a specific page, but so far I haven't been able to >>> figure out how >>> to get this information from Linux. >>> >>> Also, I've noticed that __gnttab_unmap_refs_async uses page_count, >>> shouldn't it >>> use page_ref_count instead? >>> >>> Roger. >>> >> >> In case it helps, I have continued to work on this. I notices processed >> left behind (under 4.9.27). The same issue is ongoing. >> >> # ps auxf | grep [x]vda >> root 2983 0.0 0.0 0 0 ? S 01:44 0:00 \_ >> [1.xvda1-1] >> root 5457 0.0 0.0 0 0 ? S 02:06 0:00 \_ >> [3.xvda1-1] >> root 7382 0.0 0.0 0 0 ? S 02:36 0:00 \_ >> [4.xvda1-1] >> root 9668 0.0 0.0 0 0 ? S 02:51 0:00 \_ >> [6.xvda1-1] >> root 11080 0.0 0.0 0 0 ? S 02:57 0:00 \_ >> [7.xvda1-1] >> >> # xl list >> Name ID Mem VCPUs State Time(s) >> Domain-0 0 1512 2 r----- 118.5 >> (null) 1 8 4 --p--d 43.8 >> (null) 3 8 4 --p--d 6.3 >> (null) 4 8 4 --p--d 73.4 >> (null) 6 8 4 --p--d 14.7 >> (null) 7 8 4 --p--d 30 >> >> Those all have... >> >> [root 11080]# cat wchan >> xen_blkif_schedule >> >> [root 11080]# cat stack >> [<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40 >> [<ffffffff810a0555>] kthread+0xe5/0x100 >> [<ffffffff816f1c45>] ret_from_fork+0x25/0x30 >> [<ffffffffffffffff>] 0xffffffffffffffff > > And found another reference count bug. Would you like to give the > attached patch (to be applied additionally to the previous ones) a try? > > > Juergen > This seems to have solved the issue in 4.9.28, with all three patches applied. Awesome! On my main test machine I can no longer replicate what I was originally seeing, and in dmesg I now see this flow... xen-blkback: xen_blkif_disconnect: busy xen-blkback: xen_blkif_free: delayed = 1 xen-blkback: xen_blkif_free: delayed = 0 xl list is clean, xenstore looks right. No extraneous processes left over. Thankyou Juergen, so much. Really appreciate your persistence with this. Anything I can do to help push this upstream please let me know. Feel free to add a reported-by line with my name if you think it appropriate. Regards, Glenn http://rimuhosting.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy 2017-05-16 0:49 ` Glenn Enright @ 2017-05-16 1:18 ` Steven Haigh 0 siblings, 0 replies; 28+ messages in thread From: Steven Haigh @ 2017-05-16 1:18 UTC (permalink / raw) To: glenn Cc: Juergen Gross, Andrew Cooper, Jennifer Herbert, Dietmar Hahn, xen-devel, Roger Pau Monné On 2017-05-16 10:49, Glenn Enright wrote: > On 15/05/17 21:57, Juergen Gross wrote: >> On 13/05/17 06:02, Glenn Enright wrote: >>> On 09/05/17 21:24, Roger Pau Monné wrote: >>>> On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote: >>>>> On 04/05/17 00:17, Glenn Enright wrote: >>>>>> On 04/05/17 04:58, Steven Haigh wrote: >>>>>>> On 04/05/17 01:53, Juergen Gross wrote: >>>>>>>> On 03/05/17 12:45, Steven Haigh wrote: >>>>>>>>> Just wanted to give this a little nudge now people seem to be >>>>>>>>> back on >>>>>>>>> deck... >>>>>>>> >>>>>>>> Glenn, could you please give the attached patch a try? >>>>>>>> >>>>>>>> It should be applied on top of the other correction, the old >>>>>>>> debug >>>>>>>> patch should not be applied. >>>>>>>> >>>>>>>> I have added some debug output to make sure we see what is >>>>>>>> happening. >>>>>>> >>>>>>> This patch is included in kernel-xen-4.9.26-1 >>>>>>> >>>>>>> It should be in the repos now. >>>>>>> >>>>>> >>>>>> Still seeing the same issue. Without the extra debug patch all I >>>>>> see in >>>>>> the logs after destroy is this... >>>>>> >>>>>> xen-blkback: xen_blkif_disconnect: busy >>>>>> xen-blkback: xen_blkif_free: delayed = 0 >>>>> >>>>> Hmm, to me it seems as if some grant isn't being unmapped. >>>>> >>>>> Looking at gnttab_unmap_refs_async() I wonder how this is supposed >>>>> to >>>>> work: >>>>> >>>>> I don't see how a grant would ever be unmapped in case of >>>>> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All >>>>> it >>>>> does is deferring the call to the unmap operation again and again. >>>>> Or >>>>> am I missing something here? >>>> >>>> No, I don't think you are missing anything, but I cannot see how >>>> this >>>> can be >>>> solved in a better way, unmapping a page that's still referenced is >>>> certainly >>>> not the best option, or else we risk triggering a page-fault >>>> elsewhere. >>>> >>>> IMHO, gnttab_unmap_refs_async should have a timeout, and return an >>>> error at >>>> some point. Also, I'm wondering whether there's a way to keep track >>>> of >>>> who has >>>> references on a specific page, but so far I haven't been able to >>>> figure out how >>>> to get this information from Linux. >>>> >>>> Also, I've noticed that __gnttab_unmap_refs_async uses page_count, >>>> shouldn't it >>>> use page_ref_count instead? >>>> >>>> Roger. >>>> >>> >>> In case it helps, I have continued to work on this. I notices >>> processed >>> left behind (under 4.9.27). The same issue is ongoing. >>> >>> # ps auxf | grep [x]vda >>> root 2983 0.0 0.0 0 0 ? S 01:44 0:00 \_ >>> [1.xvda1-1] >>> root 5457 0.0 0.0 0 0 ? S 02:06 0:00 \_ >>> [3.xvda1-1] >>> root 7382 0.0 0.0 0 0 ? S 02:36 0:00 \_ >>> [4.xvda1-1] >>> root 9668 0.0 0.0 0 0 ? S 02:51 0:00 \_ >>> [6.xvda1-1] >>> root 11080 0.0 0.0 0 0 ? S 02:57 0:00 \_ >>> [7.xvda1-1] >>> >>> # xl list >>> Name ID Mem VCPUs State Time(s) >>> Domain-0 0 1512 2 r----- 118.5 >>> (null) 1 8 4 --p--d 43.8 >>> (null) 3 8 4 --p--d 6.3 >>> (null) 4 8 4 --p--d 73.4 >>> (null) 6 8 4 --p--d 14.7 >>> (null) 7 8 4 --p--d 30 >>> >>> Those all have... >>> >>> [root 11080]# cat wchan >>> xen_blkif_schedule >>> >>> [root 11080]# cat stack >>> [<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40 >>> [<ffffffff810a0555>] kthread+0xe5/0x100 >>> [<ffffffff816f1c45>] ret_from_fork+0x25/0x30 >>> [<ffffffffffffffff>] 0xffffffffffffffff >> >> And found another reference count bug. Would you like to give the >> attached patch (to be applied additionally to the previous ones) a >> try? >> >> >> Juergen >> > > This seems to have solved the issue in 4.9.28, with all three patches > applied. Awesome! > > On my main test machine I can no longer replicate what I was > originally seeing, and in dmesg I now see this flow... > > xen-blkback: xen_blkif_disconnect: busy > xen-blkback: xen_blkif_free: delayed = 1 > xen-blkback: xen_blkif_free: delayed = 0 > > xl list is clean, xenstore looks right. No extraneous processes left > over. > > Thankyou Juergen, so much. Really appreciate your persistence with > this. Anything I can do to help push this upstream please let me know. > Feel free to add a reported-by line with my name if you think it > appropriate. This is good news. Juergen, Can I request a full patch set posted to the list (plz CC me) - and I'll ensure we can build the kernel with all 3 (?) patches applied and test properly. I'll build up a complete kernel with those patches and give a tested-by if all goes well. -- Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2017-05-16 1:18 UTC | newest] Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-04-11 5:25 null domains after xl destroy Glenn Enright 2017-04-11 5:59 ` Juergen Gross 2017-04-11 8:03 ` Glenn Enright 2017-04-11 9:49 ` Dietmar Hahn 2017-04-11 22:13 ` Glenn Enright 2017-04-11 22:23 ` Andrew Cooper 2017-04-11 22:45 ` Glenn Enright 2017-04-18 8:36 ` Juergen Gross 2017-04-19 1:02 ` Glenn Enright 2017-04-19 4:39 ` Juergen Gross 2017-04-19 7:16 ` Roger Pau Monné 2017-04-19 7:35 ` Juergen Gross 2017-04-19 10:09 ` Juergen Gross 2017-04-19 16:22 ` Steven Haigh 2017-04-21 8:42 ` Steven Haigh 2017-04-21 8:44 ` Juergen Gross 2017-05-01 0:55 ` Glenn Enright 2017-05-03 10:45 ` Steven Haigh 2017-05-03 13:38 ` Juergen Gross 2017-05-03 15:53 ` Juergen Gross 2017-05-03 16:58 ` Steven Haigh 2017-05-03 22:17 ` Glenn Enright 2017-05-08 9:10 ` Juergen Gross 2017-05-09 9:24 ` Roger Pau Monné 2017-05-13 4:02 ` Glenn Enright 2017-05-15 9:57 ` Juergen Gross 2017-05-16 0:49 ` Glenn Enright 2017-05-16 1:18 ` Steven Haigh
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.