* null domains after xl destroy
@ 2017-04-11 5:25 Glenn Enright
2017-04-11 5:59 ` Juergen Gross
0 siblings, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-04-11 5:25 UTC (permalink / raw)
To: xen-devel
Hi all
We are seeing an odd issue with domu domains from xl destroy, under
recent 4.9 kernels a (null) domain is left behind.
This has occurred on a variety of hardware, with no obvious commonality.
4.4.55 does not show this behavior.
On my test machine I have the following packages installed under
centos6, from https://xen.crc.id.au/
~]# rpm -qa | grep xen
xen47-licenses-4.7.2-4.el6.x86_64
xen47-4.7.2-4.el6.x86_64
kernel-xen-4.9.21-1.el6xen.x86_64
xen47-ocaml-4.7.2-4.el6.x86_64
xen47-libs-4.7.2-4.el6.x86_64
xen47-libcacard-4.7.2-4.el6.x86_64
xen47-hypervisor-4.7.2-4.el6.x86_64
xen47-runtime-4.7.2-4.el6.x86_64
kernel-xen-firmware-4.9.21-1.el6xen.x86_64
I've also replicated the issue with 4.9.17 and 4.9.20
To replicate, on a cleanly booted dom0 with one pv VM, I run the
following on the VM
{
while true; do
dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
done
}
Then on the dom0 I do this sequence to reliably get a null domain. This
occurs with oxenstored and xenstored both.
{
xl sync 1
xl destroy 1
}
xl list then renders something like ...
(null) 1 4 4 --p--d
9.8 0
From what I can see it appears to be disk related. Affected VMs all use
lvm storage for their boot disk. lvdisplay of the affected lv shows that
the lv has is being help open by something.
~]# lvdisplay test/test.img | grep open
# open 1
I've not been able to determine what that thing is as yet. I tried lsof,
dmsetup, various lv tools. Waiting for the disk to be released does not
work.
~]# xl list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 1512 2 r-----
29.0
(null) 1 4 4 --p--d
9.8
xenstore-ls reports nothing for the null domain id that I can see.
I can later start and restart the affected domain, but direct operations
on the lv such as removing it don't work since it is 'busy'.
Does anyone have thoughts on this? Happy to provide any output that
might be useful, not subscribed to this list so please CC.
Regards, Glenn
http://rimuhosting.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-11 5:25 null domains after xl destroy Glenn Enright
@ 2017-04-11 5:59 ` Juergen Gross
2017-04-11 8:03 ` Glenn Enright
0 siblings, 1 reply; 28+ messages in thread
From: Juergen Gross @ 2017-04-11 5:59 UTC (permalink / raw)
To: glenn, xen-devel
On 11/04/17 07:25, Glenn Enright wrote:
> Hi all
>
> We are seeing an odd issue with domu domains from xl destroy, under
> recent 4.9 kernels a (null) domain is left behind.
I guess this is the dom0 kernel version?
> This has occurred on a variety of hardware, with no obvious commonality.
>
> 4.4.55 does not show this behavior.
>
> On my test machine I have the following packages installed under
> centos6, from https://xen.crc.id.au/
>
> ~]# rpm -qa | grep xen
> xen47-licenses-4.7.2-4.el6.x86_64
> xen47-4.7.2-4.el6.x86_64
> kernel-xen-4.9.21-1.el6xen.x86_64
> xen47-ocaml-4.7.2-4.el6.x86_64
> xen47-libs-4.7.2-4.el6.x86_64
> xen47-libcacard-4.7.2-4.el6.x86_64
> xen47-hypervisor-4.7.2-4.el6.x86_64
> xen47-runtime-4.7.2-4.el6.x86_64
> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
>
> I've also replicated the issue with 4.9.17 and 4.9.20
>
> To replicate, on a cleanly booted dom0 with one pv VM, I run the
> following on the VM
>
> {
> while true; do
> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
> done
> }
>
> Then on the dom0 I do this sequence to reliably get a null domain. This
> occurs with oxenstored and xenstored both.
>
> {
> xl sync 1
> xl destroy 1
> }
>
> xl list then renders something like ...
>
> (null) 1 4 4 --p--d
> 9.8 0
Something is referencing the domain, e.g. some of its memory pages are
still mapped by dom0.
> From what I can see it appears to be disk related. Affected VMs all use
> lvm storage for their boot disk. lvdisplay of the affected lv shows that
> the lv has is being help open by something.
How are the disks configured? Especially the backend type is important.
>
> ~]# lvdisplay test/test.img | grep open
> # open 1
>
> I've not been able to determine what that thing is as yet. I tried lsof,
> dmsetup, various lv tools. Waiting for the disk to be released does not
> work.
>
> ~]# xl list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 1512 2 r-----
> 29.0
> (null) 1 4 4 --p--d
> 9.8
>
> xenstore-ls reports nothing for the null domain id that I can see.
Any qemu process related to the domain still running?
Any dom0 kernel messages related to Xen?
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-11 5:59 ` Juergen Gross
@ 2017-04-11 8:03 ` Glenn Enright
2017-04-11 9:49 ` Dietmar Hahn
0 siblings, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-04-11 8:03 UTC (permalink / raw)
To: Juergen Gross, xen-devel
On 11/04/17 17:59, Juergen Gross wrote:
> On 11/04/17 07:25, Glenn Enright wrote:
>> Hi all
>>
>> We are seeing an odd issue with domu domains from xl destroy, under
>> recent 4.9 kernels a (null) domain is left behind.
>
> I guess this is the dom0 kernel version?
>
>> This has occurred on a variety of hardware, with no obvious commonality.
>>
>> 4.4.55 does not show this behavior.
>>
>> On my test machine I have the following packages installed under
>> centos6, from https://xen.crc.id.au/
>>
>> ~]# rpm -qa | grep xen
>> xen47-licenses-4.7.2-4.el6.x86_64
>> xen47-4.7.2-4.el6.x86_64
>> kernel-xen-4.9.21-1.el6xen.x86_64
>> xen47-ocaml-4.7.2-4.el6.x86_64
>> xen47-libs-4.7.2-4.el6.x86_64
>> xen47-libcacard-4.7.2-4.el6.x86_64
>> xen47-hypervisor-4.7.2-4.el6.x86_64
>> xen47-runtime-4.7.2-4.el6.x86_64
>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
>>
>> I've also replicated the issue with 4.9.17 and 4.9.20
>>
>> To replicate, on a cleanly booted dom0 with one pv VM, I run the
>> following on the VM
>>
>> {
>> while true; do
>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
>> done
>> }
>>
>> Then on the dom0 I do this sequence to reliably get a null domain. This
>> occurs with oxenstored and xenstored both.
>>
>> {
>> xl sync 1
>> xl destroy 1
>> }
>>
>> xl list then renders something like ...
>>
>> (null) 1 4 4 --p--d
>> 9.8 0
>
> Something is referencing the domain, e.g. some of its memory pages are
> still mapped by dom0.
>
>> From what I can see it appears to be disk related. Affected VMs all use
>> lvm storage for their boot disk. lvdisplay of the affected lv shows that
>> the lv has is being help open by something.
>
> How are the disks configured? Especially the backend type is important.
>
>>
>> ~]# lvdisplay test/test.img | grep open
>> # open 1
>>
>> I've not been able to determine what that thing is as yet. I tried lsof,
>> dmsetup, various lv tools. Waiting for the disk to be released does not
>> work.
>>
>> ~]# xl list
>> Name ID Mem VCPUs State
>> Time(s)
>> Domain-0 0 1512 2 r-----
>> 29.0
>> (null) 1 4 4 --p--d
>> 9.8
>>
>> xenstore-ls reports nothing for the null domain id that I can see.
>
> Any qemu process related to the domain still running?
>
> Any dom0 kernel messages related to Xen?
>
>
> Juergen
>
Yep, 4.9 dom0 kernel
Typically we see an xl process running, but that has already gone away
in this case. The domU is a PV guest using phy definition, the basic
startup is like this...
xl -v create -f paramfile extra="console=hvc0 elevator=noop
xen-blkfront.max=64"
There are no qemu processes or threads anywhere I can see.
I dont see any meaningful messages in the linux kernel log, and nothing
at all in the hypervisor log. Here is output from the dom0 starting and
then stopping a domU using the above mechanism
br0: port 2(vif3.0) entered disabled state
br0: port 2(vif4.0) entered blocking state
br0: port 2(vif4.0) entered disabled state
device vif4.0 entered promiscuous mode
IPv6: ADDRCONF(NETDEV_UP): vif4.0: link is not ready
xen-blkback: backend/vbd/4/51713: using 2 queues, protocol 1
(x86_64-abi) persistent grants
xen-blkback: backend/vbd/4/51721: using 2 queues, protocol 1
(x86_64-abi) persistent grants
vif vif-4-0 vif4.0: Guest Rx ready
IPv6: ADDRCONF(NETDEV_CHANGE): vif4.0: link becomes ready
br0: port 2(vif4.0) entered blocking state
br0: port 2(vif4.0) entered forwarding state
br0: port 2(vif4.0) entered disabled state
br0: port 2(vif4.0) entered disabled state
device vif4.0 left promiscuous mode
br0: port 2(vif4.0) entered disabled state
... here is xl info ...
host : xxxxxxxxxxxx
release : 4.9.21-1.el6xen.x86_64
version : #1 SMP Sat Apr 8 18:03:45 AEST 2017
machine : x86_64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2394
hw_caps :
b7ebfbff:0000e3bd:20100800:00000001:00000000:00000000:00000000:00000000
virt_caps :
total_memory : 8190
free_memory : 6577
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 7
xen_extra : .2
xen_version : 4.7.2
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : dom0_mem=1512M cpufreq=xen dom0_max_vcpus=2
dom0_vcpus_pin log_lvl=all guest_loglvl=all vcpu_migration_delay=1000
cc_compiler : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17)
cc_compile_by : mockbuild
cc_compile_domain : (none)
cc_compile_date : Mon Apr 3 12:17:20 AEST 2017
build_id : 0ec32d14d7c34e5d9deaaf6e3b7ea0c8006d68fa
xend_config_format : 4
# cat /proc/cmdline
ro root=UUID=xxxxxxxxxx rd_MD_UUID=xxxxxxxxxxxx rd_NO_LUKS
KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_MD_UUID=xxxxxxxxxxxxx
SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet
pcie_aspm=off panic=30 max_loop=64 dm_mod.use_blk_mq=y xen-blkfront.max=64
The domu is using an lvm on top of a md raid1 array, on direct connected
HDDs. Nothing special hardware wise. The disk line for that domU looks
functionally like...
disk = [ 'phy:/dev/testlv/test.img,xvda1,w' ]
I would appreciate any suggestions on how to increase the debug level in
a relevant way or where to look to get more useful information on what
is happening.
To clarify the actual shutdown sequence that causes problems...
# xl sysrq $id s
# xl destroy $id
Regards, Glenn
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-11 8:03 ` Glenn Enright
@ 2017-04-11 9:49 ` Dietmar Hahn
2017-04-11 22:13 ` Glenn Enright
0 siblings, 1 reply; 28+ messages in thread
From: Dietmar Hahn @ 2017-04-11 9:49 UTC (permalink / raw)
To: xen-devel, glenn; +Cc: Juergen Gross
Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright:
> On 11/04/17 17:59, Juergen Gross wrote:
> > On 11/04/17 07:25, Glenn Enright wrote:
> >> Hi all
> >>
> >> We are seeing an odd issue with domu domains from xl destroy, under
> >> recent 4.9 kernels a (null) domain is left behind.
> >
> > I guess this is the dom0 kernel version?
> >
> >> This has occurred on a variety of hardware, with no obvious commonality.
> >>
> >> 4.4.55 does not show this behavior.
> >>
> >> On my test machine I have the following packages installed under
> >> centos6, from https://xen.crc.id.au/
> >>
> >> ~]# rpm -qa | grep xen
> >> xen47-licenses-4.7.2-4.el6.x86_64
> >> xen47-4.7.2-4.el6.x86_64
> >> kernel-xen-4.9.21-1.el6xen.x86_64
> >> xen47-ocaml-4.7.2-4.el6.x86_64
> >> xen47-libs-4.7.2-4.el6.x86_64
> >> xen47-libcacard-4.7.2-4.el6.x86_64
> >> xen47-hypervisor-4.7.2-4.el6.x86_64
> >> xen47-runtime-4.7.2-4.el6.x86_64
> >> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
> >>
> >> I've also replicated the issue with 4.9.17 and 4.9.20
> >>
> >> To replicate, on a cleanly booted dom0 with one pv VM, I run the
> >> following on the VM
> >>
> >> {
> >> while true; do
> >> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
> >> done
> >> }
> >>
> >> Then on the dom0 I do this sequence to reliably get a null domain. This
> >> occurs with oxenstored and xenstored both.
> >>
> >> {
> >> xl sync 1
> >> xl destroy 1
> >> }
> >>
> >> xl list then renders something like ...
> >>
> >> (null) 1 4 4 --p--d
> >> 9.8 0
> >
> > Something is referencing the domain, e.g. some of its memory pages are
> > still mapped by dom0.
You can try
# xl debug-keys q
and further
# xl dmesg
to see the output of the previous command. The 'q' dumps domain
(and guest debug) info.
# xl debug-keys h
prints all possible parameters for more informations.
Dietmar.
> >
> >> From what I can see it appears to be disk related. Affected VMs all use
> >> lvm storage for their boot disk. lvdisplay of the affected lv shows that
> >> the lv has is being help open by something.
> >
> > How are the disks configured? Especially the backend type is important.
> >
> >>
> >> ~]# lvdisplay test/test.img | grep open
> >> # open 1
> >>
> >> I've not been able to determine what that thing is as yet. I tried lsof,
> >> dmsetup, various lv tools. Waiting for the disk to be released does not
> >> work.
> >>
> >> ~]# xl list
> >> Name ID Mem VCPUs State
> >> Time(s)
> >> Domain-0 0 1512 2 r-----
> >> 29.0
> >> (null) 1 4 4 --p--d
> >> 9.8
> >>
> >> xenstore-ls reports nothing for the null domain id that I can see.
> >
> > Any qemu process related to the domain still running?
> >
> > Any dom0 kernel messages related to Xen?
> >
> >
> > Juergen
> >
>
> Yep, 4.9 dom0 kernel
>
> Typically we see an xl process running, but that has already gone away
> in this case. The domU is a PV guest using phy definition, the basic
> startup is like this...
>
> xl -v create -f paramfile extra="console=hvc0 elevator=noop
> xen-blkfront.max=64"
>
> There are no qemu processes or threads anywhere I can see.
>
> I dont see any meaningful messages in the linux kernel log, and nothing
> at all in the hypervisor log. Here is output from the dom0 starting and
> then stopping a domU using the above mechanism
>
> br0: port 2(vif3.0) entered disabled state
> br0: port 2(vif4.0) entered blocking state
> br0: port 2(vif4.0) entered disabled state
> device vif4.0 entered promiscuous mode
> IPv6: ADDRCONF(NETDEV_UP): vif4.0: link is not ready
> xen-blkback: backend/vbd/4/51713: using 2 queues, protocol 1
> (x86_64-abi) persistent grants
> xen-blkback: backend/vbd/4/51721: using 2 queues, protocol 1
> (x86_64-abi) persistent grants
> vif vif-4-0 vif4.0: Guest Rx ready
> IPv6: ADDRCONF(NETDEV_CHANGE): vif4.0: link becomes ready
> br0: port 2(vif4.0) entered blocking state
> br0: port 2(vif4.0) entered forwarding state
> br0: port 2(vif4.0) entered disabled state
> br0: port 2(vif4.0) entered disabled state
> device vif4.0 left promiscuous mode
> br0: port 2(vif4.0) entered disabled state
>
> ... here is xl info ...
>
> host : xxxxxxxxxxxx
> release : 4.9.21-1.el6xen.x86_64
> version : #1 SMP Sat Apr 8 18:03:45 AEST 2017
> machine : x86_64
> nr_cpus : 4
> max_cpu_id : 3
> nr_nodes : 1
> cores_per_socket : 4
> threads_per_core : 1
> cpu_mhz : 2394
> hw_caps :
> b7ebfbff:0000e3bd:20100800:00000001:00000000:00000000:00000000:00000000
> virt_caps :
> total_memory : 8190
> free_memory : 6577
> sharing_freed_memory : 0
> sharing_used_memory : 0
> outstanding_claims : 0
> free_cpus : 0
> xen_major : 4
> xen_minor : 7
> xen_extra : .2
> xen_version : 4.7.2
> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xffff800000000000
> xen_changeset :
> xen_commandline : dom0_mem=1512M cpufreq=xen dom0_max_vcpus=2
> dom0_vcpus_pin log_lvl=all guest_loglvl=all vcpu_migration_delay=1000
> cc_compiler : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17)
> cc_compile_by : mockbuild
> cc_compile_domain : (none)
> cc_compile_date : Mon Apr 3 12:17:20 AEST 2017
> build_id : 0ec32d14d7c34e5d9deaaf6e3b7ea0c8006d68fa
> xend_config_format : 4
>
>
> # cat /proc/cmdline
> ro root=UUID=xxxxxxxxxx rd_MD_UUID=xxxxxxxxxxxx rd_NO_LUKS
> KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_MD_UUID=xxxxxxxxxxxxx
> SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet
> pcie_aspm=off panic=30 max_loop=64 dm_mod.use_blk_mq=y xen-blkfront.max=64
>
> The domu is using an lvm on top of a md raid1 array, on direct connected
> HDDs. Nothing special hardware wise. The disk line for that domU looks
> functionally like...
>
> disk = [ 'phy:/dev/testlv/test.img,xvda1,w' ]
>
> I would appreciate any suggestions on how to increase the debug level in
> a relevant way or where to look to get more useful information on what
> is happening.
>
> To clarify the actual shutdown sequence that causes problems...
>
> # xl sysrq $id s
> # xl destroy $id
>
>
> Regards, Glenn
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
--
Company details: http://ts.fujitsu.com/imprint.html
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-11 9:49 ` Dietmar Hahn
@ 2017-04-11 22:13 ` Glenn Enright
2017-04-11 22:23 ` Andrew Cooper
0 siblings, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-04-11 22:13 UTC (permalink / raw)
To: Dietmar Hahn, xen-devel; +Cc: Juergen Gross
On 11/04/17 21:49, Dietmar Hahn wrote:
> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright:
>> On 11/04/17 17:59, Juergen Gross wrote:
>>> On 11/04/17 07:25, Glenn Enright wrote:
>>>> Hi all
>>>>
>>>> We are seeing an odd issue with domu domains from xl destroy, under
>>>> recent 4.9 kernels a (null) domain is left behind.
>>>
>>> I guess this is the dom0 kernel version?
>>>
>>>> This has occurred on a variety of hardware, with no obvious commonality.
>>>>
>>>> 4.4.55 does not show this behavior.
>>>>
>>>> On my test machine I have the following packages installed under
>>>> centos6, from https://xen.crc.id.au/
>>>>
>>>> ~]# rpm -qa | grep xen
>>>> xen47-licenses-4.7.2-4.el6.x86_64
>>>> xen47-4.7.2-4.el6.x86_64
>>>> kernel-xen-4.9.21-1.el6xen.x86_64
>>>> xen47-ocaml-4.7.2-4.el6.x86_64
>>>> xen47-libs-4.7.2-4.el6.x86_64
>>>> xen47-libcacard-4.7.2-4.el6.x86_64
>>>> xen47-hypervisor-4.7.2-4.el6.x86_64
>>>> xen47-runtime-4.7.2-4.el6.x86_64
>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
>>>>
>>>> I've also replicated the issue with 4.9.17 and 4.9.20
>>>>
>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the
>>>> following on the VM
>>>>
>>>> {
>>>> while true; do
>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
>>>> done
>>>> }
>>>>
>>>> Then on the dom0 I do this sequence to reliably get a null domain. This
>>>> occurs with oxenstored and xenstored both.
>>>>
>>>> {
>>>> xl sync 1
>>>> xl destroy 1
>>>> }
>>>>
>>>> xl list then renders something like ...
>>>>
>>>> (null) 1 4 4 --p--d
>>>> 9.8 0
>>>
>>> Something is referencing the domain, e.g. some of its memory pages are
>>> still mapped by dom0.
>
> You can try
> # xl debug-keys q
> and further
> # xl dmesg
> to see the output of the previous command. The 'q' dumps domain
> (and guest debug) info.
> # xl debug-keys h
> prints all possible parameters for more informations.
>
> Dietmar.
>
I've done this as requested, below is the output.
(XEN) 'q' pressed -> dumping domain info (now=0x92:D6C271CE)
(XEN) General information for domain 0:
(XEN) refcnt=3 dying=0 pause_count=0
(XEN) nr_pages=387072 xenheap_pages=5 shared_pages=0 paged_pages=0
dirty_cpus={0-1} max_pages=4294967295
(XEN) handle=00000000-0000-0000-0000-000000000000 vm_assist=0000000d
(XEN) Rangesets belonging to domain 0:
(XEN) I/O Ports { 0-1f, 22-3f, 44-60, 62-9f, a2-cfb, d00-1007,
100c-ffff }
(XEN) log-dirty { }
(XEN) Interrupts { 1-30 }
(XEN) I/O Memory { 0-fedff, fef00-ffffff }
(XEN) Memory pages belonging to domain 0:
(XEN) DomPage list too long to display
(XEN) XenPage 000000000020e9c5: caf=c000000000000002,
taf=7400000000000002
(XEN) XenPage 000000000020e9c4: caf=c000000000000001,
taf=7400000000000001
(XEN) XenPage 000000000020e9c3: caf=c000000000000001,
taf=7400000000000001
(XEN) XenPage 000000000020e9c2: caf=c000000000000001,
taf=7400000000000001
(XEN) XenPage 00000000000e7d2e: caf=c000000000000002,
taf=7400000000000002
(XEN) NODE affinity for domain 0: [0]
(XEN) VCPU information and callbacks for domain 0:
(XEN) VCPU0: CPU0 [has=T] poll=0 upcall_pend=01 upcall_mask=00
dirty_cpus={0}
(XEN) cpu_hard_affinity={0} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=0
(XEN) No periodic timer
(XEN) VCPU1: CPU1 [has=T] poll=0 upcall_pend=00 upcall_mask=00
dirty_cpus={1}
(XEN) cpu_hard_affinity={1} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=0
(XEN) No periodic timer
(XEN) General information for domain 1:
(XEN) refcnt=1 dying=2 pause_count=2
(XEN) nr_pages=2114 xenheap_pages=0 shared_pages=0 paged_pages=0
dirty_cpus={} max_pages=1280256
(XEN) handle=a481c2eb-31e3-4ae6-9809-290e746c8eec vm_assist=0000000d
(XEN) Rangesets belonging to domain 1:
(XEN) I/O Ports { }
(XEN) log-dirty { }
(XEN) Interrupts { }
(XEN) I/O Memory { }
(XEN) Memory pages belonging to domain 1:
(XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001
(XEN) NODE affinity for domain 1: [0]
(XEN) VCPU information and callbacks for domain 1:
(XEN) VCPU0: CPU0 [has=F] poll=0 upcall_pend=00 upcall_mask=01
dirty_cpus={}
(XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=0
(XEN) No periodic timer
(XEN) VCPU1: CPU1 [has=F] poll=0 upcall_pend=00 upcall_mask=01
dirty_cpus={}
(XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=0
(XEN) No periodic timer
(XEN) VCPU2: CPU2 [has=F] poll=0 upcall_pend=00 upcall_mask=01
dirty_cpus={}
(XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=1
(XEN) No periodic timer
(XEN) VCPU3: CPU3 [has=F] poll=0 upcall_pend=00 upcall_mask=01
dirty_cpus={}
(XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=0
(XEN) No periodic timer
(XEN) Notifying guest 0:0 (virq 1, port 4)
(XEN) Notifying guest 0:1 (virq 1, port 10)
(XEN) Notifying guest 1:0 (virq 1, port 0)
(XEN) Notifying guest 1:1 (virq 1, port 0)
(XEN) Notifying guest 1:2 (virq 1, port 0)
(XEN) Notifying guest 1:3 (virq 1, port 0)
(XEN) Shared frames 0 -- Saved frames 0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-11 22:13 ` Glenn Enright
@ 2017-04-11 22:23 ` Andrew Cooper
2017-04-11 22:45 ` Glenn Enright
0 siblings, 1 reply; 28+ messages in thread
From: Andrew Cooper @ 2017-04-11 22:23 UTC (permalink / raw)
To: glenn, Dietmar Hahn, xen-devel; +Cc: Juergen Gross
On 11/04/2017 23:13, Glenn Enright wrote:
> On 11/04/17 21:49, Dietmar Hahn wrote:
>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright:
>>> On 11/04/17 17:59, Juergen Gross wrote:
>>>> On 11/04/17 07:25, Glenn Enright wrote:
>>>>> Hi all
>>>>>
>>>>> We are seeing an odd issue with domu domains from xl destroy, under
>>>>> recent 4.9 kernels a (null) domain is left behind.
>>>>
>>>> I guess this is the dom0 kernel version?
>>>>
>>>>> This has occurred on a variety of hardware, with no obvious
>>>>> commonality.
>>>>>
>>>>> 4.4.55 does not show this behavior.
>>>>>
>>>>> On my test machine I have the following packages installed under
>>>>> centos6, from https://xen.crc.id.au/
>>>>>
>>>>> ~]# rpm -qa | grep xen
>>>>> xen47-licenses-4.7.2-4.el6.x86_64
>>>>> xen47-4.7.2-4.el6.x86_64
>>>>> kernel-xen-4.9.21-1.el6xen.x86_64
>>>>> xen47-ocaml-4.7.2-4.el6.x86_64
>>>>> xen47-libs-4.7.2-4.el6.x86_64
>>>>> xen47-libcacard-4.7.2-4.el6.x86_64
>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64
>>>>> xen47-runtime-4.7.2-4.el6.x86_64
>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
>>>>>
>>>>> I've also replicated the issue with 4.9.17 and 4.9.20
>>>>>
>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the
>>>>> following on the VM
>>>>>
>>>>> {
>>>>> while true; do
>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
>>>>> done
>>>>> }
>>>>>
>>>>> Then on the dom0 I do this sequence to reliably get a null domain.
>>>>> This
>>>>> occurs with oxenstored and xenstored both.
>>>>>
>>>>> {
>>>>> xl sync 1
>>>>> xl destroy 1
>>>>> }
>>>>>
>>>>> xl list then renders something like ...
>>>>>
>>>>> (null) 1 4 4 --p--d
>>>>> 9.8 0
>>>>
>>>> Something is referencing the domain, e.g. some of its memory pages are
>>>> still mapped by dom0.
>>
>> You can try
>> # xl debug-keys q
>> and further
>> # xl dmesg
>> to see the output of the previous command. The 'q' dumps domain
>> (and guest debug) info.
>> # xl debug-keys h
>> prints all possible parameters for more informations.
>>
>> Dietmar.
>>
>
> I've done this as requested, below is the output.
>
> <snip>
> (XEN) Memory pages belonging to domain 1:
> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001
> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001
There are 16 pages still referenced from somewhere.
Can you grab all of `xenstore-ls -f` in dom0? There is probably a
backend still attached.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-11 22:23 ` Andrew Cooper
@ 2017-04-11 22:45 ` Glenn Enright
2017-04-18 8:36 ` Juergen Gross
0 siblings, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-04-11 22:45 UTC (permalink / raw)
To: Andrew Cooper, Dietmar Hahn, xen-devel; +Cc: Juergen Gross
On 12/04/17 10:23, Andrew Cooper wrote:
> On 11/04/2017 23:13, Glenn Enright wrote:
>> On 11/04/17 21:49, Dietmar Hahn wrote:
>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright:
>>>> On 11/04/17 17:59, Juergen Gross wrote:
>>>>> On 11/04/17 07:25, Glenn Enright wrote:
>>>>>> Hi all
>>>>>>
>>>>>> We are seeing an odd issue with domu domains from xl destroy, under
>>>>>> recent 4.9 kernels a (null) domain is left behind.
>>>>>
>>>>> I guess this is the dom0 kernel version?
>>>>>
>>>>>> This has occurred on a variety of hardware, with no obvious
>>>>>> commonality.
>>>>>>
>>>>>> 4.4.55 does not show this behavior.
>>>>>>
>>>>>> On my test machine I have the following packages installed under
>>>>>> centos6, from https://xen.crc.id.au/
>>>>>>
>>>>>> ~]# rpm -qa | grep xen
>>>>>> xen47-licenses-4.7.2-4.el6.x86_64
>>>>>> xen47-4.7.2-4.el6.x86_64
>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64
>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64
>>>>>> xen47-libs-4.7.2-4.el6.x86_64
>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64
>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64
>>>>>> xen47-runtime-4.7.2-4.el6.x86_64
>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
>>>>>>
>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20
>>>>>>
>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the
>>>>>> following on the VM
>>>>>>
>>>>>> {
>>>>>> while true; do
>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
>>>>>> done
>>>>>> }
>>>>>>
>>>>>> Then on the dom0 I do this sequence to reliably get a null domain.
>>>>>> This
>>>>>> occurs with oxenstored and xenstored both.
>>>>>>
>>>>>> {
>>>>>> xl sync 1
>>>>>> xl destroy 1
>>>>>> }
>>>>>>
>>>>>> xl list then renders something like ...
>>>>>>
>>>>>> (null) 1 4 4 --p--d
>>>>>> 9.8 0
>>>>>
>>>>> Something is referencing the domain, e.g. some of its memory pages are
>>>>> still mapped by dom0.
>>>
>>> You can try
>>> # xl debug-keys q
>>> and further
>>> # xl dmesg
>>> to see the output of the previous command. The 'q' dumps domain
>>> (and guest debug) info.
>>> # xl debug-keys h
>>> prints all possible parameters for more informations.
>>>
>>> Dietmar.
>>>
>>
>> I've done this as requested, below is the output.
>>
>> <snip>
>> (XEN) Memory pages belonging to domain 1:
>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001
>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001
>
> There are 16 pages still referenced from somewhere.
>
> Can you grab all of `xenstore-ls -f` in dom0? There is probably a
> backend still attached.
>
> ~Andrew
>
Note this is under oxenstored presently.
# xenstore-ls -f
/local = ""
/local/domain = ""
/local/domain/0 = ""
/local/domain/0/control = ""
/local/domain/0/control/feature-poweroff = "1"
/local/domain/0/control/feature-reboot = "1"
/local/domain/0/control/feature-suspend = "1"
/local/domain/0/domid = "0"
/local/domain/0/name = "Domain-0"
/vm = ""
/libxl = ""
# xl list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 1512 2 r-----
35.0
(null) 1 8 4 --p--d
8.5
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-11 22:45 ` Glenn Enright
@ 2017-04-18 8:36 ` Juergen Gross
2017-04-19 1:02 ` Glenn Enright
0 siblings, 1 reply; 28+ messages in thread
From: Juergen Gross @ 2017-04-18 8:36 UTC (permalink / raw)
To: glenn, Andrew Cooper, Dietmar Hahn, xen-devel
[-- Attachment #1: Type: text/plain, Size: 3823 bytes --]
On 12/04/17 00:45, Glenn Enright wrote:
> On 12/04/17 10:23, Andrew Cooper wrote:
>> On 11/04/2017 23:13, Glenn Enright wrote:
>>> On 11/04/17 21:49, Dietmar Hahn wrote:
>>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright:
>>>>> On 11/04/17 17:59, Juergen Gross wrote:
>>>>>> On 11/04/17 07:25, Glenn Enright wrote:
>>>>>>> Hi all
>>>>>>>
>>>>>>> We are seeing an odd issue with domu domains from xl destroy, under
>>>>>>> recent 4.9 kernels a (null) domain is left behind.
>>>>>>
>>>>>> I guess this is the dom0 kernel version?
>>>>>>
>>>>>>> This has occurred on a variety of hardware, with no obvious
>>>>>>> commonality.
>>>>>>>
>>>>>>> 4.4.55 does not show this behavior.
>>>>>>>
>>>>>>> On my test machine I have the following packages installed under
>>>>>>> centos6, from https://xen.crc.id.au/
>>>>>>>
>>>>>>> ~]# rpm -qa | grep xen
>>>>>>> xen47-licenses-4.7.2-4.el6.x86_64
>>>>>>> xen47-4.7.2-4.el6.x86_64
>>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64
>>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64
>>>>>>> xen47-libs-4.7.2-4.el6.x86_64
>>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64
>>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64
>>>>>>> xen47-runtime-4.7.2-4.el6.x86_64
>>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
>>>>>>>
>>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20
>>>>>>>
>>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the
>>>>>>> following on the VM
>>>>>>>
>>>>>>> {
>>>>>>> while true; do
>>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
>>>>>>> done
>>>>>>> }
>>>>>>>
>>>>>>> Then on the dom0 I do this sequence to reliably get a null domain.
>>>>>>> This
>>>>>>> occurs with oxenstored and xenstored both.
>>>>>>>
>>>>>>> {
>>>>>>> xl sync 1
>>>>>>> xl destroy 1
>>>>>>> }
>>>>>>>
>>>>>>> xl list then renders something like ...
>>>>>>>
>>>>>>> (null) 1 4 4
>>>>>>> --p--d
>>>>>>> 9.8 0
>>>>>>
>>>>>> Something is referencing the domain, e.g. some of its memory pages
>>>>>> are
>>>>>> still mapped by dom0.
>>>>
>>>> You can try
>>>> # xl debug-keys q
>>>> and further
>>>> # xl dmesg
>>>> to see the output of the previous command. The 'q' dumps domain
>>>> (and guest debug) info.
>>>> # xl debug-keys h
>>>> prints all possible parameters for more informations.
>>>>
>>>> Dietmar.
>>>>
>>>
>>> I've done this as requested, below is the output.
>>>
>>> <snip>
>>> (XEN) Memory pages belonging to domain 1:
>>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001
>>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001
>>
>> There are 16 pages still referenced from somewhere.
Just a wild guess: could you please try the attached kernel patch? This
might give us some more diagnostic data...
Juergen
[-- Attachment #2: debug.patch --]
[-- Type: text/x-patch, Size: 1426 bytes --]
diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c
index 8fe61b5dc5a6..304d5d130e0c 100644
--- a/drivers/block/xen-blkback/xenbus.c
+++ b/drivers/block/xen-blkback/xenbus.c
@@ -313,7 +313,7 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif)
static void xen_blkif_free(struct xen_blkif *blkif)
{
- xen_blkif_disconnect(blkif);
+ WARN_ON(xen_blkif_disconnect(blkif));
xen_vbd_free(&blkif->vbd);
/* Make sure everything is drained before shutting down */
@@ -505,7 +505,7 @@ static int xen_blkbk_remove(struct xenbus_device *dev)
dev_set_drvdata(&dev->dev, NULL);
if (be->blkif)
- xen_blkif_disconnect(be->blkif);
+ WARN_ON(xen_blkif_disconnect(be->blkif));
/* Put the reference we set in xen_blkif_alloc(). */
xen_blkif_put(be->blkif);
@@ -792,7 +792,7 @@ static void frontend_changed(struct xenbus_device *dev,
* Clean up so that memory resources can be used by
* other devices. connect_ring reported already error.
*/
- xen_blkif_disconnect(be->blkif);
+ WARN_ON(xen_blkif_disconnect(be->blkif));
break;
}
xen_update_blkif_status(be->blkif);
@@ -803,7 +803,7 @@ static void frontend_changed(struct xenbus_device *dev,
break;
case XenbusStateClosed:
- xen_blkif_disconnect(be->blkif);
+ WARN_ON(xen_blkif_disconnect(be->blkif));
xenbus_switch_state(dev, XenbusStateClosed);
if (xenbus_dev_is_online(dev))
break;
[-- Attachment #3: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-18 8:36 ` Juergen Gross
@ 2017-04-19 1:02 ` Glenn Enright
2017-04-19 4:39 ` Juergen Gross
0 siblings, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-04-19 1:02 UTC (permalink / raw)
To: Juergen Gross, Andrew Cooper, Dietmar Hahn, xen-devel
On 18/04/17 20:36, Juergen Gross wrote:
> On 12/04/17 00:45, Glenn Enright wrote:
>> On 12/04/17 10:23, Andrew Cooper wrote:
>>> On 11/04/2017 23:13, Glenn Enright wrote:
>>>> On 11/04/17 21:49, Dietmar Hahn wrote:
>>>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright:
>>>>>> On 11/04/17 17:59, Juergen Gross wrote:
>>>>>>> On 11/04/17 07:25, Glenn Enright wrote:
>>>>>>>> Hi all
>>>>>>>>
>>>>>>>> We are seeing an odd issue with domu domains from xl destroy, under
>>>>>>>> recent 4.9 kernels a (null) domain is left behind.
>>>>>>>
>>>>>>> I guess this is the dom0 kernel version?
>>>>>>>
>>>>>>>> This has occurred on a variety of hardware, with no obvious
>>>>>>>> commonality.
>>>>>>>>
>>>>>>>> 4.4.55 does not show this behavior.
>>>>>>>>
>>>>>>>> On my test machine I have the following packages installed under
>>>>>>>> centos6, from https://xen.crc.id.au/
>>>>>>>>
>>>>>>>> ~]# rpm -qa | grep xen
>>>>>>>> xen47-licenses-4.7.2-4.el6.x86_64
>>>>>>>> xen47-4.7.2-4.el6.x86_64
>>>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64
>>>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64
>>>>>>>> xen47-libs-4.7.2-4.el6.x86_64
>>>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64
>>>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64
>>>>>>>> xen47-runtime-4.7.2-4.el6.x86_64
>>>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
>>>>>>>>
>>>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20
>>>>>>>>
>>>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the
>>>>>>>> following on the VM
>>>>>>>>
>>>>>>>> {
>>>>>>>> while true; do
>>>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
>>>>>>>> done
>>>>>>>> }
>>>>>>>>
>>>>>>>> Then on the dom0 I do this sequence to reliably get a null domain.
>>>>>>>> This
>>>>>>>> occurs with oxenstored and xenstored both.
>>>>>>>>
>>>>>>>> {
>>>>>>>> xl sync 1
>>>>>>>> xl destroy 1
>>>>>>>> }
>>>>>>>>
>>>>>>>> xl list then renders something like ...
>>>>>>>>
>>>>>>>> (null) 1 4 4
>>>>>>>> --p--d
>>>>>>>> 9.8 0
>>>>>>>
>>>>>>> Something is referencing the domain, e.g. some of its memory pages
>>>>>>> are
>>>>>>> still mapped by dom0.
>>>>>
>>>>> You can try
>>>>> # xl debug-keys q
>>>>> and further
>>>>> # xl dmesg
>>>>> to see the output of the previous command. The 'q' dumps domain
>>>>> (and guest debug) info.
>>>>> # xl debug-keys h
>>>>> prints all possible parameters for more informations.
>>>>>
>>>>> Dietmar.
>>>>>
>>>>
>>>> I've done this as requested, below is the output.
>>>>
>>>> <snip>
>>>> (XEN) Memory pages belonging to domain 1:
>>>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001
>>>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001
>>>
>>> There are 16 pages still referenced from somewhere.
>
> Just a wild guess: could you please try the attached kernel patch? This
> might give us some more diagnostic data...
>
>
> Juergen
>
Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
shows the issue. When replicating the leak I now see this trace (via
dmesg). Hopefully that is useful.
Please note, I'm going to be offline next week, but am keen to keep on
with this, it may just be a while before I followup is all.
Regards, Glenn
http://rimuhosting.com
------------[ cut here ]------------
WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
xen_blkbk_remove+0x138/0x140
Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus
i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy
dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
Call Trace:
[<ffffffff8136b61f>] dump_stack+0x67/0x98
[<ffffffff8108007d>] __warn+0xfd/0x120
[<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
[<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
[<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
[<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
[<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
[<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
[<ffffffff814b93a2>] device_del+0x112/0x210
[<ffffffff81448113>] ? xenbus_read+0x53/0x70
[<ffffffff814b94c2>] device_unregister+0x22/0x60
[<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
[<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
[<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
[<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
[<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
[<ffffffff81449fe0>] frontend_changed+0x10/0x20
[<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
[<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
[<ffffffff816ed93a>] ? schedule+0x3a/0xa0
[<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
[<ffffffff810c0c5d>] ? complete+0x4d/0x60
[<ffffffff81447760>] ? split+0xf0/0xf0
[<ffffffff810a051d>] kthread+0xcd/0xf0
[<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
[<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
[<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
[<ffffffff816f1b45>] ret_from_fork+0x25/0x30
---[ end trace ee097287c9865a62 ]---
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-19 1:02 ` Glenn Enright
@ 2017-04-19 4:39 ` Juergen Gross
2017-04-19 7:16 ` Roger Pau Monné
0 siblings, 1 reply; 28+ messages in thread
From: Juergen Gross @ 2017-04-19 4:39 UTC (permalink / raw)
To: glenn, xen-devel, Konrad Rzeszutek Wilk, Roger Pau Monné
Cc: Andrew Cooper, Dietmar Hahn
On 19/04/17 03:02, Glenn Enright wrote:
> On 18/04/17 20:36, Juergen Gross wrote:
>> On 12/04/17 00:45, Glenn Enright wrote:
>>> On 12/04/17 10:23, Andrew Cooper wrote:
>>>> On 11/04/2017 23:13, Glenn Enright wrote:
>>>>> On 11/04/17 21:49, Dietmar Hahn wrote:
>>>>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright:
>>>>>>> On 11/04/17 17:59, Juergen Gross wrote:
>>>>>>>> On 11/04/17 07:25, Glenn Enright wrote:
>>>>>>>>> Hi all
>>>>>>>>>
>>>>>>>>> We are seeing an odd issue with domu domains from xl destroy,
>>>>>>>>> under
>>>>>>>>> recent 4.9 kernels a (null) domain is left behind.
>>>>>>>>
>>>>>>>> I guess this is the dom0 kernel version?
>>>>>>>>
>>>>>>>>> This has occurred on a variety of hardware, with no obvious
>>>>>>>>> commonality.
>>>>>>>>>
>>>>>>>>> 4.4.55 does not show this behavior.
>>>>>>>>>
>>>>>>>>> On my test machine I have the following packages installed under
>>>>>>>>> centos6, from https://xen.crc.id.au/
>>>>>>>>>
>>>>>>>>> ~]# rpm -qa | grep xen
>>>>>>>>> xen47-licenses-4.7.2-4.el6.x86_64
>>>>>>>>> xen47-4.7.2-4.el6.x86_64
>>>>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64
>>>>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64
>>>>>>>>> xen47-libs-4.7.2-4.el6.x86_64
>>>>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64
>>>>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64
>>>>>>>>> xen47-runtime-4.7.2-4.el6.x86_64
>>>>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
>>>>>>>>>
>>>>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20
>>>>>>>>>
>>>>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the
>>>>>>>>> following on the VM
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> while true; do
>>>>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
>>>>>>>>> done
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Then on the dom0 I do this sequence to reliably get a null domain.
>>>>>>>>> This
>>>>>>>>> occurs with oxenstored and xenstored both.
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> xl sync 1
>>>>>>>>> xl destroy 1
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> xl list then renders something like ...
>>>>>>>>>
>>>>>>>>> (null) 1 4 4
>>>>>>>>> --p--d
>>>>>>>>> 9.8 0
>>>>>>>>
>>>>>>>> Something is referencing the domain, e.g. some of its memory pages
>>>>>>>> are
>>>>>>>> still mapped by dom0.
>>>>>>
>>>>>> You can try
>>>>>> # xl debug-keys q
>>>>>> and further
>>>>>> # xl dmesg
>>>>>> to see the output of the previous command. The 'q' dumps domain
>>>>>> (and guest debug) info.
>>>>>> # xl debug-keys h
>>>>>> prints all possible parameters for more informations.
>>>>>>
>>>>>> Dietmar.
>>>>>>
>>>>>
>>>>> I've done this as requested, below is the output.
>>>>>
>>>>> <snip>
>>>>> (XEN) Memory pages belonging to domain 1:
>>>>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001
>>>>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001
>>>>
>>>> There are 16 pages still referenced from somewhere.
>>
>> Just a wild guess: could you please try the attached kernel patch? This
>> might give us some more diagnostic data...
>>
>>
>> Juergen
>>
>
> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
> shows the issue. When replicating the leak I now see this trace (via
> dmesg). Hopefully that is useful.
>
> Please note, I'm going to be offline next week, but am keen to keep on
> with this, it may just be a while before I followup is all.
>
> Regards, Glenn
> http://rimuhosting.com
>
>
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
> xen_blkbk_remove+0x138/0x140
> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus
> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy
> dm_mirror dm_region_hash dm_log dm_mod
> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
> Call Trace:
> [<ffffffff8136b61f>] dump_stack+0x67/0x98
> [<ffffffff8108007d>] __warn+0xfd/0x120
> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
> [<ffffffff814b93a2>] device_del+0x112/0x210
> [<ffffffff81448113>] ? xenbus_read+0x53/0x70
> [<ffffffff814b94c2>] device_unregister+0x22/0x60
> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
> [<ffffffff81449fe0>] frontend_changed+0x10/0x20
> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
> [<ffffffff810c0c5d>] ? complete+0x4d/0x60
> [<ffffffff81447760>] ? split+0xf0/0xf0
> [<ffffffff810a051d>] kthread+0xcd/0xf0
> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
> ---[ end trace ee097287c9865a62 ]---
Konrad, Roger,
this was triggered by a debug patch in xen_blkbk_remove():
if (be->blkif)
- xen_blkif_disconnect(be->blkif);
+ WARN_ON(xen_blkif_disconnect(be->blkif));
So I guess we need something like xen_blk_drain_io() in case of calls to
xen_blkif_disconnect() which are not allowed to fail (either at the call
sites of xen_blkif_disconnect() or in this function depending on a new
boolean parameter indicating it should wait for outstanding I/Os).
I can try a patch, but I'd appreciate if you could confirm this wouldn't
add further problems...
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-19 4:39 ` Juergen Gross
@ 2017-04-19 7:16 ` Roger Pau Monné
2017-04-19 7:35 ` Juergen Gross
2017-04-19 10:09 ` Juergen Gross
0 siblings, 2 replies; 28+ messages in thread
From: Roger Pau Monné @ 2017-04-19 7:16 UTC (permalink / raw)
To: Juergen Gross; +Cc: Andrew Cooper, xen-devel, Dietmar Hahn, glenn
On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
> On 19/04/17 03:02, Glenn Enright wrote:
> > On 18/04/17 20:36, Juergen Gross wrote:
> >> On 12/04/17 00:45, Glenn Enright wrote:
> >>> On 12/04/17 10:23, Andrew Cooper wrote:
> >>>> On 11/04/2017 23:13, Glenn Enright wrote:
> >>>>> On 11/04/17 21:49, Dietmar Hahn wrote:
> >>>>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright:
> >>>>>>> On 11/04/17 17:59, Juergen Gross wrote:
> >>>>>>>> On 11/04/17 07:25, Glenn Enright wrote:
> >>>>>>>>> Hi all
> >>>>>>>>>
> >>>>>>>>> We are seeing an odd issue with domu domains from xl destroy,
> >>>>>>>>> under
> >>>>>>>>> recent 4.9 kernels a (null) domain is left behind.
> >>>>>>>>
> >>>>>>>> I guess this is the dom0 kernel version?
> >>>>>>>>
> >>>>>>>>> This has occurred on a variety of hardware, with no obvious
> >>>>>>>>> commonality.
> >>>>>>>>>
> >>>>>>>>> 4.4.55 does not show this behavior.
> >>>>>>>>>
> >>>>>>>>> On my test machine I have the following packages installed under
> >>>>>>>>> centos6, from https://xen.crc.id.au/
> >>>>>>>>>
> >>>>>>>>> ~]# rpm -qa | grep xen
> >>>>>>>>> xen47-licenses-4.7.2-4.el6.x86_64
> >>>>>>>>> xen47-4.7.2-4.el6.x86_64
> >>>>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64
> >>>>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64
> >>>>>>>>> xen47-libs-4.7.2-4.el6.x86_64
> >>>>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64
> >>>>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64
> >>>>>>>>> xen47-runtime-4.7.2-4.el6.x86_64
> >>>>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
> >>>>>>>>>
> >>>>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20
> >>>>>>>>>
> >>>>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the
> >>>>>>>>> following on the VM
> >>>>>>>>>
> >>>>>>>>> {
> >>>>>>>>> while true; do
> >>>>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
> >>>>>>>>> done
> >>>>>>>>> }
> >>>>>>>>>
> >>>>>>>>> Then on the dom0 I do this sequence to reliably get a null domain.
> >>>>>>>>> This
> >>>>>>>>> occurs with oxenstored and xenstored both.
> >>>>>>>>>
> >>>>>>>>> {
> >>>>>>>>> xl sync 1
> >>>>>>>>> xl destroy 1
> >>>>>>>>> }
> >>>>>>>>>
> >>>>>>>>> xl list then renders something like ...
> >>>>>>>>>
> >>>>>>>>> (null) 1 4 4
> >>>>>>>>> --p--d
> >>>>>>>>> 9.8 0
> >>>>>>>>
> >>>>>>>> Something is referencing the domain, e.g. some of its memory pages
> >>>>>>>> are
> >>>>>>>> still mapped by dom0.
> >>>>>>
> >>>>>> You can try
> >>>>>> # xl debug-keys q
> >>>>>> and further
> >>>>>> # xl dmesg
> >>>>>> to see the output of the previous command. The 'q' dumps domain
> >>>>>> (and guest debug) info.
> >>>>>> # xl debug-keys h
> >>>>>> prints all possible parameters for more informations.
> >>>>>>
> >>>>>> Dietmar.
> >>>>>>
> >>>>>
> >>>>> I've done this as requested, below is the output.
> >>>>>
> >>>>> <snip>
> >>>>> (XEN) Memory pages belonging to domain 1:
> >>>>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001
> >>>>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001
> >>>>
> >>>> There are 16 pages still referenced from somewhere.
> >>
> >> Just a wild guess: could you please try the attached kernel patch? This
> >> might give us some more diagnostic data...
> >>
> >>
> >> Juergen
> >>
> >
> > Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
> > shows the issue. When replicating the leak I now see this trace (via
> > dmesg). Hopefully that is useful.
> >
> > Please note, I'm going to be offline next week, but am keen to keep on
> > with this, it may just be a while before I followup is all.
> >
> > Regards, Glenn
> > http://rimuhosting.com
> >
> >
> > ------------[ cut here ]------------
> > WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
> > xen_blkbk_remove+0x138/0x140
> > Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
> > xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
> > ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
> > iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
> > nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
> > ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus
> > i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy
> > dm_mirror dm_region_hash dm_log dm_mod
> > CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
> > Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
> > ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
> > 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
> > ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
> > Call Trace:
> > [<ffffffff8136b61f>] dump_stack+0x67/0x98
> > [<ffffffff8108007d>] __warn+0xfd/0x120
> > [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
> > [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
> > [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
> > [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
> > [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
> > [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
> > [<ffffffff814b93a2>] device_del+0x112/0x210
> > [<ffffffff81448113>] ? xenbus_read+0x53/0x70
> > [<ffffffff814b94c2>] device_unregister+0x22/0x60
> > [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
> > [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
> > [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
> > [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
> > [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
> > [<ffffffff81449fe0>] frontend_changed+0x10/0x20
> > [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
> > [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
> > [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
> > [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
> > [<ffffffff810c0c5d>] ? complete+0x4d/0x60
> > [<ffffffff81447760>] ? split+0xf0/0xf0
> > [<ffffffff810a051d>] kthread+0xcd/0xf0
> > [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
> > [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
> > [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
> > [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
> > ---[ end trace ee097287c9865a62 ]---
>
> Konrad, Roger,
>
> this was triggered by a debug patch in xen_blkbk_remove():
>
> if (be->blkif)
> - xen_blkif_disconnect(be->blkif);
> + WARN_ON(xen_blkif_disconnect(be->blkif));
>
> So I guess we need something like xen_blk_drain_io() in case of calls to
> xen_blkif_disconnect() which are not allowed to fail (either at the call
> sites of xen_blkif_disconnect() or in this function depending on a new
> boolean parameter indicating it should wait for outstanding I/Os).
>
> I can try a patch, but I'd appreciate if you could confirm this wouldn't
> add further problems...
Hello,
Thanks for debugging this, the easiest solution seems to be to replace the
ring->inflight atomic_read check in xen_blkif_disconnect with a call to
xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to
prevent further issues like this one).
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-19 7:16 ` Roger Pau Monné
@ 2017-04-19 7:35 ` Juergen Gross
2017-04-19 10:09 ` Juergen Gross
1 sibling, 0 replies; 28+ messages in thread
From: Juergen Gross @ 2017-04-19 7:35 UTC (permalink / raw)
To: Roger Pau Monné; +Cc: Andrew Cooper, xen-devel, Dietmar Hahn, glenn
On 19/04/17 09:16, Roger Pau Monné wrote:
> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
>> On 19/04/17 03:02, Glenn Enright wrote:
>>> On 18/04/17 20:36, Juergen Gross wrote:
>>>> On 12/04/17 00:45, Glenn Enright wrote:
>>>>> On 12/04/17 10:23, Andrew Cooper wrote:
>>>>>> On 11/04/2017 23:13, Glenn Enright wrote:
>>>>>>> On 11/04/17 21:49, Dietmar Hahn wrote:
>>>>>>>> Am Dienstag, 11. April 2017, 20:03:14 schrieb Glenn Enright:
>>>>>>>>> On 11/04/17 17:59, Juergen Gross wrote:
>>>>>>>>>> On 11/04/17 07:25, Glenn Enright wrote:
>>>>>>>>>>> Hi all
>>>>>>>>>>>
>>>>>>>>>>> We are seeing an odd issue with domu domains from xl destroy,
>>>>>>>>>>> under
>>>>>>>>>>> recent 4.9 kernels a (null) domain is left behind.
>>>>>>>>>>
>>>>>>>>>> I guess this is the dom0 kernel version?
>>>>>>>>>>
>>>>>>>>>>> This has occurred on a variety of hardware, with no obvious
>>>>>>>>>>> commonality.
>>>>>>>>>>>
>>>>>>>>>>> 4.4.55 does not show this behavior.
>>>>>>>>>>>
>>>>>>>>>>> On my test machine I have the following packages installed under
>>>>>>>>>>> centos6, from https://xen.crc.id.au/
>>>>>>>>>>>
>>>>>>>>>>> ~]# rpm -qa | grep xen
>>>>>>>>>>> xen47-licenses-4.7.2-4.el6.x86_64
>>>>>>>>>>> xen47-4.7.2-4.el6.x86_64
>>>>>>>>>>> kernel-xen-4.9.21-1.el6xen.x86_64
>>>>>>>>>>> xen47-ocaml-4.7.2-4.el6.x86_64
>>>>>>>>>>> xen47-libs-4.7.2-4.el6.x86_64
>>>>>>>>>>> xen47-libcacard-4.7.2-4.el6.x86_64
>>>>>>>>>>> xen47-hypervisor-4.7.2-4.el6.x86_64
>>>>>>>>>>> xen47-runtime-4.7.2-4.el6.x86_64
>>>>>>>>>>> kernel-xen-firmware-4.9.21-1.el6xen.x86_64
>>>>>>>>>>>
>>>>>>>>>>> I've also replicated the issue with 4.9.17 and 4.9.20
>>>>>>>>>>>
>>>>>>>>>>> To replicate, on a cleanly booted dom0 with one pv VM, I run the
>>>>>>>>>>> following on the VM
>>>>>>>>>>>
>>>>>>>>>>> {
>>>>>>>>>>> while true; do
>>>>>>>>>>> dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
>>>>>>>>>>> done
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> Then on the dom0 I do this sequence to reliably get a null domain.
>>>>>>>>>>> This
>>>>>>>>>>> occurs with oxenstored and xenstored both.
>>>>>>>>>>>
>>>>>>>>>>> {
>>>>>>>>>>> xl sync 1
>>>>>>>>>>> xl destroy 1
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> xl list then renders something like ...
>>>>>>>>>>>
>>>>>>>>>>> (null) 1 4 4
>>>>>>>>>>> --p--d
>>>>>>>>>>> 9.8 0
>>>>>>>>>>
>>>>>>>>>> Something is referencing the domain, e.g. some of its memory pages
>>>>>>>>>> are
>>>>>>>>>> still mapped by dom0.
>>>>>>>>
>>>>>>>> You can try
>>>>>>>> # xl debug-keys q
>>>>>>>> and further
>>>>>>>> # xl dmesg
>>>>>>>> to see the output of the previous command. The 'q' dumps domain
>>>>>>>> (and guest debug) info.
>>>>>>>> # xl debug-keys h
>>>>>>>> prints all possible parameters for more informations.
>>>>>>>>
>>>>>>>> Dietmar.
>>>>>>>>
>>>>>>>
>>>>>>> I've done this as requested, below is the output.
>>>>>>>
>>>>>>> <snip>
>>>>>>> (XEN) Memory pages belonging to domain 1:
>>>>>>> (XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001
>>>>>>> (XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001
>>>>>>
>>>>>> There are 16 pages still referenced from somewhere.
>>>>
>>>> Just a wild guess: could you please try the attached kernel patch? This
>>>> might give us some more diagnostic data...
>>>>
>>>>
>>>> Juergen
>>>>
>>>
>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
>>> shows the issue. When replicating the leak I now see this trace (via
>>> dmesg). Hopefully that is useful.
>>>
>>> Please note, I'm going to be offline next week, but am keen to keep on
>>> with this, it may just be a while before I followup is all.
>>>
>>> Regards, Glenn
>>> http://rimuhosting.com
>>>
>>>
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
>>> xen_blkbk_remove+0x138/0x140
>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus
>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy
>>> dm_mirror dm_region_hash dm_log dm_mod
>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
>>> Call Trace:
>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98
>>> [<ffffffff8108007d>] __warn+0xfd/0x120
>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
>>> [<ffffffff814b93a2>] device_del+0x112/0x210
>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70
>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60
>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20
>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60
>>> [<ffffffff81447760>] ? split+0xf0/0xf0
>>> [<ffffffff810a051d>] kthread+0xcd/0xf0
>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
>>> ---[ end trace ee097287c9865a62 ]---
>>
>> Konrad, Roger,
>>
>> this was triggered by a debug patch in xen_blkbk_remove():
>>
>> if (be->blkif)
>> - xen_blkif_disconnect(be->blkif);
>> + WARN_ON(xen_blkif_disconnect(be->blkif));
>>
>> So I guess we need something like xen_blk_drain_io() in case of calls to
>> xen_blkif_disconnect() which are not allowed to fail (either at the call
>> sites of xen_blkif_disconnect() or in this function depending on a new
>> boolean parameter indicating it should wait for outstanding I/Os).
>>
>> I can try a patch, but I'd appreciate if you could confirm this wouldn't
>> add further problems...
>
> Hello,
>
> Thanks for debugging this, the easiest solution seems to be to replace the
> ring->inflight atomic_read check in xen_blkif_disconnect with a call to
> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to
> prevent further issues like this one).
Nah, this isn't going to work. Or at least it won't work as it was
designed to. :-)
The main problem seems to be that xen_blkif_get/put() are used for
multiple purposes: they shouldn't be used by xen_blkif_alloc_rings() and
xen_blkif_disconnect() as they will prohibit xen_blkif_deferred_free()
being called when an I/O is terminated and xen_blkif_put() is meant to
free all remaining resources.
I'll write a patch to correct this.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-19 7:16 ` Roger Pau Monné
2017-04-19 7:35 ` Juergen Gross
@ 2017-04-19 10:09 ` Juergen Gross
2017-04-19 16:22 ` Steven Haigh
2017-05-01 0:55 ` Glenn Enright
1 sibling, 2 replies; 28+ messages in thread
From: Juergen Gross @ 2017-04-19 10:09 UTC (permalink / raw)
To: glenn; +Cc: Andrew Cooper, xen-devel, Dietmar Hahn, Roger Pau Monné
[-- Attachment #1: Type: text/plain, Size: 4099 bytes --]
On 19/04/17 09:16, Roger Pau Monné wrote:
> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
>> On 19/04/17 03:02, Glenn Enright wrote:
>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
>>> shows the issue. When replicating the leak I now see this trace (via
>>> dmesg). Hopefully that is useful.
>>>
>>> Please note, I'm going to be offline next week, but am keen to keep on
>>> with this, it may just be a while before I followup is all.
>>>
>>> Regards, Glenn
>>> http://rimuhosting.com
>>>
>>>
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
>>> xen_blkbk_remove+0x138/0x140
>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus
>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy
>>> dm_mirror dm_region_hash dm_log dm_mod
>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
>>> Call Trace:
>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98
>>> [<ffffffff8108007d>] __warn+0xfd/0x120
>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
>>> [<ffffffff814b93a2>] device_del+0x112/0x210
>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70
>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60
>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20
>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60
>>> [<ffffffff81447760>] ? split+0xf0/0xf0
>>> [<ffffffff810a051d>] kthread+0xcd/0xf0
>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
>>> ---[ end trace ee097287c9865a62 ]---
>>
>> Konrad, Roger,
>>
>> this was triggered by a debug patch in xen_blkbk_remove():
>>
>> if (be->blkif)
>> - xen_blkif_disconnect(be->blkif);
>> + WARN_ON(xen_blkif_disconnect(be->blkif));
>>
>> So I guess we need something like xen_blk_drain_io() in case of calls to
>> xen_blkif_disconnect() which are not allowed to fail (either at the call
>> sites of xen_blkif_disconnect() or in this function depending on a new
>> boolean parameter indicating it should wait for outstanding I/Os).
>>
>> I can try a patch, but I'd appreciate if you could confirm this wouldn't
>> add further problems...
>
> Hello,
>
> Thanks for debugging this, the easiest solution seems to be to replace the
> ring->inflight atomic_read check in xen_blkif_disconnect with a call to
> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to
> prevent further issues like this one).
Glenn,
can you please try the attached patch (in dom0)?
Juergen
[-- Attachment #2: blkback.patch --]
[-- Type: text/x-patch, Size: 2698 bytes --]
commit fd4252549f5f3e4de6d887d9ae4c4d7f35d7de52
Author: Juergen Gross <jgross@suse.com>
Date: Wed Apr 19 11:19:51 2017 +0200
xen/blkback: fix disconnect while I/Os in flight
Today disconnecting xen-blkback is broken in case there are still
I/Os in flight: xen_blkif_disconnect() will bail out early without
releasing all resources in the hope it will be called again when
the last request has terminated. This, however, won't happen as
xen_blkif_free() won't be called on termination of the last running
request: xen_blkif_put() won't decrement the blkif refcnt to 0 as
xen_blkif_disconnect() didn't finish before thus some xen_blkif_put()
calls in xen_blkif_disconnect() didn't happen.
To solve this deadlock xen_blkif_disconnect() and
xen_blkif_alloc_rings() shouldn't use xen_blkif_put() and
xen_blkif_get() but use some other way to do their accounting of
resources.
This at once fixes another error in xen_blkif_disconnect(): when it
returned early with -EBUSY for another ring than 0 it would call
xen_blkif_put() again for already handled rings on a subsequent call.
This will lead to inconsistencies in the refcnt handling.
Cc: stable@vger.kernel.org
diff --git a/drivers/block/xen-blkback/common.h b/drivers/block/xen-blkback/common.h
index dea61f6ab8cb..953f38802333 100644
--- a/drivers/block/xen-blkback/common.h
+++ b/drivers/block/xen-blkback/common.h
@@ -281,6 +281,7 @@ struct xen_blkif_ring {
wait_queue_head_t wq;
atomic_t inflight;
+ int active;
/* One thread per blkif ring. */
struct task_struct *xenblkd;
unsigned int waiting_reqs;
diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c
index 8fe61b5dc5a6..411d2ded2456 100644
--- a/drivers/block/xen-blkback/xenbus.c
+++ b/drivers/block/xen-blkback/xenbus.c
@@ -159,7 +159,7 @@ static int xen_blkif_alloc_rings(struct xen_blkif *blkif)
init_waitqueue_head(&ring->shutdown_wq);
ring->blkif = blkif;
ring->st_print = jiffies;
- xen_blkif_get(blkif);
+ ring->active = 1;
}
return 0;
@@ -249,6 +249,9 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif)
struct xen_blkif_ring *ring = &blkif->rings[r];
unsigned int i = 0;
+ if (!ring->active)
+ continue;
+
if (ring->xenblkd) {
kthread_stop(ring->xenblkd);
wake_up(&ring->shutdown_wq);
@@ -296,7 +299,7 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif)
BUG_ON(ring->free_pages_num != 0);
BUG_ON(ring->persistent_gnt_c != 0);
WARN_ON(i != (XEN_BLKIF_REQS_PER_PAGE * blkif->nr_ring_pages));
- xen_blkif_put(blkif);
+ ring->active = 0;
}
blkif->nr_ring_pages = 0;
/*
[-- Attachment #3: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-19 10:09 ` Juergen Gross
@ 2017-04-19 16:22 ` Steven Haigh
2017-04-21 8:42 ` Steven Haigh
2017-05-01 0:55 ` Glenn Enright
1 sibling, 1 reply; 28+ messages in thread
From: Steven Haigh @ 2017-04-19 16:22 UTC (permalink / raw)
To: Juergen Gross, glenn
Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 4754 bytes --]
On 19/04/17 20:09, Juergen Gross wrote:
> On 19/04/17 09:16, Roger Pau Monné wrote:
>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
>>> On 19/04/17 03:02, Glenn Enright wrote:
>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
>>>> shows the issue. When replicating the leak I now see this trace (via
>>>> dmesg). Hopefully that is useful.
>>>>
>>>> Please note, I'm going to be offline next week, but am keen to keep on
>>>> with this, it may just be a while before I followup is all.
>>>>
>>>> Regards, Glenn
>>>> http://rimuhosting.com
>>>>
>>>>
>>>> ------------[ cut here ]------------
>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
>>>> xen_blkbk_remove+0x138/0x140
>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus
>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy
>>>> dm_mirror dm_region_hash dm_log dm_mod
>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
>>>> Call Trace:
>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98
>>>> [<ffffffff8108007d>] __warn+0xfd/0x120
>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
>>>> [<ffffffff814b93a2>] device_del+0x112/0x210
>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70
>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60
>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20
>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60
>>>> [<ffffffff81447760>] ? split+0xf0/0xf0
>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0
>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
>>>> ---[ end trace ee097287c9865a62 ]---
>>>
>>> Konrad, Roger,
>>>
>>> this was triggered by a debug patch in xen_blkbk_remove():
>>>
>>> if (be->blkif)
>>> - xen_blkif_disconnect(be->blkif);
>>> + WARN_ON(xen_blkif_disconnect(be->blkif));
>>>
>>> So I guess we need something like xen_blk_drain_io() in case of calls to
>>> xen_blkif_disconnect() which are not allowed to fail (either at the call
>>> sites of xen_blkif_disconnect() or in this function depending on a new
>>> boolean parameter indicating it should wait for outstanding I/Os).
>>>
>>> I can try a patch, but I'd appreciate if you could confirm this wouldn't
>>> add further problems...
>>
>> Hello,
>>
>> Thanks for debugging this, the easiest solution seems to be to replace the
>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to
>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to
>> prevent further issues like this one).
>
> Glenn,
>
> can you please try the attached patch (in dom0)?
For what its worth, I have applied this in kernel package 4.9.23-2 as
follows:
* Wed Apr 19 2017 Steven Haigh <netwiz@crc.id.au> - 4.9.23-2
- xen/blkback: fix disconnect while I/Os in flight
Its available from any 'in sync' mirror:
https://xen.crc.id.au/downloads/
Feedback welcome for both mine and Juergen's sake.
--
Steven Haigh
Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-19 16:22 ` Steven Haigh
@ 2017-04-21 8:42 ` Steven Haigh
2017-04-21 8:44 ` Juergen Gross
0 siblings, 1 reply; 28+ messages in thread
From: Steven Haigh @ 2017-04-21 8:42 UTC (permalink / raw)
To: Juergen Gross, glenn
Cc: Andrew Cooper, xen-devel, Dietmar Hahn, Roger Pau Monné
[-- Attachment #1.1.1: Type: text/plain, Size: 4853 bytes --]
On 20/04/17 02:22, Steven Haigh wrote:
> On 19/04/17 20:09, Juergen Gross wrote:
>> On 19/04/17 09:16, Roger Pau Monné wrote:
>>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
>>>> On 19/04/17 03:02, Glenn Enright wrote:
>>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
>>>>> shows the issue. When replicating the leak I now see this trace (via
>>>>> dmesg). Hopefully that is useful.
>>>>>
>>>>> Please note, I'm going to be offline next week, but am keen to keep on
>>>>> with this, it may just be a while before I followup is all.
>>>>>
>>>>> Regards, Glenn
>>>>> http://rimuhosting.com
>>>>>
>>>>>
>>>>> ------------[ cut here ]------------
>>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
>>>>> xen_blkbk_remove+0x138/0x140
>>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
>>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
>>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
>>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus
>>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy
>>>>> dm_mirror dm_region_hash dm_log dm_mod
>>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
>>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
>>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
>>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
>>>>> Call Trace:
>>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98
>>>>> [<ffffffff8108007d>] __warn+0xfd/0x120
>>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
>>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
>>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
>>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
>>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
>>>>> [<ffffffff814b93a2>] device_del+0x112/0x210
>>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70
>>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60
>>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20
>>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
>>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
>>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60
>>>>> [<ffffffff81447760>] ? split+0xf0/0xf0
>>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0
>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
>>>>> ---[ end trace ee097287c9865a62 ]---
>>>>
>>>> Konrad, Roger,
>>>>
>>>> this was triggered by a debug patch in xen_blkbk_remove():
>>>>
>>>> if (be->blkif)
>>>> - xen_blkif_disconnect(be->blkif);
>>>> + WARN_ON(xen_blkif_disconnect(be->blkif));
>>>>
>>>> So I guess we need something like xen_blk_drain_io() in case of calls to
>>>> xen_blkif_disconnect() which are not allowed to fail (either at the call
>>>> sites of xen_blkif_disconnect() or in this function depending on a new
>>>> boolean parameter indicating it should wait for outstanding I/Os).
>>>>
>>>> I can try a patch, but I'd appreciate if you could confirm this wouldn't
>>>> add further problems...
>>>
>>> Hello,
>>>
>>> Thanks for debugging this, the easiest solution seems to be to replace the
>>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to
>>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to
>>> prevent further issues like this one).
>>
>> Glenn,
>>
>> can you please try the attached patch (in dom0)?
Tested-by: Steven Haigh <netwiz@crc.id.au>
I've tried specifically with 4.9.23 and can no long make this occur in
my scenario. Also built with 4.9.24 and expecting similar results.
I'm aware Glenn has a much wider test schedule and number of systems
than me, however my testing is successful.
--
Steven Haigh
Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-21 8:42 ` Steven Haigh
@ 2017-04-21 8:44 ` Juergen Gross
0 siblings, 0 replies; 28+ messages in thread
From: Juergen Gross @ 2017-04-21 8:44 UTC (permalink / raw)
To: Steven Haigh, glenn
Cc: Andrew Cooper, xen-devel, Dietmar Hahn, Roger Pau Monné
On 21/04/17 10:42, Steven Haigh wrote:
> On 20/04/17 02:22, Steven Haigh wrote:
>> On 19/04/17 20:09, Juergen Gross wrote:
>>> On 19/04/17 09:16, Roger Pau Monné wrote:
>>>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
>>>>> On 19/04/17 03:02, Glenn Enright wrote:
>>>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
>>>>>> shows the issue. When replicating the leak I now see this trace (via
>>>>>> dmesg). Hopefully that is useful.
>>>>>>
>>>>>> Please note, I'm going to be offline next week, but am keen to keep on
>>>>>> with this, it may just be a while before I followup is all.
>>>>>>
>>>>>> Regards, Glenn
>>>>>> http://rimuhosting.com
>>>>>>
>>>>>>
>>>>>> ------------[ cut here ]------------
>>>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
>>>>>> xen_blkbk_remove+0x138/0x140
>>>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
>>>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>>>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
>>>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>>>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
>>>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus
>>>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy
>>>>>> dm_mirror dm_region_hash dm_log dm_mod
>>>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
>>>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
>>>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
>>>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
>>>>>> Call Trace:
>>>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98
>>>>>> [<ffffffff8108007d>] __warn+0xfd/0x120
>>>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
>>>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
>>>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
>>>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
>>>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
>>>>>> [<ffffffff814b93a2>] device_del+0x112/0x210
>>>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70
>>>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60
>>>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
>>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
>>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20
>>>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
>>>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
>>>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
>>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60
>>>>>> [<ffffffff81447760>] ? split+0xf0/0xf0
>>>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0
>>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
>>>>>> ---[ end trace ee097287c9865a62 ]---
>>>>>
>>>>> Konrad, Roger,
>>>>>
>>>>> this was triggered by a debug patch in xen_blkbk_remove():
>>>>>
>>>>> if (be->blkif)
>>>>> - xen_blkif_disconnect(be->blkif);
>>>>> + WARN_ON(xen_blkif_disconnect(be->blkif));
>>>>>
>>>>> So I guess we need something like xen_blk_drain_io() in case of calls to
>>>>> xen_blkif_disconnect() which are not allowed to fail (either at the call
>>>>> sites of xen_blkif_disconnect() or in this function depending on a new
>>>>> boolean parameter indicating it should wait for outstanding I/Os).
>>>>>
>>>>> I can try a patch, but I'd appreciate if you could confirm this wouldn't
>>>>> add further problems...
>>>>
>>>> Hello,
>>>>
>>>> Thanks for debugging this, the easiest solution seems to be to replace the
>>>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to
>>>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to
>>>> prevent further issues like this one).
>>>
>>> Glenn,
>>>
>>> can you please try the attached patch (in dom0)?
>
> Tested-by: Steven Haigh <netwiz@crc.id.au>
>
> I've tried specifically with 4.9.23 and can no long make this occur in
> my scenario. Also built with 4.9.24 and expecting similar results.
Thanks for testing!
Will post the patch officially now.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-04-19 10:09 ` Juergen Gross
2017-04-19 16:22 ` Steven Haigh
@ 2017-05-01 0:55 ` Glenn Enright
2017-05-03 10:45 ` Steven Haigh
1 sibling, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-05-01 0:55 UTC (permalink / raw)
To: Juergen Gross
Cc: Andrew Cooper, xen-devel, Dietmar Hahn, Roger Pau Monné
On 19/04/17 22:09, Juergen Gross wrote:
> On 19/04/17 09:16, Roger Pau Monné wrote:
>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
>>> On 19/04/17 03:02, Glenn Enright wrote:
>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
>>>> shows the issue. When replicating the leak I now see this trace (via
>>>> dmesg). Hopefully that is useful.
>>>>
>>>> Please note, I'm going to be offline next week, but am keen to keep on
>>>> with this, it may just be a while before I followup is all.
>>>>
>>>> Regards, Glenn
>>>> http://rimuhosting.com
>>>>
>>>>
>>>> ------------[ cut here ]------------
>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
>>>> xen_blkbk_remove+0x138/0x140
>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_smbus
>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci floppy
>>>> dm_mirror dm_region_hash dm_log dm_mod
>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
>>>> Call Trace:
>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98
>>>> [<ffffffff8108007d>] __warn+0xfd/0x120
>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
>>>> [<ffffffff814b93a2>] device_del+0x112/0x210
>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70
>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60
>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20
>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60
>>>> [<ffffffff81447760>] ? split+0xf0/0xf0
>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0
>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
>>>> ---[ end trace ee097287c9865a62 ]---
>>>
>>> Konrad, Roger,
>>>
>>> this was triggered by a debug patch in xen_blkbk_remove():
>>>
>>> if (be->blkif)
>>> - xen_blkif_disconnect(be->blkif);
>>> + WARN_ON(xen_blkif_disconnect(be->blkif));
>>>
>>> So I guess we need something like xen_blk_drain_io() in case of calls to
>>> xen_blkif_disconnect() which are not allowed to fail (either at the call
>>> sites of xen_blkif_disconnect() or in this function depending on a new
>>> boolean parameter indicating it should wait for outstanding I/Os).
>>>
>>> I can try a patch, but I'd appreciate if you could confirm this wouldn't
>>> add further problems...
>>
>> Hello,
>>
>> Thanks for debugging this, the easiest solution seems to be to replace the
>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to
>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void (to
>> prevent further issues like this one).
>
> Glenn,
>
> can you please try the attached patch (in dom0)?
>
>
> Juergen
>
(resending with full CC list)
I'm back. After testing unfortunately I'm still seeing the leak. The
below trace is with the debug patch applied as well under 4.9.25. It
looks very similar to me. I am still able to replicate this reliably.
Regards, Glenn
http://rimuhosting.com
------------[ cut here ]------------
WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:511
xen_blkbk_remove+0x138/0x140
Modules linked in: ebt_ip xen_pciback xen_netback xen_gntalloc
xen_gntdev xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
ipv6 crc_ccitt ppdev parport_pc parport serio_raw i2c_i801 i2c_smbus
i2c_core sg e1000e ptp pps_core i3000_edac edac_core raid1 sd_mod ahci
libahci floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.25-1.el6xen.x86_64 #1
Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
ffffc90040cfbb98 ffffffff8136b76f 0000000000000013 0000000000000000
0000000000000000 0000000000000000 ffffc90040cfbbe8 ffffffff8108007d
ffffea0000141720 000001ff41334434 ffff880000000001 ffff88004d3aedc0
Call Trace:
[<ffffffff8136b76f>] dump_stack+0x67/0x98
[<ffffffff8108007d>] __warn+0xfd/0x120
[<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
[<ffffffff814ec0a8>] xen_blkbk_remove+0x138/0x140
[<ffffffff81449b07>] xenbus_dev_remove+0x47/0xa0
[<ffffffff814bd2b4>] __device_release_driver+0xb4/0x160
[<ffffffff814bd38d>] device_release_driver+0x2d/0x40
[<ffffffff814bc2b4>] bus_remove_device+0x124/0x190
[<ffffffff814b9682>] device_del+0x112/0x210
[<ffffffff81448423>] ? xenbus_read+0x53/0x70
[<ffffffff814b97a2>] device_unregister+0x22/0x60
[<ffffffff814eda9d>] frontend_changed+0xad/0x4c0
[<ffffffff81449e67>] xenbus_otherend_changed+0xc7/0x140
[<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20
[<ffffffff8144a2f0>] frontend_changed+0x10/0x20
[<ffffffff81447b0c>] xenwatch_thread+0x9c/0x140
[<ffffffff810bffb0>] ? woken_wake_function+0x20/0x20
[<ffffffff816ed98a>] ? schedule+0x3a/0xa0
[<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20
[<ffffffff810c0c6d>] ? complete+0x4d/0x60
[<ffffffff81447a70>] ? split+0xf0/0xf0
[<ffffffff810a0535>] kthread+0xe5/0x100
[<ffffffff810a051d>] ? kthread+0xcd/0x100
[<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
[<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
[<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
[<ffffffff816f1bc5>] ret_from_fork+0x25/0x30
---[ end trace ea3a48c80e4ad79d ]---
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-01 0:55 ` Glenn Enright
@ 2017-05-03 10:45 ` Steven Haigh
2017-05-03 13:38 ` Juergen Gross
2017-05-03 15:53 ` Juergen Gross
0 siblings, 2 replies; 28+ messages in thread
From: Steven Haigh @ 2017-05-03 10:45 UTC (permalink / raw)
To: glenn, Juergen Gross
Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 7725 bytes --]
Just wanted to give this a little nudge now people seem to be back on
deck...
On 01/05/17 10:55, Glenn Enright wrote:
> On 19/04/17 22:09, Juergen Gross wrote:
>> On 19/04/17 09:16, Roger Pau Monné wrote:
>>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
>>>> On 19/04/17 03:02, Glenn Enright wrote:
>>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
>>>>> shows the issue. When replicating the leak I now see this trace (via
>>>>> dmesg). Hopefully that is useful.
>>>>>
>>>>> Please note, I'm going to be offline next week, but am keen to keep on
>>>>> with this, it may just be a while before I followup is all.
>>>>>
>>>>> Regards, Glenn
>>>>> http://rimuhosting.com
>>>>>
>>>>>
>>>>> ------------[ cut here ]------------
>>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
>>>>> xen_blkbk_remove+0x138/0x140
>>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
>>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state
>>>>> iptable_security
>>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp
>>>>> llc
>>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801
>>>>> i2c_smbus
>>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci
>>>>> floppy
>>>>> dm_mirror dm_region_hash dm_log dm_mod
>>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
>>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
>>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
>>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
>>>>> Call Trace:
>>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98
>>>>> [<ffffffff8108007d>] __warn+0xfd/0x120
>>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
>>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
>>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
>>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
>>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
>>>>> [<ffffffff814b93a2>] device_del+0x112/0x210
>>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70
>>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60
>>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20
>>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
>>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
>>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60
>>>>> [<ffffffff81447760>] ? split+0xf0/0xf0
>>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0
>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
>>>>> ---[ end trace ee097287c9865a62 ]---
>>>>
>>>> Konrad, Roger,
>>>>
>>>> this was triggered by a debug patch in xen_blkbk_remove():
>>>>
>>>> if (be->blkif)
>>>> - xen_blkif_disconnect(be->blkif);
>>>> + WARN_ON(xen_blkif_disconnect(be->blkif));
>>>>
>>>> So I guess we need something like xen_blk_drain_io() in case of
>>>> calls to
>>>> xen_blkif_disconnect() which are not allowed to fail (either at the
>>>> call
>>>> sites of xen_blkif_disconnect() or in this function depending on a new
>>>> boolean parameter indicating it should wait for outstanding I/Os).
>>>>
>>>> I can try a patch, but I'd appreciate if you could confirm this
>>>> wouldn't
>>>> add further problems...
>>>
>>> Hello,
>>>
>>> Thanks for debugging this, the easiest solution seems to be to
>>> replace the
>>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to
>>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void
>>> (to
>>> prevent further issues like this one).
>>
>> Glenn,
>>
>> can you please try the attached patch (in dom0)?
>>
>>
>> Juergen
>>
>
> (resending with full CC list)
>
> I'm back. After testing unfortunately I'm still seeing the leak. The
> below trace is with the debug patch applied as well under 4.9.25. It
> looks very similar to me. I am still able to replicate this reliably.
>
> Regards, Glenn
> http://rimuhosting.com
>
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:511
> xen_blkbk_remove+0x138/0x140
> Modules linked in: ebt_ip xen_pciback xen_netback xen_gntalloc
> xen_gntdev xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
> iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
> ipv6 crc_ccitt ppdev parport_pc parport serio_raw i2c_i801 i2c_smbus
> i2c_core sg e1000e ptp pps_core i3000_edac edac_core raid1 sd_mod ahci
> libahci floppy dm_mirror dm_region_hash dm_log dm_mod
> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.25-1.el6xen.x86_64 #1
> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
> ffffc90040cfbb98 ffffffff8136b76f 0000000000000013 0000000000000000
> 0000000000000000 0000000000000000 ffffc90040cfbbe8 ffffffff8108007d
> ffffea0000141720 000001ff41334434 ffff880000000001 ffff88004d3aedc0
> Call Trace:
> [<ffffffff8136b76f>] dump_stack+0x67/0x98
> [<ffffffff8108007d>] __warn+0xfd/0x120
> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
> [<ffffffff814ec0a8>] xen_blkbk_remove+0x138/0x140
> [<ffffffff81449b07>] xenbus_dev_remove+0x47/0xa0
> [<ffffffff814bd2b4>] __device_release_driver+0xb4/0x160
> [<ffffffff814bd38d>] device_release_driver+0x2d/0x40
> [<ffffffff814bc2b4>] bus_remove_device+0x124/0x190
> [<ffffffff814b9682>] device_del+0x112/0x210
> [<ffffffff81448423>] ? xenbus_read+0x53/0x70
> [<ffffffff814b97a2>] device_unregister+0x22/0x60
> [<ffffffff814eda9d>] frontend_changed+0xad/0x4c0
> [<ffffffff81449e67>] xenbus_otherend_changed+0xc7/0x140
> [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20
> [<ffffffff8144a2f0>] frontend_changed+0x10/0x20
> [<ffffffff81447b0c>] xenwatch_thread+0x9c/0x140
> [<ffffffff810bffb0>] ? woken_wake_function+0x20/0x20
> [<ffffffff816ed98a>] ? schedule+0x3a/0xa0
> [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20
> [<ffffffff810c0c6d>] ? complete+0x4d/0x60
> [<ffffffff81447a70>] ? split+0xf0/0xf0
> [<ffffffff810a0535>] kthread+0xe5/0x100
> [<ffffffff810a051d>] ? kthread+0xcd/0x100
> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
> [<ffffffff816f1bc5>] ret_from_fork+0x25/0x30
> ---[ end trace ea3a48c80e4ad79d ]---
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
--
Steven Haigh
Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-03 10:45 ` Steven Haigh
@ 2017-05-03 13:38 ` Juergen Gross
2017-05-03 15:53 ` Juergen Gross
1 sibling, 0 replies; 28+ messages in thread
From: Juergen Gross @ 2017-05-03 13:38 UTC (permalink / raw)
To: Steven Haigh, glenn
Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel
On 03/05/17 12:45, Steven Haigh wrote:
> Just wanted to give this a little nudge now people seem to be back on
> deck...
Things seem to be more complicated than I thought.
There are clearly paths leading to use-after-free scenarios, e.g. the
one of the backtrace below:
xen_blkbk_remove() will free be regardless of the return value of
xen_blkif_disconnect(). If -EBUSY is returned xen_blkif_disconnect()
will be called again at the end of an I/O still in progress, leading
to a call of xenbus_unmap_ring_vfree(blkif->be->dev, ...) with be
already freed...
As Roger already said: this is a complete mess.
Working on a patch now...
BTW: Glenn, the debug patch isn't important any longer. It was just
meant to locate the problem which is now known.
Juergen
>
> On 01/05/17 10:55, Glenn Enright wrote:
>> On 19/04/17 22:09, Juergen Gross wrote:
>>> On 19/04/17 09:16, Roger Pau Monné wrote:
>>>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
>>>>> On 19/04/17 03:02, Glenn Enright wrote:
>>>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
>>>>>> shows the issue. When replicating the leak I now see this trace (via
>>>>>> dmesg). Hopefully that is useful.
>>>>>>
>>>>>> Please note, I'm going to be offline next week, but am keen to keep on
>>>>>> with this, it may just be a while before I followup is all.
>>>>>>
>>>>>> Regards, Glenn
>>>>>> http://rimuhosting.com
>>>>>>
>>>>>>
>>>>>> ------------[ cut here ]------------
>>>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
>>>>>> xen_blkbk_remove+0x138/0x140
>>>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
>>>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>>>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state
>>>>>> iptable_security
>>>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>>>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp
>>>>>> llc
>>>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801
>>>>>> i2c_smbus
>>>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci
>>>>>> floppy
>>>>>> dm_mirror dm_region_hash dm_log dm_mod
>>>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
>>>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
>>>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
>>>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
>>>>>> Call Trace:
>>>>>> [<ffffffff8136b61f>] dump_stack+0x67/0x98
>>>>>> [<ffffffff8108007d>] __warn+0xfd/0x120
>>>>>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>>>>> [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
>>>>>> [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
>>>>>> [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
>>>>>> [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
>>>>>> [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
>>>>>> [<ffffffff814b93a2>] device_del+0x112/0x210
>>>>>> [<ffffffff81448113>] ? xenbus_read+0x53/0x70
>>>>>> [<ffffffff814b94c2>] device_unregister+0x22/0x60
>>>>>> [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
>>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>>> [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
>>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>>> [<ffffffff81449fe0>] frontend_changed+0x10/0x20
>>>>>> [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
>>>>>> [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
>>>>>> [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
>>>>>> [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>>> [<ffffffff810c0c5d>] ? complete+0x4d/0x60
>>>>>> [<ffffffff81447760>] ? split+0xf0/0xf0
>>>>>> [<ffffffff810a051d>] kthread+0xcd/0xf0
>>>>>> [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>>> [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
>>>>>> ---[ end trace ee097287c9865a62 ]---
>>>>>
>>>>> Konrad, Roger,
>>>>>
>>>>> this was triggered by a debug patch in xen_blkbk_remove():
>>>>>
>>>>> if (be->blkif)
>>>>> - xen_blkif_disconnect(be->blkif);
>>>>> + WARN_ON(xen_blkif_disconnect(be->blkif));
>>>>>
>>>>> So I guess we need something like xen_blk_drain_io() in case of
>>>>> calls to
>>>>> xen_blkif_disconnect() which are not allowed to fail (either at the
>>>>> call
>>>>> sites of xen_blkif_disconnect() or in this function depending on a new
>>>>> boolean parameter indicating it should wait for outstanding I/Os).
>>>>>
>>>>> I can try a patch, but I'd appreciate if you could confirm this
>>>>> wouldn't
>>>>> add further problems...
>>>>
>>>> Hello,
>>>>
>>>> Thanks for debugging this, the easiest solution seems to be to
>>>> replace the
>>>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to
>>>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void
>>>> (to
>>>> prevent further issues like this one).
>>>
>>> Glenn,
>>>
>>> can you please try the attached patch (in dom0)?
>>>
>>>
>>> Juergen
>>>
>>
>> (resending with full CC list)
>>
>> I'm back. After testing unfortunately I'm still seeing the leak. The
>> below trace is with the debug patch applied as well under 4.9.25. It
>> looks very similar to me. I am still able to replicate this reliably.
>>
>> Regards, Glenn
>> http://rimuhosting.com
>>
>> ------------[ cut here ]------------
>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:511
>> xen_blkbk_remove+0x138/0x140
>> Modules linked in: ebt_ip xen_pciback xen_netback xen_gntalloc
>> xen_gntdev xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
>> iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw i2c_i801 i2c_smbus
>> i2c_core sg e1000e ptp pps_core i3000_edac edac_core raid1 sd_mod ahci
>> libahci floppy dm_mirror dm_region_hash dm_log dm_mod
>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.25-1.el6xen.x86_64 #1
>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>> ffffc90040cfbb98 ffffffff8136b76f 0000000000000013 0000000000000000
>> 0000000000000000 0000000000000000 ffffc90040cfbbe8 ffffffff8108007d
>> ffffea0000141720 000001ff41334434 ffff880000000001 ffff88004d3aedc0
>> Call Trace:
>> [<ffffffff8136b76f>] dump_stack+0x67/0x98
>> [<ffffffff8108007d>] __warn+0xfd/0x120
>> [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>> [<ffffffff814ec0a8>] xen_blkbk_remove+0x138/0x140
>> [<ffffffff81449b07>] xenbus_dev_remove+0x47/0xa0
>> [<ffffffff814bd2b4>] __device_release_driver+0xb4/0x160
>> [<ffffffff814bd38d>] device_release_driver+0x2d/0x40
>> [<ffffffff814bc2b4>] bus_remove_device+0x124/0x190
>> [<ffffffff814b9682>] device_del+0x112/0x210
>> [<ffffffff81448423>] ? xenbus_read+0x53/0x70
>> [<ffffffff814b97a2>] device_unregister+0x22/0x60
>> [<ffffffff814eda9d>] frontend_changed+0xad/0x4c0
>> [<ffffffff81449e67>] xenbus_otherend_changed+0xc7/0x140
>> [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>> [<ffffffff8144a2f0>] frontend_changed+0x10/0x20
>> [<ffffffff81447b0c>] xenwatch_thread+0x9c/0x140
>> [<ffffffff810bffb0>] ? woken_wake_function+0x20/0x20
>> [<ffffffff816ed98a>] ? schedule+0x3a/0xa0
>> [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>> [<ffffffff810c0c6d>] ? complete+0x4d/0x60
>> [<ffffffff81447a70>] ? split+0xf0/0xf0
>> [<ffffffff810a0535>] kthread+0xe5/0x100
>> [<ffffffff810a051d>] ? kthread+0xcd/0x100
>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>> [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>> [<ffffffff816f1bc5>] ret_from_fork+0x25/0x30
>> ---[ end trace ea3a48c80e4ad79d ]---
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-03 10:45 ` Steven Haigh
2017-05-03 13:38 ` Juergen Gross
@ 2017-05-03 15:53 ` Juergen Gross
2017-05-03 16:58 ` Steven Haigh
1 sibling, 1 reply; 28+ messages in thread
From: Juergen Gross @ 2017-05-03 15:53 UTC (permalink / raw)
To: Steven Haigh, glenn
Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel
[-- Attachment #1: Type: text/plain, Size: 356 bytes --]
On 03/05/17 12:45, Steven Haigh wrote:
> Just wanted to give this a little nudge now people seem to be back on
> deck...
Glenn, could you please give the attached patch a try?
It should be applied on top of the other correction, the old debug
patch should not be applied.
I have added some debug output to make sure we see what is happening.
Juergen
[-- Attachment #2: blk.patch --]
[-- Type: text/x-patch, Size: 1969 bytes --]
commit 246aaf60bd934b7571944b98a31078d519d637c6
Author: Juergen Gross <jgross@suse.com>
Date: Wed May 3 15:57:18 2017 +0200
xen/blkback: don't free be structure too early
The be structure must nor be freed when freeing the blkif structure
isn't done. Otherwise a use-after-free of be when unmapping the ring
used for communicating with the frontend will occur in case of a
late call of xenblk_disconnect() (e.g. due to an I/O still active
when trying to disconnect).
diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c
index 411d2ded2456..0614fb294e2b 100644
--- a/drivers/block/xen-blkback/xenbus.c
+++ b/drivers/block/xen-blkback/xenbus.c
@@ -33,6 +33,7 @@ struct backend_info {
unsigned major;
unsigned minor;
char *mode;
+ int delayed;
};
static struct kmem_cache *xen_blkif_cachep;
@@ -262,8 +263,11 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif)
* don't have any discard_io or other_io requests. So, checking
* for inflight IO is enough.
*/
- if (atomic_read(&ring->inflight) > 0)
+ if (atomic_read(&ring->inflight) > 0) {
+ pr_warn("xen_blkif_disconnect: busy\n");
+ blkif->be->delayed = 1;
return -EBUSY;
+ }
if (ring->irq) {
unbind_from_irqhandler(ring->irq, ring);
@@ -315,9 +319,11 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif)
static void xen_blkif_free(struct xen_blkif *blkif)
{
-
- xen_blkif_disconnect(blkif);
+ pr_warn("xen_blkif_free: delayed = %d\n", blkif->be->delayed);
+ WARN_ON(xen_blkif_disconnect(blkif));
xen_vbd_free(&blkif->vbd);
+ kfree(blkif->be->mode);
+ kfree(blkif->be);
/* Make sure everything is drained before shutting down */
kmem_cache_free(xen_blkif_cachep, blkif);
@@ -512,8 +518,7 @@ static int xen_blkbk_remove(struct xenbus_device *dev)
/* Put the reference we set in xen_blkif_alloc(). */
xen_blkif_put(be->blkif);
- kfree(be->mode);
- kfree(be);
+
return 0;
}
[-- Attachment #3: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-03 15:53 ` Juergen Gross
@ 2017-05-03 16:58 ` Steven Haigh
2017-05-03 22:17 ` Glenn Enright
0 siblings, 1 reply; 28+ messages in thread
From: Steven Haigh @ 2017-05-03 16:58 UTC (permalink / raw)
To: Juergen Gross, glenn
Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 613 bytes --]
On 04/05/17 01:53, Juergen Gross wrote:
> On 03/05/17 12:45, Steven Haigh wrote:
>> Just wanted to give this a little nudge now people seem to be back on
>> deck...
>
> Glenn, could you please give the attached patch a try?
>
> It should be applied on top of the other correction, the old debug
> patch should not be applied.
>
> I have added some debug output to make sure we see what is happening.
This patch is included in kernel-xen-4.9.26-1
It should be in the repos now.
--
Steven Haigh
Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-03 16:58 ` Steven Haigh
@ 2017-05-03 22:17 ` Glenn Enright
2017-05-08 9:10 ` Juergen Gross
0 siblings, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-05-03 22:17 UTC (permalink / raw)
To: Steven Haigh, Juergen Gross
Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel
On 04/05/17 04:58, Steven Haigh wrote:
> On 04/05/17 01:53, Juergen Gross wrote:
>> On 03/05/17 12:45, Steven Haigh wrote:
>>> Just wanted to give this a little nudge now people seem to be back on
>>> deck...
>>
>> Glenn, could you please give the attached patch a try?
>>
>> It should be applied on top of the other correction, the old debug
>> patch should not be applied.
>>
>> I have added some debug output to make sure we see what is happening.
>
> This patch is included in kernel-xen-4.9.26-1
>
> It should be in the repos now.
>
Still seeing the same issue. Without the extra debug patch all I see in
the logs after destroy is this...
xen-blkback: xen_blkif_disconnect: busy
xen-blkback: xen_blkif_free: delayed = 0
br0: port 2(vif1.0) entered disabled state
br0: port 2(vif1.0) entered disabled state
device vif1.0 left promiscuous mode
br0: port 2(vif1.0) entered disabled state
Without the dd running in the domU, the domU exits cleanly on destroy
and I don't see the busy message from above, eg
xen-blkback: xen_blkif_free: delayed = 0
xen-blkback: xen_blkif_free: delayed = 0
br0: port 2(vif5.0) entered disabled state
br0: port 2(vif5.0) entered disabled state
device vif5.0 left promiscuous mode
br0: port 2(vif5.0) entered disabled state
Regards, Glenn
http://rimuhosting.com
{ domid=1; xl sysrq $domid s; sleep 2; xl destroy $domid; }
# xl list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 1512 2 r-----
37.9
(null) 1 8 4 --p--d
9.5
# xl inf
host : host480.rimuhosting.com
release : 4.9.26-1.el6xen.x86_64
version : #1 SMP Thu May 4 02:07:34 AEST 2017
machine : x86_64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2394
hw_caps :
b7ebfbff:0000e3bd:20100800:00000001:00000000:00000000:00000000:00000000
virt_caps :
total_memory : 8190
free_memory : 6573
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 7
xen_extra : .2
xen_version : 4.7.2
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : dom0_mem=1512M cpufreq=xen dom0_max_vcpus=2
dom0_vcpus_pin loglvl=debug vcpu_migration_delay=1000
cc_compiler : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
cc_compile_by : mockbuild
cc_compile_domain : xen.crc.id.au
cc_compile_date : Wed Apr 19 18:17:37 AEST 2017
build_id : d5f616d8c9d8b5decfdbdca9d19eacb60c666418
xend_config_format : 4
(XEN) 'q' pressed -> dumping domain info (now=0x22E:795D0BA5)
(XEN) General information for domain 0:
(XEN) refcnt=3 dying=0 pause_count=0
(XEN) nr_pages=387072 xenheap_pages=5 shared_pages=0 paged_pages=0
dirty_cpus={0-1} max_pages=4294967295
(XEN) handle=00000000-0000-0000-0000-000000000000 vm_assist=0000000d
(XEN) Rangesets belonging to domain 0:
(XEN) I/O Ports { 0-1f, 22-3f, 44-60, 62-9f, a2-cfb, d00-1007,
100c-ffff }
(XEN) log-dirty { }
(XEN) Interrupts { 1-30 }
(XEN) I/O Memory { 0-fedff, fef00-ffffff }
(XEN) Memory pages belonging to domain 0:
(XEN) DomPage list too long to display
(XEN) XenPage 000000000020e9c4: caf=c000000000000002,
taf=7400000000000002
(XEN) XenPage 000000000020e9c3: caf=c000000000000001,
taf=7400000000000001
(XEN) XenPage 000000000020e9c2: caf=c000000000000001,
taf=7400000000000001
(XEN) XenPage 000000000020e9c1: caf=c000000000000001,
taf=7400000000000001
(XEN) XenPage 00000000000e7d2e: caf=c000000000000002,
taf=7400000000000002
(XEN) NODE affinity for domain 0: [0]
(XEN) VCPU information and callbacks for domain 0:
(XEN) VCPU0: CPU0 [has=T] poll=0 upcall_pend=00 upcall_mask=00
dirty_cpus={0}
(XEN) cpu_hard_affinity={0} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=0
(XEN) No periodic timer
(XEN) VCPU1: CPU1 [has=T] poll=0 upcall_pend=00 upcall_mask=01
dirty_cpus={1}
(XEN) cpu_hard_affinity={1} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=0
(XEN) No periodic timer
(XEN) General information for domain 1:
(XEN) refcnt=1 dying=2 pause_count=2
(XEN) nr_pages=2113 xenheap_pages=0 shared_pages=0 paged_pages=0
dirty_cpus={} max_pages=1280256
(XEN) handle=7a5642fc-2372-4174-9508-aae2ad1f6908 vm_assist=0000000d
(XEN) Rangesets belonging to domain 1:
(XEN) I/O Ports { }
(XEN) log-dirty { }
(XEN) Interrupts { }
(XEN) I/O Memory { }
(XEN) Memory pages belonging to domain 1:
(XEN) DomPage 0000000000071c00: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c01: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c02: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c03: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c04: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c05: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c06: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c07: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c08: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c09: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0a: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0b: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0c: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0d: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0e: caf=00000001, taf=7400000000000001
(XEN) DomPage 0000000000071c0f: caf=00000001, taf=7400000000000001
(XEN) NODE affinity for domain 1: [0]
(XEN) VCPU information and callbacks for domain 1:
(XEN) VCPU0: CPU0 [has=F] poll=0 upcall_pend=00 upcall_mask=01
dirty_cpus={}
(XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=0
(XEN) No periodic timer
(XEN) VCPU1: CPU1 [has=F] poll=0 upcall_pend=00 upcall_mask=01
dirty_cpus={}
(XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=1
(XEN) No periodic timer
(XEN) VCPU2: CPU2 [has=F] poll=0 upcall_pend=00 upcall_mask=01
dirty_cpus={}
(XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=0
(XEN) No periodic timer
(XEN) VCPU3: CPU3 [has=F] poll=0 upcall_pend=00 upcall_mask=01
dirty_cpus={}
(XEN) cpu_hard_affinity={0-3} cpu_soft_affinity={0-3}
(XEN) pause_count=0 pause_flags=1
(XEN) No periodic timer
(XEN) Notifying guest 0:0 (virq 1, port 4)
(XEN) Notifying guest 0:1 (virq 1, port 10)
(XEN) Notifying guest 1:0 (virq 1, port 0)
(XEN) Notifying guest 1:1 (virq 1, port 0)
(XEN) Notifying guest 1:2 (virq 1, port 0)
(XEN) Notifying guest 1:3 (virq 1, port 0)
(XEN) Shared frames 0 -- Saved frames 0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-03 22:17 ` Glenn Enright
@ 2017-05-08 9:10 ` Juergen Gross
2017-05-09 9:24 ` Roger Pau Monné
0 siblings, 1 reply; 28+ messages in thread
From: Juergen Gross @ 2017-05-08 9:10 UTC (permalink / raw)
To: glenn, Steven Haigh, Jennifer Herbert
Cc: Andrew Cooper, Roger Pau Monné, Dietmar Hahn, xen-devel
On 04/05/17 00:17, Glenn Enright wrote:
> On 04/05/17 04:58, Steven Haigh wrote:
>> On 04/05/17 01:53, Juergen Gross wrote:
>>> On 03/05/17 12:45, Steven Haigh wrote:
>>>> Just wanted to give this a little nudge now people seem to be back on
>>>> deck...
>>>
>>> Glenn, could you please give the attached patch a try?
>>>
>>> It should be applied on top of the other correction, the old debug
>>> patch should not be applied.
>>>
>>> I have added some debug output to make sure we see what is happening.
>>
>> This patch is included in kernel-xen-4.9.26-1
>>
>> It should be in the repos now.
>>
>
> Still seeing the same issue. Without the extra debug patch all I see in
> the logs after destroy is this...
>
> xen-blkback: xen_blkif_disconnect: busy
> xen-blkback: xen_blkif_free: delayed = 0
Hmm, to me it seems as if some grant isn't being unmapped.
Looking at gnttab_unmap_refs_async() I wonder how this is supposed to
work:
I don't see how a grant would ever be unmapped in case of
page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it
does is deferring the call to the unmap operation again and again. Or
am I missing something here?
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-08 9:10 ` Juergen Gross
@ 2017-05-09 9:24 ` Roger Pau Monné
2017-05-13 4:02 ` Glenn Enright
0 siblings, 1 reply; 28+ messages in thread
From: Roger Pau Monné @ 2017-05-09 9:24 UTC (permalink / raw)
To: Juergen Gross
Cc: Steven Haigh, Andrew Cooper, Jennifer Herbert, Dietmar Hahn,
xen-devel, glenn
On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote:
> On 04/05/17 00:17, Glenn Enright wrote:
> > On 04/05/17 04:58, Steven Haigh wrote:
> >> On 04/05/17 01:53, Juergen Gross wrote:
> >>> On 03/05/17 12:45, Steven Haigh wrote:
> >>>> Just wanted to give this a little nudge now people seem to be back on
> >>>> deck...
> >>>
> >>> Glenn, could you please give the attached patch a try?
> >>>
> >>> It should be applied on top of the other correction, the old debug
> >>> patch should not be applied.
> >>>
> >>> I have added some debug output to make sure we see what is happening.
> >>
> >> This patch is included in kernel-xen-4.9.26-1
> >>
> >> It should be in the repos now.
> >>
> >
> > Still seeing the same issue. Without the extra debug patch all I see in
> > the logs after destroy is this...
> >
> > xen-blkback: xen_blkif_disconnect: busy
> > xen-blkback: xen_blkif_free: delayed = 0
>
> Hmm, to me it seems as if some grant isn't being unmapped.
>
> Looking at gnttab_unmap_refs_async() I wonder how this is supposed to
> work:
>
> I don't see how a grant would ever be unmapped in case of
> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it
> does is deferring the call to the unmap operation again and again. Or
> am I missing something here?
No, I don't think you are missing anything, but I cannot see how this can be
solved in a better way, unmapping a page that's still referenced is certainly
not the best option, or else we risk triggering a page-fault elsewhere.
IMHO, gnttab_unmap_refs_async should have a timeout, and return an error at
some point. Also, I'm wondering whether there's a way to keep track of who has
references on a specific page, but so far I haven't been able to figure out how
to get this information from Linux.
Also, I've noticed that __gnttab_unmap_refs_async uses page_count, shouldn't it
use page_ref_count instead?
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-09 9:24 ` Roger Pau Monné
@ 2017-05-13 4:02 ` Glenn Enright
2017-05-15 9:57 ` Juergen Gross
0 siblings, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-05-13 4:02 UTC (permalink / raw)
To: Roger Pau Monné, Juergen Gross
Cc: Jennifer Herbert, xen-devel, Steven Haigh, Dietmar Hahn, Andrew Cooper
On 09/05/17 21:24, Roger Pau Monné wrote:
> On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote:
>> On 04/05/17 00:17, Glenn Enright wrote:
>>> On 04/05/17 04:58, Steven Haigh wrote:
>>>> On 04/05/17 01:53, Juergen Gross wrote:
>>>>> On 03/05/17 12:45, Steven Haigh wrote:
>>>>>> Just wanted to give this a little nudge now people seem to be back on
>>>>>> deck...
>>>>>
>>>>> Glenn, could you please give the attached patch a try?
>>>>>
>>>>> It should be applied on top of the other correction, the old debug
>>>>> patch should not be applied.
>>>>>
>>>>> I have added some debug output to make sure we see what is happening.
>>>>
>>>> This patch is included in kernel-xen-4.9.26-1
>>>>
>>>> It should be in the repos now.
>>>>
>>>
>>> Still seeing the same issue. Without the extra debug patch all I see in
>>> the logs after destroy is this...
>>>
>>> xen-blkback: xen_blkif_disconnect: busy
>>> xen-blkback: xen_blkif_free: delayed = 0
>>
>> Hmm, to me it seems as if some grant isn't being unmapped.
>>
>> Looking at gnttab_unmap_refs_async() I wonder how this is supposed to
>> work:
>>
>> I don't see how a grant would ever be unmapped in case of
>> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it
>> does is deferring the call to the unmap operation again and again. Or
>> am I missing something here?
>
> No, I don't think you are missing anything, but I cannot see how this can be
> solved in a better way, unmapping a page that's still referenced is certainly
> not the best option, or else we risk triggering a page-fault elsewhere.
>
> IMHO, gnttab_unmap_refs_async should have a timeout, and return an error at
> some point. Also, I'm wondering whether there's a way to keep track of who has
> references on a specific page, but so far I haven't been able to figure out how
> to get this information from Linux.
>
> Also, I've noticed that __gnttab_unmap_refs_async uses page_count, shouldn't it
> use page_ref_count instead?
>
> Roger.
>
In case it helps, I have continued to work on this. I notices processed
left behind (under 4.9.27). The same issue is ongoing.
# ps auxf | grep [x]vda
root 2983 0.0 0.0 0 0 ? S 01:44 0:00 \_
[1.xvda1-1]
root 5457 0.0 0.0 0 0 ? S 02:06 0:00 \_
[3.xvda1-1]
root 7382 0.0 0.0 0 0 ? S 02:36 0:00 \_
[4.xvda1-1]
root 9668 0.0 0.0 0 0 ? S 02:51 0:00 \_
[6.xvda1-1]
root 11080 0.0 0.0 0 0 ? S 02:57 0:00 \_
[7.xvda1-1]
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1512 2 r----- 118.5
(null) 1 8 4 --p--d 43.8
(null) 3 8 4 --p--d 6.3
(null) 4 8 4 --p--d 73.4
(null) 6 8 4 --p--d 14.7
(null) 7 8 4 --p--d 30
Those all have...
[root 11080]# cat wchan
xen_blkif_schedule
[root 11080]# cat stack
[<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40
[<ffffffff810a0555>] kthread+0xe5/0x100
[<ffffffff816f1c45>] ret_from_fork+0x25/0x30
[<ffffffffffffffff>] 0xffffffffffffffff
So I can give anything that is in the /proc/pid space. 'maps' and
'pagemap' are empty fwiw.
I also have perf tools installed on this box in case anyone has anything
I can run with that which might help.
A reminder I can replicate by doing xl destroy $domU while running this
inside the domU. Has anyone else been able to repeat this?
{
while true; do
dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
done
}
This does not work in 4.4 kernels, so there is possibly a regression of
some sort?
Regards, Glenn
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-13 4:02 ` Glenn Enright
@ 2017-05-15 9:57 ` Juergen Gross
2017-05-16 0:49 ` Glenn Enright
0 siblings, 1 reply; 28+ messages in thread
From: Juergen Gross @ 2017-05-15 9:57 UTC (permalink / raw)
To: glenn, Roger Pau Monné
Cc: Jennifer Herbert, xen-devel, Steven Haigh, Dietmar Hahn, Andrew Cooper
[-- Attachment #1: Type: text/plain, Size: 3641 bytes --]
On 13/05/17 06:02, Glenn Enright wrote:
> On 09/05/17 21:24, Roger Pau Monné wrote:
>> On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote:
>>> On 04/05/17 00:17, Glenn Enright wrote:
>>>> On 04/05/17 04:58, Steven Haigh wrote:
>>>>> On 04/05/17 01:53, Juergen Gross wrote:
>>>>>> On 03/05/17 12:45, Steven Haigh wrote:
>>>>>>> Just wanted to give this a little nudge now people seem to be
>>>>>>> back on
>>>>>>> deck...
>>>>>>
>>>>>> Glenn, could you please give the attached patch a try?
>>>>>>
>>>>>> It should be applied on top of the other correction, the old debug
>>>>>> patch should not be applied.
>>>>>>
>>>>>> I have added some debug output to make sure we see what is happening.
>>>>>
>>>>> This patch is included in kernel-xen-4.9.26-1
>>>>>
>>>>> It should be in the repos now.
>>>>>
>>>>
>>>> Still seeing the same issue. Without the extra debug patch all I see in
>>>> the logs after destroy is this...
>>>>
>>>> xen-blkback: xen_blkif_disconnect: busy
>>>> xen-blkback: xen_blkif_free: delayed = 0
>>>
>>> Hmm, to me it seems as if some grant isn't being unmapped.
>>>
>>> Looking at gnttab_unmap_refs_async() I wonder how this is supposed to
>>> work:
>>>
>>> I don't see how a grant would ever be unmapped in case of
>>> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it
>>> does is deferring the call to the unmap operation again and again. Or
>>> am I missing something here?
>>
>> No, I don't think you are missing anything, but I cannot see how this
>> can be
>> solved in a better way, unmapping a page that's still referenced is
>> certainly
>> not the best option, or else we risk triggering a page-fault elsewhere.
>>
>> IMHO, gnttab_unmap_refs_async should have a timeout, and return an
>> error at
>> some point. Also, I'm wondering whether there's a way to keep track of
>> who has
>> references on a specific page, but so far I haven't been able to
>> figure out how
>> to get this information from Linux.
>>
>> Also, I've noticed that __gnttab_unmap_refs_async uses page_count,
>> shouldn't it
>> use page_ref_count instead?
>>
>> Roger.
>>
>
> In case it helps, I have continued to work on this. I notices processed
> left behind (under 4.9.27). The same issue is ongoing.
>
> # ps auxf | grep [x]vda
> root 2983 0.0 0.0 0 0 ? S 01:44 0:00 \_
> [1.xvda1-1]
> root 5457 0.0 0.0 0 0 ? S 02:06 0:00 \_
> [3.xvda1-1]
> root 7382 0.0 0.0 0 0 ? S 02:36 0:00 \_
> [4.xvda1-1]
> root 9668 0.0 0.0 0 0 ? S 02:51 0:00 \_
> [6.xvda1-1]
> root 11080 0.0 0.0 0 0 ? S 02:57 0:00 \_
> [7.xvda1-1]
>
> # xl list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 1512 2 r----- 118.5
> (null) 1 8 4 --p--d 43.8
> (null) 3 8 4 --p--d 6.3
> (null) 4 8 4 --p--d 73.4
> (null) 6 8 4 --p--d 14.7
> (null) 7 8 4 --p--d 30
>
> Those all have...
>
> [root 11080]# cat wchan
> xen_blkif_schedule
>
> [root 11080]# cat stack
> [<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40
> [<ffffffff810a0555>] kthread+0xe5/0x100
> [<ffffffff816f1c45>] ret_from_fork+0x25/0x30
> [<ffffffffffffffff>] 0xffffffffffffffff
And found another reference count bug. Would you like to give the
attached patch (to be applied additionally to the previous ones) a try?
Juergen
[-- Attachment #2: 0003-xen-blkback-don-t-use-xen_blkif_get-in-xen-blkback-k.patch --]
[-- Type: text/x-patch, Size: 1850 bytes --]
>From ef37fe77b7b1b1d733011df5615634e543e21c4f Mon Sep 17 00:00:00 2001
From: Juergen Gross <jgross@suse.com>
Date: Mon, 15 May 2017 11:41:18 +0200
Subject: [PATCH 3/3] xen/blkback: don't use xen_blkif_get() in xen-blkback
kthread
There is no need to use xen_blkif_get()/xen_blkif_put() in the kthread
of xen-blkback. Thread stopping is synchronous and using the blkif
reference counting in the kthread will avoid to ever let the reference
count drop to zero at the end of an I/O running concurrent to
disconnecting and multiple rings.
Setting ring->xenblkd to NULL after stopping the kthread isn't needed
as the kthread does this already.
Signed-off-by: Juergen Gross <jgross@suse.com>
---
drivers/block/xen-blkback/blkback.c | 3 ---
drivers/block/xen-blkback/xenbus.c | 1 -
2 files changed, 4 deletions(-)
diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
index 726c32e35db9..6b14c509f3c7 100644
--- a/drivers/block/xen-blkback/blkback.c
+++ b/drivers/block/xen-blkback/blkback.c
@@ -609,8 +609,6 @@ int xen_blkif_schedule(void *arg)
unsigned long timeout;
int ret;
- xen_blkif_get(blkif);
-
set_freezable();
while (!kthread_should_stop()) {
if (try_to_freeze())
@@ -665,7 +663,6 @@ int xen_blkif_schedule(void *arg)
print_stats(ring);
ring->xenblkd = NULL;
- xen_blkif_put(blkif);
return 0;
}
diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c
index ec7eaeee3765..9f8d136d7636 100644
--- a/drivers/block/xen-blkback/xenbus.c
+++ b/drivers/block/xen-blkback/xenbus.c
@@ -255,7 +255,6 @@ static int xen_blkif_disconnect(struct xen_blkif *blkif)
if (ring->xenblkd) {
kthread_stop(ring->xenblkd);
wake_up(&ring->shutdown_wq);
- ring->xenblkd = NULL;
}
/* The above kthread_stop() guarantees that at this point we
--
2.12.0
[-- Attachment #3: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-15 9:57 ` Juergen Gross
@ 2017-05-16 0:49 ` Glenn Enright
2017-05-16 1:18 ` Steven Haigh
0 siblings, 1 reply; 28+ messages in thread
From: Glenn Enright @ 2017-05-16 0:49 UTC (permalink / raw)
To: Juergen Gross, Roger Pau Monné
Cc: Jennifer Herbert, xen-devel, Steven Haigh, Dietmar Hahn, Andrew Cooper
On 15/05/17 21:57, Juergen Gross wrote:
> On 13/05/17 06:02, Glenn Enright wrote:
>> On 09/05/17 21:24, Roger Pau Monné wrote:
>>> On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote:
>>>> On 04/05/17 00:17, Glenn Enright wrote:
>>>>> On 04/05/17 04:58, Steven Haigh wrote:
>>>>>> On 04/05/17 01:53, Juergen Gross wrote:
>>>>>>> On 03/05/17 12:45, Steven Haigh wrote:
>>>>>>>> Just wanted to give this a little nudge now people seem to be
>>>>>>>> back on
>>>>>>>> deck...
>>>>>>>
>>>>>>> Glenn, could you please give the attached patch a try?
>>>>>>>
>>>>>>> It should be applied on top of the other correction, the old debug
>>>>>>> patch should not be applied.
>>>>>>>
>>>>>>> I have added some debug output to make sure we see what is happening.
>>>>>>
>>>>>> This patch is included in kernel-xen-4.9.26-1
>>>>>>
>>>>>> It should be in the repos now.
>>>>>>
>>>>>
>>>>> Still seeing the same issue. Without the extra debug patch all I see in
>>>>> the logs after destroy is this...
>>>>>
>>>>> xen-blkback: xen_blkif_disconnect: busy
>>>>> xen-blkback: xen_blkif_free: delayed = 0
>>>>
>>>> Hmm, to me it seems as if some grant isn't being unmapped.
>>>>
>>>> Looking at gnttab_unmap_refs_async() I wonder how this is supposed to
>>>> work:
>>>>
>>>> I don't see how a grant would ever be unmapped in case of
>>>> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All it
>>>> does is deferring the call to the unmap operation again and again. Or
>>>> am I missing something here?
>>>
>>> No, I don't think you are missing anything, but I cannot see how this
>>> can be
>>> solved in a better way, unmapping a page that's still referenced is
>>> certainly
>>> not the best option, or else we risk triggering a page-fault elsewhere.
>>>
>>> IMHO, gnttab_unmap_refs_async should have a timeout, and return an
>>> error at
>>> some point. Also, I'm wondering whether there's a way to keep track of
>>> who has
>>> references on a specific page, but so far I haven't been able to
>>> figure out how
>>> to get this information from Linux.
>>>
>>> Also, I've noticed that __gnttab_unmap_refs_async uses page_count,
>>> shouldn't it
>>> use page_ref_count instead?
>>>
>>> Roger.
>>>
>>
>> In case it helps, I have continued to work on this. I notices processed
>> left behind (under 4.9.27). The same issue is ongoing.
>>
>> # ps auxf | grep [x]vda
>> root 2983 0.0 0.0 0 0 ? S 01:44 0:00 \_
>> [1.xvda1-1]
>> root 5457 0.0 0.0 0 0 ? S 02:06 0:00 \_
>> [3.xvda1-1]
>> root 7382 0.0 0.0 0 0 ? S 02:36 0:00 \_
>> [4.xvda1-1]
>> root 9668 0.0 0.0 0 0 ? S 02:51 0:00 \_
>> [6.xvda1-1]
>> root 11080 0.0 0.0 0 0 ? S 02:57 0:00 \_
>> [7.xvda1-1]
>>
>> # xl list
>> Name ID Mem VCPUs State Time(s)
>> Domain-0 0 1512 2 r----- 118.5
>> (null) 1 8 4 --p--d 43.8
>> (null) 3 8 4 --p--d 6.3
>> (null) 4 8 4 --p--d 73.4
>> (null) 6 8 4 --p--d 14.7
>> (null) 7 8 4 --p--d 30
>>
>> Those all have...
>>
>> [root 11080]# cat wchan
>> xen_blkif_schedule
>>
>> [root 11080]# cat stack
>> [<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40
>> [<ffffffff810a0555>] kthread+0xe5/0x100
>> [<ffffffff816f1c45>] ret_from_fork+0x25/0x30
>> [<ffffffffffffffff>] 0xffffffffffffffff
>
> And found another reference count bug. Would you like to give the
> attached patch (to be applied additionally to the previous ones) a try?
>
>
> Juergen
>
This seems to have solved the issue in 4.9.28, with all three patches
applied. Awesome!
On my main test machine I can no longer replicate what I was originally
seeing, and in dmesg I now see this flow...
xen-blkback: xen_blkif_disconnect: busy
xen-blkback: xen_blkif_free: delayed = 1
xen-blkback: xen_blkif_free: delayed = 0
xl list is clean, xenstore looks right. No extraneous processes left over.
Thankyou Juergen, so much. Really appreciate your persistence with this.
Anything I can do to help push this upstream please let me know. Feel
free to add a reported-by line with my name if you think it appropriate.
Regards, Glenn
http://rimuhosting.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: null domains after xl destroy
2017-05-16 0:49 ` Glenn Enright
@ 2017-05-16 1:18 ` Steven Haigh
0 siblings, 0 replies; 28+ messages in thread
From: Steven Haigh @ 2017-05-16 1:18 UTC (permalink / raw)
To: glenn
Cc: Juergen Gross, Andrew Cooper, Jennifer Herbert, Dietmar Hahn,
xen-devel, Roger Pau Monné
On 2017-05-16 10:49, Glenn Enright wrote:
> On 15/05/17 21:57, Juergen Gross wrote:
>> On 13/05/17 06:02, Glenn Enright wrote:
>>> On 09/05/17 21:24, Roger Pau Monné wrote:
>>>> On Mon, May 08, 2017 at 11:10:24AM +0200, Juergen Gross wrote:
>>>>> On 04/05/17 00:17, Glenn Enright wrote:
>>>>>> On 04/05/17 04:58, Steven Haigh wrote:
>>>>>>> On 04/05/17 01:53, Juergen Gross wrote:
>>>>>>>> On 03/05/17 12:45, Steven Haigh wrote:
>>>>>>>>> Just wanted to give this a little nudge now people seem to be
>>>>>>>>> back on
>>>>>>>>> deck...
>>>>>>>>
>>>>>>>> Glenn, could you please give the attached patch a try?
>>>>>>>>
>>>>>>>> It should be applied on top of the other correction, the old
>>>>>>>> debug
>>>>>>>> patch should not be applied.
>>>>>>>>
>>>>>>>> I have added some debug output to make sure we see what is
>>>>>>>> happening.
>>>>>>>
>>>>>>> This patch is included in kernel-xen-4.9.26-1
>>>>>>>
>>>>>>> It should be in the repos now.
>>>>>>>
>>>>>>
>>>>>> Still seeing the same issue. Without the extra debug patch all I
>>>>>> see in
>>>>>> the logs after destroy is this...
>>>>>>
>>>>>> xen-blkback: xen_blkif_disconnect: busy
>>>>>> xen-blkback: xen_blkif_free: delayed = 0
>>>>>
>>>>> Hmm, to me it seems as if some grant isn't being unmapped.
>>>>>
>>>>> Looking at gnttab_unmap_refs_async() I wonder how this is supposed
>>>>> to
>>>>> work:
>>>>>
>>>>> I don't see how a grant would ever be unmapped in case of
>>>>> page_count(item->pages[pc]) > 1 in __gnttab_unmap_refs_async(). All
>>>>> it
>>>>> does is deferring the call to the unmap operation again and again.
>>>>> Or
>>>>> am I missing something here?
>>>>
>>>> No, I don't think you are missing anything, but I cannot see how
>>>> this
>>>> can be
>>>> solved in a better way, unmapping a page that's still referenced is
>>>> certainly
>>>> not the best option, or else we risk triggering a page-fault
>>>> elsewhere.
>>>>
>>>> IMHO, gnttab_unmap_refs_async should have a timeout, and return an
>>>> error at
>>>> some point. Also, I'm wondering whether there's a way to keep track
>>>> of
>>>> who has
>>>> references on a specific page, but so far I haven't been able to
>>>> figure out how
>>>> to get this information from Linux.
>>>>
>>>> Also, I've noticed that __gnttab_unmap_refs_async uses page_count,
>>>> shouldn't it
>>>> use page_ref_count instead?
>>>>
>>>> Roger.
>>>>
>>>
>>> In case it helps, I have continued to work on this. I notices
>>> processed
>>> left behind (under 4.9.27). The same issue is ongoing.
>>>
>>> # ps auxf | grep [x]vda
>>> root 2983 0.0 0.0 0 0 ? S 01:44 0:00 \_
>>> [1.xvda1-1]
>>> root 5457 0.0 0.0 0 0 ? S 02:06 0:00 \_
>>> [3.xvda1-1]
>>> root 7382 0.0 0.0 0 0 ? S 02:36 0:00 \_
>>> [4.xvda1-1]
>>> root 9668 0.0 0.0 0 0 ? S 02:51 0:00 \_
>>> [6.xvda1-1]
>>> root 11080 0.0 0.0 0 0 ? S 02:57 0:00 \_
>>> [7.xvda1-1]
>>>
>>> # xl list
>>> Name ID Mem VCPUs State Time(s)
>>> Domain-0 0 1512 2 r----- 118.5
>>> (null) 1 8 4 --p--d 43.8
>>> (null) 3 8 4 --p--d 6.3
>>> (null) 4 8 4 --p--d 73.4
>>> (null) 6 8 4 --p--d 14.7
>>> (null) 7 8 4 --p--d 30
>>>
>>> Those all have...
>>>
>>> [root 11080]# cat wchan
>>> xen_blkif_schedule
>>>
>>> [root 11080]# cat stack
>>> [<ffffffff814eaee8>] xen_blkif_schedule+0x418/0xb40
>>> [<ffffffff810a0555>] kthread+0xe5/0x100
>>> [<ffffffff816f1c45>] ret_from_fork+0x25/0x30
>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>
>> And found another reference count bug. Would you like to give the
>> attached patch (to be applied additionally to the previous ones) a
>> try?
>>
>>
>> Juergen
>>
>
> This seems to have solved the issue in 4.9.28, with all three patches
> applied. Awesome!
>
> On my main test machine I can no longer replicate what I was
> originally seeing, and in dmesg I now see this flow...
>
> xen-blkback: xen_blkif_disconnect: busy
> xen-blkback: xen_blkif_free: delayed = 1
> xen-blkback: xen_blkif_free: delayed = 0
>
> xl list is clean, xenstore looks right. No extraneous processes left
> over.
>
> Thankyou Juergen, so much. Really appreciate your persistence with
> this. Anything I can do to help push this upstream please let me know.
> Feel free to add a reported-by line with my name if you think it
> appropriate.
This is good news.
Juergen, Can I request a full patch set posted to the list (plz CC me) -
and I'll ensure we can build the kernel with all 3 (?) patches applied
and test properly.
I'll build up a complete kernel with those patches and give a tested-by
if all goes well.
--
Steven Haigh
Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2017-05-16 1:18 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-04-11 5:25 null domains after xl destroy Glenn Enright
2017-04-11 5:59 ` Juergen Gross
2017-04-11 8:03 ` Glenn Enright
2017-04-11 9:49 ` Dietmar Hahn
2017-04-11 22:13 ` Glenn Enright
2017-04-11 22:23 ` Andrew Cooper
2017-04-11 22:45 ` Glenn Enright
2017-04-18 8:36 ` Juergen Gross
2017-04-19 1:02 ` Glenn Enright
2017-04-19 4:39 ` Juergen Gross
2017-04-19 7:16 ` Roger Pau Monné
2017-04-19 7:35 ` Juergen Gross
2017-04-19 10:09 ` Juergen Gross
2017-04-19 16:22 ` Steven Haigh
2017-04-21 8:42 ` Steven Haigh
2017-04-21 8:44 ` Juergen Gross
2017-05-01 0:55 ` Glenn Enright
2017-05-03 10:45 ` Steven Haigh
2017-05-03 13:38 ` Juergen Gross
2017-05-03 15:53 ` Juergen Gross
2017-05-03 16:58 ` Steven Haigh
2017-05-03 22:17 ` Glenn Enright
2017-05-08 9:10 ` Juergen Gross
2017-05-09 9:24 ` Roger Pau Monné
2017-05-13 4:02 ` Glenn Enright
2017-05-15 9:57 ` Juergen Gross
2017-05-16 0:49 ` Glenn Enright
2017-05-16 1:18 ` Steven Haigh
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.