From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven Haigh Subject: Re: null domains after xl destroy Date: Fri, 21 Apr 2017 18:42:30 +1000 Message-ID: References: <78571a7b-62ec-b046-02e3-3d6739b779a6@rimuhosting.com> <95efee87-6925-5376-e347-55e438c90212@suse.com> <70eae378-2392-bd82-670a-5dafff58c259@rimuhosting.com> <3385656.IoOB642KYU@amur> <6e150a33-576b-5cf8-7abc-2cba584602ff@citrix.com> <05cd7b43-153a-8b51-8fd9-e8ae4a8b5287@rimuhosting.com> <06829f8f-def6-4822-c18a-877d8633556c@suse.com> <034c9f96-1bfe-6793-68a7-9b070676971a@suse.com> <20170419071624.6enfeemielfqhqw2@dhcp-3-128.uk.xensource.com> <0b981374-700b-f26a-9504-583bad046f7d@suse.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3536654765257384334==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Juergen Gross , glenn@rimuhosting.com Cc: Andrew Cooper , xen-devel@lists.xen.org, Dietmar Hahn , =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= List-Id: xen-devel@lists.xenproject.org This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --===============3536654765257384334== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Cjcimlq1xKrMEKgllt7BL1MmsH5FFSUbA" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Cjcimlq1xKrMEKgllt7BL1MmsH5FFSUbA Content-Type: multipart/mixed; boundary="XuwmW2tddguUf1qWGpcGgB1nkf6rbKoMh"; protected-headers="v1" From: Steven Haigh To: Juergen Gross , glenn@rimuhosting.com Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= , Dietmar Hahn , xen-devel@lists.xen.org Message-ID: Subject: Re: null domains after xl destroy References: <78571a7b-62ec-b046-02e3-3d6739b779a6@rimuhosting.com> <95efee87-6925-5376-e347-55e438c90212@suse.com> <70eae378-2392-bd82-670a-5dafff58c259@rimuhosting.com> <3385656.IoOB642KYU@amur> <6e150a33-576b-5cf8-7abc-2cba584602ff@citrix.com> <05cd7b43-153a-8b51-8fd9-e8ae4a8b5287@rimuhosting.com> <06829f8f-def6-4822-c18a-877d8633556c@suse.com> <034c9f96-1bfe-6793-68a7-9b070676971a@suse.com> <20170419071624.6enfeemielfqhqw2@dhcp-3-128.uk.xensource.com> <0b981374-700b-f26a-9504-583bad046f7d@suse.com> In-Reply-To: --XuwmW2tddguUf1qWGpcGgB1nkf6rbKoMh Content-Type: text/plain; charset=windows-1252 Content-Language: en-AU Content-Transfer-Encoding: quoted-printable On 20/04/17 02:22, Steven Haigh wrote: > On 19/04/17 20:09, Juergen Gross wrote: >> On 19/04/17 09:16, Roger Pau Monn=E9 wrote: >>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote: >>>> On 19/04/17 03:02, Glenn Enright wrote: >>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which st= ill >>>>> shows the issue. When replicating the leak I now see this trace (vi= a >>>>> dmesg). Hopefully that is useful. >>>>> >>>>> Please note, I'm going to be offline next week, but am keen to keep= on >>>>> with this, it may just be a while before I followup is all. >>>>> >>>>> Regards, Glenn >>>>> http://rimuhosting.com >>>>> >>>>> >>>>> ------------[ cut here ]------------ >>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508 >>>>> xen_blkbk_remove+0x138/0x140 >>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev >>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4 >>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_sec= urity >>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 >>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp= llc >>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801 i2c_s= mbus >>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci f= loppy >>>>> dm_mirror dm_region_hash dm_log dm_mod >>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1= >>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007 >>>>> ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 000000000000000= 0 >>>>> 0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007= d >>>>> ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac= 0 >>>>> Call Trace: >>>>> [] dump_stack+0x67/0x98 >>>>> [] __warn+0xfd/0x120 >>>>> [] warn_slowpath_null+0x1d/0x20 >>>>> [] xen_blkbk_remove+0x138/0x140 >>>>> [] xenbus_dev_remove+0x47/0xa0 >>>>> [] __device_release_driver+0xb4/0x160 >>>>> [] device_release_driver+0x2d/0x40 >>>>> [] bus_remove_device+0x124/0x190 >>>>> [] device_del+0x112/0x210 >>>>> [] ? xenbus_read+0x53/0x70 >>>>> [] device_unregister+0x22/0x60 >>>>> [] frontend_changed+0xad/0x4c0 >>>>> [] ? schedule_tail+0x1e/0xc0 >>>>> [] xenbus_otherend_changed+0xc7/0x140 >>>>> [] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>> [] ? schedule_tail+0x1e/0xc0 >>>>> [] frontend_changed+0x10/0x20 >>>>> [] xenwatch_thread+0x9c/0x140 >>>>> [] ? woken_wake_function+0x20/0x20 >>>>> [] ? schedule+0x3a/0xa0 >>>>> [] ? _raw_spin_unlock_irqrestore+0x16/0x20 >>>>> [] ? complete+0x4d/0x60 >>>>> [] ? split+0xf0/0xf0 >>>>> [] kthread+0xcd/0xf0 >>>>> [] ? schedule_tail+0x1e/0xc0 >>>>> [] ? __kthread_init_worker+0x40/0x40 >>>>> [] ? __kthread_init_worker+0x40/0x40 >>>>> [] ret_from_fork+0x25/0x30 >>>>> ---[ end trace ee097287c9865a62 ]--- >>>> >>>> Konrad, Roger, >>>> >>>> this was triggered by a debug patch in xen_blkbk_remove(): >>>> >>>> if (be->blkif) >>>> - xen_blkif_disconnect(be->blkif); >>>> + WARN_ON(xen_blkif_disconnect(be->blkif)); >>>> >>>> So I guess we need something like xen_blk_drain_io() in case of call= s to >>>> xen_blkif_disconnect() which are not allowed to fail (either at the = call >>>> sites of xen_blkif_disconnect() or in this function depending on a n= ew >>>> boolean parameter indicating it should wait for outstanding I/Os). >>>> >>>> I can try a patch, but I'd appreciate if you could confirm this woul= dn't >>>> add further problems... >>> >>> Hello, >>> >>> Thanks for debugging this, the easiest solution seems to be to replac= e the >>> ring->inflight atomic_read check in xen_blkif_disconnect with a call = to >>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void= (to >>> prevent further issues like this one). >> >> Glenn, >> >> can you please try the attached patch (in dom0)? Tested-by: Steven Haigh I've tried specifically with 4.9.23 and can no long make this occur in my scenario. Also built with 4.9.24 and expecting similar results. I'm aware Glenn has a much wider test schedule and number of systems than me, however my testing is successful. --=20 Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 --XuwmW2tddguUf1qWGpcGgB1nkf6rbKoMh-- --Cjcimlq1xKrMEKgllt7BL1MmsH5FFSUbA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJY+cX6AAoJEEGvNdV6fTHcgbcP/0M8qPm4JWb/roJOjH2lyHaW KV7jRZ1La5bLhuHOEzx7MJHxd6/fZ9H8+pnRK8EoMf9LPt4iTwWTPW56wrvLlMvJ 4+j1on0Nu3N6FRFtsaUckqNiGlMHqaXTtsH1hcrc+IJ6bNOa2R+sG0cB/AQdSYSj gfQXitkXD5ousjpPejkhYtMOx8410bRXyJ0ue+/WH0p4n8gdudDpxfqUxNzgGbxY rfxiL9YuX0hoSNsF4gyD8VuPpNqBEEGtJ02MScCEetC2zZvf4lrLkRZboDM32aHy mmzkPi4r1mcDQs4Hd8s4w2wIrQmjcweMU4uFfN9utsrJ/lSAS1vrFz/qVjS7vN8O xyZCfEgoNiG+bLa2TyC88vlI+O9yztghDkhoBFrF/7NUEQ/brQhkFNc/Eb6A/5hx laJgQrQhQP/LtOObRqnB/vDcyPupwTXuze58bWyz7ZtUvCYBEq0M+dwPxuzoFxzI RSuyaee606meZsyId/DMVr3RYMrcoLTWg+RqlikUuZXB0jqRYzKkmguUZqzDIW8S GxQSBR+swaDHUZPJYamJm7vd4wLI42m3nNVAwPjvVmKSWjxSVMp0N0BE6P5iQeJi CWlwk2/WM35cmHPdxQA/Nj4bSUYkcKH7PZDeGyfLJ4N/gDndkpJMeEQaKa1fKlHB RQILPgOgPkC7m34/htt6 =mY+M -----END PGP SIGNATURE----- --Cjcimlq1xKrMEKgllt7BL1MmsH5FFSUbA-- --===============3536654765257384334== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============3536654765257384334==--