From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stefan Bader Subject: Re: Strange PVM spinlock case revisited (nailed down) Date: Thu, 14 Feb 2013 15:21:11 +0100 Message-ID: <511CF2D7.6090004@canonical.com> References: <51151ABF.1070007@canonical.com> <511CC4D8.3060203@canonical.com> <511CDBF302000078000BE2D7@nat28.tlf.novell.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7694255929101596411==" Return-path: In-Reply-To: <511CDBF302000078000BE2D7@nat28.tlf.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich Cc: Andrew Cooper , Ian Campbell , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --===============7694255929101596411== Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="------------enig915666D3C4E4B680CA8FFC0C" This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig915666D3C4E4B680CA8FFC0C Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 14.02.2013 12:43, Jan Beulich wrote: >>>> On 14.02.13 at 12:04, Stefan Bader wrot= e: >> There was a bit more on the stack but try_to_wake_up seemed a believab= le=20 >> current >> path. Even more so after looking into the function: >> >> #ifdef CONFIG_SMP >> /* >> * If the owning (remote) cpu is still in the middle of schedu= le() with >> * this task as prev, wait until its done referencing the task= =2E >> */ >> while (p->on_cpu) { >> #ifdef __ARCH_WANT_INTERRUPTS_ON_CTXSW >> /* >> * In case the architecture enables interrupts in >> * context_switch(), we cannot busy wait, since that >> * would lead to deadlocks when an interrupt hits and >> * tries to wake up @prev. So bail and do a complete >> * remote wakeup. >> */ >> if (ttwu_activate_remote(p, wake_flags)) >> goto stat; >> #else >> cpu_relax(); >> #endif >> >> And prying out the task in question from the stack, it was one current= ly >> being accounted for VCPU 6 and on_cpu. VCPU 6 is one of the other wait= ers=20 >> for >> the runq lock of VCPU 1. Which would get picked up as soon as this cal= lback=20 >> is >> done. Unfortunately we "stay awhile... stay forever!". >=20 > When analyzing a similar problem with our old PV ticket lock > implementation (and the interrupt re-enabling there when possible > before going into polling mode), it was precisely this open coded > lock construct that caused problems for us. Back then I didn't, > however, realize that this would even affect the simplistic byte > locks used by the pv-ops Xen code (or else I would have pointed > this out). >=20 > Being relatively certain that with our new implementation we don't > have any such problems, I can only recommend against dropping > the re-enabling of interrupts - this needlessly introduces high > interrupt latencies in a broader range of cases. Instead, the > interactions ought to be fixed properly. And it's time for using > ticket locks in the Xen code too... >=20 Not sure what *your new* implementation is. ;) I am more concerned about = the currently released/stable kernels which potentially have this problem. I = agree that a proper solution is preferable. Ticket locks likely have a much low= er chance of hitting this as part of the current problem is that lower numbe= r VCPUs are preferred in unlock_slow. In the end, though, we would need something that could go upstream (and f= rom there back into stable). So it should not be too complicated. Then a prop= er solution can be applied. Having more understanding now, I had a different idea. Not sure this is foolproof and surely is causing some more active spinning. But it looks l= ike I also can keep the system from locking up (v3.2 kernel and the testcase) i= f I change xen_spin_unlock_slow to send the wakeup IPI to _all_ spinners inst= ead of only the first one found. -Stefan --------------enig915666D3C4E4B680CA8FFC0C Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCgAGBQJRHPLXAAoJEOhnXe7L7s6jtTcP/3xGjsOfdIWtrctq4+F361+U 5wIC4In0i6CTxmP5qAKaLsNJjj02h9HZkZ2qYg763Svs1KYqX2pZlZowk6E+sS3X jR11JU56MMtR4Wyf8VTtCZcEBGvbPzC4ygtbveTLOQ+w2QlrQLTWqmYiqE9AiG+N gP+0XG4ICoP0B6kP/UnqDDXqJH0eBhao44WRfN0IHODTJgXxRLox1crvwB9rc7dM wUTco9BMy4xapAdxiNQRJsK7oHk+i+yZzKAEWg9nNFIyOzA7icaW5kws+E3Sp8Da xgg0DjN7kBYMUg1O3jp5mkfb8MAC5QOz02zI0tp6xX8DA7NsWht5zvgQG2L6NhDj 9oAVSFnY//TeZMBozPpjaJ88vL0RIXebbn7oEkUA96C9GH/hCw4cPJaYzIpHhCs4 VWadGN2Y22x6bFqjYcS1VRHPcvokdeblw3odmBW0CxYZ4AXtWBbzH7+8KPXd1BAb c65Npgt0iLVfCKcxeUs4NfnVSn9ijvSyvPwbRgf2a3A4pEeNNQu3cn62d3f1gxg1 RRkh+HysySM0xT3ZB8LIWrnrKkwveTOeto9g08FSd1P2fiUQ4SXtjbU9UZcgj5+v tmeuQ/B9Ci+7U3oEgwG2wRY+X1FlD6ngcvQxEmr3WSOsf8/+IS5rrfzDrYRAwik+ FvAcu0ybws2vxMyYYsVv =WCR+ -----END PGP SIGNATURE----- --------------enig915666D3C4E4B680CA8FFC0C-- --===============7694255929101596411== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============7694255929101596411==--