From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Tian, Kevin" Subject: RE: RE: Kernel BUG at arch/x86/mm/tlb.c:61 Date: Fri, 29 Apr 2011 09:57:11 +0800 Message-ID: <625BA99ED14B2D499DC4E29D8138F1505C843BB382@shsmsx502.ccr.corp.intel.com> References: , , , , , , , , , , , , , <4DA3438A.6070503@goop.org>, , , , , , <20110412100000.GA15647@dumpdata.com>, , , , , , , , , , <4DA8B715.9080508@goop.org>, , <625BA99ED14B2D499DC4E29D8138F1505C7F2C5185@shsmsx502.ccr.corp.intel.com>, <4DB9F845.6020204@goop.org>, <625BA99ED14B2D499DC4E29D8138F1505C843BB27A@shsmsx502.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0574505600==" Return-path: In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: MaoXiaoyun , "jeremy@goop.org" Cc: xen devel , "giamteckchoon@gmail.com" , "konrad.wilk@oracle.com" List-Id: xen-devel@lists.xenproject.org --===============0574505600== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_625BA99ED14B2D499DC4E29D8138F1505C843BB382shsmsx502ccrc_" --_000_625BA99ED14B2D499DC4E29D8138F1505C843BB382shsmsx502ccrc_ Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 T0ssIHRoYW5rcyBmb3IgdGhlIHVwZGF0ZS4gSaGvbGwgc2VuZCBvdXQgdGhlIHBhdGNoIHRoZW4N Cg0KVGhhbmtzDQpLZXZpbg0KDQpGcm9tOiBNYW9YaWFveXVuIFttYWlsdG86dGlubnljbG91ZEBo b3RtYWlsLmNvbV0NClNlbnQ6IEZyaWRheSwgQXByaWwgMjksIDIwMTEgOTo1MSBBTQ0KVG86IFRp YW4sIEtldmluOyBqZXJlbXlAZ29vcC5vcmcNCkNjOiB4ZW4gZGV2ZWw7IGdpYW10ZWNrY2hvb25A Z21haWwuY29tOyBrb25yYWQud2lsa0BvcmFjbGUuY29tDQpTdWJqZWN0OiBSRTogW1hlbi1kZXZl bF0gUkU6IEtlcm5lbCBCVUcgYXQgYXJjaC94ODYvbW0vdGxiLmM6NjENCg0KDQo+IEZyb206IGtl dmluLnRpYW5AaW50ZWwuY29tPG1haWx0bzprZXZpbi50aWFuQGludGVsLmNvbT4NCj4gVG86IGpl cmVteUBnb29wLm9yZzxtYWlsdG86amVyZW15QGdvb3Aub3JnPg0KPiBDQzogdGlubnljbG91ZEBo b3RtYWlsLmNvbTxtYWlsdG86dGlubnljbG91ZEBob3RtYWlsLmNvbT47IHhlbi1kZXZlbEBsaXN0 cy54ZW5zb3VyY2UuY29tPG1haWx0bzp4ZW4tZGV2ZWxAbGlzdHMueGVuc291cmNlLmNvbT47IGdp YW10ZWNrY2hvb25AZ21haWwuY29tPG1haWx0bzpnaWFtdGVja2Nob29uQGdtYWlsLmNvbT47IGtv bnJhZC53aWxrQG9yYWNsZS5jb208bWFpbHRvOmtvbnJhZC53aWxrQG9yYWNsZS5jb20+DQo+IERh dGU6IEZyaSwgMjkgQXByIDIwMTEgMDg6MTk6NDQgKzA4MDANCj4gU3ViamVjdDogUkU6IFtYZW4t ZGV2ZWxdIFJFOiBLZXJuZWwgQlVHIGF0IGFyY2gveDg2L21tL3RsYi5jOjYxDQo+DQo+ID4gRnJv bTogSmVyZW15IEZpdHpoYXJkaW5nZSBbbWFpbHRvOmplcmVteUBnb29wLm9yZ108bWFpbHRvOltt YWlsdG86amVyZW15QGdvb3Aub3JnXT4NCj4gPiBTZW50OiBGcmlkYXksIEFwcmlsIDI5LCAyMDEx IDc6MjkgQU0NCj4gPg0KPiA+IE9uIDA0LzI1LzIwMTEgMTA6NTIgUE0sIFRpYW4sIEtldmluIHdy b3RlOg0KPiA+ID4+IEZyb206IE1hb1hpYW95dW4NCj4gPiA+PiBTZW50OiBNb25kYXksIEFwcmls IDI1LCAyMDExIDExOjE1IEFNDQo+ID4gPj4+IERhdGU6IEZyaSwgMTUgQXByIDIwMTEgMTQ6MjI6 MjkgLTA3MDANCj4gPiA+Pj4gRnJvbTogamVyZW15QGdvb3Aub3JnPG1haWx0bzpqZXJlbXlAZ29v cC5vcmc+DQo+ID4gPj4+IFRvOiB0aW5ueWNsb3VkQGhvdG1haWwuY29tPG1haWx0bzp0aW5ueWNs b3VkQGhvdG1haWwuY29tPg0KPiA+ID4+PiBDQzogZ2lhbXRlY2tjaG9vbkBnbWFpbC5jb208bWFp bHRvOmdpYW10ZWNrY2hvb25AZ21haWwuY29tPjsgeGVuLWRldmVsQGxpc3RzLnhlbnNvdXJjZS5j b208bWFpbHRvOnhlbi1kZXZlbEBsaXN0cy54ZW5zb3VyY2UuY29tPjsNCj4gPiA+Pj4ga29ucmFk LndpbGtAb3JhY2xlLmNvbTxtYWlsdG86a29ucmFkLndpbGtAb3JhY2xlLmNvbT4NCj4gPiA+Pj4g U3ViamVjdDogUmU6IEtlcm5lbCBCVUcgYXQgYXJjaC94ODYvbW0vdGxiLmM6NjENCj4gPiA+Pj4N Cj4gPiA+Pj4gT24gMDQvMTUvMjAxMSAwNToyMyBBTSwgTWFvWGlhb3l1biB3cm90ZToNCj4gPiA+ Pj4+IEhpo7oNCj4gPiA+Pj4+DQo+ID4gPj4+PiBDb3VsZCB0aGUgY3Jhc2ggcmVsYXRlZCB0byB0 aGlzIHBhdGNoID8NCj4gPiA+Pj4+IGh0dHA6Ly9naXQua2VybmVsLm9yZy8/cD1saW51eC9rZXJu ZWwvZ2l0L2plcmVteS94ZW4uZ2l0O2E9Y29tbWl0ZGkNCj4gPiA+Pj4+IGZmO2g9NDViZmQ3YmZj NmNmMzJmOGU2MGJiOTFiMzIzNDlmMGI1MDkwZWVhMw0KPiA+ID4+Pj4NCj4gPiA+Pj4+IFNpbmNl IG5vdyBUTEIgc3RhdGUgY2hhbmdlIHRvIFRMQlNUQVRFX09LKG1tdV9jb250ZXh0Lmg6NDApIGlz DQo+ID4gPj4+PiBiZWZvcmUgY3B1bWFza19jbGVhcl9jcHUobGluZSA0OSkuDQo+ID4gPj4+PiBD b3VsZCBpdCBwb3NzaWJsZSB0aGF0IHJpZ2h0IGFmdGVyIGV4ZWN1dGUgbGluZSA0MCBvZg0KPiA+ ID4+Pj4gbW11X2NvbnRleHQuaCwgQ1BVIHJldmljZSBJUEkgZnJvbSBvdGhlciBDUFUgdG8gZmx1 c2ggdGhlIG1tLCBhbmQNCj4gPiA+Pj4+IHdoZW4gaW4gaW50ZXJydXB0LCBmaW5kIHRoZSBUTEIg c3RhdGUgaGFwcGVuZWQgdG8gYmUgVExCU1RBVEVfT0suDQo+ID4gPj4+PiBXaGljaCBjb25mbGlj dHMuDQo+ID4gPj4+IERvZXMgcmV2ZXJ0aW5nIGl0IGhlbHA/DQo+ID4gPj4+DQo+ID4gPj4+IEoN Cj4gPiA+Pg0KPiA+ID4+IEhpIEplcmVteToNCj4gPiA+Pg0KPiA+ID4+IFRoZSBsYXN0ZXN0IHRl c3QgcmVzdWx0IHNob3dzIHRoZSByZXZlcnRpbmcgZGlkbid0IGhlbHAuDQo+ID4gPj4gS2VybmVs IHBhbmljIGV4YWN0bHkgYXQgdGhlIHNhbWUgcGxhY2UgaW4gdGxiLmMuDQo+ID4gPj4NCj4gPiA+ PiBJIGhhdmUgcXVlc3Rpb24gYWJvdXQgVExCIHN0YXRlLCBmcm9tIHRoZSBzdGFjaywNCj4gPiA+ PiB4ZW5fZG9faHlwZXJ2aXNvcl9jYWxsYmFjay0+IHhlbl9ldnRjaG5fZG9fdXBjYWxsLT4uLi4N Cj4gPiA+PiAtPmRyb3Bfb3RoZXJfbW1fcmVmDQo+ID4gPj4NCj4gPiA+PiBXaGF0IGNwdV90bGJz dGF0ZS5zdGF0ZSBzaG91bGQgYmUsIGNvdWxkIFRMQlNUQVRFX09LIG9yDQo+ID4gVExCU1RBVEVf TEFaWSBhbGwgYmUgcG9zc2libGU/DQo+ID4gPj4gVGhhdCBpcyBhZnRlciBhIGh5cGVyY2FsbCBm cm9tIHVzZXJzcGFjZSwgc3RhdGUgd2lsbCBiZSBUTEJTVEFURV9PSywNCj4gPiBhbmQNCj4gPiA+ PiBpZiBmcm9tIGtlcm5lbCBzcGFjZSwgc3RhdGUgd2lsbCBiZSBUTEJTVEFURV9MQVpFID8NCj4g PiA+Pg0KPiA+ID4+IHRoYW5rcy4NCj4gPiA+IGl0IGxvb2tzIGEgYnVnIGluIGRyb3Bfb3RoZXJf bW1fcmVmIGltcGxlbWVudGF0aW9uLCB0aGF0IGN1cnJlbnQgVExCDQo+ID4gPiBzdGF0ZSBzaG91 bGQgYmUgY2hlY2tlZCBiZWZvcmUgaW52b2tpbmcgbGVhdmVfbW0oKS4gVGhlcmUncyBhIHdpbmRv dw0KPiA+IGJldHdlZW4gYmVsb3cgbGluZXMgb2YgY29kZToNCj4gPiA+DQo+ID4gPiA8eGVuX2Ry b3BfbW1fcmVmPg0KPiA+ID4gLyogR2V0IHRoZSAib2ZmaWNpYWwiIHNldCBvZiBjcHVzIHJlZmVy cmluZyB0byBvdXIgcGFnZXRhYmxlLiAqLw0KPiA+ID4gaWYgKCFhbGxvY19jcHVtYXNrX3Zhcigm bWFzaywgR0ZQX0FUT01JQykpIHsNCj4gPiA+IGZvcl9lYWNoX29ubGluZV9jcHUoY3B1KSB7DQo+ ID4gPiBpZiAoIWNwdW1hc2tfdGVzdF9jcHUoY3B1LA0KPiA+IG1tX2NwdW1hc2sobW0pKQ0KPiA+ ID4gJiYgcGVyX2NwdSh4ZW5fY3VycmVudF9jcjMsIGNwdSkgIT0NCj4gPiBfX3BhKG1tLT5wZ2Qp KQ0KPiA+ID4gY29udGludWU7DQo+ID4gPiBzbXBfY2FsbF9mdW5jdGlvbl9zaW5nbGUoY3B1LA0K PiA+IGRyb3Bfb3RoZXJfbW1fcmVmLCBtbSwgMSk7DQo+ID4gPiB9DQo+ID4gPiByZXR1cm47DQo+ ID4gPiB9DQo+ID4gPg0KPiA+ID4gdGhlcmUncyBjaGFuY2UgdGhhdCB3aGVuIHNtcF9jYWxsX2Z1 bmN0aW9uX3NpbmdsZSBpcyBpbnZva2VkLCBhY3R1YWwNCj4gPiA+IFRMQiBzdGF0ZSBoYXMgYmVl biB1cGRhdGVkIGluIHRoZSBvdGhlciBjcHUuIFRoZSB1cHN0cmVhbSBrZXJuZWwgcGF0Y2gNCj4g PiA+IHlvdSByZWZlcnJlZCB0byBlYXJsaWVyIGp1c3QgbWFrZXMgdGhpcyBidWcgZXhwb3NlZCBt b3JlIGVhc2lseS4gQnV0DQo+ID4gPiBldmVuIHdpdGhvdXQgdGhpcyBwYXRjaCwgeW91IG1heSBz dGlsbCBzdWZmZXIgc3VjaCBpc3N1ZSB3aGljaCBpcyB3aHkgcmV2ZXJ0aW5nDQo+ID4gdGhlIHBh dGNoIGRvZXNuJ3QgaGVscC4NCj4gPiA+DQo+ID4gPiBDb3VsZCB5b3UgdHJ5IGFkZGluZyBhIGNo ZWNrIGluIGRyb3Bfb3RoZXJfbW1fcmVmPw0KPiA+ID4NCj4gPiA+IGlmIChhY3RpdmVfbW0gPT0g bW0gJiYgcGVyY3B1X3JlYWQoY3B1X3RsYnN0YXRlLnN0YXRlKSAhPQ0KPiA+IFRMQlNUQVRFX09L KQ0KPiA+ID4gbGVhdmVfbW0oc21wX3Byb2Nlc3Nvcl9pZCgpKTsNCj4gPiA+DQo+ID4gPiBvbmNl IHRoZSBpbnRlcnJ1cHRlZCBjb250ZXh0IGhhcyBUTEJTVEFURV9PSywgaXQgaW1wbGljYXRlcyB0 aGF0IGxhdGVyDQo+ID4gPiBpdCB3aWxsIGhhbmRsZSB0aGUgVExCIGZsdXNoIGFuZCB0aHVzIG5v IG5lZWQgZm9yIGxlYXZlX21tIGZyb20NCj4gPiA+IGludGVycnVwdCBoYW5kbGVyLCBhbmQgdGhh dCdzIHRoZSBhc3N1bXB0aW9uIG9mIGRvaW5nIGxlYXZlX21tLg0KPiA+DQo+ID4gVGhhdCBzZWVt cyByZWFzb25hYmxlLiBNYW9YaWFveXVuLCBkb2VzIGl0IGZpeCB0aGUgYnVnIGZvciB5b3U/DQo+ ID4NCj4gPiBLZXZpbiwgY291bGQgeW91IHN1Ym1pdCB0aGlzIGFzIGEgcHJvcGVyIHBhdGNoPw0K PiA+DQo+DQo+IEknbSB3YWl0aW5nIGZvciBYaWFveXVuJ3MgdGVzdCByZXN1bHQgYmVmb3JlIHN1 Ym1pdHRpbmcgYSBwcm9wZXIgcGF0Y2gsIHNpbmNlIHRoaXMNCj4gcGFydCBvZiBsb2dpYyBpcyB0 cmlja3kgYW5kIGhpcyB0ZXN0IGNhbiBtYWtlIHN1cmUgd2UgZG9uJ3Qgb3Zlcmxvb2sgc29tZSBj b3JuZXINCj4gY2FzZXMuIDotKQ0KPg0KDQpJIHRoaW5rIGl0IHdvcmtzLiBUaGUgdGVzdCBoYXMg YmVlbiBydW5uaW5nIG92ZXIgNzAgaG91cnMgc3VjY2Vzc2Z1bGx5Lg0KTXkgcGxhbiBpcyBydW4g b25lIHdlZWsuDQoNClRoYW5rcy4NCg0KPiBUaGFua3MNCj4gS2V2aW4NCg== --_000_625BA99ED14B2D499DC4E29D8138F1505C843BB382shsmsx502ccrc_ Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable

OK, thanks for the update. I=A1=AFll send out the patch then=

 

Thanks

<= p class=3DMsoNormal>Kevin

 

From: MaoXiaoyun [mailt= o:tinnycloud@hotmail.com]
Sent: Friday, April 29, 2011 9:51 AMTo: Tian, Kevin; jeremy@goop.org
Cc: xen devel; giamteck= choon@gmail.com; konrad.wilk@oracle.com
Subject: RE: [Xen-devel] = RE: Kernel BUG at arch/x86/mm/tlb.c:61

 

 > From: kevin.tian@intel.com
> To: jeremy@goop.org
&g= t; CC: tinnycloud@hotmail.com= ; xen-devel@lists.xensourc= e.com; giamteckchoon@gmail.c= om; konrad.wilk@oracle.com
> Date: Fri, 29 Apr 2011 08:19:44 +0800
> Subject: RE: [Xen-d= evel] RE: Kernel BUG at arch/x86/mm/tlb.c:61
>
> > From: Je= remy Fitzhardinge [mailto:jerem= y@goop.org]
> > Sent: Friday, April 29, 2011 7:29 AM
> &= gt;
> > On 04/25/2011 10:52 PM, Tian, Kevin wrote:
> > &= gt;> From: MaoXiaoyun
> > >> Sent: Monday, April 25, 2011= 11:15 AM
> > >>> Date: Fri, 15 Apr 2011 14:22:29 -0700> > >>> From: jeremy@go= op.org
> > >>> To: tinnycloud@hotmail.com
> > >>> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com;
&= gt; > >>> konrad.wilk= @oracle.com
> > >>> Subject: Re: Kernel BUG at arch/x= 86/mm/tlb.c:61
> > >>>
> > >>> On 04/15= /2011 05:23 AM, MaoXiaoyun wrote:
> > >>>> Hi=A3=BA
> > >>>>
> >= ; >>>> Could the crash related to this patch ?
> > >= ;>>> http://git.kernel.org/?p=3Dlinux/kernel/git/jeremy/x= en.git;a=3Dcommitdi
> > >>>> ff;h=3D45bfd7bfc6cf32= f8e60bb91b32349f0b5090eea3
> > >>>>
> > >&= gt;>> Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is<= br>> > >>>> before cpumask_clear_cpu(line 49).
> &g= t; >>>> Could it possible that right after execute line 40 of> > >>>> mmu_context.h, CPU revice IPI from other CPU t= o flush the mm, and
> > >>>> when in interrupt, find t= he TLB state happened to be TLBSTATE_OK.
> > >>>> Whic= h conflicts.
> > >>> Does reverting it help?
> >= >>>
> > >>> J
> > >>
> >= ; >> Hi Jeremy:
> > >>
> > >> The laste= st test result shows the reverting didn't help.
> > >> Kerne= l panic exactly at the same place in tlb.c.
> > >>
> &= gt; >> I have question about TLB state, from the stack,
> > = >> xen_do_hypervisor_callback-> xen_evtchn_do_upcall->...
&g= t; > >> ->drop_other_mm_ref
> > >>
> > = >> What cpu_tlbstate.state should be, could TLBSTATE_OK or
> &g= t; TLBSTATE_LAZY all be possible?
> > >> That is after a hyp= ercall from userspace, state will be TLBSTATE_OK,
> > and
> = > >> if from kernel space, state will be TLBSTATE_LAZE ?
> &= gt; >>
> > >> thanks.
> > > it looks a bug= in drop_other_mm_ref implementation, that current TLB
> > > st= ate should be checked before invoking leave_mm(). There's a window
> = > between below lines of code:
> > >
> > > <x= en_drop_mm_ref>
> > > /* Get the "official" set of= cpus referring to our pagetable. */
> > > if (!alloc_cpumask_v= ar(&mask, GFP_ATOMIC)) {
> > > for_each_online_cpu(cpu) {> > > if (!cpumask_test_cpu(cpu,
> > mm_cpumask(mm))> > > && per_cpu(xen_current_cr3, cpu) !=3D
> > = __pa(mm->pgd))
> > > continue;
> > > smp_call_fu= nction_single(cpu,
> > drop_other_mm_ref, mm, 1);
> > >= ; }
> > > return;
> > > }
> > >
>= > > there's chance that when smp_call_function_single is invoked, ac= tual
> > > TLB state has been updated in the other cpu. The ups= tream kernel patch
> > > you referred to earlier just makes thi= s bug exposed more easily. But
> > > even without this patch, y= ou may still suffer such issue which is why reverting
> > the patc= h doesn't help.
> > >
> > > Could you try adding a = check in drop_other_mm_ref?
> > >
> > > if (active_= mm =3D=3D mm && percpu_read(cpu_tlbstate.state) !=3D
> > T= LBSTATE_OK)
> > > leave_mm(smp_processor_id());
> > &g= t;
> > > once the interrupted context has TLBSTATE_OK, it impli= cates that later
> > > it will handle the TLB flush and thus no= need for leave_mm from
> > > interrupt handler, and that's the= assumption of doing leave_mm.
> >
> > That seems reason= able. MaoXiaoyun, does it fix the bug for you?
> >
> > K= evin, could you submit this as a proper patch?
> >
>
&g= t; I'm waiting for Xiaoyun's test result before submitting a proper patch, = since this
> part of logic is tricky and his test can make sure we do= n't overlook some corner
> cases. :-)
>
 
I think i= t works. The test has been running over 70 hours successfully.
My plan i= s run one week.
 
Thanks.
 
> Thanks
> Kevi= n

= --_000_625BA99ED14B2D499DC4E29D8138F1505C843BB382shsmsx502ccrc_-- --===============0574505600== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0574505600==--