* Re: kernel crashes on commit @ 2015-01-20 21:45 Mkrtchyan, Tigran 2015-01-21 10:04 ` Mkrtchyan, Tigran 0 siblings, 1 reply; 13+ messages in thread From: Mkrtchyan, Tigran @ 2015-01-20 21:45 UTC (permalink / raw) To: Weston Andros Adamson; +Cc: Linux NFS Mailing List SSB3aWxsIGNoZWNrIHRvbW9ycm93IHdpdGggUkhFTCA2IGtlcm5lbCBhbmQgbGV0IHlvdSBrbm93 bi4KClRoYW5rcywKICBUaWdyYW5PbiBKYW4gMjAsIDIwMTUgOTo0MyBQTSwgV2VzdG9uIEFuZHJv cyBBZGFtc29uIDxkcm9zQHByaW1hcnlkYXRhLmNvbT4gd3JvdGU6Cj4KPgo+ID4gT24gSmFuIDIw LCAyMDE1LCBhdCAyOjIyIFBNLCBNa3J0Y2h5YW4sIFRpZ3JhbiA8dGlncmFuLm1rcnRjaHlhbkBk ZXN5LmRlPiB3cm90ZTogCj4gPiAKPiA+IEhpIERyb3MsIAo+ID4gCj4gPiBkbyB5b3UgcmVmZXIg dG8gdGhpcyBjb21taXQgCj4gPiAKPiA+IGh0dHA6Ly9naXQubGludXgtbmZzLm9yZy8/cD1kcm9z L2xpbnV4LW5mcy5naXQ7YT1jb21taXQ7aD1kMjAxYzRkZTUxOGMxZDYxN2FhMjE2NjY0ODY5ZmEz MjlkNTYyZDdkID8gCj4KPiBZZXMsIHRoYXTigJlzIHRoZSBwYXRjaCBJIHdhcyB0YWxraW5nIGFi b3V0LiBHb29kIGZpbmQsIEkgd2FzIGFib3V0IHRvIGdvIGxvb2tpbmcgZm9yIGl0LiAKPgo+IElz IHRoYXQgcGF0Y2ggaW4gdGhlIGtlcm5lbHMgeW914oCZcmUgdGVzdGluZz8gCj4KPiAtZHJvcyAK Pgo+ID4gLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAtLS0tLSAKPiA+PiBGcm9tOiAiV2VzdG9uIEFu ZHJvcyBBZGFtc29uIiA8ZHJvc0BwcmltYXJ5ZGF0YS5jb20+IAo+ID4+IFRvOiAiVGlncmFuIE1r cnRjaHlhbiIgPHRpZ3Jhbi5ta3J0Y2h5YW5AZGVzeS5kZT4gCj4gPj4gQ2M6ICJsaW51eC1uZnMg bGlzdCIgPGxpbnV4LW5mc0B2Z2VyLmtlcm5lbC5vcmc+IAo+ID4+IFNlbnQ6IFR1ZXNkYXksIEph bnVhcnkgMjAsIDIwMTUgMzozNzo0OSBQTSAKPiA+PiBTdWJqZWN0OiBSZToga2VybmVsIGNyYXNo ZXMgb24gY29tbWl0IAo+ID4gCj4gPj4+IE9uIEphbiAyMCwgMjAxNSwgYXQgOTowMCBBTSwgTWty dGNoeWFuLCBUaWdyYW4gPHRpZ3Jhbi5ta3J0Y2h5YW5AZGVzeS5kZT4gd3JvdGU6IAo+ID4+PiAK PiA+Pj4gCj4gPj4+IAo+ID4+PiBEZWFyIGZlbGxvd3MsIAo+ID4+PiAKPiA+Pj4gc2luY2Ugd2Ug aGF2ZSBlbmFibGVkIGNvbW1pdCB0aHJvdWdoIERTIGNvZGUgd2UgCj4gPj4+IHBlcm1hbmVudGx5 IG9ic2VydmUga2VybmVsIGNyYXNoZXMgd2l0aCBSSEVMNi83IGFuZCB1YnVudHUgMTQuMDQ6IAo+ ID4+PiAKPiA+Pj4gCj4gPj4+IDwxPkJVRzogdW5hYmxlIHRvIGhhbmRsZSBrZXJuZWwgcGFnaW5n IHJlcXVlc3QgYXQgMDAwMDAwMDBkYzM2NDkxMyAKPiA+Pj4gPDE+SVA6IFs8ZmZmZmZmZmZhMDJi NDVkZj5dIG5mc19pbml0X2NvbW1pdCsweDFmLzB4ZjAgW25mc10gCj4gPj4+IDw0PlBHRCA2Mzkz YWUwNjcgUFVEIDAgCj4gPj4+IDw0Pk9vcHM6IDAwMDAgWyMxXSBTTVAgCj4gPj4+IDw0Pmxhc3Qg c3lzZnMgZmlsZTogL3N5cy9kZXZpY2VzL3N5c3RlbS9jcHUvb25saW5lIAo+ID4+PiA8ND5DUFUg MSAKPiA+Pj4gPDQ+TW9kdWxlcyBsaW5rZWQgaW46IHZmYXQgZmF0IHVzYl9zdG9yYWdlIG1wdDNz YXMgbXB0MnNhcyByYWlkX2NsYXNzIG1wdGN0bCAKPiA+Pj4gaXBtaV9kZXZpbnRmIGRlbGxfcmJ1 IG9wZW5hZnMoUCkoVSkgYXV0b2YgCj4gPj4+IHM0IG5mc19sYXlvdXRfbmZzdjQxX2ZpbGVzIG5m cyBsb2NrZCBmc2NhY2hlIGF1dGhfcnBjZ3NzIG5mc19hY2wgc3VucnBjIGJvbmRpbmcgCj4gPj4+ IDgwMjFxIGdhcnAgc3RwIGxsYyBpcHY2IHBvd2VyX21ldGVyIGFjIAo+ID4+PiBwaV9pcG1pIGlw bWlfc2kgaXBtaV9tc2doYW5kbGVyIGlUQ09fd2R0IGlUQ09fdmVuZG9yX3N1cHBvcnQgbWljcm9j b2RlIGRjZGJhcyBzZyAKPiA+Pj4gYm54MiBscGNfaWNoIG1mZF9jb3JlIGk3Y29yZV9lZGFjIGVk YSAKPiA+Pj4gY19jb3JlIGV4dDQgamJkMiBtYmNhY2hlIHNkX21vZCBjcmNfdDEwZGlmIHdtaSBw YXRhX2FjcGkgYXRhX2dlbmVyaWMgYXRhX3BpaXggCj4gPj4+IG1wdHNhcyBtcHRzY3NpaCBtcHRi YXNlIHNjc2lfdHJhbnNwb3J0X3MgCj4gPj4+IGFzIGRtX21pcnJvciBkbV9yZWdpb25faGFzaCBk bV9sb2cgZG1fbW9kIFtsYXN0IHVubG9hZGVkOiBzY3NpX3dhaXRfc2Nhbl0gCj4gPj4+IDw0PiAK PiA+Pj4gPDQ+UGlkOiAxODIwOSwgY29tbTogZmx1c2gtMDoxOSBUYWludGVkOiBQwqDCoMKgwqDC oMKgwqDCoMKgwqAgLS0tLS0tLS0tLS0tLS0tIAo+ID4+PiAyLjYuMzItNTA0LjMuMy5lbDYueDg2 XzY0ICMxIERlbGwgSW5jLiBQb3dlckVkZ2UgTTYxMC8wTjU4Mk0gCj4gPj4+IDw0PlJJUDogMDAx MDpbPGZmZmZmZmZmYTAyYjQ1ZGY+XcKgIFs8ZmZmZmZmZmZhMDJiNDVkZj5dIAo+ID4+PiBuZnNf aW5pdF9jb21taXQrMHgxZi8weGYwIFtuZnNdIAo+ID4+PiA8ND5SU1A6IDAwMTg6ZmZmZjg4MDYz OTg4ZGEzMMKgIEVGTEFHUzogMDAwMTAyNDYgCj4gPj4+IDw0PlJBWDogZmZmZjg4MDYzOTg4ZGI2 MCBSQlg6IGZmZmY4ODAwOWM0OTIwNDAgUkNYOiBmZmZmODgwNjM5ODhkYjMwIAo+ID4+PiA8ND5S RFg6IDAwMDAwMDAwMDAwMDAwMDAgUlNJOiBmZmZmODgwNjM5ODhkYjYwIFJESTogMDAwMDAwMDBk YzM2NDkwMyAKPiA+Pj4gPDQ+UkJQOiBmZmZmODgwNjM5ODhkYTQwIFIwODogZmZmZjg4MDYzOTg4 ZGE5MCBSMDk6IGY5YWEzN2ZhYTI1NGQ0MDQgCj4gPj4+IDw0PlIxMDogMDAwMDAwMDAwMDAwMDAx MCBSMTE6IDAwMDAwMDAwMDAwMDAwMDAgUjEyOiAwMDAwMDAwMDAwMDAwMDAxIAo+ID4+PiA8ND5S MTM6IGZmZmY4ODAzMzlmMzNhMDAgUjE0OiBmZmZmODgwNjM5ODhkYjMwIFIxNTogZmZmZjg4MDYz OTg4ZDhjOCAKPiA+Pj4gPDQ+RlM6wqAgMDAwMDAwMDAwMDAwMDAwMCgwMDAwKSBHUzpmZmZmODgw MDI4MjAwMDAwKDAwMDApIGtubEdTOjAwMDAwMDAwMDAwMDAwMDAgCj4gPj4+IDw0PkNTOsKgIDAw MTAgRFM6IDAwMTggRVM6IDAwMTggQ1IwOiAwMDAwMDAwMDgwMDUwMDNiIAo+ID4+PiA8ND5DUjI6 IDAwMDAwMDAwZGMzNjQ5MTMgQ1IzOiAwMDAwMDAwNjM5ZmJiMDAwIENSNDogMDAwMDAwMDAwMDAw MDdlMCAKPiA+Pj4gPDQ+RFIwOiAwMDAwMDAwMDAwMDAwMDAwIERSMTogMDAwMDAwMDAwMDAwMDAw MCBEUjI6IDAwMDAwMDAwMDAwMDAwMDAgCj4gPj4+IDw0PkRSMzogMDAwMDAwMDAwMDAwMDAwMCBE UjY6IDAwMDAwMDAwZmZmZjBmZjAgRFI3OiAwMDAwMDAwMDAwMDAwNDAwIAo+ID4+PiA8ND5Qcm9j ZXNzIGZsdXNoLTA6MTkgKHBpZDogMTgyMDksIHRocmVhZGluZm8gZmZmZjg4MDYzOTg4YzAwMCwg dGFzayAKPiA+Pj4gZmZmZjg4MDYzODM3YzA0MCkgCj4gPj4+IDw0PlN0YWNrOiAKPiA+Pj4gPDQ+ IDAwMDAwMDAwMDAwMDAwMDAgZmZmZjg4MDA5YzQ5MjA0MCBmZmZmODgwNjM5ODhkYWQwIGZmZmZm ZmZmYTAzMWZkYjcgCj4gPj4+IDw0PjxkPiBmZmZmODgwNjM4MzdjNWY4IGZmZmY4ODA2Mzk4OGRh OTAgZmZmZjg4MDBhNmUzNDYwMCBmZmZmODgwMzM3ZjJhOTUwIAo+ID4+PiA8ND48ZD4gZmZmZjg4 MDYzN2M5OTQ4OCAwMDAwMDAwMDM3ZjJhOTQwIGZmZmY4ODA2Mzk4OGRiNjAgMDAwMDAwMDAwMDAw MDAwMCAKPiA+Pj4gPDQ+Q2FsbCBUcmFjZTogCj4gPj4+IDw0PiBbPGZmZmZmZmZmYTAzMWZkYjc+ XSBmaWxlbGF5b3V0X2NvbW1pdF9wYWdlbGlzdCsweDI3Ny8weDNjMCAKPiA+Pj4gW25mc19sYXlv dXRfbmZzdjQxX2ZpbGVzXSAKPiA+Pj4gPDQ+IFs8ZmZmZmZmZmZhMDJiNjEzYj5dIG5mc19nZW5l cmljX2NvbW1pdF9saXN0KzB4YWIvMHgxMDAgW25mc10gCj4gPj4+IDw0PiBbPGZmZmZmZmZmYTAy YjYyN2M+XSBuZnNfY29tbWl0X2lub2RlKzB4ZWMvMHgxNTAgW25mc10gCj4gPj4+IDw0PiBbPGZm ZmZmZmZmYTAyYjZhYWI+XSBuZnNfd3JpdGVfaW5vZGUrMHhhYi8weDEwMCBbbmZzXSAKPiA+Pj4g PDQ+IFs8ZmZmZmZmZmY4MTFiYWVkYz5dIHdyaXRlYmFja19zaW5nbGVfaW5vZGUrMHgyMGMvMHgy OTAgCj4gPj4+IDw0PiBbPGZmZmZmZmZmODExYmIxYWQ+XSB3cml0ZWJhY2tfc2JfaW5vZGVzKzB4 YmQvMHgxNzAgCj4gPj4+IDw0PiBbPGZmZmZmZmZmODExYmIzMGI+XSB3cml0ZWJhY2tfaW5vZGVz X3diKzB4YWIvMHgxYjAgCj4gPj4+IDw0PiBbPGZmZmZmZmZmODExYmI3MDM+XSB3Yl93cml0ZWJh Y2srMHgyZjMvMHg0MTAgCj4gPj4+IDw0PiBbPGZmZmZmZmZmODEwMGI5Y2U+XSA/IGNvbW1vbl9p bnRlcnJ1cHQrMHhlLzB4MTMgCj4gPj4+IDw0PiBbPGZmZmZmZmZmODEwODgwNjI+XSA/IGRlbF90 aW1lcl9zeW5jKzB4MjIvMHgzMCAKPiA+Pj4gPDQ+IFs8ZmZmZmZmZmY4MTFiYjljNT5dIHdiX2Rv X3dyaXRlYmFjaysweDFhNS8weDI0MCAKPiA+Pj4gPDQ+IFs8ZmZmZmZmZmY4MTFiYmFjMz5dIGJk aV93cml0ZWJhY2tfdGFzaysweDYzLzB4MWIwIAo+ID4+PiA8ND4gWzxmZmZmZmZmZjgxMDllOTg3 Pl0gPyBiaXRfd2FpdHF1ZXVlKzB4MTcvMHhkMCAKPiA+Pj4gPDQ+IFs8ZmZmZmZmZmY4MTE0ODM2 MD5dID8gYmRpX3N0YXJ0X2ZuKzB4MC8weDEwMCAKPiA+Pj4gPDQ+IFs8ZmZmZmZmZmY4MTE0ODNl Nj5dIGJkaV9zdGFydF9mbisweDg2LzB4MTAwIAo+ID4+PiA8ND4gWzxmZmZmZmZmZjgxMTQ4MzYw Pl0gPyBiZGlfc3RhcnRfZm4rMHgwLzB4MTAwIAo+ID4+PiA8ND4gWzxmZmZmZmZmZjgxMDllNjZl Pl0ga3RocmVhZCsweDllLzB4YzAgCj4gPj4+IDw0PiBbPGZmZmZmZmZmODEwMGMyMGE+XSBjaGls ZF9yaXArMHhhLzB4MjAgCj4gPj4+IDw0PiBbPGZmZmZmZmZmODEwOWU1ZDA+XSA/IGt0aHJlYWQr MHgwLzB4YzAgCj4gPj4+IDw0PiBbPGZmZmZmZmZmODEwMGMyMDA+XSA/IGNoaWxkX3JpcCsweDAv MHgyMCAKPiA+Pj4gPDQ+Q29kZTogYzMgNjYgNjYgMmUgMGYgMWYgODQgMDAgMDAgMDAgMDAgMDAg NTUgNDggODkgZTUgNTMgNDggODMgZWMgMDggMGYgMWYgNDQgCj4gPj4+IDAwIDAwIDQ4IDhiIDA2 IDQ4IDg5IGZiIDQ4IDhiIDc4IDE4IDQ4IDM5IGM2IDQ4IDhiIDdmIDQwIDw0OD4gOGIgN2YgMTAg NzQgMmIgNGMgCj4gPj4+IDhiIDgzIGM4IDAxIDAwIDAwIDRjIDhiIDRlIDA4IDRjIDhkIDkzIGM4 IAo+ID4+PiA8MT5SSVDCoCBbPGZmZmZmZmZmYTAyYjQ1ZGY+XSBuZnNfaW5pdF9jb21taXQrMHgx Zi8weGYwIFtuZnNdIAo+ID4+PiA8ND4gUlNQIDxmZmZmODgwNjM5ODhkYTMwPiAKPiA+Pj4gPDQ+ Q1IyOiAwMDAwMDAwMGRjMzY0OTEzIAo+ID4+PiAKPiA+Pj4gCj4gPj4+IEkgaGF2ZSB2bWNvcmUg ZmlsZSBhcyB3ZWxsLCBzbyBsZXQgbWUga25vdyBpZiB5b3UgbmVlZCBzb21lIG1vcmUgaW5mb3Jt YXRpb24uIAo+ID4+PiAKPiA+PiAKPiA+PiBIaSBUaWdyYW4hIAo+ID4+IAo+ID4+IEhhdmUgeW91 IHRyaWVkIGEgcmVjZW50IHVwc3RyZWFtIGtlcm5lbD8gSUlSQyBJIGZpeGVkIGEgc2VlbWluZyBz aW1pbGFyIAo+ID4+IGZpbGVsYXlvdXQgCj4gPj4gY29tbWl0IGlzc3VlIG5vdCB0b28gbG9uZyBh Z28uIAo+ID4+IAo+ID4+IFRoZSBmaWxlbGF5b3V0IGNvbW1pdCBwYXRoIHNlZW1zIHRvIGhhdmUg YmVlbiBicm9rZW4gZm9yIGEgd2hpbGUgLSBtb3N0bHkgCj4gPj4gYmVjYXVzZSAKPiA+PiBhbGwg dGhlIGZpbGVsYXlvdXQgc2VydmVycyAodGhhdCBJIGtub3cgb2YpIHVzZSBzdGFibGUgd3JpdGVz LCBzbyB0aGF0IGNvZGUgcGF0aCAKPiA+PiB3ZW50IAo+ID4+IHVudGVzdGVkLi4uIAo+ID4+IAo+ ID4+IC1kcm9zIAo+Cg== ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-01-20 21:45 kernel crashes on commit Mkrtchyan, Tigran @ 2015-01-21 10:04 ` Mkrtchyan, Tigran 2015-01-21 15:20 ` Weston Andros Adamson 0 siblings, 1 reply; 13+ messages in thread From: Mkrtchyan, Tigran @ 2015-01-21 10:04 UTC (permalink / raw) To: Weston Andros Adamson; +Cc: Linux NFS Mailing List, Myklebust, Trond Hi Dros, after adopting patch for RHEL6 kernel, it works. We have to push it into stable fixes. Do you know the procedure? Thanks, Tigran. ----- Original Message ----- > From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de> > To: "Weston Andros Adamson" <dros@primarydata.com> > Cc: "Linux NFS Mailing List" <linux-nfs@vger.kernel.org> > Sent: Tuesday, January 20, 2015 10:45:22 PM > Subject: Re: kernel crashes on commit > I will check tomorrow with RHEL 6 kernel and let you known. > > Thanks, > TigranOn Jan 20, 2015 9:43 PM, Weston Andros Adamson <dros@primarydata.com> > wrote: >> >> >> > On Jan 20, 2015, at 2:22 PM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: >> > >> > Hi Dros, >> > >> > do you refer to this commit >> > >> > http://git.linux-nfs.org/?p=dros/linux-nfs.git;a=commit;h=d201c4de518c1d617aa216664869fa329d562d7d >> > ? >> >> Yes, that’s the patch I was talking about. Good find, I was about to go looking >> for it. >> >> Is that patch in the kernels you’re testing? >> >> -dros >> >> > ----- Original Message ----- >> >> From: "Weston Andros Adamson" <dros@primarydata.com> >> >> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de> >> >> Cc: "linux-nfs list" <linux-nfs@vger.kernel.org> >> >> Sent: Tuesday, January 20, 2015 3:37:49 PM >> >> Subject: Re: kernel crashes on commit >> > >> >>> On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: >> >>> >> >>> >> >>> >> >>> Dear fellows, >> >>> >> >>> since we have enabled commit through DS code we >> >>> permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: >> >>> >> >>> >> >>> <1>BUG: unable to handle kernel paging request at 00000000dc364913 >> >>> <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >> >>> <4>PGD 6393ae067 PUD 0 >> >>> <4>Oops: 0000 [#1] SMP >> >>> <4>last sysfs file: /sys/devices/system/cpu/online >> >>> <4>CPU 1 >> >>> <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl >> >>> ipmi_devintf dell_rbu openafs(P)(U) autof >> >>> s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding >> >>> 8021q garp stp llc ipv6 power_meter ac >> >>> pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg >> >>> bnx2 lpc_ich mfd_core i7core_edac eda >> >>> c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix >> >>> mptsas mptscsih mptbase scsi_transport_s >> >>> as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] >> >>> <4> >> >>> <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- >> >>> 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M >> >>> <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] >> >>> nfs_init_commit+0x1f/0xf0 [nfs] >> >>> <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 >> >>> <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 >> >>> <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 >> >>> <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 >> >>> <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 >> >>> <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 >> >>> <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 >> >>> <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >> >>> <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 >> >>> <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> >>> <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >> >>> <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task >> >>> ffff88063837c040) >> >>> <4>Stack: >> >>> <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 >> >>> <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 >> >>> <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 >> >>> <4>Call Trace: >> >>> <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 >> >>> [nfs_layout_nfsv41_files] >> >>> <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] >> >>> <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] >> >>> <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] >> >>> <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 >> >>> <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 >> >>> <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 >> >>> <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 >> >>> <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 >> >>> <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 >> >>> <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 >> >>> <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 >> >>> <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 >> >>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >> >>> <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 >> >>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >> >>> <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 >> >>> <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 >> >>> <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 >> >>> <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 >> >>> <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 >> >>> 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c >> >>> 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 >> >>> <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >> >>> <4> RSP <ffff88063988da30> >> >>> <4>CR2: 00000000dc364913 >> >>> >> >>> >> >>> I have vmcore file as well, so let me know if you need some more information. >> >>> >> >> >> >> Hi Tigran! >> >> >> >> Have you tried a recent upstream kernel? IIRC I fixed a seeming similar >> >> filelayout >> >> commit issue not too long ago. >> >> >> >> The filelayout commit path seems to have been broken for a while - mostly >> >> because >> >> all the filelayout servers (that I know of) use stable writes, so that code path >> >> went >> >> untested... >> >> >> >> -dros >> > N�����r��y���b�X��ǧv�^�){.n�+����{���"��^n�r���z���h����&���G���h�(�階�ݢj"���m�����z�ޖ���f���h���~�m� ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-01-21 10:04 ` Mkrtchyan, Tigran @ 2015-01-21 15:20 ` Weston Andros Adamson 2015-03-26 9:28 ` Benjamin Coddington 0 siblings, 1 reply; 13+ messages in thread From: Weston Andros Adamson @ 2015-01-21 15:20 UTC (permalink / raw) To: Tigran Mkrtchyan, Steve Dickson; +Cc: linux-nfs list, Trond Myklebust > On Jan 21, 2015, at 5:04 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > > Hi Dros, > > after adopting patch for RHEL6 kernel, it works. Great! > We have to push it into stable fixes. Do you know > the procedure? I normally bug Steve D ;) -dros > ----- Original Message ----- >> From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de> >> To: "Weston Andros Adamson" <dros@primarydata.com> >> Cc: "Linux NFS Mailing List" <linux-nfs@vger.kernel.org> >> Sent: Tuesday, January 20, 2015 10:45:22 PM >> Subject: Re: kernel crashes on commit > >> I will check tomorrow with RHEL 6 kernel and let you known. >> >> Thanks, >> TigranOn Jan 20, 2015 9:43 PM, Weston Andros Adamson <dros@primarydata.com> >> wrote: >>> >>> >>>> On Jan 20, 2015, at 2:22 PM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: >>>> >>>> Hi Dros, >>>> >>>> do you refer to this commit >>>> >>>> http://git.linux-nfs.org/?p=dros/linux-nfs.git;a=commit;h=d201c4de518c1d617aa216664869fa329d562d7d >>>> ? >>> >>> Yes, that’s the patch I was talking about. Good find, I was about to go looking >>> for it. >>> >>> Is that patch in the kernels you’re testing? >>> >>> -dros >>> >>>> ----- Original Message ----- >>>>> From: "Weston Andros Adamson" <dros@primarydata.com> >>>>> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de> >>>>> Cc: "linux-nfs list" <linux-nfs@vger.kernel.org> >>>>> Sent: Tuesday, January 20, 2015 3:37:49 PM >>>>> Subject: Re: kernel crashes on commit >>>> >>>>>> On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: >>>>>> >>>>>> >>>>>> >>>>>> Dear fellows, >>>>>> >>>>>> since we have enabled commit through DS code we >>>>>> permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: >>>>>> >>>>>> >>>>>> <1>BUG: unable to handle kernel paging request at 00000000dc364913 >>>>>> <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >>>>>> <4>PGD 6393ae067 PUD 0 >>>>>> <4>Oops: 0000 [#1] SMP >>>>>> <4>last sysfs file: /sys/devices/system/cpu/online >>>>>> <4>CPU 1 >>>>>> <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl >>>>>> ipmi_devintf dell_rbu openafs(P)(U) autof >>>>>> s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding >>>>>> 8021q garp stp llc ipv6 power_meter ac >>>>>> pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg >>>>>> bnx2 lpc_ich mfd_core i7core_edac eda >>>>>> c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix >>>>>> mptsas mptscsih mptbase scsi_transport_s >>>>>> as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] >>>>>> <4> >>>>>> <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- >>>>>> 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M >>>>>> <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] >>>>>> nfs_init_commit+0x1f/0xf0 [nfs] >>>>>> <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 >>>>>> <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 >>>>>> <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 >>>>>> <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 >>>>>> <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 >>>>>> <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 >>>>>> <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 >>>>>> <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >>>>>> <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 >>>>>> <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>>> <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>>> <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task >>>>>> ffff88063837c040) >>>>>> <4>Stack: >>>>>> <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 >>>>>> <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 >>>>>> <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 >>>>>> <4>Call Trace: >>>>>> <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 >>>>>> [nfs_layout_nfsv41_files] >>>>>> <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] >>>>>> <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] >>>>>> <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] >>>>>> <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 >>>>>> <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 >>>>>> <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 >>>>>> <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 >>>>>> <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 >>>>>> <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 >>>>>> <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 >>>>>> <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 >>>>>> <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >>>>>> <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >>>>>> <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 >>>>>> <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 >>>>>> <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 >>>>>> <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 >>>>>> <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 >>>>>> 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c >>>>>> 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 >>>>>> <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >>>>>> <4> RSP <ffff88063988da30> >>>>>> <4>CR2: 00000000dc364913 >>>>>> >>>>>> >>>>>> I have vmcore file as well, so let me know if you need some more information. >>>>>> >>>>> >>>>> Hi Tigran! >>>>> >>>>> Have you tried a recent upstream kernel? IIRC I fixed a seeming similar >>>>> filelayout >>>>> commit issue not too long ago. >>>>> >>>>> The filelayout commit path seems to have been broken for a while - mostly >>>>> because >>>>> all the filelayout servers (that I know of) use stable writes, so that code path >>>>> went >>>>> untested... >>>>> >>>>> -dros >>> >> N�����r��y���b�X��ǧv�^�){.n�+����{���"��^n�r���z���h����&���G���h�(�階�ݢj"���m�����z�ޖ���f���h���~�m� ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-01-21 15:20 ` Weston Andros Adamson @ 2015-03-26 9:28 ` Benjamin Coddington 2015-03-26 10:11 ` Mkrtchyan, Tigran 0 siblings, 1 reply; 13+ messages in thread From: Benjamin Coddington @ 2015-03-26 9:28 UTC (permalink / raw) To: Weston Andros Adamson Cc: Tigran Mkrtchyan, Steve Dickson, linux-nfs list, Trond Myklebust [-- Attachment #1: Type: text/plain, Size: 7065 bytes --] On Wed, 21 Jan 2015, Weston Andros Adamson wrote: > > > On Jan 21, 2015, at 5:04 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > > > > Hi Dros, > > > > after adopting patch for RHEL6 kernel, it works. > > Great! > > > We have to push it into stable fixes. Do you know > > the procedure? > > I normally bug Steve D ;) Or you can open a bug against RHEL6; it should get picked up quickly, now that you've done all the work. Ben > > > ----- Original Message ----- > >> From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de> > >> To: "Weston Andros Adamson" <dros@primarydata.com> > >> Cc: "Linux NFS Mailing List" <linux-nfs@vger.kernel.org> > >> Sent: Tuesday, January 20, 2015 10:45:22 PM > >> Subject: Re: kernel crashes on commit > > > >> I will check tomorrow with RHEL 6 kernel and let you known. > >> > >> Thanks, > >> TigranOn Jan 20, 2015 9:43 PM, Weston Andros Adamson <dros@primarydata.com> > >> wrote: > >>> > >>> > >>>> On Jan 20, 2015, at 2:22 PM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > >>>> > >>>> Hi Dros, > >>>> > >>>> do you refer to this commit > >>>> > >>>> http://git.linux-nfs.org/?p=dros/linux-nfs.git;a=commit;h=d201c4de518c1d617aa216664869fa329d562d7d > >>>> ? > >>> > >>> Yes, that’s the patch I was talking about. Good find, I was about to go looking > >>> for it. > >>> > >>> Is that patch in the kernels you’re testing? > >>> > >>> -dros > >>> > >>>> ----- Original Message ----- > >>>>> From: "Weston Andros Adamson" <dros@primarydata.com> > >>>>> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de> > >>>>> Cc: "linux-nfs list" <linux-nfs@vger.kernel.org> > >>>>> Sent: Tuesday, January 20, 2015 3:37:49 PM > >>>>> Subject: Re: kernel crashes on commit > >>>> > >>>>>> On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> Dear fellows, > >>>>>> > >>>>>> since we have enabled commit through DS code we > >>>>>> permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: > >>>>>> > >>>>>> > >>>>>> <1>BUG: unable to handle kernel paging request at 00000000dc364913 > >>>>>> <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > >>>>>> <4>PGD 6393ae067 PUD 0 > >>>>>> <4>Oops: 0000 [#1] SMP > >>>>>> <4>last sysfs file: /sys/devices/system/cpu/online > >>>>>> <4>CPU 1 > >>>>>> <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl > >>>>>> ipmi_devintf dell_rbu openafs(P)(U) autof > >>>>>> s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding > >>>>>> 8021q garp stp llc ipv6 power_meter ac > >>>>>> pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg > >>>>>> bnx2 lpc_ich mfd_core i7core_edac eda > >>>>>> c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix > >>>>>> mptsas mptscsih mptbase scsi_transport_s > >>>>>> as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] > >>>>>> <4> > >>>>>> <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- > >>>>>> 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M > >>>>>> <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] > >>>>>> nfs_init_commit+0x1f/0xf0 [nfs] > >>>>>> <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 > >>>>>> <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 > >>>>>> <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 > >>>>>> <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 > >>>>>> <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 > >>>>>> <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 > >>>>>> <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > >>>>>> <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > >>>>>> <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 > >>>>>> <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > >>>>>> <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > >>>>>> <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task > >>>>>> ffff88063837c040) > >>>>>> <4>Stack: > >>>>>> <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 > >>>>>> <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 > >>>>>> <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 > >>>>>> <4>Call Trace: > >>>>>> <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 > >>>>>> [nfs_layout_nfsv41_files] > >>>>>> <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] > >>>>>> <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] > >>>>>> <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] > >>>>>> <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 > >>>>>> <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 > >>>>>> <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 > >>>>>> <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 > >>>>>> <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 > >>>>>> <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 > >>>>>> <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 > >>>>>> <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 > >>>>>> <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 > >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > >>>>>> <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 > >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > >>>>>> <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 > >>>>>> <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 > >>>>>> <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 > >>>>>> <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 > >>>>>> <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 > >>>>>> 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c > >>>>>> 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 > >>>>>> <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > >>>>>> <4> RSP <ffff88063988da30> > >>>>>> <4>CR2: 00000000dc364913 > >>>>>> > >>>>>> > >>>>>> I have vmcore file as well, so let me know if you need some more information. > >>>>>> > >>>>> > >>>>> Hi Tigran! > >>>>> > >>>>> Have you tried a recent upstream kernel? IIRC I fixed a seeming similar > >>>>> filelayout > >>>>> commit issue not too long ago. > >>>>> > >>>>> The filelayout commit path seems to have been broken for a while - mostly > >>>>> because > >>>>> all the filelayout servers (that I know of) use stable writes, so that code path > >>>>> went > >>>>> untested... > >>>>> > >>>>> -dros > >>> > >> N???????????????r??????y?????????b???X??????ǧv???^???){.n???+????????????{?????????"??????^n???r?????????z?????????h????????????&?????????G?????????h???(???階???ݢj"?????????m???????????????z???ޖ?????????f?????????h?????????~???m??? > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-03-26 9:28 ` Benjamin Coddington @ 2015-03-26 10:11 ` Mkrtchyan, Tigran 2015-03-26 12:00 ` Benjamin Coddington 2015-06-26 12:33 ` Benjamin Coddington 0 siblings, 2 replies; 13+ messages in thread From: Mkrtchyan, Tigran @ 2015-03-26 10:11 UTC (permalink / raw) To: Benjamin Coddington Cc: Weston Andros Adamson, Steve Dickson, linux-nfs list, Trond Myklebust the bug was submitted and fixed in rhel 6.7 https://bugzilla.redhat.com/show_bug.cgi?id=3D1184394 I don't know about rhel7, which is affected as well. Tigran. ----- Original Message ----- > From: "Benjamin Coddington" <bcodding@redhat.com> > To: "Weston Andros Adamson" <dros@primarydata.com> > Cc: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>, "Steve Dickson" <Steve= D@redhat.com>, "linux-nfs list" > <linux-nfs@vger.kernel.org>, "Trond Myklebust" <trond.myklebust@primaryda= ta.com> > Sent: Thursday, March 26, 2015 10:28:59 AM > Subject: Re: kernel crashes on commit > On Wed, 21 Jan 2015, Weston Andros Adamson wrote: >=20 >> >> > On Jan 21, 2015, at 5:04 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.= de> wrote: >> > >> > Hi Dros, >> > >> > after adopting patch for RHEL6 kernel, it works. >> >> Great! >> >> > We have to push it into stable fixes. Do you know >> > the procedure? >> >> I normally bug Steve D ;) >=20 > Or you can open a bug against RHEL6; it should get picked up quickly, now > that you've done all the work. >=20 > Ben >=20 >> >> > ----- Original Message ----- >> >> From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de> >> >> To: "Weston Andros Adamson" <dros@primarydata.com> >> >> Cc: "Linux NFS Mailing List" <linux-nfs@vger.kernel.org> >> >> Sent: Tuesday, January 20, 2015 10:45:22 PM >> >> Subject: Re: kernel crashes on commit >> > >> >> I will check tomorrow with RHEL 6 kernel and let you known. >> >> >> >> Thanks, >> >> TigranOn Jan 20, 2015 9:43 PM, Weston Andros Adamson <dros@primarydat= a.com> >> >> wrote: >> >>> >> >>> >> >>>> On Jan 20, 2015, at 2:22 PM, Mkrtchyan, Tigran <tigran.mkrtchyan@de= sy.de> wrote: >> >>>> >> >>>> Hi Dros, >> >>>> >> >>>> do you refer to this commit >> >>>> >> >>>> http://git.linux-nfs.org/?p=3Ddros/linux-nfs.git;a=3Dcommit;h=3Dd20= 1c4de518c1d617aa216664869fa329d562d7d >> >>>> ? >> >>> >> >>> Yes, that=E2=80=99s the patch I was talking about. Good find, I was = about to go looking >> >>> for it. >> >>> >> >>> Is that patch in the kernels you=E2=80=99re testing? >> >>> >> >>> -dros >> >>> >> >>>> ----- Original Message ----- >> >>>>> From: "Weston Andros Adamson" <dros@primarydata.com> >> >>>>> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de> >> >>>>> Cc: "linux-nfs list" <linux-nfs@vger.kernel.org> >> >>>>> Sent: Tuesday, January 20, 2015 3:37:49 PM >> >>>>> Subject: Re: kernel crashes on commit >> >>>> >> >>>>>> On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@= desy.de> wrote: >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> Dear fellows, >> >>>>>> >> >>>>>> since we have enabled commit through DS code we >> >>>>>> permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: >> >>>>>> >> >>>>>> >> >>>>>> <1>BUG: unable to handle kernel paging request at 00000000dc36491= 3 >> >>>>>> <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >> >>>>>> <4>PGD 6393ae067 PUD 0 >> >>>>>> <4>Oops: 0000 [#1] SMP >> >>>>>> <4>last sysfs file: /sys/devices/system/cpu/online >> >>>>>> <4>CPU 1 >> >>>>>> <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_c= lass mptctl >> >>>>>> ipmi_devintf dell_rbu openafs(P)(U) autof >> >>>>>> s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl = sunrpc bonding >> >>>>>> 8021q garp stp llc ipv6 power_meter ac >> >>>>>> pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support micr= ocode dcdbas sg >> >>>>>> bnx2 lpc_ich mfd_core i7core_edac eda >> >>>>>> c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_gene= ric ata_piix >> >>>>>> mptsas mptscsih mptbase scsi_transport_s >> >>>>>> as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wa= it_scan] >> >>>>>> <4> >> >>>>>> <4>Pid: 18209, comm: flush-0:19 Tainted: P ------------= --- >> >>>>>> 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M >> >>>>>> <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] >> >>>>>> nfs_init_commit+0x1f/0xf0 [nfs] >> >>>>>> <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 >> >>>>>> <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988d= b30 >> >>>>>> <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364= 903 >> >>>>>> <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d= 404 >> >>>>>> <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000= 001 >> >>>>>> <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d= 8c8 >> >>>>>> <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:00= 00000000000000 >> >>>>>> <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >> >>>>>> <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 0000000000000= 7e0 >> >>>>>> <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000= 000 >> >>>>>> <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000= 400 >> >>>>>> <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, t= ask >> >>>>>> ffff88063837c040) >> >>>>>> <4>Stack: >> >>>>>> <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa0= 31fdb7 >> >>>>>> <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880= 337f2a950 >> >>>>>> <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000= 000000000 >> >>>>>> <4>Call Trace: >> >>>>>> <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 >> >>>>>> [nfs_layout_nfsv41_files] >> >>>>>> <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] >> >>>>>> <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] >> >>>>>> <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] >> >>>>>> <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 >> >>>>>> <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 >> >>>>>> <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 >> >>>>>> <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 >> >>>>>> <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 >> >>>>>> <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 >> >>>>>> <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 >> >>>>>> <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 >> >>>>>> <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 >> >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >> >>>>>> <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 >> >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >> >>>>>> <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 >> >>>>>> <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 >> >>>>>> <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 >> >>>>>> <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 >> >>>>>> <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83= ec 08 0f 1f 44 >> >>>>>> 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b = 7f 10 74 2b 4c >> >>>>>> 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 >> >>>>>> <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >> >>>>>> <4> RSP <ffff88063988da30> >> >>>>>> <4>CR2: 00000000dc364913 >> >>>>>> >> >>>>>> >> >>>>>> I have vmcore file as well, so let me know if you need some more = information. >> >>>>>> >> >>>>> >> >>>>> Hi Tigran! >> >>>>> >> >>>>> Have you tried a recent upstream kernel? IIRC I fixed a seeming si= milar >> >>>>> filelayout >> >>>>> commit issue not too long ago. >> >>>>> >> >>>>> The filelayout commit path seems to have been broken for a while -= mostly >> >>>>> because >> >>>>> all the filelayout servers (that I know of) use stable writes, so = that code path >> >>>>> went >> >>>>> untested... >> >>>>> >> >>>>> -dros >> >>> >> >> N???????????????r??????y?????????b???X??????=C7=A7v???^???)=DE=BA{.n?= ??+????????????{?????????"??????^n???r?????????z?????????h????????????&????= ?????G?????????h???(???=E9=9A=8E???=DD=A2j"?????????m???????????????z???=DE= =96?????????f?????????h?????????~???m??? >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-03-26 10:11 ` Mkrtchyan, Tigran @ 2015-03-26 12:00 ` Benjamin Coddington 2015-06-26 12:33 ` Benjamin Coddington 1 sibling, 0 replies; 13+ messages in thread From: Benjamin Coddington @ 2015-03-26 12:00 UTC (permalink / raw) To: Mkrtchyan, Tigran Cc: Weston Andros Adamson, Steve Dickson, linux-nfs list, Trond Myklebust [-- Attachment #1: Type: text/plain, Size: 8661 bytes --] On Thu, 26 Mar 2015, Mkrtchyan, Tigran wrote: > the bug was submitted and fixed in rhel 6.7 > > https://bugzilla.redhat.com/show_bug.cgi?id=1184394 > > I don't know about rhel7, which is affected as well. Ah! Thanks for pointing that out. RHEL7 is missing d201c4d pnfs: fix race in filelayout commit path Which will hopefully make it into 7.2; working on it here: https://bugzilla.redhat.com/show_bug.cgi?id=1111712 We probably want it in earlier RHEL7 too.. I'll see about that. Ben > ----- Original Message ----- > > From: "Benjamin Coddington" <bcodding@redhat.com> > > To: "Weston Andros Adamson" <dros@primarydata.com> > > Cc: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>, "Steve Dickson" <SteveD@redhat.com>, "linux-nfs list" > > <linux-nfs@vger.kernel.org>, "Trond Myklebust" <trond.myklebust@primarydata.com> > > Sent: Thursday, March 26, 2015 10:28:59 AM > > Subject: Re: kernel crashes on commit > > > On Wed, 21 Jan 2015, Weston Andros Adamson wrote: > > > >> > >> > On Jan 21, 2015, at 5:04 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > >> > > >> > Hi Dros, > >> > > >> > after adopting patch for RHEL6 kernel, it works. > >> > >> Great! > >> > >> > We have to push it into stable fixes. Do you know > >> > the procedure? > >> > >> I normally bug Steve D ;) > > > > Or you can open a bug against RHEL6; it should get picked up quickly, now > > that you've done all the work. > > > > Ben > > > >> > >> > ----- Original Message ----- > >> >> From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de> > >> >> To: "Weston Andros Adamson" <dros@primarydata.com> > >> >> Cc: "Linux NFS Mailing List" <linux-nfs@vger.kernel.org> > >> >> Sent: Tuesday, January 20, 2015 10:45:22 PM > >> >> Subject: Re: kernel crashes on commit > >> > > >> >> I will check tomorrow with RHEL 6 kernel and let you known. > >> >> > >> >> Thanks, > >> >> TigranOn Jan 20, 2015 9:43 PM, Weston Andros Adamson <dros@primarydata.com> > >> >> wrote: > >> >>> > >> >>> > >> >>>> On Jan 20, 2015, at 2:22 PM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > >> >>>> > >> >>>> Hi Dros, > >> >>>> > >> >>>> do you refer to this commit > >> >>>> > >> >>>> http://git.linux-nfs.org/?p=dros/linux-nfs.git;a=commit;h=d201c4de518c1d617aa216664869fa329d562d7d > >> >>>> ? > >> >>> > >> >>> Yes, that’s the patch I was talking about. Good find, I was about to go looking > >> >>> for it. > >> >>> > >> >>> Is that patch in the kernels you’re testing? > >> >>> > >> >>> -dros > >> >>> > >> >>>> ----- Original Message ----- > >> >>>>> From: "Weston Andros Adamson" <dros@primarydata.com> > >> >>>>> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de> > >> >>>>> Cc: "linux-nfs list" <linux-nfs@vger.kernel.org> > >> >>>>> Sent: Tuesday, January 20, 2015 3:37:49 PM > >> >>>>> Subject: Re: kernel crashes on commit > >> >>>> > >> >>>>>> On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> Dear fellows, > >> >>>>>> > >> >>>>>> since we have enabled commit through DS code we > >> >>>>>> permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: > >> >>>>>> > >> >>>>>> > >> >>>>>> <1>BUG: unable to handle kernel paging request at 00000000dc364913 > >> >>>>>> <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > >> >>>>>> <4>PGD 6393ae067 PUD 0 > >> >>>>>> <4>Oops: 0000 [#1] SMP > >> >>>>>> <4>last sysfs file: /sys/devices/system/cpu/online > >> >>>>>> <4>CPU 1 > >> >>>>>> <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl > >> >>>>>> ipmi_devintf dell_rbu openafs(P)(U) autof > >> >>>>>> s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding > >> >>>>>> 8021q garp stp llc ipv6 power_meter ac > >> >>>>>> pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg > >> >>>>>> bnx2 lpc_ich mfd_core i7core_edac eda > >> >>>>>> c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix > >> >>>>>> mptsas mptscsih mptbase scsi_transport_s > >> >>>>>> as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] > >> >>>>>> <4> > >> >>>>>> <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- > >> >>>>>> 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M > >> >>>>>> <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] > >> >>>>>> nfs_init_commit+0x1f/0xf0 [nfs] > >> >>>>>> <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 > >> >>>>>> <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 > >> >>>>>> <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 > >> >>>>>> <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 > >> >>>>>> <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 > >> >>>>>> <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 > >> >>>>>> <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > >> >>>>>> <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > >> >>>>>> <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 > >> >>>>>> <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > >> >>>>>> <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > >> >>>>>> <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task > >> >>>>>> ffff88063837c040) > >> >>>>>> <4>Stack: > >> >>>>>> <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 > >> >>>>>> <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 > >> >>>>>> <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 > >> >>>>>> <4>Call Trace: > >> >>>>>> <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 > >> >>>>>> [nfs_layout_nfsv41_files] > >> >>>>>> <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] > >> >>>>>> <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] > >> >>>>>> <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] > >> >>>>>> <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 > >> >>>>>> <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 > >> >>>>>> <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 > >> >>>>>> <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 > >> >>>>>> <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 > >> >>>>>> <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 > >> >>>>>> <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 > >> >>>>>> <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 > >> >>>>>> <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 > >> >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > >> >>>>>> <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 > >> >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > >> >>>>>> <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 > >> >>>>>> <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 > >> >>>>>> <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 > >> >>>>>> <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 > >> >>>>>> <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 > >> >>>>>> 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c > >> >>>>>> 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 > >> >>>>>> <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > >> >>>>>> <4> RSP <ffff88063988da30> > >> >>>>>> <4>CR2: 00000000dc364913 > >> >>>>>> > >> >>>>>> > >> >>>>>> I have vmcore file as well, so let me know if you need some more information. > >> >>>>>> > >> >>>>> > >> >>>>> Hi Tigran! > >> >>>>> > >> >>>>> Have you tried a recent upstream kernel? IIRC I fixed a seeming similar > >> >>>>> filelayout > >> >>>>> commit issue not too long ago. > >> >>>>> > >> >>>>> The filelayout commit path seems to have been broken for a while - mostly > >> >>>>> because > >> >>>>> all the filelayout servers (that I know of) use stable writes, so that code path > >> >>>>> went > >> >>>>> untested... > >> >>>>> > >> >>>>> -dros > >> >>> > >> >> N???????????????r??????y?????????b???X??????ǧv???^???){.n???+????????????{?????????"??????^n???r?????????z?????????h????????????&?????????G?????????h???(???階???ݢj"?????????m???????????????z???ޖ?????????f?????????h?????????~???m??? > >> > >> -- > >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-03-26 10:11 ` Mkrtchyan, Tigran 2015-03-26 12:00 ` Benjamin Coddington @ 2015-06-26 12:33 ` Benjamin Coddington 1 sibling, 0 replies; 13+ messages in thread From: Benjamin Coddington @ 2015-06-26 12:33 UTC (permalink / raw) To: Mkrtchyan, Tigran Cc: Weston Andros Adamson, Steve Dickson, linux-nfs list, Trond Myklebust [-- Attachment #1: Type: text/plain, Size: 8414 bytes --] Fixed in RHEL7.1 Ben On Thu, 26 Mar 2015, Mkrtchyan, Tigran wrote: > the bug was submitted and fixed in rhel 6.7 > > https://bugzilla.redhat.com/show_bug.cgi?id=1184394 > > I don't know about rhel7, which is affected as well. > > Tigran. > > ----- Original Message ----- > > From: "Benjamin Coddington" <bcodding@redhat.com> > > To: "Weston Andros Adamson" <dros@primarydata.com> > > Cc: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>, "Steve Dickson" <SteveD@redhat.com>, "linux-nfs list" > > <linux-nfs@vger.kernel.org>, "Trond Myklebust" <trond.myklebust@primarydata.com> > > Sent: Thursday, March 26, 2015 10:28:59 AM > > Subject: Re: kernel crashes on commit > > > On Wed, 21 Jan 2015, Weston Andros Adamson wrote: > > > >> > >> > On Jan 21, 2015, at 5:04 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > >> > > >> > Hi Dros, > >> > > >> > after adopting patch for RHEL6 kernel, it works. > >> > >> Great! > >> > >> > We have to push it into stable fixes. Do you know > >> > the procedure? > >> > >> I normally bug Steve D ;) > > > > Or you can open a bug against RHEL6; it should get picked up quickly, now > > that you've done all the work. > > > > Ben > > > >> > >> > ----- Original Message ----- > >> >> From: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de> > >> >> To: "Weston Andros Adamson" <dros@primarydata.com> > >> >> Cc: "Linux NFS Mailing List" <linux-nfs@vger.kernel.org> > >> >> Sent: Tuesday, January 20, 2015 10:45:22 PM > >> >> Subject: Re: kernel crashes on commit > >> > > >> >> I will check tomorrow with RHEL 6 kernel and let you known. > >> >> > >> >> Thanks, > >> >> TigranOn Jan 20, 2015 9:43 PM, Weston Andros Adamson <dros@primarydata.com> > >> >> wrote: > >> >>> > >> >>> > >> >>>> On Jan 20, 2015, at 2:22 PM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > >> >>>> > >> >>>> Hi Dros, > >> >>>> > >> >>>> do you refer to this commit > >> >>>> > >> >>>> http://git.linux-nfs.org/?p=dros/linux-nfs.git;a=commit;h=d201c4de518c1d617aa216664869fa329d562d7d > >> >>>> ? > >> >>> > >> >>> Yes, that’s the patch I was talking about. Good find, I was about to go looking > >> >>> for it. > >> >>> > >> >>> Is that patch in the kernels you’re testing? > >> >>> > >> >>> -dros > >> >>> > >> >>>> ----- Original Message ----- > >> >>>>> From: "Weston Andros Adamson" <dros@primarydata.com> > >> >>>>> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de> > >> >>>>> Cc: "linux-nfs list" <linux-nfs@vger.kernel.org> > >> >>>>> Sent: Tuesday, January 20, 2015 3:37:49 PM > >> >>>>> Subject: Re: kernel crashes on commit > >> >>>> > >> >>>>>> On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> Dear fellows, > >> >>>>>> > >> >>>>>> since we have enabled commit through DS code we > >> >>>>>> permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: > >> >>>>>> > >> >>>>>> > >> >>>>>> <1>BUG: unable to handle kernel paging request at 00000000dc364913 > >> >>>>>> <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > >> >>>>>> <4>PGD 6393ae067 PUD 0 > >> >>>>>> <4>Oops: 0000 [#1] SMP > >> >>>>>> <4>last sysfs file: /sys/devices/system/cpu/online > >> >>>>>> <4>CPU 1 > >> >>>>>> <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl > >> >>>>>> ipmi_devintf dell_rbu openafs(P)(U) autof > >> >>>>>> s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding > >> >>>>>> 8021q garp stp llc ipv6 power_meter ac > >> >>>>>> pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg > >> >>>>>> bnx2 lpc_ich mfd_core i7core_edac eda > >> >>>>>> c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix > >> >>>>>> mptsas mptscsih mptbase scsi_transport_s > >> >>>>>> as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] > >> >>>>>> <4> > >> >>>>>> <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- > >> >>>>>> 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M > >> >>>>>> <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] > >> >>>>>> nfs_init_commit+0x1f/0xf0 [nfs] > >> >>>>>> <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 > >> >>>>>> <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 > >> >>>>>> <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 > >> >>>>>> <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 > >> >>>>>> <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 > >> >>>>>> <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 > >> >>>>>> <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > >> >>>>>> <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > >> >>>>>> <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 > >> >>>>>> <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > >> >>>>>> <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > >> >>>>>> <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task > >> >>>>>> ffff88063837c040) > >> >>>>>> <4>Stack: > >> >>>>>> <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 > >> >>>>>> <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 > >> >>>>>> <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 > >> >>>>>> <4>Call Trace: > >> >>>>>> <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 > >> >>>>>> [nfs_layout_nfsv41_files] > >> >>>>>> <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] > >> >>>>>> <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] > >> >>>>>> <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] > >> >>>>>> <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 > >> >>>>>> <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 > >> >>>>>> <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 > >> >>>>>> <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 > >> >>>>>> <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 > >> >>>>>> <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 > >> >>>>>> <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 > >> >>>>>> <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 > >> >>>>>> <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 > >> >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > >> >>>>>> <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 > >> >>>>>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > >> >>>>>> <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 > >> >>>>>> <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 > >> >>>>>> <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 > >> >>>>>> <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 > >> >>>>>> <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 > >> >>>>>> 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c > >> >>>>>> 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 > >> >>>>>> <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > >> >>>>>> <4> RSP <ffff88063988da30> > >> >>>>>> <4>CR2: 00000000dc364913 > >> >>>>>> > >> >>>>>> > >> >>>>>> I have vmcore file as well, so let me know if you need some more information. > >> >>>>>> > >> >>>>> > >> >>>>> Hi Tigran! > >> >>>>> > >> >>>>> Have you tried a recent upstream kernel? IIRC I fixed a seeming similar > >> >>>>> filelayout > >> >>>>> commit issue not too long ago. > >> >>>>> > >> >>>>> The filelayout commit path seems to have been broken for a while - mostly > >> >>>>> because > >> >>>>> all the filelayout servers (that I know of) use stable writes, so that code path > >> >>>>> went > >> >>>>> untested... > >> >>>>> > >> >>>>> -dros > >> >>> > >> >> N???????????????r??????y?????????b???X??????ǧv???^???){.n???+????????????{?????????"??????^n???r?????????z?????????h????????????&?????????G?????????h???(???階???ݢj"?????????m???????????????z???ޖ?????????f?????????h?????????~???m??? > >> > >> -- > >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 13+ messages in thread
* kernel crashes on commit @ 2015-01-20 14:00 Mkrtchyan, Tigran 2015-01-20 14:37 ` Weston Andros Adamson 2015-01-20 16:53 ` Peng Tao 0 siblings, 2 replies; 13+ messages in thread From: Mkrtchyan, Tigran @ 2015-01-20 14:00 UTC (permalink / raw) To: linux-nfs Dear fellows, since we have enabled commit through DS code we permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: <1>BUG: unable to handle kernel paging request at 00000000dc364913 <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] <4>PGD 6393ae067 PUD 0 <4>Oops: 0000 [#1] SMP <4>last sysfs file: /sys/devices/system/cpu/online <4>CPU 1 <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl ipmi_devintf dell_rbu openafs(P)(U) autof s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding 8021q garp stp llc ipv6 power_meter ac pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg bnx2 lpc_ich mfd_core i7core_edac eda c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix mptsas mptscsih mptbase scsi_transport_s as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] <4> <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task ffff88063837c040) <4>Stack: <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 <4>Call Trace: <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 [nfs_layout_nfsv41_files] <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] <4> RSP <ffff88063988da30> <4>CR2: 00000000dc364913 I have vmcore file as well, so let me know if you need some more information. Tigran. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-01-20 14:00 Mkrtchyan, Tigran @ 2015-01-20 14:37 ` Weston Andros Adamson 2015-01-20 16:13 ` Mkrtchyan, Tigran 2015-01-20 19:22 ` Mkrtchyan, Tigran 2015-01-20 16:53 ` Peng Tao 1 sibling, 2 replies; 13+ messages in thread From: Weston Andros Adamson @ 2015-01-20 14:37 UTC (permalink / raw) To: Tigran Mkrtchyan; +Cc: linux-nfs list > On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > > > > Dear fellows, > > since we have enabled commit through DS code we > permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: > > > <1>BUG: unable to handle kernel paging request at 00000000dc364913 > <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > <4>PGD 6393ae067 PUD 0 > <4>Oops: 0000 [#1] SMP > <4>last sysfs file: /sys/devices/system/cpu/online > <4>CPU 1 > <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl ipmi_devintf dell_rbu openafs(P)(U) autof > s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding 8021q garp stp llc ipv6 power_meter ac > pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg bnx2 lpc_ich mfd_core i7core_edac eda > c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix mptsas mptscsih mptbase scsi_transport_s > as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] > <4> > <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M > <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 > <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 > <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 > <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 > <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 > <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 > <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 > <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task ffff88063837c040) > <4>Stack: > <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 > <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 > <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 > <4>Call Trace: > <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 [nfs_layout_nfsv41_files] > <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] > <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] > <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] > <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 > <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 > <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 > <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 > <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 > <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 > <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 > <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 > <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 > <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 > <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 > <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 > <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 > <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 > <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 > <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > <4> RSP <ffff88063988da30> > <4>CR2: 00000000dc364913 > > > I have vmcore file as well, so let me know if you need some more information. > Hi Tigran! Have you tried a recent upstream kernel? IIRC I fixed a seeming similar filelayout commit issue not too long ago. The filelayout commit path seems to have been broken for a while - mostly because all the filelayout servers (that I know of) use stable writes, so that code path went untested... -dros ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-01-20 14:37 ` Weston Andros Adamson @ 2015-01-20 16:13 ` Mkrtchyan, Tigran 2015-01-20 19:22 ` Mkrtchyan, Tigran 1 sibling, 0 replies; 13+ messages in thread From: Mkrtchyan, Tigran @ 2015-01-20 16:13 UTC (permalink / raw) To: Weston Andros Adamson; +Cc: linux-nfs list ----- Original Message ----- > From: "Weston Andros Adamson" <dros@primarydata.com> > To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de> > Cc: "linux-nfs list" <linux-nfs@vger.kernel.org> > Sent: Tuesday, January 20, 2015 3:37:49 PM > Subject: Re: kernel crashes on commit >> On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: >> >> >> >> Dear fellows, >> >> since we have enabled commit through DS code we >> permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: >> >> >> <1>BUG: unable to handle kernel paging request at 00000000dc364913 >> <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >> <4>PGD 6393ae067 PUD 0 >> <4>Oops: 0000 [#1] SMP >> <4>last sysfs file: /sys/devices/system/cpu/online >> <4>CPU 1 >> <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl >> ipmi_devintf dell_rbu openafs(P)(U) autof >> s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding >> 8021q garp stp llc ipv6 power_meter ac >> pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg >> bnx2 lpc_ich mfd_core i7core_edac eda >> c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix >> mptsas mptscsih mptbase scsi_transport_s >> as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] >> <4> >> <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- >> 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M >> <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] >> nfs_init_commit+0x1f/0xf0 [nfs] >> <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 >> <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 >> <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 >> <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 >> <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 >> <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 >> <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 >> <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >> <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 >> <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >> <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task >> ffff88063837c040) >> <4>Stack: >> <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 >> <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 >> <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 >> <4>Call Trace: >> <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 >> [nfs_layout_nfsv41_files] >> <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] >> <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] >> <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] >> <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 >> <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 >> <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 >> <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 >> <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 >> <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 >> <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 >> <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 >> <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 >> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >> <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 >> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >> <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 >> <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 >> <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 >> <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 >> <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 >> 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c >> 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 >> <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >> <4> RSP <ffff88063988da30> >> <4>CR2: 00000000dc364913 >> >> >> I have vmcore file as well, so let me know if you need some more information. >> > > Hi Tigran! Hi Dros, > > Have you tried a recent upstream kernel? IIRC I fixed a seeming similar > filelayout > commit issue not too long ago. In did, looks like my fedora-21 desktop survives massive writes, when other hosts crashing. Now I recall seeing you mail, but can't find it any more. Could you please point to it? > > The filelayout commit path seems to have been broken for a while - mostly > because > all the filelayout servers (that I know of) use stable writes, so that code path > went > untested... We have enabled unstable write in December. Tigran. > > -dros ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-01-20 14:37 ` Weston Andros Adamson 2015-01-20 16:13 ` Mkrtchyan, Tigran @ 2015-01-20 19:22 ` Mkrtchyan, Tigran 2015-01-20 20:42 ` Weston Andros Adamson 1 sibling, 1 reply; 13+ messages in thread From: Mkrtchyan, Tigran @ 2015-01-20 19:22 UTC (permalink / raw) To: Weston Andros Adamson; +Cc: linux-nfs list Hi Dros, do you refer to this commit http://git.linux-nfs.org/?p=dros/linux-nfs.git;a=commit;h=d201c4de518c1d617aa216664869fa329d562d7d ? Tigran. ----- Original Message ----- > From: "Weston Andros Adamson" <dros@primarydata.com> > To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de> > Cc: "linux-nfs list" <linux-nfs@vger.kernel.org> > Sent: Tuesday, January 20, 2015 3:37:49 PM > Subject: Re: kernel crashes on commit >> On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: >> >> >> >> Dear fellows, >> >> since we have enabled commit through DS code we >> permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: >> >> >> <1>BUG: unable to handle kernel paging request at 00000000dc364913 >> <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >> <4>PGD 6393ae067 PUD 0 >> <4>Oops: 0000 [#1] SMP >> <4>last sysfs file: /sys/devices/system/cpu/online >> <4>CPU 1 >> <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl >> ipmi_devintf dell_rbu openafs(P)(U) autof >> s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding >> 8021q garp stp llc ipv6 power_meter ac >> pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg >> bnx2 lpc_ich mfd_core i7core_edac eda >> c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix >> mptsas mptscsih mptbase scsi_transport_s >> as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] >> <4> >> <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- >> 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M >> <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] >> nfs_init_commit+0x1f/0xf0 [nfs] >> <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 >> <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 >> <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 >> <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 >> <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 >> <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 >> <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 >> <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >> <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 >> <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >> <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task >> ffff88063837c040) >> <4>Stack: >> <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 >> <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 >> <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 >> <4>Call Trace: >> <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 >> [nfs_layout_nfsv41_files] >> <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] >> <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] >> <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] >> <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 >> <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 >> <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 >> <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 >> <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 >> <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 >> <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 >> <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 >> <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 >> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >> <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 >> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >> <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 >> <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 >> <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 >> <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 >> <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 >> 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c >> 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 >> <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >> <4> RSP <ffff88063988da30> >> <4>CR2: 00000000dc364913 >> >> >> I have vmcore file as well, so let me know if you need some more information. >> > > Hi Tigran! > > Have you tried a recent upstream kernel? IIRC I fixed a seeming similar > filelayout > commit issue not too long ago. > > The filelayout commit path seems to have been broken for a while - mostly > because > all the filelayout servers (that I know of) use stable writes, so that code path > went > untested... > > -dros ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-01-20 19:22 ` Mkrtchyan, Tigran @ 2015-01-20 20:42 ` Weston Andros Adamson 0 siblings, 0 replies; 13+ messages in thread From: Weston Andros Adamson @ 2015-01-20 20:42 UTC (permalink / raw) To: Tigran Mkrtchyan; +Cc: linux-nfs list > On Jan 20, 2015, at 2:22 PM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > > Hi Dros, > > do you refer to this commit > > http://git.linux-nfs.org/?p=dros/linux-nfs.git;a=commit;h=d201c4de518c1d617aa216664869fa329d562d7d ? Yes, that’s the patch I was talking about. Good find, I was about to go looking for it. Is that patch in the kernels you’re testing? -dros > ----- Original Message ----- >> From: "Weston Andros Adamson" <dros@primarydata.com> >> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de> >> Cc: "linux-nfs list" <linux-nfs@vger.kernel.org> >> Sent: Tuesday, January 20, 2015 3:37:49 PM >> Subject: Re: kernel crashes on commit > >>> On Jan 20, 2015, at 9:00 AM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: >>> >>> >>> >>> Dear fellows, >>> >>> since we have enabled commit through DS code we >>> permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: >>> >>> >>> <1>BUG: unable to handle kernel paging request at 00000000dc364913 >>> <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >>> <4>PGD 6393ae067 PUD 0 >>> <4>Oops: 0000 [#1] SMP >>> <4>last sysfs file: /sys/devices/system/cpu/online >>> <4>CPU 1 >>> <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl >>> ipmi_devintf dell_rbu openafs(P)(U) autof >>> s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding >>> 8021q garp stp llc ipv6 power_meter ac >>> pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg >>> bnx2 lpc_ich mfd_core i7core_edac eda >>> c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix >>> mptsas mptscsih mptbase scsi_transport_s >>> as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] >>> <4> >>> <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- >>> 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M >>> <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] >>> nfs_init_commit+0x1f/0xf0 [nfs] >>> <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 >>> <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 >>> <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 >>> <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 >>> <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 >>> <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 >>> <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 >>> <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >>> <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 >>> <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>> <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>> <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task >>> ffff88063837c040) >>> <4>Stack: >>> <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 >>> <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 >>> <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 >>> <4>Call Trace: >>> <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 >>> [nfs_layout_nfsv41_files] >>> <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] >>> <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] >>> <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] >>> <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 >>> <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 >>> <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 >>> <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 >>> <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 >>> <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 >>> <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 >>> <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 >>> <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 >>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >>> <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 >>> <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 >>> <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 >>> <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 >>> <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 >>> <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 >>> <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 >>> 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c >>> 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 >>> <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] >>> <4> RSP <ffff88063988da30> >>> <4>CR2: 00000000dc364913 >>> >>> >>> I have vmcore file as well, so let me know if you need some more information. >>> >> >> Hi Tigran! >> >> Have you tried a recent upstream kernel? IIRC I fixed a seeming similar >> filelayout >> commit issue not too long ago. >> >> The filelayout commit path seems to have been broken for a while - mostly >> because >> all the filelayout servers (that I know of) use stable writes, so that code path >> went >> untested... >> >> -dros ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: kernel crashes on commit 2015-01-20 14:00 Mkrtchyan, Tigran 2015-01-20 14:37 ` Weston Andros Adamson @ 2015-01-20 16:53 ` Peng Tao 1 sibling, 0 replies; 13+ messages in thread From: Peng Tao @ 2015-01-20 16:53 UTC (permalink / raw) To: Mkrtchyan, Tigran; +Cc: Linux NFS Mailing List On Tue, Jan 20, 2015 at 10:00 PM, Mkrtchyan, Tigran <tigran.mkrtchyan@desy.de> wrote: > > > Dear fellows, > > since we have enabled commit through DS code we > permanently observe kernel crashes with RHEL6/7 and ubuntu 14.04: > Hi Tigran, I fixed an issue for flexfiles layout driver that although client is supposed to have only one RW segment, it is possible to have multiple segments when one attached to layout header others unlinked due to layoutreturn/layoutrecall. If we don't check for this before freeing commit buckets, client might crash when accessing ds commit info as there is still a valid lseg. I'm not sure if it is the same issue but it looks similar. Please see ff_layout_free_lseg() in Tom's patchset (the 49th patch) where I take inode->i_lock and check for existing RW layouts before freeing commit info buckets. Cheers, Tao > > <1>BUG: unable to handle kernel paging request at 00000000dc364913 > <1>IP: [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > <4>PGD 6393ae067 PUD 0 > <4>Oops: 0000 [#1] SMP > <4>last sysfs file: /sys/devices/system/cpu/online > <4>CPU 1 > <4>Modules linked in: vfat fat usb_storage mpt3sas mpt2sas raid_class mptctl ipmi_devintf dell_rbu openafs(P)(U) autof > s4 nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding 8021q garp stp llc ipv6 power_meter ac > pi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support microcode dcdbas sg bnx2 lpc_ich mfd_core i7core_edac eda > c_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi pata_acpi ata_generic ata_piix mptsas mptscsih mptbase scsi_transport_s > as dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] > <4> > <4>Pid: 18209, comm: flush-0:19 Tainted: P --------------- 2.6.32-504.3.3.el6.x86_64 #1 Dell Inc. PowerEdge M610/0N582M > <4>RIP: 0010:[<ffffffffa02b45df>] [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > <4>RSP: 0018:ffff88063988da30 EFLAGS: 00010246 > <4>RAX: ffff88063988db60 RBX: ffff88009c492040 RCX: ffff88063988db30 > <4>RDX: 0000000000000000 RSI: ffff88063988db60 RDI: 00000000dc364903 > <4>RBP: ffff88063988da40 R08: ffff88063988da90 R09: f9aa37faa254d404 > <4>R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000001 > <4>R13: ffff880339f33a00 R14: ffff88063988db30 R15: ffff88063988d8c8 > <4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > <4>CR2: 00000000dc364913 CR3: 0000000639fbb000 CR4: 00000000000007e0 > <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > <4>Process flush-0:19 (pid: 18209, threadinfo ffff88063988c000, task ffff88063837c040) > <4>Stack: > <4> 0000000000000000 ffff88009c492040 ffff88063988dad0 ffffffffa031fdb7 > <4><d> ffff88063837c5f8 ffff88063988da90 ffff8800a6e34600 ffff880337f2a950 > <4><d> ffff880637c99488 0000000037f2a940 ffff88063988db60 0000000000000000 > <4>Call Trace: > <4> [<ffffffffa031fdb7>] filelayout_commit_pagelist+0x277/0x3c0 [nfs_layout_nfsv41_files] > <4> [<ffffffffa02b613b>] nfs_generic_commit_list+0xab/0x100 [nfs] > <4> [<ffffffffa02b627c>] nfs_commit_inode+0xec/0x150 [nfs] > <4> [<ffffffffa02b6aab>] nfs_write_inode+0xab/0x100 [nfs] > <4> [<ffffffff811baedc>] writeback_single_inode+0x20c/0x290 > <4> [<ffffffff811bb1ad>] writeback_sb_inodes+0xbd/0x170 > <4> [<ffffffff811bb30b>] writeback_inodes_wb+0xab/0x1b0 > <4> [<ffffffff811bb703>] wb_writeback+0x2f3/0x410 > <4> [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 > <4> [<ffffffff81088062>] ? del_timer_sync+0x22/0x30 > <4> [<ffffffff811bb9c5>] wb_do_writeback+0x1a5/0x240 > <4> [<ffffffff811bbac3>] bdi_writeback_task+0x63/0x1b0 > <4> [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0 > <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > <4> [<ffffffff811483e6>] bdi_start_fn+0x86/0x100 > <4> [<ffffffff81148360>] ? bdi_start_fn+0x0/0x100 > <4> [<ffffffff8109e66e>] kthread+0x9e/0xc0 > <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 > <4> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0 > <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 > <4>Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 00 00 48 8b 06 48 89 fb 48 8b 78 18 48 39 c6 48 8b 7f 40 <48> 8b 7f 10 74 2b 4c 8b 83 c8 01 00 00 4c 8b 4e 08 4c 8d 93 c8 > <1>RIP [<ffffffffa02b45df>] nfs_init_commit+0x1f/0xf0 [nfs] > <4> RSP <ffff88063988da30> > <4>CR2: 00000000dc364913 > > > I have vmcore file as well, so let me know if you need some more information. > > Tigran. > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2015-06-26 12:33 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-01-20 21:45 kernel crashes on commit Mkrtchyan, Tigran 2015-01-21 10:04 ` Mkrtchyan, Tigran 2015-01-21 15:20 ` Weston Andros Adamson 2015-03-26 9:28 ` Benjamin Coddington 2015-03-26 10:11 ` Mkrtchyan, Tigran 2015-03-26 12:00 ` Benjamin Coddington 2015-06-26 12:33 ` Benjamin Coddington -- strict thread matches above, loose matches on Subject: below -- 2015-01-20 14:00 Mkrtchyan, Tigran 2015-01-20 14:37 ` Weston Andros Adamson 2015-01-20 16:13 ` Mkrtchyan, Tigran 2015-01-20 19:22 ` Mkrtchyan, Tigran 2015-01-20 20:42 ` Weston Andros Adamson 2015-01-20 16:53 ` Peng Tao
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.