From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756565AbdJJTog (ORCPT ); Tue, 10 Oct 2017 15:44:36 -0400 Received: from mail.skyhub.de ([5.9.137.197]:59922 "EHLO mail.skyhub.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756531AbdJJTof (ORCPT ); Tue, 10 Oct 2017 15:44:35 -0400 Date: Tue, 10 Oct 2017 21:44:26 +0200 From: Borislav Petkov To: Jeremy Cline , Tony Luck Cc: Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , x86@kernel.org, linux-edac@vger.kernel.org, linux-kernel@vger.kernel.org, Laura Abbott Subject: Re: x86/mce: suspicious RCU usage in 4.13.4 Message-ID: <20171010194426.s7keveirclglx6vh@pd.tnic> References: <9840bf0d-6756-75eb-1a2d-d3aace235244@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <9840bf0d-6756-75eb-1a2d-d3aace235244@redhat.com> User-Agent: NeoMutt/20170113 (1.7.2) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Oct 10, 2017 at 03:00:09PM -0400, Jeremy Cline wrote: > Hello, > > A Fedora user has reported an issue about suspicious RCU usage in > dev-mcelog. It looks like perhaps the notifier call chain is not > acquiring the mce_chrdev_read_mutex? The traceback is > > [36915.633804] ============================= > [36915.633805] WARNING: suspicious RCU usage > [36915.633808] 4.13.4-301.fc27.x86_64+debug #1 Not tainted > [36915.633809] ----------------------------- > [36915.633811] arch/x86/kernel/cpu/mcheck/dev-mcelog.c:60 suspicious > mce_log_get_idx_check() usage! > [36915.633812] > other info that might help us debug this: > > [36915.633813] > rcu_scheduler_active = 2, debug_locks = 1 > [36915.633815] 3 locks held by kworker/1:2/14637: > [36915.633816] #0: ("events"){.+.+.+}, at: [] > process_one_work+0x1d0/0x6a0 > [36915.633827] #1: ((&mce_work)){+.+...}, at: [] > process_one_work+0x1d0/0x6a0 > [36915.633833] #2: ((x86_mce_decoder_chain).rwsem){++++..}, at: > [] blocking_notifier_call_chain+0x2f/0x70 > [36915.633840] > stack backtrace: > [36915.633843] CPU: 1 PID: 14637 Comm: kworker/1:2 Not tainted > 4.13.4-301.fc27.x86_64+debug #1 > [36915.633844] Hardware name: Gigabyte Technology Co., Ltd. > Z87M-D3H/Z87M-D3H, BIOS F11 08/12/2014 > [36915.633847] Workqueue: events mce_gen_pool_process > [36915.633849] Call Trace: > [36915.633854] dump_stack+0x8e/0xd6 > [36915.633858] lockdep_rcu_suspicious+0xc5/0x100 > [36915.633862] dev_mce_log+0xf6/0x1e0 > [36915.633865] notifier_call_chain+0x39/0x90 > [36915.633869] blocking_notifier_call_chain+0x49/0x70 > [36915.633873] mce_gen_pool_process+0x41/0x70 Right, so dev_mce_log() is called in process context now and thus can be greatly simplified by removing all those memory barriers and cmpxchg() fun which was for atomic context back then. And simply grab the mutex instead. IOW, something like this totally untested hunk. Tony? --- diff --git a/arch/x86/kernel/cpu/mcheck/dev-mcelog.c b/arch/x86/kernel/cpu/mcheck/dev-mcelog.c index 10cec43aac38..1dacebb6a23b 100644 --- a/arch/x86/kernel/cpu/mcheck/dev-mcelog.c +++ b/arch/x86/kernel/cpu/mcheck/dev-mcelog.c @@ -53,9 +53,10 @@ static int dev_mce_log(struct notifier_block *nb, unsigned long val, void *data) { struct mce *mce = (struct mce *)data; - unsigned int next, entry; + unsigned int entry; + + mutex_lock(&mce_chrdev_read_mutex); - wmb(); for (;;) { entry = mce_log_get_idx_check(mcelog.next); for (;;) { @@ -66,10 +67,10 @@ static int dev_mce_log(struct notifier_block *nb, unsigned long val, * interesting ones: */ if (entry >= MCE_LOG_LEN) { - set_bit(MCE_OVERFLOW, - (unsigned long *)&mcelog.flags); + set_bit(MCE_OVERFLOW, (unsigned long *)&mcelog.flags); return NOTIFY_OK; } + /* Old left over entry. Skip: */ if (mcelog.entry[entry].finished) { entry++; @@ -77,15 +78,13 @@ static int dev_mce_log(struct notifier_block *nb, unsigned long val, } break; } - smp_rmb(); - next = entry + 1; - if (cmpxchg(&mcelog.next, entry, next) == entry) - break; + mcelog.next = entry + 1; } + memcpy(mcelog.entry + entry, mce, sizeof(struct mce)); - wmb(); mcelog.entry[entry].finished = 1; - wmb(); + + mutex_unlock(&mce_chrdev_read_mutex); /* wake processes polling /dev/mcelog */ wake_up_interruptible(&mce_chrdev_wait); -- Regards/Gruss, Boris. Good mailing practices for 400: avoid top-posting and trim the reply. From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Subject: x86/mce: suspicious RCU usage in 4.13.4 From: Borislav Petkov Message-Id: <20171010194426.s7keveirclglx6vh@pd.tnic> Date: Tue, 10 Oct 2017 21:44:26 +0200 To: Jeremy Cline , Tony Luck Cc: Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , x86@kernel.org, linux-edac@vger.kernel.org, linux-kernel@vger.kernel.org, Laura Abbott List-ID: T24gVHVlLCBPY3QgMTAsIDIwMTcgYXQgMDM6MDA6MDlQTSAtMDQwMCwgSmVyZW15IENsaW5lIHdy b3RlOgo+IEhlbGxvLAo+IAo+IEEgRmVkb3JhIHVzZXIgaGFzIHJlcG9ydGVkIGFuIGlzc3VlIGFi b3V0IHN1c3BpY2lvdXMgUkNVIHVzYWdlIGluCj4gZGV2LW1jZWxvZy4gSXQgbG9va3MgbGlrZSBw ZXJoYXBzIHRoZSBub3RpZmllciBjYWxsIGNoYWluIGlzIG5vdAo+IGFjcXVpcmluZyB0aGUgbWNl X2NocmRldl9yZWFkX211dGV4PyBUaGUgdHJhY2ViYWNrIGlzCj4gCj4gWzM2OTE1LjYzMzgwNF0g PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KPiBbMzY5MTUuNjMzODA1XSBXQVJOSU5HOiBz dXNwaWNpb3VzIFJDVSB1c2FnZQo+IFszNjkxNS42MzM4MDhdIDQuMTMuNC0zMDEuZmMyNy54ODZf NjQrZGVidWcgIzEgTm90IHRhaW50ZWQKPiBbMzY5MTUuNjMzODA5XSAtLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLQo+IFszNjkxNS42MzM4MTFdIGFyY2gveDg2L2tlcm5lbC9jcHUvbWNoZWNr L2Rldi1tY2Vsb2cuYzo2MCBzdXNwaWNpb3VzCj4gbWNlX2xvZ19nZXRfaWR4X2NoZWNrKCkgdXNh Z2UhCj4gWzM2OTE1LjYzMzgxMl0KPiAgICAgICAgICAgICAgICBvdGhlciBpbmZvIHRoYXQgbWln aHQgaGVscCB1cyBkZWJ1ZyB0aGlzOgo+IAo+IFszNjkxNS42MzM4MTNdCj4gICAgICAgICAgICAg ICAgcmN1X3NjaGVkdWxlcl9hY3RpdmUgPSAyLCBkZWJ1Z19sb2NrcyA9IDEKPiBbMzY5MTUuNjMz ODE1XSAzIGxvY2tzIGhlbGQgYnkga3dvcmtlci8xOjIvMTQ2Mzc6Cj4gWzM2OTE1LjYzMzgxNl0g ICMwOiAgKCJldmVudHMiKXsuKy4rLit9LCBhdDogWzxmZmZmZmZmZmFhMGQyYWMwPl0KPiBwcm9j ZXNzX29uZV93b3JrKzB4MWQwLzB4NmEwCj4gWzM2OTE1LjYzMzgyN10gICMxOiAgKCgmbWNlX3dv cmspKXsrLisuLi59LCBhdDogWzxmZmZmZmZmZmFhMGQyYWMwPl0KPiBwcm9jZXNzX29uZV93b3Jr KzB4MWQwLzB4NmEwCj4gWzM2OTE1LjYzMzgzM10gICMyOiAgKCh4ODZfbWNlX2RlY29kZXJfY2hh aW4pLnJ3c2VtKXsrKysrLi59LCBhdDoKPiBbPGZmZmZmZmZmYWEwZGM5MmY+XSBibG9ja2luZ19u b3RpZmllcl9jYWxsX2NoYWluKzB4MmYvMHg3MAo+IFszNjkxNS42MzM4NDBdCj4gICAgICAgICAg ICAgICAgc3RhY2sgYmFja3RyYWNlOgo+IFszNjkxNS42MzM4NDNdIENQVTogMSBQSUQ6IDE0NjM3 IENvbW06IGt3b3JrZXIvMToyIE5vdCB0YWludGVkCj4gNC4xMy40LTMwMS5mYzI3Lng4Nl82NCtk ZWJ1ZyAjMQo+IFszNjkxNS42MzM4NDRdIEhhcmR3YXJlIG5hbWU6IEdpZ2FieXRlIFRlY2hub2xv Z3kgQ28uLCBMdGQuCj4gWjg3TS1EM0gvWjg3TS1EM0gsIEJJT1MgRjExIDA4LzEyLzIwMTQKPiBb MzY5MTUuNjMzODQ3XSBXb3JrcXVldWU6IGV2ZW50cyBtY2VfZ2VuX3Bvb2xfcHJvY2Vzcwo+IFsz NjkxNS42MzM4NDldIENhbGwgVHJhY2U6Cj4gWzM2OTE1LjYzMzg1NF0gIGR1bXBfc3RhY2srMHg4 ZS8weGQ2Cj4gWzM2OTE1LjYzMzg1OF0gIGxvY2tkZXBfcmN1X3N1c3BpY2lvdXMrMHhjNS8weDEw MAo+IFszNjkxNS42MzM4NjJdICBkZXZfbWNlX2xvZysweGY2LzB4MWUwCj4gWzM2OTE1LjYzMzg2 NV0gIG5vdGlmaWVyX2NhbGxfY2hhaW4rMHgzOS8weDkwCj4gWzM2OTE1LjYzMzg2OV0gIGJsb2Nr aW5nX25vdGlmaWVyX2NhbGxfY2hhaW4rMHg0OS8weDcwCj4gWzM2OTE1LjYzMzg3M10gIG1jZV9n ZW5fcG9vbF9wcm9jZXNzKzB4NDEvMHg3MAoKUmlnaHQsIHNvIGRldl9tY2VfbG9nKCkgaXMgY2Fs bGVkIGluIHByb2Nlc3MgY29udGV4dCBub3cgYW5kIHRodXMgY2FuIGJlCmdyZWF0bHkgc2ltcGxp ZmllZCBieSByZW1vdmluZyBhbGwgdGhvc2UgbWVtb3J5IGJhcnJpZXJzIGFuZCBjbXB4Y2hnKCkK ZnVuIHdoaWNoIHdhcyBmb3IgYXRvbWljIGNvbnRleHQgYmFjayB0aGVuLiBBbmQgc2ltcGx5IGdy YWIgdGhlIG11dGV4Cmluc3RlYWQuCgpJT1csIHNvbWV0aGluZyBsaWtlIHRoaXMgdG90YWxseSB1 bnRlc3RlZCBodW5rLiBUb255PwoKZGlmZiAtLWdpdCBhL2FyY2gveDg2L2tlcm5lbC9jcHUvbWNo ZWNrL2Rldi1tY2Vsb2cuYyBiL2FyY2gveDg2L2tlcm5lbC9jcHUvbWNoZWNrL2Rldi1tY2Vsb2cu YwppbmRleCAxMGNlYzQzYWFjMzguLjFkYWNlYmI2YTIzYiAxMDA2NDQKLS0tIGEvYXJjaC94ODYv a2VybmVsL2NwdS9tY2hlY2svZGV2LW1jZWxvZy5jCisrKyBiL2FyY2gveDg2L2tlcm5lbC9jcHUv bWNoZWNrL2Rldi1tY2Vsb2cuYwpAQCAtNTMsOSArNTMsMTAgQEAgc3RhdGljIGludCBkZXZfbWNl X2xvZyhzdHJ1Y3Qgbm90aWZpZXJfYmxvY2sgKm5iLCB1bnNpZ25lZCBsb25nIHZhbCwKIAkJCQl2 b2lkICpkYXRhKQogewogCXN0cnVjdCBtY2UgKm1jZSA9IChzdHJ1Y3QgbWNlICopZGF0YTsKLQl1 bnNpZ25lZCBpbnQgbmV4dCwgZW50cnk7CisJdW5zaWduZWQgaW50IGVudHJ5OworCisJbXV0ZXhf bG9jaygmbWNlX2NocmRldl9yZWFkX211dGV4KTsKIAotCXdtYigpOwogCWZvciAoOzspIHsKIAkJ ZW50cnkgPSBtY2VfbG9nX2dldF9pZHhfY2hlY2sobWNlbG9nLm5leHQpOwogCQlmb3IgKDs7KSB7 CkBAIC02NiwxMCArNjcsMTAgQEAgc3RhdGljIGludCBkZXZfbWNlX2xvZyhzdHJ1Y3Qgbm90aWZp ZXJfYmxvY2sgKm5iLCB1bnNpZ25lZCBsb25nIHZhbCwKIAkJCSAqIGludGVyZXN0aW5nIG9uZXM6 CiAJCQkgKi8KIAkJCWlmIChlbnRyeSA+PSBNQ0VfTE9HX0xFTikgewotCQkJCXNldF9iaXQoTUNF X09WRVJGTE9XLAotCQkJCQkodW5zaWduZWQgbG9uZyAqKSZtY2Vsb2cuZmxhZ3MpOworCQkJCXNl dF9iaXQoTUNFX09WRVJGTE9XLCAodW5zaWduZWQgbG9uZyAqKSZtY2Vsb2cuZmxhZ3MpOwogCQkJ CXJldHVybiBOT1RJRllfT0s7CiAJCQl9CisKIAkJCS8qIE9sZCBsZWZ0IG92ZXIgZW50cnkuIFNr aXA6ICovCiAJCQlpZiAobWNlbG9nLmVudHJ5W2VudHJ5XS5maW5pc2hlZCkgewogCQkJCWVudHJ5 Kys7CkBAIC03NywxNSArNzgsMTMgQEAgc3RhdGljIGludCBkZXZfbWNlX2xvZyhzdHJ1Y3Qgbm90 aWZpZXJfYmxvY2sgKm5iLCB1bnNpZ25lZCBsb25nIHZhbCwKIAkJCX0KIAkJCWJyZWFrOwogCQl9 Ci0JCXNtcF9ybWIoKTsKLQkJbmV4dCA9IGVudHJ5ICsgMTsKLQkJaWYgKGNtcHhjaGcoJm1jZWxv Zy5uZXh0LCBlbnRyeSwgbmV4dCkgPT0gZW50cnkpCi0JCQlicmVhazsKKwkJbWNlbG9nLm5leHQg PSBlbnRyeSArIDE7CiAJfQorCiAJbWVtY3B5KG1jZWxvZy5lbnRyeSArIGVudHJ5LCBtY2UsIHNp emVvZihzdHJ1Y3QgbWNlKSk7Ci0Jd21iKCk7CiAJbWNlbG9nLmVudHJ5W2VudHJ5XS5maW5pc2hl ZCA9IDE7Ci0Jd21iKCk7CisKKwltdXRleF91bmxvY2soJm1jZV9jaHJkZXZfcmVhZF9tdXRleCk7 CiAKIAkvKiB3YWtlIHByb2Nlc3NlcyBwb2xsaW5nIC9kZXYvbWNlbG9nICovCiAJd2FrZV91cF9p bnRlcnJ1cHRpYmxlKCZtY2VfY2hyZGV2X3dhaXQpOwo=