From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3304AC433F5 for ; Mon, 30 May 2022 15:16:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WuxzYEoCoeIuUTtaIpK21Y0cs2hugT7zbeUGfpM2AHM=; b=cyY7M0iyou/cJM f/5BO6NBDc5cgyPujwuaHJZdLvFncKDlH2bQD9lQjORKpNJ5E4l1v5sTcVFFfs/sbiFGSJ+Ry71Au 4GH8hiHh5egbr0gCc0cMB/inprb/4SLv7OTzHbEFrWwT86UVPySJJsqZKYRtFSMi6rESwgq3H9pXT r2fUIGczSLpcRK8Yie9ZP/GSpnMMvaxEXeel3I72X3O/wYy9MsdO6VIVS/z8ejjIbBDfMc7kIsL1I qCmG6hKfvGv3MIdyA0yo7jNJmX6Ulu2Z8fW4teqgj8cjDbrBbm8FRHwAvRSXlUzc2OWUiZYOmfwY8 1eQOXoQ/a3e7mCKN3/jg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nvh6y-007Sxh-AC; Mon, 30 May 2022 15:15:16 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nvh6v-007SxK-C0 for linux-arm-kernel@lists.infradead.org; Mon, 30 May 2022 15:15:15 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D47D4B80E33 for ; Mon, 30 May 2022 15:15:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 796F4C3411A for ; Mon, 30 May 2022 15:15:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1653923710; bh=u4DYqQJQ6TujLt3iM3oWJlaeTZ+o4bqF2UhuCCFWBfU=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=qDyR2jgQvHr6DG4IXqF+A4ba/CX9aYy/L3ZZ+HMMbjBgxB/q9XHf8U+5gW5pfDoIl Kv6FaoFki72kySATnUD1+cV1VJULsRbgGQF0DMROuvqaP3NGNpGX+OvNk1R+gLbWMz G6XOLmDZkoAFnQlarx3OuXkItvX86V/5sD7Rp8JFSuVORIsrobAxkP1kMTAMDO7wT5 gt/JShLDcJRGUBSu5MJawUDyhsmbXVCD9eMTC9XyfWLjL44Fd+el3HksTo1t7O3T2r QOg18iwNfUAJwSUw+LmwkinjPBtq3HgdjNCro4QE0r0Db7w7mazp1jSl3oEGb6Ioeb cHXtKatGuSOlA== Received: by mail-oi1-f170.google.com with SMTP id h188so7830121oia.2 for ; Mon, 30 May 2022 08:15:10 -0700 (PDT) X-Gm-Message-State: AOAM531ZF88AShk1WiQaAkmL84M0uSQQv/hlft+7fsp+cYWImS+rtmdV qUabyPbQ5GxZLeH1rSaJJ8p7GwTtPK2c5wKgA+E= X-Google-Smtp-Source: ABdhPJy8j1h8bMsLWsG+twABoqm+pDH1jUsw+VGzI7BTRp6Hznm894S54Lu/05c4tEgn9C2gPzWpV32tHvz4iO/UR/c= X-Received: by 2002:a05:6808:f88:b0:32b:d10f:cc6b with SMTP id o8-20020a0568080f8800b0032bd10fcc6bmr8265762oiw.228.1653923709641; Mon, 30 May 2022 08:15:09 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ard Biesheuvel Date: Mon, 30 May 2022 17:14:58 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: am335x: 5.18.x: system stalling To: Arnd Bergmann Cc: Yegor Yefremov , Tony Lindgren , Linux-OMAP , linux-clk , Stephen Boyd , Linux ARM X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220530_081513_731884_C7D3DABB X-CRM114-Status: GOOD ( 31.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gTW9uLCAzMCBNYXkgMjAyMiBhdCAxNTo1NCwgQXJuZCBCZXJnbWFubiA8YXJuZEBhcm5kYi5k ZT4gd3JvdGU6Cj4KPiBPbiBTYXQsIE1heSAyOCwgMjAyMiBhdCA5OjI4IFBNIFllZ29yIFllZnJl bW92Cj4gPHllZ29yc2xpc3RzQGdvb2dsZW1haWwuY29tPiB3cm90ZToKPiA+Cj4gPiBPbiBTYXQs IE1heSAyOCwgMjAyMiBhdCAzOjE0IFBNIEFybmQgQmVyZ21hbm4gPGFybmRAYXJuZGIuZGU+IHdy b3RlOgo+ID4gPgo+ID4gPiBPbiBTYXQsIE1heSAyOCwgMjAyMiBhdCAzOjAxIFBNIFllZ29yIFll ZnJlbW92Cj4gPiA+IDx5ZWdvcnNsaXN0c0Bnb29nbGVtYWlsLmNvbT4gd3JvdGU6Cj4gPiA+ID4g T24gU2F0LCBNYXkgMjgsIDIwMjIgYXQgMTE6MDcgQU0gQXJkIEJpZXNoZXV2ZWwgPGFyZGJAa2Vy bmVsLm9yZz4gd3JvdGU6Cj4gPiA+ID4gSW4gZmlsZSBpbmNsdWRlZCBmcm9tIC4vaW5jbHVkZS9s aW51eC9pcnFmbGFncy5oOjE3LAo+ID4gPiA+ICAgICAgICAgICAgICAgICAgZnJvbSAuL2FyY2gv YXJtL2luY2x1ZGUvYXNtL2JpdG9wcy5oOjI4LAo+ID4gPiA+ICAgICAgICAgICAgICAgICAgZnJv bSAuL2luY2x1ZGUvbGludXgvYml0b3BzLmg6MzMsCj4gPiA+ID4gICAgICAgICAgICAgICAgICBm cm9tIC4vaW5jbHVkZS9saW51eC9sb2cyLmg6MTIsCj4gPiA+ID4gICAgICAgICAgICAgICAgICBm cm9tIGtlcm5lbC9ib3VuZHMuYzoxMzoKPiA+ID4gPiAuL2FyY2gvYXJtL2luY2x1ZGUvYXNtL3Bl cmNwdS5oOiBJbiBmdW5jdGlvbiDigJhfX215X2NwdV9vZmZzZXTigJk6Cj4gPiA+ID4gLi9hcmNo L2FybS9pbmNsdWRlL2FzbS9wZXJjcHUuaDozMjo5OiBlcnJvcjog4oCYX19wZXJfY3B1X29mZnNl dOKAmQo+ID4gPiA+IHVuZGVjbGFyZWQgKGZpcnN0IHVzZSBpbiB0aGlzIGZ1bmN0aW9uKTsgZGlk IHlvdSBtZWFuCj4gPiA+ID4g4oCYX19teV9jcHVfb2Zmc2V04oCZPwo+ID4gPiA+ICAgIDMyIHwg IHJldHVybiBfX3Blcl9jcHVfb2Zmc2V0WzBdOwo+ID4gPiA+ICAgICAgIHwgICAgICAgICBefn5+ fn5+fn5+fn5+fn5+Cj4gPiA+ID4gICAgICAgfCAgICAgICAgIF9fbXlfY3B1X29mZnNldAo+ID4g PiA+IC4vYXJjaC9hcm0vaW5jbHVkZS9hc20vcGVyY3B1Lmg6MzI6OTogbm90ZTogZWFjaCB1bmRl Y2xhcmVkIGlkZW50aWZpZXIKPiA+ID4gPiBpcyByZXBvcnRlZCBvbmx5IG9uY2UgZm9yIGVhY2gg ZnVuY3Rpb24gaXQgYXBwZWFycyBpbgo+ID4gPgo+ID4gPiBJIHRoaW5rIHlvdSBqdXN0IG1pc3Nl ZCB0aGUgbGluZSBpbiBteSBwYXRjaCB0aGF0IGFkZHMgdGhlCj4gPiA+ICJleHRlcm4gdW5zaWdu ZWQgbG9uZyBfX3Blcl9jcHVfb2Zmc2V0W107IiB2YXJpYWJsZSBkZWNsYXJhdGlvbi4KPiA+Cj4g PiBTbywgSSB0cmllZCBib3RoIHZhcmlhbnRzIGFuZCBib3RoIGxlZCB0byBzdGFsbHMuCj4KPiBJ J20gcnVubmluZyBvdXQgb2YgaWRlYXMgaGVyZS4gIEdvaW5nIHRvIGJhY2sgdG8gdGhlIG9yaWdp bmFsIGJpc2VjdGlvbiwKPiBJIHJlYmFzZWQgQXJkJ3MgcGF0Y2hlcyBpbiBhIHdheSB0aGF0IHlv dSBzaG91bGQgYmUgYWJsZSB0byBidWlsZCB0aGUKPiBjb25maWcgZm9yIGVhY2ggcGF0Y2gsIGFu ZCBJIHNwbGl0IHVwIHRoZSAiQVJNOiBpbXBsZW1lbnQKPiBUSFJFQURfSU5GT19JTl9UQVNLIGZv ciB1bmlwcm9jZXNzb3Igc3lzdGVtcyIgY29tbWl0IGluIHlldAo+IGFub3RoZXIgd2F5LCBob3Bp bmcgdG8gZ2V0IHNvbWV0aGluZyBsZWZ0IG92ZXIgdGhhdCBwb2ludHMgdG8gdGhlCj4gYnVnLiBD YW4geW91IHRyeSBiaXNlY3RpbmcgdGhyb3VnaCB0aGUgdG9wIGNvbW1pdHMgb2YKPgo+IGh0dHBz Oi8va2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvc29jL3NvYy5naXQgYW0zMzV4 LXN0YWxsLXRlc3QKPgo+IHN0YXJ0aW5nIG1heWJlIHdpdGggIjUyZDI0MDg3MTc2MCBpcnFjaGlw OiBudmljOiBVc2UKPiBHRU5FUklDX0lSUV9NVUxUSV9IQU5ETEVSIiBhcyB0aGUgcGF0Y2ggdGhh dCBpcyBhbG1vc3QgY2VydGFpbmx5Cj4gZ29pbmcgdG8gYmUgb2s/Cj4KPiBBdCBzb21lIHBvaW50 IEkgZmVhciB3ZSBtYXkgaGF2ZSB0byBnaXZlIHVwIGFuZCBqdXN0IG1hcmsgdGhlIHY2K1NNUAo+ IGNvbmZpZ3VyYXRpb24gYXMgYnJva2VuLCB3aGljaCBpcyBzb21ldGhpbmcgd2UgaGF2ZSBjb25z aWRlcmVkIGluIHRoZQo+IHBhc3QgYnV0IGVuZGVkIHVwIGFsd2F5cyBrZWVwaW5nIGFyb3VuZCBm b3IgdGhlIHB1cnBvc2Ugb2YgdGVzdGluZwo+IG9tYXAycGx1c19kZWZjb25maWcgYW5kIGlteF92 Nl92N19kZWZjb25maWcuIE5vdGUgdGhhdCBvbiBwcm9kdWN0aW9uCj4gc3lzdGVtcyB5b3UgcHJv YmFibHkgZG9uJ3Qgd2FudCB0byB1c2UgdGhhdCBjb25maWcgYW53YXksIGFuZCBzaG91bGQKPiBl aXRoZXIgc3RpY2sgdG8gYSB1bmlwcm9jZXNzb3IgYnVpbGQsIG9yIGRpc2FibGUgdGhlIEFSTXY2 IHN1cHBvcnQuCj4KClllYWgsIEkgYW0gYWxzbyBydW5uaW5nIG91dCBvZiBpZGVhcy4gT25lIHF1 ZXN0aW9uLCB0aG91Z2g6IGRvZXMgdGhlClJDVSBkZXRlY3RlZCBzdGFsbCBhbHdheXMgb2NjdXIg aW4gdGhlIHNhbWUgcGxhY2U/IEkuZS4sIGhvdyBzaW1pbGFyCmFyZSB0aGUgYmFja3RyYWNlcyBv ZiB0aGUgc3RhbGxzIGJldHdlZW4gZGlmZmVyZW50IG9jY3VycmVuY2VzPwpQZXJoYXBzIHdlIGNv dWxkIG5hcnJvdyBkb3duIHdoZXJlIGluIHRoZSBjb2RlIHdlIGFyZSBzdGFsbGluZywgYW5kCmdh aW4gc29tZSBtb3JlIHVuZGVyc3RhbmRpbmcgb2YgdGhlIHJvb3QgY2F1c2UuCgpfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2VybmVsIG1h aWxpbmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xp c3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CA0AC433EF for ; Mon, 30 May 2022 15:50:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236808AbiE3Puz (ORCPT ); Mon, 30 May 2022 11:50:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242845AbiE3PuW (ORCPT ); Mon, 30 May 2022 11:50:22 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 311AC6EC55; Mon, 30 May 2022 08:15:13 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id BBBA1B80DC0; Mon, 30 May 2022 15:15:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6BBC0C385B8; Mon, 30 May 2022 15:15:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1653923710; bh=u4DYqQJQ6TujLt3iM3oWJlaeTZ+o4bqF2UhuCCFWBfU=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=qDyR2jgQvHr6DG4IXqF+A4ba/CX9aYy/L3ZZ+HMMbjBgxB/q9XHf8U+5gW5pfDoIl Kv6FaoFki72kySATnUD1+cV1VJULsRbgGQF0DMROuvqaP3NGNpGX+OvNk1R+gLbWMz G6XOLmDZkoAFnQlarx3OuXkItvX86V/5sD7Rp8JFSuVORIsrobAxkP1kMTAMDO7wT5 gt/JShLDcJRGUBSu5MJawUDyhsmbXVCD9eMTC9XyfWLjL44Fd+el3HksTo1t7O3T2r QOg18iwNfUAJwSUw+LmwkinjPBtq3HgdjNCro4QE0r0Db7w7mazp1jSl3oEGb6Ioeb cHXtKatGuSOlA== Received: by mail-oi1-f173.google.com with SMTP id r206so7106090oib.8; Mon, 30 May 2022 08:15:10 -0700 (PDT) X-Gm-Message-State: AOAM532ybMqNxJyaRLSu9rLAa/pHcvVFw7n/vj7XBOyI0W15EeZELaaj 1uUBv2FTGkBDqTLWZdRvWHeavs4291y3wgoYAbo= X-Google-Smtp-Source: ABdhPJy8j1h8bMsLWsG+twABoqm+pDH1jUsw+VGzI7BTRp6Hznm894S54Lu/05c4tEgn9C2gPzWpV32tHvz4iO/UR/c= X-Received: by 2002:a05:6808:f88:b0:32b:d10f:cc6b with SMTP id o8-20020a0568080f8800b0032bd10fcc6bmr8265762oiw.228.1653923709641; Mon, 30 May 2022 08:15:09 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ard Biesheuvel Date: Mon, 30 May 2022 17:14:58 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: am335x: 5.18.x: system stalling To: Arnd Bergmann Cc: Yegor Yefremov , Tony Lindgren , Linux-OMAP , linux-clk , Stephen Boyd , Linux ARM Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-clk@vger.kernel.org On Mon, 30 May 2022 at 15:54, Arnd Bergmann wrote: > > On Sat, May 28, 2022 at 9:28 PM Yegor Yefremov > wrote: > > > > On Sat, May 28, 2022 at 3:14 PM Arnd Bergmann wrote: > > > > > > On Sat, May 28, 2022 at 3:01 PM Yegor Yefremov > > > wrote: > > > > On Sat, May 28, 2022 at 11:07 AM Ard Biesheuvel w= rote: > > > > In file included from ./include/linux/irqflags.h:17, > > > > from ./arch/arm/include/asm/bitops.h:28, > > > > from ./include/linux/bitops.h:33, > > > > from ./include/linux/log2.h:12, > > > > from kernel/bounds.c:13: > > > > ./arch/arm/include/asm/percpu.h: In function =E2=80=98__my_cpu_offs= et=E2=80=99: > > > > ./arch/arm/include/asm/percpu.h:32:9: error: =E2=80=98__per_cpu_off= set=E2=80=99 > > > > undeclared (first use in this function); did you mean > > > > =E2=80=98__my_cpu_offset=E2=80=99? > > > > 32 | return __per_cpu_offset[0]; > > > > | ^~~~~~~~~~~~~~~~ > > > > | __my_cpu_offset > > > > ./arch/arm/include/asm/percpu.h:32:9: note: each undeclared identif= ier > > > > is reported only once for each function it appears in > > > > > > I think you just missed the line in my patch that adds the > > > "extern unsigned long __per_cpu_offset[];" variable declaration. > > > > So, I tried both variants and both led to stalls. > > I'm running out of ideas here. Going to back to the original bisection, > I rebased Ard's patches in a way that you should be able to build the > config for each patch, and I split up the "ARM: implement > THREAD_INFO_IN_TASK for uniprocessor systems" commit in yet > another way, hoping to get something left over that points to the > bug. Can you try bisecting through the top commits of > > https://kernel.org/pub/scm/linux/kernel/git/soc/soc.git am335x-stall-test > > starting maybe with "52d240871760 irqchip: nvic: Use > GENERIC_IRQ_MULTI_HANDLER" as the patch that is almost certainly > going to be ok? > > At some point I fear we may have to give up and just mark the v6+SMP > configuration as broken, which is something we have considered in the > past but ended up always keeping around for the purpose of testing > omap2plus_defconfig and imx_v6_v7_defconfig. Note that on production > systems you probably don't want to use that config anway, and should > either stick to a uniprocessor build, or disable the ARMv6 support. > Yeah, I am also running out of ideas. One question, though: does the RCU detected stall always occur in the same place? I.e., how similar are the backtraces of the stalls between different occurrences? Perhaps we could narrow down where in the code we are stalling, and gain some more understanding of the root cause.