From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FB86C433DB for ; Wed, 17 Mar 2021 08:32:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5489064FAC for ; Wed, 17 Mar 2021 08:32:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229622AbhCQIcW (ORCPT ); Wed, 17 Mar 2021 04:32:22 -0400 Received: from mout.kundenserver.de ([212.227.126.135]:56791 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229648AbhCQIcH (ORCPT ); Wed, 17 Mar 2021 04:32:07 -0400 Received: from mail-ot1-f53.google.com ([209.85.210.53]) by mrelayeu.kundenserver.de (mreue011 [213.165.67.97]) with ESMTPSA (Nemesis) id 1MJmX3-1l6pci1RrF-00K80u for ; Wed, 17 Mar 2021 09:32:05 +0100 Received: by mail-ot1-f53.google.com with SMTP id k14-20020a9d7dce0000b02901b866632f29so1030631otn.1 for ; Wed, 17 Mar 2021 01:32:05 -0700 (PDT) X-Gm-Message-State: AOAM532Q6SzNH3I5kYqlAVuVU7XvrKDjuddDznZEw18xNmKP1trDNB/0 MnP6JHH8snmrgXvSz2LECjfz5V1Z3vtgnvuolp8= X-Google-Smtp-Source: ABdhPJw0WED6GA+Cp8O4H4XC5cwO0STfMpzH/SMQfAXNc1kpYycFbPPDhLRQrHynegxHzApNDtrTXWPIyV31wwgqbdU= X-Received: by 2002:a05:6830:148c:: with SMTP id s12mr2447135otq.251.1615969924131; Wed, 17 Mar 2021 01:32:04 -0700 (PDT) MIME-Version: 1.0 References: <00000000000069802205bda22b7f@google.com> <20210316155102.GP1463@shell.armlinux.org.uk> In-Reply-To: From: Arnd Bergmann Date: Wed, 17 Mar 2021 09:31:47 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [syzbot] kernel panic: corrupted stack end in openat To: Dmitry Vyukov Cc: Russell King - ARM Linux admin , syzbot , Linus Walleij , Linux ARM , Andrew Morton , LKML , Linux-MM , syzkaller-bugs , =?UTF-8?Q?Uwe_Kleine=2DK=C3=B6nig?= Content-Type: text/plain; charset="UTF-8" X-Provags-ID: V03:K1:7yY7oMIUOBPFblG5hd/l/OPZ1U+GuravbZTThq0cXP2wARrTdw/ QCfBLHWZ1Ks4KdhaUtwoYkDaofF/qHX3S3HGUTlQVT8d5jGM1k/0EpApG8htbfzERwECy3j Lo1P0ubBtdPnYWvUXGAVl6Ngf83ZgocSDKL6Q/dXiHdS6FAhJKmcQcdcFb9SMIINSApBLYC JfPaUWUeEY4ta36zPE6Kw== X-UI-Out-Filterresults: notjunk:1;V03:K0:bW6bTBeD6v4=:V9mthI9ObXAWWEFYti1NqU mTK8UWF8ftz1Ow/gWA0/yWWTX/XGI9I4Cs2umhdUSFxnOTIPDTfWuu2Xl9vI6ZY3h9X+8R0sy Lwk3o/0yFc7CeJrzZu9BzJMgv0akO0+009ZtD84GMxVgYDxGX/TOBeEvFfZjOvzUa04VDS1N3 ZE9Oogr/GFaexVTuMQzkfpL3wqB1Nf6S2RGVVMG3MYR9pGZwgMeIwhvBf6Aj2GbSZmMMa4qCu OpxTGWVrGq6KFfnHP6Om+ez5UbfplURzol4LEgPUh/Ue8OVRz+odSksp2ASIp4mBpT8KSOsbX 5rZHsJiNMV6NNzb2hvhkDXdvdN9Y3CBsSGCMynJ0bC+TepgISghM05dUZ79lBKGFFfw939x1V 0ENj/Y4ixIh001zaF0fqKdMzH/wONJhaSuumJAJOd04CBMyC2C11GioPrpblnJXB8rzkkfkUk jg6IyXYq8Xfa0auhx9IKaxH0txnoKOZbtoKHpTxu4bRcywCIc1ajbZx7WMOdeZMZr1MEOdpKI mXaLZ4B5D36Z8EdK2w8NW481Ci0cgxZmpmufLUeEt+p/FKj+XBvASeA5qf2dHLcUg== Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Mar 17, 2021 at 8:52 AM Dmitry Vyukov wrote: > On Tue, Mar 16, 2021 at 5:28 PM Arnd Bergmann wrote: > > On Tue, Mar 16, 2021 at 5:13 PM Dmitry Vyukov wrote: > > > On Tue, Mar 16, 2021 at 5:03 PM Arnd Bergmann wrote: > > > > On Tue, Mar 16, 2021 at 4:51 PM Russell King - ARM Linux admin > > > > wrote: > > > > > On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote: > > > > > > On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov wrote: > > > > > > > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6) > > > > > > > > > > > > Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's > > > > > > the closest I have installed, and I think the Debian and Ubuntu versions > > > > > > are generally quite close in case of gcc since they are maintained by > > > > > > the same packagers. > > > > > > > > > > ... which shouldn't be a problem - that's just over 1/4 of the stack > > > > > space. Could it be the syzbot's gcc is doing something weird and > > > > > inflating the stack frames? > > > > > > > > It's possible, I think that's really unlikely given that it's just Debian's > > > > gcc, which is as close to mainline as the version I was using. > > > > > > > > Uwe's DEBUG_STACKOVERFLOW patch from a while ago might > > > > help if this was the problem though: > > > > https://lore.kernel.org/linux-arm-kernel/20200108082913.29710-1-u.kleine-koenig@pengutronix.de/ > > > > > > > > My best guess is something going wrong in the interrupt > > > > that triggered the preempt_schedule() which ended up calling > > > > task_stack_end_corrupted() in schedule_debug(), as you suggested > > > > earlier. > > > > > > FWIW I see slightly larger frames with the config: > > > > > > 073ab64 : > > > 8073ab64: e1a0c00d mov ip, sp > > > 8073ab68: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, > > > fp, ip, lr, pc} > > > 8073ab6c: e24cb004 sub fp, ip, #4 > > > 8073ab70: e24ddfa7 sub sp, sp, #668 ; 0x29c > > > > Yes, this is the one that the compiler complained about when warning > > for stack over 600 bytes. It's not called in this call chain though. > > > > > page_alloc can also do reclaim, I had the impression that reclaim can > > > be quite heavy-weight in all respects. > > > > Yes, that is another possibility. What writable file systems or swap > > do you normally have mounted that it could be writing to, and on > > what storage device? > > The root fs is ext4 on virtio-blk. > > There are also several dozens of shrinkers that can be called during reclaim: > https://elixir.bootlin.com/linux/latest/C/ident/unregister_shrinker Right, unfortunately I don't see a smoking gun there either, unless you are also using NFS or devicemapper. Implementing VMAP_STACK as you suggested earlier is probably the best way to figure out if there is an actual overrun of the stack. Alternatively, adding support for GCC_PLUGIN_STACKLEAK might also help find out if we ever get close to the limit. This is probably less work, but it might not actually help in this case. Arnd From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4F0AC433DB for ; Wed, 17 Mar 2021 08:33:54 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4903B64E76 for ; Wed, 17 Mar 2021 08:33:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4903B64E76 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arndb.de Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=8s9FHmcEHmBmynjj06j9U7Xj40dVctQWMiRLSvgJxxw=; b=dUBoee4WGNl9hbCuK/OEHcJ8c XjuWdmZbL5MZaHA40/Uqi6H5fLZz66SBQZo3tLuTahEkqbS3sqNhbzmvQGsEWwmmW0a9VzobeHwYZ QT3OIhM9zAS0ts4jaINR4sTaRVSB9VuUfnBMXziXMBSn5KdUVlzoj+ITF0b+HwIS/SaAPhZ4cUboS kQNnz+dhlM5aRG/L7o9wgFilr/fZGm1y4T+rhxB/uqTeSM6/aQXgrysp1sy2Z5qQb/yE/B6FzihIb N5lKL0AUH1jLlNxSHDIfSVhvdv4kn9QKec495XkrmIaDhvuCfk7M+fVVDxUngOZK20WEgrkVnKnru PuZUQGVcg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lMRbC-002lYI-32; Wed, 17 Mar 2021 08:32:14 +0000 Received: from mout.kundenserver.de ([212.227.126.131]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lMRb7-002lX2-99 for linux-arm-kernel@lists.infradead.org; Wed, 17 Mar 2021 08:32:11 +0000 Received: from mail-ot1-f54.google.com ([209.85.210.54]) by mrelayeu.kundenserver.de (mreue012 [213.165.67.97]) with ESMTPSA (Nemesis) id 1Mxpqo-1ldOMF3DW2-00zI6g for ; Wed, 17 Mar 2021 09:32:06 +0100 Received: by mail-ot1-f54.google.com with SMTP id h6-20020a0568300346b02901b71a850ab4so1023872ote.6 for ; Wed, 17 Mar 2021 01:32:05 -0700 (PDT) X-Gm-Message-State: AOAM530Lz0VrbZCwrdenF9CNt8XYKXKLBQlJLgDUeV3rmefI2eT7R7IH y4cAPIJfMjgGuGS0nlsBqqcFNUwTw4oIWqEKL/s= X-Google-Smtp-Source: ABdhPJw0WED6GA+Cp8O4H4XC5cwO0STfMpzH/SMQfAXNc1kpYycFbPPDhLRQrHynegxHzApNDtrTXWPIyV31wwgqbdU= X-Received: by 2002:a05:6830:148c:: with SMTP id s12mr2447135otq.251.1615969924131; Wed, 17 Mar 2021 01:32:04 -0700 (PDT) MIME-Version: 1.0 References: <00000000000069802205bda22b7f@google.com> <20210316155102.GP1463@shell.armlinux.org.uk> In-Reply-To: From: Arnd Bergmann Date: Wed, 17 Mar 2021 09:31:47 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [syzbot] kernel panic: corrupted stack end in openat To: Dmitry Vyukov Cc: Russell King - ARM Linux admin , syzbot , Linus Walleij , Linux ARM , Andrew Morton , LKML , Linux-MM , syzkaller-bugs , =?UTF-8?Q?Uwe_Kleine=2DK=C3=B6nig?= X-Provags-ID: V03:K1:OvxwipZgA3zxqc9onwMoGB/L//BM1ez4wWFMw9P3RjwRjCo5qh5 lYMx32htN05H3AUmIhmiDNys2yeqN4V6eNg1jXwZp1IIC2z/tz2oxhLtJDqTahrzavaKFRE 6wSPr4nUyRUSf77EvjtsypIXCc/LLdtmH3hnygkSwD82sS1x8R8470+rxybywNmJgwKmRTr rEqaFpqHqZ5loto6P2b0A== X-UI-Out-Filterresults: notjunk:1;V03:K0:+w/Darl3H4c=:pKpHzN9bm5MLngkTGj51Kl P5F5dcZLk14OVKF+Rws8Ytu4Lc2Omz+ZWJsrqtt81eNjltF3s+ukbRq+dW4ysTWK4iomZI+9R hlKFa5GrnhMmk3ML1fGzgHjfFwWlQgD18LkY+8eCwHrpplumz4Dh1ga/lQg0qkMkrxcWYh/IS mn+M3RIoDZV4SChqPM8O/nciG4yiAA4PHE9tb9ZF4mMS91xJ+C77NqgY8HmA5UxaUIZKm/DLW 48HPIfmvHkh8s+BBFy4qlsvhTtbHIoVx8fuG/GH+pXUrxfKEqF1kKlcmfJUtAK4F9P7vE7PG9 +9UrO6itU+yRIJ19Pji//NSZ+oB09GVA/1imflkvXlMm3OnUKYR++o5aB36fFVAvR42d8DmBS Qv9Wm7IVIVCr61Q8j+ZaSIpnYn01AgQGnGodknmBdfSPmoDZGLVab/QQmt4jvzdmTgFBUjLYm JavTmHMVVaMTbu1hVJA9JgJskF8vDC9vinqf70KOj4gcbP5daza1gKzjL6lCUB5mRjRBwuzsQ EGIWI8uC8aX23IOytnLwe4IzCsoydxS29xIn+kvlyiq1TqFfDIdCGwoLro0Fq3cQA== X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210317_083209_528315_1B865397 X-CRM114-Status: GOOD ( 33.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Mar 17, 2021 at 8:52 AM Dmitry Vyukov wrote: > On Tue, Mar 16, 2021 at 5:28 PM Arnd Bergmann wrote: > > On Tue, Mar 16, 2021 at 5:13 PM Dmitry Vyukov wrote: > > > On Tue, Mar 16, 2021 at 5:03 PM Arnd Bergmann wrote: > > > > On Tue, Mar 16, 2021 at 4:51 PM Russell King - ARM Linux admin > > > > wrote: > > > > > On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote: > > > > > > On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov wrote: > > > > > > > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6) > > > > > > > > > > > > Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's > > > > > > the closest I have installed, and I think the Debian and Ubuntu versions > > > > > > are generally quite close in case of gcc since they are maintained by > > > > > > the same packagers. > > > > > > > > > > ... which shouldn't be a problem - that's just over 1/4 of the stack > > > > > space. Could it be the syzbot's gcc is doing something weird and > > > > > inflating the stack frames? > > > > > > > > It's possible, I think that's really unlikely given that it's just Debian's > > > > gcc, which is as close to mainline as the version I was using. > > > > > > > > Uwe's DEBUG_STACKOVERFLOW patch from a while ago might > > > > help if this was the problem though: > > > > https://lore.kernel.org/linux-arm-kernel/20200108082913.29710-1-u.kleine-koenig@pengutronix.de/ > > > > > > > > My best guess is something going wrong in the interrupt > > > > that triggered the preempt_schedule() which ended up calling > > > > task_stack_end_corrupted() in schedule_debug(), as you suggested > > > > earlier. > > > > > > FWIW I see slightly larger frames with the config: > > > > > > 073ab64 : > > > 8073ab64: e1a0c00d mov ip, sp > > > 8073ab68: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, > > > fp, ip, lr, pc} > > > 8073ab6c: e24cb004 sub fp, ip, #4 > > > 8073ab70: e24ddfa7 sub sp, sp, #668 ; 0x29c > > > > Yes, this is the one that the compiler complained about when warning > > for stack over 600 bytes. It's not called in this call chain though. > > > > > page_alloc can also do reclaim, I had the impression that reclaim can > > > be quite heavy-weight in all respects. > > > > Yes, that is another possibility. What writable file systems or swap > > do you normally have mounted that it could be writing to, and on > > what storage device? > > The root fs is ext4 on virtio-blk. > > There are also several dozens of shrinkers that can be called during reclaim: > https://elixir.bootlin.com/linux/latest/C/ident/unregister_shrinker Right, unfortunately I don't see a smoking gun there either, unless you are also using NFS or devicemapper. Implementing VMAP_STACK as you suggested earlier is probably the best way to figure out if there is an actual overrun of the stack. Alternatively, adding support for GCC_PLUGIN_STACKLEAK might also help find out if we ever get close to the limit. This is probably less work, but it might not actually help in this case. Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel