From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F66EC433FE for ; Tue, 28 Dec 2021 14:39:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233857AbhL1Oj0 (ORCPT ); Tue, 28 Dec 2021 09:39:26 -0500 Received: from mail-ua1-f47.google.com ([209.85.222.47]:35469 "EHLO mail-ua1-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233497AbhL1Oj0 (ORCPT ); Tue, 28 Dec 2021 09:39:26 -0500 Received: by mail-ua1-f47.google.com with SMTP id v14so14799276uau.2; Tue, 28 Dec 2021 06:39:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=kpKebKhTPkAf5g+7wJecDZs+7FWqimgm1MM7GRPt3OU=; b=BYPeeVGBeI05WcUXpkB5nZzUWMDVpOeLgiPQciBMWfzDm0rME8mdY8A5JFrvVdl69I Gkbw641ogVTLr+C/jHw/8qc2JFJXj08/s3keuj17SJq08L7UNe+HM2FOKsXKR9Z9HCoC wJxPQjVzBRrhZUGzsIhEIHwcyGF0WKHAGSEDd1QZ6LUUqVaMOVSvjTrmkT9S2SRALCFx QCC+0Pru7A2Dbhsb7l/ITaTJmzK4pOtFrZwoW331hdfPgrCZnui57GgIBR+64PLgeqhg jGZmj6BNBe/8soTN0jRhGDVdwIlu/lr2tBaS4QRLNVT0+CaJGbfToquCU/yW7CWQmFaU 594A== X-Gm-Message-State: AOAM5317bsiTRGKvdfCKSajgvjC31EI2yMx5AzGuZBR2jh+9VG3CjHJt 8maaKVr55EyC/blVITOWxQlgidaLdSP58w== X-Google-Smtp-Source: ABdhPJxCk+nr+FHmcvl/qBgSlh3lHEc5LdIGNQBeuCphSvhWynkoZDCbv06JTchYCeUjU8a50XYxAQ== X-Received: by 2002:a67:b143:: with SMTP id z3mr6313194vsl.66.1640702365077; Tue, 28 Dec 2021 06:39:25 -0800 (PST) Received: from mail-ua1-f52.google.com (mail-ua1-f52.google.com. [209.85.222.52]) by smtp.gmail.com with ESMTPSA id x19sm3788139vsh.6.2021.12.28.06.39.24 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Dec 2021 06:39:24 -0800 (PST) Received: by mail-ua1-f52.google.com with SMTP id i6so32172900uae.6; Tue, 28 Dec 2021 06:39:24 -0800 (PST) X-Received: by 2002:a05:6102:21dc:: with SMTP id r28mr6366856vsg.57.1640702364070; Tue, 28 Dec 2021 06:39:24 -0800 (PST) MIME-Version: 1.0 References: <20211122092816.2865873-1-ardb@kernel.org> <20211122092816.2865873-8-ardb@kernel.org> <0ffc858f-27e7-6789-4be1-c4c5ad61eb9d@samsung.com> In-Reply-To: From: Geert Uytterhoeven Date: Tue, 28 Dec 2021 15:39:12 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v4 7/7] ARM: implement support for vmap'ed stacks To: Ard Biesheuvel Cc: Marek Szyprowski , Linux ARM , Russell King , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers , Tony Lindgren , Krzysztof Kozlowski , Linux Samsung SOC , Linux-Renesas Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-renesas-soc@vger.kernel.org Hi Ard, On Thu, Dec 23, 2021 at 3:30 PM Ard Biesheuvel wrote: > On Tue, 21 Dec 2021 at 22:56, Marek Szyprowski wrote: > > On 21.12.2021 17:20, Ard Biesheuvel wrote: > > > On Tue, 21 Dec 2021 at 14:51, Marek Szyprowski wrote: > > >> On 21.12.2021 14:34, Ard Biesheuvel wrote: > > >>> On Tue, 21 Dec 2021 at 12:15, Marek Szyprowski wrote: > > >>>> On 21.12.2021 11:44, Ard Biesheuvel wrote: > > >>>>> On Tue, 21 Dec 2021 at 11:39, Marek Szyprowski wrote: > > >>>>>> On 22.11.2021 10:28, Ard Biesheuvel wrote: > > >>>>>>> Wire up the generic support for managing task stack allocations via vmalloc, > > >>>>>>> and implement the entry code that detects whether we faulted because of a > > >>>>>>> stack overrun (or future stack overrun caused by pushing the pt_regs array) > > >>>>>>> > > >>>>>>> While this adds a fair amount of tricky entry asm code, it should be > > >>>>>>> noted that it only adds a TST + branch to the svc_entry path. The code > > >>>>>>> implementing the non-trivial handling of the overflow stack is emitted > > >>>>>>> out-of-line into the .text section. > > >>>>>>> > > >>>>>>> Since on ARM, we rely on do_translation_fault() to keep PMD level page > > >>>>>>> table entries that cover the vmalloc region up to date, we need to > > >>>>>>> ensure that we don't hit such a stale PMD entry when accessing the > > >>>>>>> stack. So we do a dummy read from the new stack while still running from > > >>>>>>> the old one on the context switch path, and bump the vmalloc_seq counter > > >>>>>>> when PMD level entries in the vmalloc range are modified, so that the MM > > >>>>>>> switch fetches the latest version of the entries. > > >>>>>>> > > >>>>>>> Note that we need to increase the per-mode stack by 1 word, to gain some > > >>>>>>> space to stash a GPR until we know it is safe to touch the stack. > > >>>>>>> However, due to the cacheline alignment of the struct, this does not > > >>>>>>> actually increase the memory footprint of the struct stack array at all. > > >>>>>>> > > >>>>>>> Signed-off-by: Ard Biesheuvel > > >>>>>>> Tested-by: Keith Packard > > >>>>>> This patch landed recently in linux-next 20211220 as commit a1c510d0adc6 > > >>>>>> ("ARM: implement support for vmap'ed stacks"). Sadly it breaks > > >>>>>> suspend/resume operation on all ARM 32bit Exynos SoCs. Probably the > > >>>>>> suspend/resume related code must be updated somehow (it partially works > > >>>>>> on physical addresses and disabled MMU), but I didn't analyze it yet. If > > >>>>>> you have any hints, let me know. > > >>>>>> > > >>>>> Are there any such systems in KernelCI? We caught a suspend/resume > > >>>>> related issue in development, which is why the hunk below was added. > > >>>> I think that some Exynos-based Odroids (U3 and XU3) were some time ago > > >>>> available in KernelCI, but I don't know if they are still there. > > >>>> > > >>>> > > >>>>> In general, any virt-to-phys translation involving and address on the > > >>>>> stack will become problematic. > > >>>>> > > >>>>> Could you please confirm whether the issue persists with the patch > > >>>>> applied but with CONFIG_VMAP_STACK turned off? Just so we know we are > > >>>>> looking in the right place? > > >>>> I've just checked. After disabling CONFIG_VMAP_STACK suspend/resume > > >>>> works fine both on commit a1c510d0adc6 and linux-next 20211220. > > >>>> > > >>> Thanks. Any other context you can provide beyond 'does not work' ? > > >> Well, the board properly suspends, but it doesn't wake then (tested > > >> remotely with rtcwake command). So far I cannot provide anything more. > > >> > > > Thanks. Does the below help? Or otherwise, could you try doubling the > > > size of the overflow stack at arch/arm/include/asm/thread_info.h:34? > > > > I've tried both (but not at the same time) on the current linux-next and > > none helped. This must be something else... :/ > > > > Thanks. > > As i don't have access to this hardware, I am going to have to rely on > someone who does to debug this further. The only alternative is > marking CONFIG_VMAP_STACK broken on MACH_EXYNOS but that would be > unfortunate. Wish I had seen this thread before... I've just bisected a resume after s2ram failure on R-Car Gen2 to the same commit a1c510d0adc604bb ("ARM: implement support for vmap'ed stacks") in arm/for-next. Expected output: PM: suspend entry (deep) Filesystems sync: 0.000 seconds Freezing user space processes ... (elapsed 0.010 seconds) done. OOM killer disabled. Freezing remaining freezable tasks ... (elapsed 0.009 seconds) done. Disabling non-boot CPUs ... [system suspended, this is also where it hangs on failure] Enabling non-boot CPUs ... CPU1 is up sh-eth ee700000.ethernet eth0: Link is Down Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=193) OOM killer enabled. Restarting tasks ... done. PM: suspend exit Both wake-on-LAN and wake-up by gpio-keys fail. Nothing interesting in the kernel log, cfr. above. Disabling CONFIG_VMAP_STACK fixes the issue for me. Just like arch/arm/mach-exynos/ (and others), arch/arm/mach-shmobile/ has several *.S files related to secondary CPU bringup. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE315C433F5 for ; Tue, 28 Dec 2021 14:41:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QPq5ZXDH1KlJfeRAC926x1fUsIj3s6fqdA2cDS+i0vw=; b=PCkfie1A3Y4r71 1nDvAC0Fc+qoSAEhHK4Swk4XS9H8tw9TA4Dw0XaBogX867ZgTSCe5TMo5Lq8/x7CpPYkHf1+iI7WX LCgC50JtnCAOdVGpeEp3CHhazDU3c+ufzxus7ynGGtsPcUhvslicnz4Soko2EZGa0b7Clgxqgimqp IcfnrM5Ojlau/OkeLf+idh01Lt8IvtwfDujdpXj0Xgp7MXT+5815Lt1YGhznzCxB8o/ZvKHgrreQ0 vq4fNMvfoksYniBnf25m7TvtMuf8FISxlPJaOwIrOi6gjr8kGi0UxBeFVdqSW8XK1e1c24mccn7O/ 7aK3FxUAXzGNmx6MqLpg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1n2Ddp-001BPG-3r; Tue, 28 Dec 2021 14:39:53 +0000 Received: from mail-ua1-f48.google.com ([209.85.222.48]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1n2DdP-001BFU-77 for linux-arm-kernel@lists.infradead.org; Tue, 28 Dec 2021 14:39:29 +0000 Received: by mail-ua1-f48.google.com with SMTP id u6so25052256uaq.0 for ; Tue, 28 Dec 2021 06:39:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=kpKebKhTPkAf5g+7wJecDZs+7FWqimgm1MM7GRPt3OU=; b=DX68etsbv2M0qEvx40jRSSGOT9TZFlsxcAehrc80gLpijWQVVhnB6l5Oq5o/9Peod3 0AhyulNE8PmAiOI7elLmj7K3oPLtn4oa9jK3am3a62RzlQwULjcq2ile21v95UlBevS+ IDo02BBvlyfjhYWuAGnf/nUEK6xTmpXD2uNDExqx4QP8bdWB76WQfeL9J+l2tDTQvoCL qYT6dAu3exUkEmDJQrGweqyy6wNU3AeL/HOV5n5MOumVFAqAvEXJQXNDNT78PgKPegQx bZbuYzkWrOY3BUkpnI/8oZ3YjGosMjFeoDN/LbcvHRRc4yZjuvwO3fOEwGQ+YKam9r8S KqUw== X-Gm-Message-State: AOAM5306xQMFq7DQo7+3zo0gCJmdcn+dtUM+jpOpgy76XEGC7G3b9Lx4 F1TtDWapxSanK3oxVZAliMXsAaCmBWSHiQ== X-Google-Smtp-Source: ABdhPJyq7Z3NKgWhJWwPcuwVo0nfRRVFzxJsUdcj3jbNqoFw2p0gsf7+pMiOxqi1H4sqF1VcMsZuDw== X-Received: by 2002:a67:f287:: with SMTP id m7mr6000030vsk.9.1640702364850; Tue, 28 Dec 2021 06:39:24 -0800 (PST) Received: from mail-ua1-f47.google.com (mail-ua1-f47.google.com. [209.85.222.47]) by smtp.gmail.com with ESMTPSA id p46sm3783674uad.16.2021.12.28.06.39.24 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Dec 2021 06:39:24 -0800 (PST) Received: by mail-ua1-f47.google.com with SMTP id p1so10355591uap.9 for ; Tue, 28 Dec 2021 06:39:24 -0800 (PST) X-Received: by 2002:a05:6102:21dc:: with SMTP id r28mr6366856vsg.57.1640702364070; Tue, 28 Dec 2021 06:39:24 -0800 (PST) MIME-Version: 1.0 References: <20211122092816.2865873-1-ardb@kernel.org> <20211122092816.2865873-8-ardb@kernel.org> <0ffc858f-27e7-6789-4be1-c4c5ad61eb9d@samsung.com> In-Reply-To: From: Geert Uytterhoeven Date: Tue, 28 Dec 2021 15:39:12 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v4 7/7] ARM: implement support for vmap'ed stacks To: Ard Biesheuvel Cc: Marek Szyprowski , Linux ARM , Russell King , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers , Tony Lindgren , Krzysztof Kozlowski , Linux Samsung SOC , Linux-Renesas X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211228_063927_307752_114DE320 X-CRM114-Status: GOOD ( 52.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Ard, On Thu, Dec 23, 2021 at 3:30 PM Ard Biesheuvel wrote: > On Tue, 21 Dec 2021 at 22:56, Marek Szyprowski wrote: > > On 21.12.2021 17:20, Ard Biesheuvel wrote: > > > On Tue, 21 Dec 2021 at 14:51, Marek Szyprowski wrote: > > >> On 21.12.2021 14:34, Ard Biesheuvel wrote: > > >>> On Tue, 21 Dec 2021 at 12:15, Marek Szyprowski wrote: > > >>>> On 21.12.2021 11:44, Ard Biesheuvel wrote: > > >>>>> On Tue, 21 Dec 2021 at 11:39, Marek Szyprowski wrote: > > >>>>>> On 22.11.2021 10:28, Ard Biesheuvel wrote: > > >>>>>>> Wire up the generic support for managing task stack allocations via vmalloc, > > >>>>>>> and implement the entry code that detects whether we faulted because of a > > >>>>>>> stack overrun (or future stack overrun caused by pushing the pt_regs array) > > >>>>>>> > > >>>>>>> While this adds a fair amount of tricky entry asm code, it should be > > >>>>>>> noted that it only adds a TST + branch to the svc_entry path. The code > > >>>>>>> implementing the non-trivial handling of the overflow stack is emitted > > >>>>>>> out-of-line into the .text section. > > >>>>>>> > > >>>>>>> Since on ARM, we rely on do_translation_fault() to keep PMD level page > > >>>>>>> table entries that cover the vmalloc region up to date, we need to > > >>>>>>> ensure that we don't hit such a stale PMD entry when accessing the > > >>>>>>> stack. So we do a dummy read from the new stack while still running from > > >>>>>>> the old one on the context switch path, and bump the vmalloc_seq counter > > >>>>>>> when PMD level entries in the vmalloc range are modified, so that the MM > > >>>>>>> switch fetches the latest version of the entries. > > >>>>>>> > > >>>>>>> Note that we need to increase the per-mode stack by 1 word, to gain some > > >>>>>>> space to stash a GPR until we know it is safe to touch the stack. > > >>>>>>> However, due to the cacheline alignment of the struct, this does not > > >>>>>>> actually increase the memory footprint of the struct stack array at all. > > >>>>>>> > > >>>>>>> Signed-off-by: Ard Biesheuvel > > >>>>>>> Tested-by: Keith Packard > > >>>>>> This patch landed recently in linux-next 20211220 as commit a1c510d0adc6 > > >>>>>> ("ARM: implement support for vmap'ed stacks"). Sadly it breaks > > >>>>>> suspend/resume operation on all ARM 32bit Exynos SoCs. Probably the > > >>>>>> suspend/resume related code must be updated somehow (it partially works > > >>>>>> on physical addresses and disabled MMU), but I didn't analyze it yet. If > > >>>>>> you have any hints, let me know. > > >>>>>> > > >>>>> Are there any such systems in KernelCI? We caught a suspend/resume > > >>>>> related issue in development, which is why the hunk below was added. > > >>>> I think that some Exynos-based Odroids (U3 and XU3) were some time ago > > >>>> available in KernelCI, but I don't know if they are still there. > > >>>> > > >>>> > > >>>>> In general, any virt-to-phys translation involving and address on the > > >>>>> stack will become problematic. > > >>>>> > > >>>>> Could you please confirm whether the issue persists with the patch > > >>>>> applied but with CONFIG_VMAP_STACK turned off? Just so we know we are > > >>>>> looking in the right place? > > >>>> I've just checked. After disabling CONFIG_VMAP_STACK suspend/resume > > >>>> works fine both on commit a1c510d0adc6 and linux-next 20211220. > > >>>> > > >>> Thanks. Any other context you can provide beyond 'does not work' ? > > >> Well, the board properly suspends, but it doesn't wake then (tested > > >> remotely with rtcwake command). So far I cannot provide anything more. > > >> > > > Thanks. Does the below help? Or otherwise, could you try doubling the > > > size of the overflow stack at arch/arm/include/asm/thread_info.h:34? > > > > I've tried both (but not at the same time) on the current linux-next and > > none helped. This must be something else... :/ > > > > Thanks. > > As i don't have access to this hardware, I am going to have to rely on > someone who does to debug this further. The only alternative is > marking CONFIG_VMAP_STACK broken on MACH_EXYNOS but that would be > unfortunate. Wish I had seen this thread before... I've just bisected a resume after s2ram failure on R-Car Gen2 to the same commit a1c510d0adc604bb ("ARM: implement support for vmap'ed stacks") in arm/for-next. Expected output: PM: suspend entry (deep) Filesystems sync: 0.000 seconds Freezing user space processes ... (elapsed 0.010 seconds) done. OOM killer disabled. Freezing remaining freezable tasks ... (elapsed 0.009 seconds) done. Disabling non-boot CPUs ... [system suspended, this is also where it hangs on failure] Enabling non-boot CPUs ... CPU1 is up sh-eth ee700000.ethernet eth0: Link is Down Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=193) OOM killer enabled. Restarting tasks ... done. PM: suspend exit Both wake-on-LAN and wake-up by gpio-keys fail. Nothing interesting in the kernel log, cfr. above. Disabling CONFIG_VMAP_STACK fixes the issue for me. Just like arch/arm/mach-exynos/ (and others), arch/arm/mach-shmobile/ has several *.S files related to secondary CPU bringup. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel