From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E23DC433E3 for ; Mon, 27 Jul 2020 11:05:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2D7702072E for ; Mon, 27 Jul 2020 11:05:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2D7702072E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 872FC6B0002; Mon, 27 Jul 2020 07:05:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 823356B0003; Mon, 27 Jul 2020 07:05:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C3DA6B0005; Mon, 27 Jul 2020 07:05:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0192.hostedemail.com [216.40.44.192]) by kanga.kvack.org (Postfix) with ESMTP id 523336B0002 for ; Mon, 27 Jul 2020 07:05:21 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 006511EE6 for ; Mon, 27 Jul 2020 11:05:20 +0000 (UTC) X-FDA: 77083574442.24.cows55_570fba726f60 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id C38351A4A7 for ; Mon, 27 Jul 2020 11:05:20 +0000 (UTC) X-HE-Tag: cows55_570fba726f60 X-Filterd-Recvd-Size: 4139 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Mon, 27 Jul 2020 11:05:20 +0000 (UTC) Received: from gaia (unknown [95.146.230.158]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0FB99205CB; Mon, 27 Jul 2020 11:05:16 +0000 (UTC) Date: Mon, 27 Jul 2020 12:05:14 +0100 From: Catalin Marinas To: Yu Xu Cc: Linus Torvalds , Andrew Morton , Johannes Weiner , Hillf Danton , Hugh Dickins , Josef Bacik , "Kirill A . Shutemov" , Linux-MM , mm-commits@vger.kernel.org, Will Deacon , Matthew Wilcox , yang.shi@linux.alibaba.com Subject: Re: [patch 01/15] mm/memory.c: avoid access flag update TLB flush for retried page fault Message-ID: <20200727110512.GB25400@gaia> References: <20200723211432.b31831a0df3bc2cbdae31b40@linux-foundation.org> <20200724041508.QlTbrHnfh%akpm@linux-foundation.org> <0323de82-cfbd-8506-fa9c-a702703dd654@linux.alibaba.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0323de82-cfbd-8506-fa9c-a702703dd654@linux.alibaba.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Queue-Id: C38351A4A7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jul 27, 2020 at 03:31:16PM +0800, Yu Xu wrote: > On 7/25/20 4:22 AM, Linus Torvalds wrote: > > On Fri, Jul 24, 2020 at 12:27 PM Linus Torvalds > > wrote: > > > > > > It *may* make sense to say "ok, don't bother flushing the TLB if this > > > is a retry, because we already did that originally". MAYBE. [...] > > We could say that we never need it at all for FAULT_FLAG_RETRY. That > > makes a lot of sense to me. > > > > So a patch that does something like the appended (intentionally > > whitespace-damaged) seems sensible. > > I tested your patch on our aarch64 box, with 128 online CPUs. [...] > There are two points to sum up. > > 1) the performance of page_fault3_process is restored, while the performance > of page_fault3_thread is about ~80% of the vanilla, except the case of 128 > threads. > > 2) in the case of 128 threads, test worker threads seem to get stuck, making > no progress in the iterations of mmap-write-munmap until a period of time > later. the test result is 0 because only first 16 samples are counted, and > they are all 0. This situation is easy to re-produce with large number of > threads (not necessarily 128), and the stack of one stuck thread is shown > below. > > [<0>] __switch_to+0xdc/0x150 > [<0>] wb_wait_for_completion+0x84/0xb0 > [<0>] __writeback_inodes_sb_nr+0x9c/0xe8 > [<0>] try_to_writeback_inodes_sb+0x6c/0x88 > [<0>] ext4_nonda_switch+0x90/0x98 [ext4] > [<0>] ext4_page_mkwrite+0x248/0x4c0 [ext4] > [<0>] do_page_mkwrite+0x4c/0x100 > [<0>] do_fault+0x2ac/0x3e0 > [<0>] handle_pte_fault+0xb4/0x258 > [<0>] __handle_mm_fault+0x1d8/0x3a8 > [<0>] handle_mm_fault+0x104/0x1d0 > [<0>] do_page_fault+0x16c/0x490 > [<0>] do_translation_fault+0x60/0x68 > [<0>] do_mem_abort+0x58/0x100 > [<0>] el0_da+0x24/0x28 > [<0>] 0xffffffffffffffff > > It seems quite normal, right? and I've run out of ideas. If threads get stuck here, it could be a stale TLB entry that's not flushed with Linus' patch. Since that's a write fault, I think it hits the FAULT_FLAG_TRIED case. Could you give my patch here a try as an alternative: https://lore.kernel.org/linux-mm/20200725155841.GA14490@gaia/ It leaves the spurious flush in place but only local (though note that in a guest under KVM, all local TLBIs are upgraded to inner-shareable, so you'd not get the performance benefit). -- Catalin