From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0E3FECAAD3 for ; Thu, 1 Sep 2022 22:27:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 35B9880074; Thu, 1 Sep 2022 18:27:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 30A138000D; Thu, 1 Sep 2022 18:27:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DC5F80074; Thu, 1 Sep 2022 18:27:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0E4508000D for ; Thu, 1 Sep 2022 18:27:12 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D092E16025E for ; Thu, 1 Sep 2022 22:27:11 +0000 (UTC) X-FDA: 79864953462.30.05B71D9 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf27.hostedemail.com (Postfix) with ESMTP id 8F0B440086 for ; Thu, 1 Sep 2022 22:27:11 +0000 (UTC) Received: by mail-pg1-f175.google.com with SMTP id h188so350762pgc.12 for ; Thu, 01 Sep 2022 15:27:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date; bh=Lo7nlr4ngh+9J4kFVHJNG61HSBnAKEXajh28lyOmbuQ=; b=LKowKo28eGkwfw5+o6KUV/8v86qC+HRML8ipAx9y1Tvp6Yn4pjIpRwG6NPISoNGGpL CBuoJzMvq1++2aTlnBrIj5b6mB1NbcBo3kHciadOGPxs42cKkVEypdkq+tyqDgW9zqYn qV/+Hku/vHcqYzDiPbIrzowhYjKuWZDUuFoLJAOKFsNHYHWgQ+evMxseMkT0pX8VsV8S OVsQHtt0lZa4/k4qe90CXJ3BtuFWOcBezvS4Lfbv2sJr878ozn2sndhtfv3rTVA65yPw 2G4trdn5dcmzkCF8i06w0B4ZTJ09bCqz7vHJlthhSeIAs493Z8G71dyWhEZhszpaYetl ug2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date; bh=Lo7nlr4ngh+9J4kFVHJNG61HSBnAKEXajh28lyOmbuQ=; b=8LYsdoORsPtbMAw88LcA3TlbzPx7A/tL1d49N2YHfhGWtuT9O8HJnQtjhvDkWseW+B xawTUDlfmT8EhuUsWUQNHXPgk4u7kiLFRf2gwoR9WwHFEh9tBZDpsmMwDwlRgdmkWCoc F9EOJsV5FrnlwlKLcSphyrxFeDB6qQpVPgz/V5NdHNZD/VxBUoeAFKXwWtlyGEjd167O Eb8Wfg0ypvcpA1Vv/ULr9WZl/bWShm5uktkoU/82VQwKurqfZ/8kqHWclE4eC4R3lMmu 64JZGzYwbO0w3WgYHDDJ5wcjKGhF6uRrriukDUts2WabrHrTcuyMrj8y5M5ilR5+KFDI oe4Q== X-Gm-Message-State: ACgBeo3MXW9CNRrGNKN1hBNdQNyLUdsmz7hWakuIyNB8vZsB+X0ATLrm dWfd0rXnf8zrFvjQmavQ5rc= X-Google-Smtp-Source: AA6agR7+/t+ApeGA/lwoDZbioFjKzHBNd4dPLG/4QJjwOWxP23ckzXzZ6g1CYM+FOF4LbevEGKEK9A== X-Received: by 2002:a05:6a00:2918:b0:536:3967:5d08 with SMTP id cg24-20020a056a00291800b0053639675d08mr33083798pfb.68.1662071230457; Thu, 01 Sep 2022 15:27:10 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id u10-20020a17090341ca00b0016b81679c1fsm68494ple.216.2022.09.01.15.27.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:27:09 -0700 (PDT) From: Yang Shi To: david@redhat.com, peterx@redhat.com, kirill.shutemov@linux.intel.com, jhubbard@nvidia.com, jgg@nvidia.com, hughd@google.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH] mm: gup: fix the fast GUP race against THP collapse Date: Thu, 1 Sep 2022 15:27:07 -0700 Message-Id: <20220901222707.477402-1-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662071231; a=rsa-sha256; cv=none; b=LqrzoCNg34amA8Nc0Frb2vwgnWNgI3Z8/WX7nwI5HSUL2vX08IeF7B5G0zNbYsC9dwMrEH ivTX8yiRR3CcFe4VCEUd76fdix47oy/xbQyEST4B1CoqWMHNlGTijcDMYtvaH+Fagc7Mjc GVs47fB39gt/8+b7h0OA42DuCSoTEWI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=LKowKo28; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of shy828301@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=shy828301@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662071231; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Lo7nlr4ngh+9J4kFVHJNG61HSBnAKEXajh28lyOmbuQ=; b=Nj1Dceh0qomUPWZQgASIajNaB1XhfFdeIcS9JP5aBlWfkKs63kXCPsnMJzpt6CVgzmBY2u 9mgH3C+djFRvum3mhpRhJC9Gnbe3PTGcNnLhc2NFULYlcluGzSqBKCQTOHkJqbUBaZ3P+l 4QmUIfYQ/5qAfVKsXR0pl8u4tTVihNc= X-Rspamd-Server: rspam09 Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=LKowKo28; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of shy828301@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspam-User: X-Stat-Signature: yw9r3ea9foe1na7y3zobt8e5rr7jypza X-Rspamd-Queue-Id: 8F0B440086 X-HE-Tag: 1662071231-510854 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since general RCU GUP fast was introduced in commit 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer sufficient to handle concurrent GUP-fast in all cases, it only handles traditional IPI-based GUP-fast correctly. On architectures that send an IPI broadcast on TLB flush, it works as expected. But on the architectures that do not use IPI to broadcast TLB flush, it may have the below race: CPU A CPU B THP collapse fast GUP gup_pmd_range() <-- see valid pmd gup_pte_range() <-- work on pte pmdp_collapse_flush() <-- clear pmd and flush __collapse_huge_page_isolate() check page pinned <-- before GUP bump refcount pin the page check PTE <-- no change __collapse_huge_page_copy() copy data to huge page ptep_clear() install huge pmd for the huge page return the stale page discard the stale page The race could be fixed by checking whether PMD is changed or not after taking the page pin in fast GUP, just like what it does for PTE. If the PMD is changed it means there may be parallel THP collapse, so GUP should back off. Also update the stale comment about serializing against fast GUP in khugepaged. Fixes: 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()") Signed-off-by: Yang Shi --- mm/gup.c | 30 ++++++++++++++++++++++++------ mm/khugepaged.c | 10 ++++++---- 2 files changed, 30 insertions(+), 10 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index f3fc1f08d90c..4365b2811269 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2380,8 +2380,9 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { struct dev_pagemap *pgmap = NULL; int nr_start = *nr, ret = 0; @@ -2423,7 +2424,23 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap; } - if (unlikely(pte_val(pte) != pte_val(*ptep))) { + /* + * THP collapse conceptually does: + * 1. Clear and flush PMD + * 2. Check the base page refcount + * 3. Copy data to huge page + * 4. Clear PTE + * 5. Discard the base page + * + * So fast GUP may race with THP collapse then pin and + * return an old page since TLB flush is no longer sufficient + * to serialize against fast GUP. + * + * Check PMD, if it is changed just back off since it + * means there may be parallel THP collapse. + */ + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || + unlikely(pte_val(pte) != pte_val(*ptep))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } @@ -2470,8 +2487,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, * get_user_pages_fast_only implementation that can pin pages. Thus it's still * useful to have gup_huge_pmd even if we can't operate on ptes. */ -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { return 0; } @@ -2791,7 +2809,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!gup_huge_pd(__hugepd(pmd_val(pmd)), addr, PMD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) + } else if (!gup_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) return 0; } while (pmdp++, addr = next, addr != end); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2d74cf01f694..518b49095db3 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1049,10 +1049,12 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* - * After this gup_fast can't run anymore. This also removes - * any huge TLB entry from the CPU so we won't allow - * huge and small TLB entries for the same virtual address - * to avoid the risk of CPU bugs in that area. + * This removes any huge TLB entry from the CPU so we won't allow + * huge and small TLB entries for the same virtual address to + * avoid the risk of CPU bugs in that area. + * + * Parallel fast GUP is fine since fast GUP will back off when + * it detects PMD is changed. */ _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl); -- 2.26.3