From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0D0FC3F2D1 for ; Thu, 5 Mar 2020 00:27:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 654E120870 for ; Thu, 5 Mar 2020 00:27:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="cnKRrjDF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 654E120870 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 513856B0007; Wed, 4 Mar 2020 19:27:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 406FB6B000C; Wed, 4 Mar 2020 19:27:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07F136B0008; Wed, 4 Mar 2020 19:27:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0115.hostedemail.com [216.40.44.115]) by kanga.kvack.org (Postfix) with ESMTP id D5C8A6B0007 for ; Wed, 4 Mar 2020 19:27:40 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 803BF4DBE for ; Thu, 5 Mar 2020 00:27:40 +0000 (UTC) X-FDA: 76559420280.15.stamp77_8b7a4385b622 X-HE-Tag: stamp77_8b7a4385b622 X-Filterd-Recvd-Size: 10586 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Mar 2020 00:27:39 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0250NFaM110949; Thu, 5 Mar 2020 00:27:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=mime-version : message-id : date : from : to : cc : subject : references : in-reply-to : content-type : content-transfer-encoding; s=corp-2020-01-29; bh=hBxzAlCC7cKYY6O2ccn7Bt/YrNwCFCMbLn0fbwBhh8A=; b=cnKRrjDFfNsYfaC4U+Y5Kl4hzX29asxhe6bR9PD8gJy2oFlX5Fk6zsan2gB/TFDLZhNv gXh8lz06T+WTwUMKUlKIoXs5Xa0KmrRTrfNGzhj77NTAf293CMU4uevPTK+p6idcCwLy 4hf7pe5TkdETLLC+myU+MXWV0mkkzMwTVNRfxHwEN+7fEtrHMk8Xt3XA+SLbCqrqjKR9 +hIUav06NLFUurdrc2Ma4G2UiPTwxnqU2eRtIVN+V+uStyYzkRvNz6riQgV47K7PXb9r 1PoO9k0vsfe6WqiCGYjVioQA03trDEfZHwn2vEUVohwK5aTGCbdyhdGJTSda5UCzGNUx vw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 2yghn3dk69-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 05 Mar 2020 00:27:33 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0250GZup062975; Thu, 5 Mar 2020 00:27:33 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userp3020.oracle.com with ESMTP id 2yg1p8wer2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 05 Mar 2020 00:27:32 +0000 Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 0250RV8O025605; Thu, 5 Mar 2020 00:27:31 GMT Received: from monkey.oracle.com (/71.63.128.209) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 04 Mar 2020 16:27:00 -0800 MIME-Version: 1.0 Message-ID: <20200305002650.160855-3-mike.kravetz@oracle.com> Date: Wed, 4 Mar 2020 16:26:50 -0800 (PST) From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Hugh Dickins , Naoya Horiguchi , "Aneesh Kumar K . V" , Andrea Arcangeli , "Kirill A . Shutemov" , Davidlohr Bueso , Prakash Sangappa , Andrew Morton , Mike Kravetz Subject: [PATCH 2/2] hugetlbfs: Use i_mmap_rwsem to address page fault/truncate race References: <20200305002650.160855-1-mike.kravetz@oracle.com> In-Reply-To: <20200305002650.160855-1-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.24.1 Content-Type: text/plain; charset=ascii X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9550 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 mlxlogscore=999 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2003050000 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9550 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 phishscore=0 spamscore=0 impostorscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 lowpriorityscore=0 priorityscore=1501 bulkscore=0 clxscore=1015 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2003050000 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: hugetlbfs page faults can race with truncate and hole punch operations. Current code in the page fault path attempts to handle this by 'backing out' operations if we encounter the race. One obvious omission in the current code is removing a page newly added to the page cache. This is pretty straight forward to address, but there is a more subtle and difficult issue of backing out hugetlb reservations. To handle this correctly, the 'reservation state' before page allocation needs to be noted so that it can be properly backed out. There are four distinct possibilities for reservation state: shared/reserved, shared/no-resv, private/reserved and private/no-resv. Backing out a reservation may require memory allocation which could fail so that needs to be taken into account as well. Instead of writing the required complicated code for this rare occurrence, just eliminate the race. i_mmap_rwsem is now held in read mode for the duration of page fault processing. Hold i_mmap_rwsem in write mode when modifying i_size. In this way, truncation can not proceed when page faults are being processed. In addition, i_size will not change during fault processing so a single check can be made to ensure faults are not beyond (proposed) end of file. Faults can still race with hole punch, but that race is handled by existing code and the use of hugetlb_fault_mutex. With this modification, checks for races with truncation in the page fault path can be simplified and removed. remove_inode_hugepages no longer needs to take hugetlb_fault_mutex in the case of truncation. Comments are expanded to explain reasoning behind locking. Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 32 ++++++++++++++++++++++---------- mm/hugetlb.c | 23 +++++++++++------------ 2 files changed, 33 insertions(+), 22 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index ce9d354ea5c2..863bae2393d2 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -393,10 +393,9 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, p= goff_t start, pgoff_t end) * In this case, we first scan the range and release found pages. * After releasing pages, hugetlb_unreserve_pages cleans up region/reser= v * maps and global counts. Page faults can not race with truncation - * in this routine. hugetlb_no_page() prevents page faults in the - * truncated range. It checks i_size before allocation, and again after - * with the page table lock for the page held. The same lock must be - * acquired to unmap a page. + * in this routine. hugetlb_no_page() holds i_mmap_rwsem and prevents + * page faults in the truncated range by checking i_size. i_size is + * modified while holding i_mmap_rwsem. * hole punch is indicated if end is not LLONG_MAX * In the hole punch case we scan the range and release found pages. * Only when releasing a page is the associated region/reserv map @@ -434,9 +433,17 @@ static void remove_inode_hugepages(struct inode *ino= de, loff_t lstart, struct page *page =3D pvec.pages[i]; u32 hash; =20 - index =3D page->index; - hash =3D hugetlb_fault_mutex_hash(mapping, index); - mutex_lock(&hugetlb_fault_mutex_table[hash]); + if (!truncate_op) { + /* + * Only need to hold the fault mutex in the + * hole punch case. This prevents races with + * page faults. Races are not possible in the + * case of truncation. + */ + index =3D page->index; + hash =3D hugetlb_fault_mutex_hash(mapping, index); + mutex_lock(&hugetlb_fault_mutex_table[hash]); + } =20 /* * If page is mapped, it was faulted in after being @@ -479,7 +486,8 @@ static void remove_inode_hugepages(struct inode *inod= e, loff_t lstart, } =20 unlock_page(page); - mutex_unlock(&hugetlb_fault_mutex_table[hash]); + if (!truncate_op) + mutex_unlock(&hugetlb_fault_mutex_table[hash]); } huge_pagevec_release(&pvec); cond_resched(); @@ -517,8 +525,8 @@ static int hugetlb_vmtruncate(struct inode *inode, lo= ff_t offset) BUG_ON(offset & ~huge_page_mask(h)); pgoff =3D offset >> PAGE_SHIFT; =20 - i_size_write(inode, offset); i_mmap_lock_write(mapping); + i_size_write(inode, offset); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0); i_mmap_unlock_write(mapping); @@ -640,7 +648,11 @@ static long hugetlbfs_fallocate(struct file *file, i= nt mode, loff_t offset, /* addr is the offset within the file (zero based) */ addr =3D index * hpage_size; =20 - /* mutex taken here, fault path and hole punch */ + /* + * fault mutex taken here, protects against fault path + * and hole punch. inode_lock previously taken protects + * against truncation. + */ hash =3D hugetlb_fault_mutex_hash(mapping, index); mutex_lock(&hugetlb_fault_mutex_table[hash]); =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b6156eac926d..71dcdbe2d32d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4172,16 +4172,17 @@ static vm_fault_t hugetlb_no_page(struct mm_struc= t *mm, } =20 /* - * Use page lock to guard against racing truncation - * before we get page_table_lock. + * We can not race with truncation due to holding i_mmap_rwsem. + * i_size is modified when holding i_mmap_rwsem, so check here + * once for faults beyond end of file. */ + size =3D i_size_read(mapping->host) >> huge_page_shift(h); + if (idx >=3D size) + goto out; + retry: page =3D find_lock_page(mapping, idx); if (!page) { - size =3D i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >=3D size) - goto out; - /* * Check for page in userfault range */ @@ -4287,10 +4288,6 @@ static vm_fault_t hugetlb_no_page(struct mm_struct= *mm, } =20 ptl =3D huge_pte_lock(h, mm, ptep); - size =3D i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >=3D size) - goto backout; - ret =3D 0; if (!huge_pte_none(huge_ptep_get(ptep))) goto backout; @@ -4394,8 +4391,10 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, str= uct vm_area_struct *vma, =20 /* * Acquire i_mmap_rwsem before calling huge_pte_alloc and hold - * until finished with ptep. This prevents huge_pmd_unshare from - * being called elsewhere and making the ptep no longer valid. + * until finished with ptep. This serves two purposes: + * 1) It prevents huge_pmd_unshare from being called elsewhere + * and making the ptep no longer valid. + * 2) It synchronizes us with i_size modifications during truncation. * * ptep could have already be assigned via huge_pte_offset. That * is OK, as huge_pte_alloc will return the same value unless --=20 2.24.1