From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 488EAC433B4 for ; Tue, 11 May 2021 00:33:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A43F161613 for ; Tue, 11 May 2021 00:33:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A43F161613 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 035346B0071; Mon, 10 May 2021 20:33:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F270C6B0072; Mon, 10 May 2021 20:33:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D047E6B0073; Mon, 10 May 2021 20:33:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0200.hostedemail.com [216.40.44.200]) by kanga.kvack.org (Postfix) with ESMTP id ADCF76B0071 for ; Mon, 10 May 2021 20:33:24 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 5A212181AF5FA for ; Tue, 11 May 2021 00:33:24 +0000 (UTC) X-FDA: 78127076328.26.97A23B5 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf13.hostedemail.com (Postfix) with ESMTP id 86952E000125 for ; Tue, 11 May 2021 00:33:08 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 14B0OuNk065927; Tue, 11 May 2021 00:33:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=subject : to : references : from : message-id : date : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2020-01-29; bh=fU6t47fIXngx4+oMa1WhwdKihy/MqUnuC0w490dhpAg=; b=HGt+8wEAQuwDMcbQI0VYra7xtmQ5YohMoQyA6RDTW+P+RsGqi1d3RNxtrGfdYY+TVpwb IqRkZQHEnURWLiwq0AOYOGT8QCMI5146Jrl4K3Lu531eUo6TBQpxnCrKiiEvfJnZye0K MgOSkqrEqamU2C/Vlyc8zEgFX7bi6MD5nk7TWghOK2zXkv8nQykCsH4NSFKCr5veMRIw 6k2tuivScjlwvPMXvRLbhMfaOYexEBr7W1714gLDmsL9Lyl/FAHVMPrc3kPKsw00v7Ly ZvQ/mjGXlMxVMyx5++BXzlRDJvddeLGYEnpPl+4w/7Mv+L+YtJSS0wddyS2EjCXiI3K0 3g== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 38e285c9ar-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 11 May 2021 00:33:20 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 14B0Pn40112233; Tue, 11 May 2021 00:33:20 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2041.outbound.protection.outlook.com [104.47.73.41]) by aserp3030.oracle.com with ESMTP id 38e5pw9m9g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 11 May 2021 00:33:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EPLbPZFqqvzB4P2qas/48YyOvZRzeCh/mReBrsNaAlHkReIWYK6QBJF8kHfhntENvuinzHADs78f95GtQgtBEmgXILX0c8y6TO4WWLNrZ7+U/Dxs5l+PPW8GuujC9zjseg1qlWV9jG5DcNPKJCL5CirJy6fC9f7+SP7KHKPkAzCkUqhTZrBm87YT0qLU9ogksDDuQrwppi/LJyIO9T4K33Sj2bQSckPZ10JJeEP3onbHnTzRqBJ3COnlZ/2F3ehhii1oprslw0cggE8MPuofo1pphwavIt55neoaoINHj7fNijk7VYACc/6NM0CaZtF24lKsYVdN/Qt1xvJCmwzIsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fU6t47fIXngx4+oMa1WhwdKihy/MqUnuC0w490dhpAg=; b=StW36B+qHBTqHFjMJ+kjIDgpY0EV/mPRLMXoyDEZ4v7NZllS+DqqTcJ3p98MpRgyX/xdtSzszmNv45V8ZmmvM40cXz5o733ciskM5Y5ZSscHYwG8Rxe60m6322FZ1KSWjmRoOaMG4FheK/oQymCZLCkF7ldE/fmEBEw0ZOGfE8fRQU6akPSGD9knrMDUl9CsZPmS0Z0p+J5f/Z2vc707K6OmJQakIKO1WCPEC3cR1XbKfaKhSpVi4QElV13C1xSzTZrl9TSeHUZzRanQRT/aha6UFdTQfMRSBr/+QA4IV9AAW0YK1zpF93jtDldpRZKrvtKOmDGiwmJtDn4Zo/KxUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fU6t47fIXngx4+oMa1WhwdKihy/MqUnuC0w490dhpAg=; b=ASS0DlR2+90QznMpTiJbrto5syFVi7CryoLyOB4W5wuYDMUzPYqX2tyf+fPFh5N71vT0SDOHrbY6Ax1zzjvXnOHJREaZj36dmH24AoZ7ZEd3ybid5U6WDDSm8nUHARgzC6KUwTadHXQe62j2UiI5CB+Jiw2cKxH5ERkAI10tS9o= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3685.namprd10.prod.outlook.com (2603:10b6:a03:124::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.24; Tue, 11 May 2021 00:33:17 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::4407:2ff6:c0a:5d90]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::4407:2ff6:c0a:5d90%8]) with mapi id 15.20.4108.031; Tue, 11 May 2021 00:33:17 +0000 Subject: Re: resv_huge_page underflow with userfaultfd test To: Mina Almasry , Linux-MM , Axel Rasmussen , aarcange@redhat.com, peterx@redhat.com References: From: Mike Kravetz Message-ID: Date: Mon, 10 May 2021 17:33:14 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.1.1 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO2PR04CA0145.namprd04.prod.outlook.com (2603:10b6:104::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.2.112] (50.38.35.18) by CO2PR04CA0145.namprd04.prod.outlook.com (2603:10b6:104::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.25 via Frontend Transport; Tue, 11 May 2021 00:33:17 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6a59d61a-3819-40ec-0ffe-08d914145ef8 X-MS-TrafficTypeDiagnostic: BYAPR10MB3685: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BiXcQM4JPT4e5GTvG2YQSWJ9FKfpE273A1hvX2z/fr4ZWOEiWBNXKqblS0FaxOQWhzIkVgUNuYD/+krAXl6q0rWVOx5dOULbvYMn3Wz0KUkBHQyKY/ZiE7hrP0W5uBkZcyBdzd8nPRC2o6OeovZyVvwC4mLvW4ynBE243K9wSszzM325NVYTSRYlpuKcIDRHe+bkkg4kr9I0ocAfgVrRBeoI5dDL12YUiYbhwkXFzqOPWoMvwSaWSuZI2VAJA7JoCFwpCPyV1vXhzKLN47mC2WEqNf8Ja4fxzs1Y2Pbj9inT6HLQoIv5+2to9kTepQPlUoYPH6PxBDAyFzFuUdZQLg3M0jW9222YBlcV1W8VEn2qrRYeXa1BrBchCT7vuq/ztux5sPnCI9L+OlIOu5KCTLPnPfvU9vPx9FG+SlxNO5tKeQfDeo3t7cv35au40CW5Qit4M2KwALTu53F765opiJO6nC9yHF1zSgEzbxooVUaPLXdPcv2G1WdkZwK+SRmSO9tM2zeOKkpTQi9VmsqgtX/pXSAy2LrO2masMyWgGYPP7Y/bE7K4Egl1m123gLE7+9f2mBSjPczIXolJ0jC+UYhl51eYw2EZw+CoAJYfKWFcUbp17G105z+x0G695+LBnDCIc1zjtjvnk/f9mpdQSRPa14UmvhTeN/i8auSKQGlKYf9BvhOFk0f/hvI+CfthR+hhaa0/QIDpnUvq6LHWJMrNHDFFBvhAgXq4m3+PJ5w= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(346002)(39860400002)(396003)(376002)(36756003)(6666004)(478600001)(110136005)(31696002)(2906002)(52116002)(31686004)(16576012)(316002)(83380400001)(8676002)(16526019)(86362001)(66476007)(6486002)(53546011)(66946007)(38100700002)(956004)(66556008)(44832011)(5660300002)(186003)(45080400002)(8936002)(2616005)(26005)(38350700002)(14583001)(43740500002)(45980500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?QkNPTUdIdU05QzMxTGg0bkUyazZtRzN2eXEwYXQ0b0FBWFI1V01UNVRWSDZr?= =?utf-8?B?Rys5UC9oa1VQUWJVd0IwRUF1VGFTd3BDdGRvaytuTGhBcHpHTXkycm5MRlB0?= =?utf-8?B?V3RWOFdYYVh2aWRlcEh5SmhkZk1YSm9INzlGZUUybHNnUHJYOTd0NlRXUkQv?= =?utf-8?B?eWdhN3FhS2V0ZS9JTy8yVkFFWlB2QWxLR25VRW9DdVcvZUlvZUlmZktPWU15?= =?utf-8?B?eEJYUmprbmZDTVBIam0zVWVhSVd2aHlDNVZLa29uYmlVY3pMY2lNb3p4OGdG?= =?utf-8?B?M3U2dVNUQ2M4VEZ1Ui96dUVSV1F1aHBCODdNbjZpdWdQTjBRNlF5T1N0QlVp?= =?utf-8?B?TU9ES3htUmhvWHF6TXFTZSt6YnlTN0YvekJpQnFXL1Fxd0loeEdwUUtneFRz?= =?utf-8?B?NER6WGx1KzI1K2syNCsxNnZQQ3VNcjIySWx5eldhdkc5VHNENTJlUndGT0Qr?= =?utf-8?B?Zk1YK3g3OTdJT2xkTk5FRENOVit4bG5sL09teVFQVTZUOTg1TEJWTXpwcWpS?= =?utf-8?B?emx6aVF1OVVLNmZsaGVSOVptcDk3VjU2amNRM29PZkduWHdHV0RuVUhpVDRJ?= =?utf-8?B?QmNicGlPMUxDZWVQNkYyeXFRV2dHaFBGdCt5MjJRZDZsMjBCQ1RGanJweWtE?= =?utf-8?B?REF5ZkEvNGJJSXFFMTJNMzFQRGdNU29sYlhMOGdvT2pieFVtN0xzdERmaW5U?= =?utf-8?B?ei9oUkpnYjdzb2JFRmVybUZndFZyQmJhb0JrMkxmd3lrOHpCWDZVdkk3cFRR?= =?utf-8?B?NlYzK3diOHNuaHcxME1FNE1qaEJidVVVMFJBQzdzSkJuMDBJTmEreFcrSzJ0?= =?utf-8?B?VW5ndCtpdUJ1U2t0NDlDYXB3eDQyMlZuQUVGODUrUnVESTJORmFoTUVDb093?= =?utf-8?B?R2NKRkhKZGp2alovb25ZcG1sL2pVSWE3NnlNZlVmTWordDRuUGx2NFNXeDBT?= =?utf-8?B?aG96dkxKUjBnK204ZXhGbEt3Q2J5bW56UEMrc2taLzRvaUJZTnlNZGlBbTl2?= =?utf-8?B?K3o4c2R1T0puRGc2b1E4c24zTWhiVDBFcFNEN09yS2g5ZVloL2dvTWVhUHZC?= =?utf-8?B?MVV5UmloRHp0TE5yTVlDRCtDRVF2QlhFbWhOOHZwVWsvblIvK0MxcWl2RGVk?= =?utf-8?B?Y0hBWkVoY0E3Tm5wYlJYSG04RlY4clJvTEJtUHQ4Smd3M1N6SExCNGlzcy9p?= =?utf-8?B?ZXY4T0lSUkVVdUZjNHVyeEdReEdadkg1UU53VXlsa1VNWjN6emN2SHUvd3kr?= =?utf-8?B?WlZGeGJEajRJSExPTDhWalFlSVZLQkl2L21qcVN3SGhGQnZkUWJ4V0d5K01O?= =?utf-8?B?VHZ1V3hBZy9meHhjbDBKRG1JdEN1T3lvWFU3QUtubmpIODJuWmhXbGx1RzZx?= =?utf-8?B?K3E5eVArZjJ4ME12MDMzcFArcFcxZ215QWRZZVl1TW53ZzJkNDdjck9WdDVX?= =?utf-8?B?SXpxamVkTGYzYkhlVGFEMGZjZjBkOHV1YXNGcVhoeHBaZWovMUdOL3pnQ0ZY?= =?utf-8?B?U0FyMDZXS0R6elhTUGg4Y0cwM1NISmVxeDFOSGdjQTN5RWN6ZDhmZDhYUm1D?= =?utf-8?B?UDlOSExLNWdqRnpvenFaSTI5L1NsZGplMUFuUnNBbUpJeml6djhLKy9OUDgx?= =?utf-8?B?enNFR1V4MHdmczA2N291dlFkZTZtcDhEU2tVdi8yaXFtdk1TR0ppMTJZTi9P?= =?utf-8?B?TmtXMDhLd0FSMnR3bkhYYlZKdGF1MjVjYU1SUHJTaUtTU1ZLeUR6NW95Q2hF?= =?utf-8?Q?KA/a/eXIaZH7/nQmrN0m1z6sozReC0Nb4M4vVGQ?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6a59d61a-3819-40ec-0ffe-08d914145ef8 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 May 2021 00:33:17.7670 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vlUD6cZU1OxIozv7bLnD3U6Ps1hfQv/W84akg5jNbhsjH0QsSMsccXkW2pVEz/Y5AJZQu8nBVYGOYKn8+z/0Rg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3685 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9980 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2105110000 X-Proofpoint-GUID: BkSucTuU8oaMcOSKVFSL760jHsJhLy-e X-Proofpoint-ORIG-GUID: BkSucTuU8oaMcOSKVFSL760jHsJhLy-e X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9980 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 lowpriorityscore=0 priorityscore=1501 spamscore=0 clxscore=1015 impostorscore=0 phishscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2105110000 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=HGt+8wEA; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ASS0DlR2; spf=none (imf13.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 156.151.31.86) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 86952E000125 X-Stat-Signature: fyx6edef3wgrk1gt9jxcah1uwbw3pfpn Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=userp2130.oracle.com; client-ip=156.151.31.86 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620693188-803375 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 5/7/21 2:21 PM, Mina Almasry wrote: > Hi folks, > > I ran into a bug that I'm not sure how to solve so I'm wondering if > anyone has suggestions on what the issue could be and how to > investigate. I added the WARN_ON_ONCE() here to catch instances of > resv_huge_pages underflowing: > > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > index 629aa4c2259c..7d763eed650f 100644 > --- a/mm/hugetlb.c > +++ b/mm/hugetlb.c > @@ -1165,7 +1165,21 @@ static struct page > *dequeue_huge_page_vma(struct hstate *h, > page = dequeue_huge_page_nodemask(h, gfp_mask, nid, nodemask); > if (page && !avoid_reserve && vma_has_reserves(vma, chg)) { > SetHPageRestoreReserve(page); > + WARN_ON_ONCE(!h->resv_huge_pages); > h->resv_huge_pages--; > } > > And ran the userfaultfd selftests like so: > > echo 1024 > /proc/sys/vm/nr_hugepages > mkdir -p /mnt/huge > mount -t hugetlbfs none /mnt/huge > ./tools/testings/selftests/vm/userfaultfd hugetlb_shared 1024 200 > /mnt/huge/userfaultfd_test > > And run into this warning indicating this test does discover an underflow: > > [ 11.163403] ------------[ cut here ]------------ > [ 11.163404] WARNING: CPU: 0 PID: 237 at mm/hugetlb.c:1178 > alloc_huge_page+0x558/0x5a0 > [ 11.163413] Modules linked in: > [ 11.163419] CPU: 0 PID: 237 Comm: userfaultfd Not tainted 5.12.0-dbg-DEV #135 > [ 11.163424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS 1.14.0-2 04/01/2014 > [ 11.163429] RIP: 0010:alloc_huge_page+0x558/0x5a0 > [ 11.163432] Code: b0 00 0f 85 3d ff ff ff e9 2a ff ff ff be 01 00 > 00 00 48 89 df e8 18 e7 ff ff 48 f7 d8 4c 89 ef 48 89 c6 e8 da d7 ff > ff eb 8c <0f> 0b 4d 8b 85 c0 00 00 00 e9 95 fd ff ff e8 35 59 84 00 4c > 897 > [ 11.163434] RSP: 0018:ffff94bb0073fc80 EFLAGS: 00010046 > [ 11.163436] RAX: 0000000000000080 RBX: 0000000000000000 RCX: 5fa252c406a76700 > [ 11.163438] RDX: c0000000ffff7fff RSI: 0000000000000004 RDI: > 0000000000017ffd > [ 11.163439] RBP: ffff94bb0073fcf8 R08: 0000000000000000 R09: > ffffffff9813ba70 > [ 11.163440] R10: 00000000ffff7fff R11: 0000000000000000 R12: > ffff8ac7800558c8 > [ 11.163442] R13: ffffffff993f8880 R14: 00007f0dfa200000 R15: > ffffed85453e0000 > [ 11.163443] FS: 00007f0d731fc700(0000) GS:ffff8acba9400000(0000) > knlGS:0000000000000000 > [ 11.163445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 11.163448] CR2: 00007f0e65e00028 CR3: 0000000108d50003 CR4: > 0000000000370ef0 > [ 11.163452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 11.163453] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 11.163455] Call Trace: > [ 11.163468] hugetlb_mcopy_atomic_pte+0xcb/0x450 > [ 11.163477] mcopy_atomic+0xa08/0xd60 > [ 11.163480] ? __might_fault+0x56/0x80 > [ 11.163493] userfaultfd_ioctl+0xb18/0xd60 > [ 11.163502] __se_sys_ioctl+0x77/0xc0 > [ 11.163507] __x64_sys_ioctl+0x1d/0x20 > [ 11.163510] do_syscall_64+0x3f/0x80 > [ 11.163515] entry_SYSCALL_64_after_hwframe+0x44/0xae > [ 11.163519] RIP: 0033:0x45ec87 > [ 11.163531] Code: 3c 1c 48 f7 d8 49 39 c4 72 b8 e8 64 63 03 00 85 > c0 78 bd 48 83 c4 08 4c 89 e0 5b 41 5c c3 0f 1f 44 00 00 b8 10 00 00 > 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 > 018 > [ 11.163532] RSP: 002b:00007f0d731fc248 EFLAGS: 00000206 ORIG_RAX: > 0000000000000010 > [ 11.163534] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000000000045ec87 > [ 11.163536] RDX: 00007f0d731fc290 RSI: 00000000c028aa03 RDI: 0000000000000004 > [ 11.163537] RBP: 00007f0d731fc270 R08: 00000000004022b3 R09: 00007f0d731fc700 > [ 11.163538] R10: 00007f0d731fc9d0 R11: 0000000000000206 R12: 00007fff610cd82e > [ 11.163539] R13: 00007fff610cd82f R14: 00007f0d731fc400 R15: 0000000001002000 > [ 11.163549] irq event stamp: 722 > [ 11.163550] hardirqs last enabled at (721): [] > kmem_cache_alloc_trace+0x1db/0x370 > [ 11.163558] hardirqs last disabled at (722): [] > _raw_spin_lock_irq+0x32/0x80 > [ 11.163560] softirqs last enabled at (130): [] > __irq_exit_rcu+0xf6/0x100 > [ 11.163564] softirqs last disabled at (125): [] > __irq_exit_rcu+0xf6/0x100 > [ 11.163567] ---[ end trace 358ac5c76c211ea1 ]--- > > Debugging further I find the resv_huge_pages underflows by 1 > temporarily during the run of the test multiple times, but a > __free_huge_page() is always subsequently called that overflows it > back to 0. resv_huge_pages is always 0 at the end of the test. I've > initially looked at this as I suspected a problem in the > resv_huge_pages accounting, but seems the resv_huge_pages accounting > is fine in itself as it correctly decrements resv_huge_pages when a > page is allocated from reservation and correctly increments it back up > when that page is freed. > > I'm not that familiar with the userfaultfd/hugetlb code so I was > hoping to solicit some suggestions for what the issue could be. Things > I've tried so far: > > - Adding code that prevents resv_huge_pages to underflow causes the > test to fail, so it seems in this test the calling code actually > expects to be able to temporarily allocate 1 more page than the VMA > has reserved, which seems like a bug maybe? > - Modifying hugetlb_mcopy_atomic_pte() to not use reserved pages > causes the test to fail again. Doin that and overprovisioning > /proc/sys/vm/nr_hugepages causes the test to pass again but I'm not > sure that's right (not familiar with the code). > - The failure gets reproduced as far back as 5.11, so it doesn't seem > to be related to any recent changes. Hi Mina, I am fairly confident the issue is with hugetlb_mcopy_atomic_pte. It does not detect/handle the case where a page cache page already exists when in MCOPY_ATOMIC_NORMAL mode. If you add a printk/warning after the failure of huge_add_to_page_cache, these will generally correspond to the underflow. From a reservation POV, if the page exists in the cache the reservation was already consumed. The call to alloc_huge_page will 'consume' another reservation which can lead to the underflow. As you noted, this underflow gets cleaned up in the error path. However, we should prevent it from happening as we do not want anyone making decisions on that underflow value. hugetlb_mcopy_atomic_pte should check for a page in the cache and if it exists use it in MCOPY_ATOMIC_NORMAL. This code is quite tricky and my first simple attempt at this did not work. I am happy to continue working on this. However, if you or anyone else want to jump in and fix feel free. -- Mike Kravetz