From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7662C433ED for ; Tue, 11 May 2021 19:08:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3C5E861585 for ; Tue, 11 May 2021 19:08:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3C5E861585 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 734076B006E; Tue, 11 May 2021 15:08:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BD676B0072; Tue, 11 May 2021 15:08:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 440D16B006E; Tue, 11 May 2021 15:08:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0039.hostedemail.com [216.40.44.39]) by kanga.kvack.org (Postfix) with ESMTP id 154536B006E for ; Tue, 11 May 2021 15:08:49 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id CC36818026B1A for ; Tue, 11 May 2021 19:08:48 +0000 (UTC) X-FDA: 78129887136.15.75EA7ED Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by imf30.hostedemail.com (Postfix) with ESMTP id 4E7B7E000129 for ; Tue, 11 May 2021 19:08:19 +0000 (UTC) Received: by mail-pg1-f173.google.com with SMTP id m37so16442387pgb.8 for ; Tue, 11 May 2021 12:08:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=H2K0TBlrsfZ30oqWS+C1tUlRFe3Ic29TcICemWDaFNg=; b=poy3OxZv/OA4HVG2CzYIsVcpK4XdRLbz9yGFRfM4GCdQRGfIfaTQ8de922Puv8j4hu vQajQuzIjOxrZqnSdTL2xQvuGDvU6Hl+bSVIUjmBW80ZNPvDSVnauBksljJ+RLKreE6K DiylF6/5Kah4iQ7y1g3FmYZf6ttuQPmXXxe8vSUxhN6s1/8JKd3W4zC61A8ZAjLwyMfM nMW6zFyM9rRs0K9r6yXlMzt20waVFk+Ealtd74XlqDMZF0YxRI73aybeHRczCQb/VTf+ Iu4Lf9m5BTyG4Z/8y9TVunnkX1q01TW1ZDEN7ris+QaHlgctnMxrVtbq16GbsCbXw5Kr tXIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=H2K0TBlrsfZ30oqWS+C1tUlRFe3Ic29TcICemWDaFNg=; b=Gb1Rt/Vi0LtuRKIpO2Dkec9uYuJbNg64eAoOkGvi7sP9cm7JMXm2VRlXCpp13c0HI+ y9tftVaCkrhAlvkAQDtKCQtzEI3vGRs8q50I8CEzBhbrPjB86ZXqSRuM3Ts9qMLTSGpH rR5D+7uNIHOBCibQ/5728NoEiUOp22kEFzxDYi4NUT4lk+/R51V39qyOawgZkum9VyST 5tjGVSOB4bccBnU2kwt0Zvu3YlxKdilT4+tLSH06a3PJGrlQ7pfqafXeSUecP8UfG5Ik OoQI1Jo/HQ/z36KULg87Tdx/o25hnLH3HkCxFO00LzqjgBmEyQO0JVschmGps1uxZceO ppTA== X-Gm-Message-State: AOAM5309esUJxNGvJ61xWV/6a/UKOWn+bnzLjQdZPubkDq66IbLFzUuU vlRIIQI8bCbYcR+mmjlJmqqLcixClFbPIb7anXGYHA== X-Google-Smtp-Source: ABdhPJwt3qT1MotsDLuFgOVBXp2R55i9DO3YpDtceA46PCp1/76PkDxnjUCMGW9+Ydrc5lFuZ5QeRZkIVj/SizyIb8U= X-Received: by 2002:a63:2115:: with SMTP id h21mr32568932pgh.191.1620760127319; Tue, 11 May 2021 12:08:47 -0700 (PDT) MIME-Version: 1.0 References: <5d4eb1c7-0e46-7b71-b692-72cee179e0aa@oracle.com> In-Reply-To: <5d4eb1c7-0e46-7b71-b692-72cee179e0aa@oracle.com> From: Mina Almasry Date: Tue, 11 May 2021 12:08:36 -0700 Message-ID: Subject: Re: resv_huge_page underflow with userfaultfd test To: Mike Kravetz Cc: Axel Rasmussen , Linux-MM , Andrea Arcangeli , Peter Xu Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=poy3OxZv; spf=pass (imf30.hostedemail.com: domain of almasrymina@google.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=almasrymina@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 4E7B7E000129 X-Stat-Signature: q7q5eysh416ijuhc4zarikoi5jz1noia Received-SPF: none (google.com>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=mail-pg1-f173.google.com; client-ip=209.85.215.173 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620760099-511956 X-Bogosity: Ham, tests=bogofilter, spamicity=0.009081, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, May 11, 2021 at 9:38 AM Mike Kravetz wrote: > > On 5/10/21 11:45 PM, Axel Rasmussen wrote: > > Thanks for the investigation, Mike! > > > > Mina, since hugetlb_mcopy_atomic_pte is specific to userfaultfd, I'm > > happy to take a deeper look at it this week as well. > > > > For context, we have seen the WARN_ON Mina described trigger in > > production before, but were never able to reproduce it. The > > userfaultfd self test turns out to reproduce it reliably, so the > > thinking up to this point was that it just happened to reproduce some > > non-userfaultfd-specific issue. But from Mike's description, it seems > > this bug is very specific to userfaultfd after all. :) > > Certainly, this case is userfaultfd specific. > > However, I too recall seeing 'transient' underflows in the past. Pretty > sure this was not userfaultfd specific. Specifically, when working on > commit 22146c3ce989 "hugetlbfs: dirty pages as they are added to pagecache" > I recall seeing transient underflow. After fixing the issue in 22146c3ce989, > I could not reproduce transient underflows and stopped looking for the > cause. We added code to a production kernel in an attmempt to catch > the issue: > https://github.com/oracle/linux-uek/commit/bd697676290d91762ef2bf79832f653b44e6f83b#diff-fb6066ca63d9afdc2e3660c85e2dcc04cf31b37900ca9df2a1019ee8fa80dce0 > We also have local changes that detect resv_huge_pages underflowing, and they trigger in production every few months and we don't use userfaultfd in production. We haven't been able to reproduce these issues though, and we run an older version of the kernel so I haven't had anything useful to share until now. I'm (quite tentatively to be honest) considering proposing adding some of these WARN_ONCE_ON() to the kernel after I fix this issue and soliciting reproducers from the community to fix these underflows. I'm hesitant because I'm not confident I can fix the ensuing bug reports in a timely manner and the volume of reports may be large. But we can cross that bridge when this issue is figured out :) > I'll try running some non-userfaultfd specific tests to see if I can > reproduce. > -- > Mike Kravetz