From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20826C433FE for ; Mon, 27 Sep 2021 17:34:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C8F7D60E08 for ; Mon, 27 Sep 2021 17:34:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C8F7D60E08 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 6270D6B0071; Mon, 27 Sep 2021 13:34:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D73C6B0072; Mon, 27 Sep 2021 13:34:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49FB96B0073; Mon, 27 Sep 2021 13:34:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id 3B6C06B0071 for ; Mon, 27 Sep 2021 13:34:43 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id EBA2B8249980 for ; Mon, 27 Sep 2021 17:34:42 +0000 (UTC) X-FDA: 78634053204.14.A95631F Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) by imf06.hostedemail.com (Postfix) with ESMTP id AFB01801A89C for ; Mon, 27 Sep 2021 17:34:42 +0000 (UTC) Received: by mail-io1-f53.google.com with SMTP id n71so23900805iod.0 for ; Mon, 27 Sep 2021 10:34:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Ju0i6KmuCXf0Y8f28twVBwRvi9cYB/Cv7YJM46OaRDg=; b=Xb2P1lUNNyTkjRvl/Od0jpIuYij3PEuS44FpO6RCeuYQARbhXQAlrzHisVd7g21dx+ i31HTNXkTxYl2cxhmyakx2CSMtkGqvj2uh8eFi4kLLRd1PhJ/4n6Jotw/2ywe/P7Lrb3 VGCJ9o7MjPX8YO55/Ej1uZ2039hZ4U7DhMjFQmHEYYvzrOU+xXfOs8dRU1dDr+cV6Nwz 9GCHY+941Nw0t4puq7Ff5Qjv5egduRn9q9YHH6+5efJUqgZueZQ3uFRrRWiyWMujfl6V 0e+J2UlyjyQufjlc6BcHb3n2gqrzII+2AT0fgzeVRTaZDOi/6p3M7t4nm+MBjgtuhfVO Jmxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Ju0i6KmuCXf0Y8f28twVBwRvi9cYB/Cv7YJM46OaRDg=; b=TegXQpxo0/EfszKczTNNE4Fi4vpZVu6b8sfPiHzyE3ScxQTe7r2HVp79VuiXfqrdas ftVWwLVEYAfNNQ0hBZLInh8YO4rDUyt+ePo70vec89i/XbKMx/orHolrN2ouMr4s8UR+ RI6pfNw0mMHLpspwEuT7IAuIDVc6fxcdFvsSsShIuPWTRwYPcZKmdk1Y7ElQMtluZzmf SFVCGgu85MsLIPiTPPuS31xfk6Bouy4hfIBqJh+aOns11g0v39+9tEq7U5pHnADaUKpw NW4Oj4qBHUPMPOafk+ClGzkyVtFXmZGmQSKUkMXHs/RprSU7fbtt6jwsGc15uJ9WEYqv NeCw== X-Gm-Message-State: AOAM531SY44On0JqZiAPRcoRrd6NiHMkY6Kd5Cum0i2MtrwuMl0JFf2t pk+WGgmbsaSNVoyDzvDsjPC1GnKQ8jO/KRXk0QOs5A== X-Google-Smtp-Source: ABdhPJzbJA1evZE5AUAluhy7MYFmrXxwibyO9sgFGsJIj89IVO2Sx27SLZ1c5HaP9OmFZ66G2up63sYwKhVA/GUqEUM= X-Received: by 2002:a5e:db0c:: with SMTP id q12mr699484iop.32.1632764081836; Mon, 27 Sep 2021 10:34:41 -0700 (PDT) MIME-Version: 1.0 References: <20210923232512.210092-1-peterx@redhat.com> In-Reply-To: From: Axel Rasmussen Date: Mon, 27 Sep 2021 10:34:06 -0700 Message-ID: Subject: Re: [PATCH] mm/userfaultfd: selftests: Fix memory corruption with thp enabled To: Peter Xu Cc: Linux MM , LKML , Andrew Morton , Andrea Arcangeli , Nadav Amit , Li Wang Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Xb2P1lUN; spf=pass (imf06.hostedemail.com: domain of axelrasmussen@google.com designates 209.85.166.53 as permitted sender) smtp.mailfrom=axelrasmussen@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AFB01801A89C X-Stat-Signature: dt5fbtp8eufnekyfaurfy518diugueht X-HE-Tag: 1632764082-260427 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Sep 24, 2021 at 12:59 PM Peter Xu wrote: > > On Fri, Sep 24, 2021 at 10:21:30AM -0700, Axel Rasmussen wrote: > > On Thu, Sep 23, 2021 at 4:25 PM Peter Xu wrote: > > > > > > In RHEL's gating selftests we've encountered memory corruption in the uffd > > > event test even with upstream kernel: > > > > > > # ./userfaultfd anon 128 4 > > > nr_pages: 32768, nr_pages_per_cpu: 32768 > > > bounces: 3, mode: rnd racing read, userfaults: 6240 missing (6240) 14729 wp (14729) > > > bounces: 2, mode: racing read, userfaults: 1444 missing (1444) 28877 wp (28877) > > > bounces: 1, mode: rnd read, userfaults: 6055 missing (6055) 14699 wp (14699) > > > bounces: 0, mode: read, userfaults: 82 missing (82) 25196 wp (25196) > > > testing uffd-wp with pagemap (pgsize=4096): done > > > testing uffd-wp with pagemap (pgsize=2097152): done > > > testing events (fork, remap, remove): ERROR: nr 32427 memory corruption 0 1 (errno=0, line=963) > > > ERROR: faulting process failed (errno=0, line=1117) > > > > > > It can be easily reproduced when global thp enabled, which is the default for > > > RHEL. > > > > > > It's also known as a side effect of commit 0db282ba2c12 ("selftest: use mmap > > > instead of posix_memalign to allocate memory", 2021-07-23), which is imho right > > > itself on using mmap() to make sure the addresses will be untagged even on arm. > > > > > > The problem is, for each test we allocate buffers using two allocate_area() > > > calls. We assumed these two buffers won't affect each other, however they > > > could, because mmap() could have found that the two buffers are near each other > > > and having the same VMA flags, so they got merged into one VMA. > > > > > > It won't be a big problem if thp is not enabled, but when thp is agressively > > > enabled it means when initializing the src buffer it could accidentally setup > > > part of the dest buffer too when there's a shared THP that overlaps the two > > > regions. Then some of the dest buffer won't be able to be trapped by > > > userfaultfd missing mode, then it'll cause memory corruption as described. > > > > > > To fix it, do release_pages() after initializing the src buffer. > > > > But, if I understand correctly, release_pages() will just free the > > physical pages, but not touch the VMA(s). So, with the right > > max_ptes_none setting, why couldn't khugepaged just decide to > > re-collapse (with zero pages) immediately after we release the pages, > > causing the same problem? It seems to me this change just > > significantly narrows the race window (which explains why we see less > > of the issue), but doesn't fix it fundamentally. > > Did you mean you can reproduce the issue even with this patch? No, I haven't actually seen this happen after the patch. I suspect with this patch the window for it to happen is small, so reproducing may be hard. But from a theoretical standpoint, I don't see why it couldn't happen. > > It is a good point anyway, indeed I don't see anything stops it from happening. > > I wanted to prepare a v2 by releasing the pages after uffdio registration where > we'll do the vma split, but it won't simply work because release_pages() will > cause the process to hang death since that test registers with EVENT_REMOVE, > and release_pages() upon the thp will trigger synchronous EVENT_REMOVE which > cannot be handled by anyone. > > Another solution is to map some PROT_NONE regions between the buffers, to make > sure they won't share a VMA. I'll need to think more about which is better.. One possibility would be to MADV_NOHUGEPAGE the regions, which at least would fix the immediate flakiness. Then we could spend some time adding a test case which specifically targets THP interactions? (I do think we want test coverage of that in the end, but with the current tests it's kind of "accidental".) > > -- > Peter Xu >