From: Peter Xu <peterx@redhat.com>
To: Axel Rasmussen <axelrasmussen@google.com>
Cc: Linux MM <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Andrea Arcangeli <aarcange@redhat.com>,
Nadav Amit <nadav.amit@gmail.com>, Li Wang <liwan@redhat.com>
Subject: Re: [PATCH] mm/userfaultfd: selftests: Fix memory corruption with thp enabled
Date: Fri, 24 Sep 2021 15:59:32 -0400 [thread overview]
Message-ID: <YU4uJLJHsHLVsS2w@t490s> (raw)
In-Reply-To: <CAJHvVci8ig4WCY2aK-GvmHP=Uw3s8DrGahk-Yh37GX2kR35kww@mail.gmail.com>
On Fri, Sep 24, 2021 at 10:21:30AM -0700, Axel Rasmussen wrote:
> On Thu, Sep 23, 2021 at 4:25 PM Peter Xu <peterx@redhat.com> wrote:
> >
> > In RHEL's gating selftests we've encountered memory corruption in the uffd
> > event test even with upstream kernel:
> >
> > # ./userfaultfd anon 128 4
> > nr_pages: 32768, nr_pages_per_cpu: 32768
> > bounces: 3, mode: rnd racing read, userfaults: 6240 missing (6240) 14729 wp (14729)
> > bounces: 2, mode: racing read, userfaults: 1444 missing (1444) 28877 wp (28877)
> > bounces: 1, mode: rnd read, userfaults: 6055 missing (6055) 14699 wp (14699)
> > bounces: 0, mode: read, userfaults: 82 missing (82) 25196 wp (25196)
> > testing uffd-wp with pagemap (pgsize=4096): done
> > testing uffd-wp with pagemap (pgsize=2097152): done
> > testing events (fork, remap, remove): ERROR: nr 32427 memory corruption 0 1 (errno=0, line=963)
> > ERROR: faulting process failed (errno=0, line=1117)
> >
> > It can be easily reproduced when global thp enabled, which is the default for
> > RHEL.
> >
> > It's also known as a side effect of commit 0db282ba2c12 ("selftest: use mmap
> > instead of posix_memalign to allocate memory", 2021-07-23), which is imho right
> > itself on using mmap() to make sure the addresses will be untagged even on arm.
> >
> > The problem is, for each test we allocate buffers using two allocate_area()
> > calls. We assumed these two buffers won't affect each other, however they
> > could, because mmap() could have found that the two buffers are near each other
> > and having the same VMA flags, so they got merged into one VMA.
> >
> > It won't be a big problem if thp is not enabled, but when thp is agressively
> > enabled it means when initializing the src buffer it could accidentally setup
> > part of the dest buffer too when there's a shared THP that overlaps the two
> > regions. Then some of the dest buffer won't be able to be trapped by
> > userfaultfd missing mode, then it'll cause memory corruption as described.
> >
> > To fix it, do release_pages() after initializing the src buffer.
>
> But, if I understand correctly, release_pages() will just free the
> physical pages, but not touch the VMA(s). So, with the right
> max_ptes_none setting, why couldn't khugepaged just decide to
> re-collapse (with zero pages) immediately after we release the pages,
> causing the same problem? It seems to me this change just
> significantly narrows the race window (which explains why we see less
> of the issue), but doesn't fix it fundamentally.
Did you mean you can reproduce the issue even with this patch?
It is a good point anyway, indeed I don't see anything stops it from happening.
I wanted to prepare a v2 by releasing the pages after uffdio registration where
we'll do the vma split, but it won't simply work because release_pages() will
cause the process to hang death since that test registers with EVENT_REMOVE,
and release_pages() upon the thp will trigger synchronous EVENT_REMOVE which
cannot be handled by anyone.
Another solution is to map some PROT_NONE regions between the buffers, to make
sure they won't share a VMA. I'll need to think more about which is better..
--
Peter Xu
next prev parent reply other threads:[~2021-09-24 19:59 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-23 23:25 [PATCH] mm/userfaultfd: selftests: Fix memory corruption with thp enabled Peter Xu
2021-09-24 2:19 ` Andrew Morton
2021-09-24 14:14 ` Peter Xu
2021-09-24 17:21 ` Axel Rasmussen
2021-09-24 19:59 ` Peter Xu [this message]
2021-09-27 17:34 ` Axel Rasmussen
2021-09-27 17:39 ` Peter Xu
2021-09-27 17:36 ` Peter Xu
2021-09-27 17:49 ` Axel Rasmussen
2021-09-27 20:37 ` Peter Xu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YU4uJLJHsHLVsS2w@t490s \
--to=peterx@redhat.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=liwan@redhat.com \
--cc=nadav.amit@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).