From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71C81C43461 for ; Tue, 15 Sep 2020 16:06:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0B97921D43 for ; Tue, 15 Sep 2020 16:06:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="GSQ3sEbX" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B97921D43 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 84CEC6B0101; Tue, 15 Sep 2020 12:06:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FE586B0103; Tue, 15 Sep 2020 12:06:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C3A66B0104; Tue, 15 Sep 2020 12:06:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0251.hostedemail.com [216.40.44.251]) by kanga.kvack.org (Postfix) with ESMTP id 520F46B0101 for ; Tue, 15 Sep 2020 12:06:11 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DF0F2364A for ; Tue, 15 Sep 2020 16:06:10 +0000 (UTC) X-FDA: 77265772500.05.snow78_3516d7427112 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id CD04618045D2B for ; Tue, 15 Sep 2020 16:06:05 +0000 (UTC) X-HE-Tag: snow78_3516d7427112 X-Filterd-Recvd-Size: 6416 Received: from mail-io1-f68.google.com (mail-io1-f68.google.com [209.85.166.68]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Tue, 15 Sep 2020 16:05:55 +0000 (UTC) Received: by mail-io1-f68.google.com with SMTP id g128so4620770iof.11 for ; Tue, 15 Sep 2020 09:05:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=+a5NFihdkcuOmZsUsCh8QwUNs+nWQ0s/6KK3dY3Krbw=; b=GSQ3sEbX697YRpg9CR2VlGc9+xfZ7fD+HMnmCgWxGr8y26h7AIYMdFiC5L67kEG3Qr zo3aojd+RHgDTXqiDB/BQbcFLRwLETAOLTDJVWhaWtCwVa0JNhIvcohuUiYH8R7CUu95 Nt53Cd8til24T0tg2ON7Gb1FM8we+nJBGx/Sc7MULK9Am2b0EPlwyrl6bOTtMLzT4xf6 qJ0JTST6QVfKgHY7AqX7VdI7KJ4LzlLt/DX4QTTXkjFpH32zVwHquqXc0+l5yTedosKj laaJ6/nmAZHETdQBJKhVuQ1b3fdq4xJqAy8dsnyu4x+jIxS+lZQrQR0Ywl6zgmjBYour U2vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=+a5NFihdkcuOmZsUsCh8QwUNs+nWQ0s/6KK3dY3Krbw=; b=iUW5RRcSsT0hiXUIyBI1oTGf0GSWX8DTytutvftFmO4+E8WBQK7UyNdiq6TNVIePI9 vLGnv8sbLp3nyPOdbxddvr29WFCZhNxVzhqDr2ODN9XAq7Na8K/OFGtRNAhubtSr949/ QUt6Ka9fqV4l6WiUOPghiGbAuGWUnE4IbKZwZ4Pn2DsFGyL+2o5GJDeJp1lm9ubvnRxT W2PixdKiVrytaNnYThDNQ2S42TsJuwNjUMJSiOt6sQWfiTOqEdeo+Q7LSC3jSS81q2rp bDzbN1bfUx28g5g4wX2YQZBCcXBm0muHklA3drwioFdSO2Yn7LJOsZxS0MOWnhRBM/5+ pGRA== X-Gm-Message-State: AOAM533JLoLO/fIO9uWRw4thlR5iJj+EWk2VkneyxbRjPoOgG1f1gQfF D38Q3t1AuY5BUYqTcRH8+4XKDA== X-Google-Smtp-Source: ABdhPJwATFo46SJSoq0UDVdvowEpZdaKacMf18zCy3iPw2KHtIxWNhiYVraqVhi8N0+dWtAdnkopCg== X-Received: by 2002:a5e:8a08:: with SMTP id d8mr15385257iok.95.1600185955222; Tue, 15 Sep 2020 09:05:55 -0700 (PDT) Received: from ziepe.ca ([206.223.160.26]) by smtp.gmail.com with ESMTPSA id m19sm9155980ila.40.2020.09.15.09.05.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 09:05:54 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kIDSr-006UEB-Ha; Tue, 15 Sep 2020 13:05:53 -0300 Date: Tue, 15 Sep 2020 13:05:53 -0300 From: Jason Gunthorpe To: Peter Xu Cc: Linus Torvalds , Leon Romanovsky , Linux-MM , Linux Kernel Mailing List , "Maya B . Gokhale" , Yang Shi , Marty Mcfadden , Kirill Shutemov , Oleg Nesterov , Jann Horn , Jan Kara , Kirill Tkhai , Andrea Arcangeli , Christoph Hellwig , Andrew Morton Subject: Re: [PATCH 1/4] mm: Trial do_wp_page() simplification Message-ID: <20200915160553.GJ1221970@ziepe.ca> References: <20200821234958.7896-1-peterx@redhat.com> <20200821234958.7896-2-peterx@redhat.com> <20200914143829.GA1424636@nvidia.com> <20200914183436.GD30881@xz-x1> <20200914211515.GA5901@xz-x1> <20200914225542.GO904879@nvidia.com> <20200914232851.GH1221970@ziepe.ca> <20200915145040.GA2949@xz-x1> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200915145040.GA2949@xz-x1> X-Rspamd-Queue-Id: CD04618045D2B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Sep 15, 2020 at 10:50:40AM -0400, Peter Xu wrote: > On Mon, Sep 14, 2020 at 08:28:51PM -0300, Jason Gunthorpe wrote: > > Yes, this stuff does pin_user_pages_fast() and MADV_DONTFORK > > together. It sets FOLL_FORCE and FOLL_WRITE to get an exclusive copy > > of the page and MADV_DONTFORK was needed to ensure that a future fork > > doesn't establish a COW that would break the DMA by moving the > > physical page over to the fork. DMA should stay with the process that > > called pin_user_pages_fast() (Is MADV_DONTFORK still needed with > > recent years work to GUP/etc? It is a pretty terrible ancient thing) > > ... Now I'm more confused on what has happened. I'm going to try to confirm that the MADV_DONTFORK is actually being done by userspace properly, more later. > It means, as long as the rdma region has VM_WRITE set (which I think of no > reason on why it shouldn't...), then it should have the write bit in the COWed > page entry. If so, the page should be stable and I don't undersdand why > another COW could even trigger and how the code path in the "trial cow" patch > is triggered. All the regions the test are doing DMA to will be simple process writable anonymous VMA's from malloc() > Or, the VMA is without VM_WRITE due to some reason? Sorry I probably know > nothing about RDMA, more information on that side might help too. E.g., is the > hardware going to walk the software process page table too when doing RDMA (or > is IOMMU page table used, or none)? It does pin_user_pages_fast(), gets a list of DMA addresses for the pages and then programs the hardware. The pin remains for a very long time and the HW does DMA to those pages independently. Userspace will write to the memory and trigger DMA reads and HW will do DMA writes and trigger something close to an eventfd to let userspace know to check the DMA'd data. Very similar to how an in-kernel driver works. It is similar to VFIO in how it uses pin_user_pages_fast(). Symptoms look to be like the DMA's are not arriving. As before, the requirement is that once a process as done pin_user_pages() the physical page stays with the process. If there is a fork() and a COW then the current memory stays with the original process and the fork'd child gets the copy. MADV_DONTFORK is expected to ensure this.. We haven't been able to narrow to a reproduction that doesn't require alot of hardware unfortunately. It seems oddly sensitive, maybe due to memory layout triggering a COW.. Jason