From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49FBBC433E0 for ; Mon, 22 Jun 2020 23:55:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F3CAF20727 for ; Mon, 22 Jun 2020 23:55:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MK3YZsIq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F3CAF20727 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6D7896B0002; Mon, 22 Jun 2020 19:55:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 662066B0003; Mon, 22 Jun 2020 19:55:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 551056B0005; Mon, 22 Jun 2020 19:55:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id 3B5A86B0002 for ; Mon, 22 Jun 2020 19:55:15 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id AFE47701D0 for ; Mon, 22 Jun 2020 23:55:14 +0000 (UTC) X-FDA: 76958506548.20.rake45_210af2326e37 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 869DA18106BF8 for ; Mon, 22 Jun 2020 23:55:14 +0000 (UTC) X-HE-Tag: rake45_210af2326e37 X-Filterd-Recvd-Size: 7176 Received: from mail-ej1-f66.google.com (mail-ej1-f66.google.com [209.85.218.66]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Mon, 22 Jun 2020 23:55:13 +0000 (UTC) Received: by mail-ej1-f66.google.com with SMTP id n24so19794393ejd.0 for ; Mon, 22 Jun 2020 16:55:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=NjMcZXprv/nOI4forFhKBTk/9F1/kItqecbA0f3zzT8=; b=MK3YZsIqGYoDjbxLtQy2a2wo4ruMmS1w9QUl9/b2H1N34YqISWMm4seVahEk6vXpOI SgBWMKclzauV/+pl9tGCeOaTNW0t2+0vC53zrCUIN6UO6sMVq+xOis8Nnm32NMx8suxR UhCmTiQTZQhtWbpw2vcER4mouZPvsT8VZEXNEeATbbrxzozgbjOPBthCdAw+infZucxe zkUG4Enjj+ZXGXRDXzuhcOIhk127PUmHwYCqIOvVFSxdxnzVTp8DEZ/aH2mR5XH5+Ans xmUT27IlXZQdKBsq/7FrGNFly2GxAGf/6aYN5rJVJtKl2tow0KSBSoBWJGhErIvYIYDX NbSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=NjMcZXprv/nOI4forFhKBTk/9F1/kItqecbA0f3zzT8=; b=qyTz7Y+XVjI8nKm8fypr0v3R1XfB4xRj0L75mC+pk6OU5be5lVjkOhq+5qMoPz3aTg 4wGWEQKumzQir0rHQ46/G2I66G4vbC09fadaIg8YunYnhsxvx1vpOnqEk48OexpsjOGG QMa8FSpi5pYj31tcF7inojkR7jX6V/nPwyxaFts5GveApzAUrc9lWB02Aobssa6BEwnN BXtM50UFcWEH7vd3JwZRoFRIjCiCe4icPx0UE6QEERWFM3FvQWpOVUZQg5nzZrske73O YbxrFTLLBRCDvRHz+DEfqxY1XU5QMzLKZUPGl+ZK1FyE+RQlBfD+4dt2Cg2cGO3lygKT 8pUw== X-Gm-Message-State: AOAM532BTFXBAKYSQSTFru2AQdfIQLSpxpXyQiDiA3ehC1TouoNtl/31 fewt3Tc11wTJyWYoOampQuyBi3Pr5owUCVeY5Js= X-Google-Smtp-Source: ABdhPJwQzUdtQVA8N+epe6jO74BqD4xLZlDw/rWnWRA1sAy4igezZryH7Kv9srM9ru85/edGlgH35lOeiLAECSYhKcc= X-Received: by 2002:a17:906:2b81:: with SMTP id m1mr17303645ejg.488.1592870112660; Mon, 22 Jun 2020 16:55:12 -0700 (PDT) MIME-Version: 1.0 References: <20200619215649.32297-1-rcampbell@nvidia.com> <20200619215649.32297-14-rcampbell@nvidia.com> <4C364E23-0716-4D59-85A1-0C293B86BC2C@nvidia.com> In-Reply-To: From: Yang Shi Date: Mon, 22 Jun 2020 16:54:45 -0700 Message-ID: Subject: Re: [PATCH 13/16] mm: support THP migration to device private memory To: John Hubbard Cc: Zi Yan , Ralph Campbell , nouveau@lists.freedesktop.org, linux-rdma@vger.kernel.org, Linux MM , linux-kselftest@vger.kernel.org, Linux Kernel Mailing List , Jerome Glisse , Christoph Hellwig , Jason Gunthorpe , Ben Skeggs , Andrew Morton , Shuah Khan , "Huang, Ying" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 869DA18106BF8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 22, 2020 at 4:02 PM John Hubbard wrote: > > On 2020-06-22 15:33, Yang Shi wrote: > > On Mon, Jun 22, 2020 at 3:30 PM Yang Shi wrote: > >> On Mon, Jun 22, 2020 at 2:53 PM Zi Yan wrote: > >>> On 22 Jun 2020, at 17:31, Ralph Campbell wrote: > >>>> On 6/22/20 1:10 PM, Zi Yan wrote: > >>>>> On 22 Jun 2020, at 15:36, Ralph Campbell wrote: > >>>>>> On 6/21/20 4:20 PM, Zi Yan wrote: > >>>>>>> On 19 Jun 2020, at 17:56, Ralph Campbell wrote: > ... > >>> Ying(cc=E2=80=99d) developed the code to swapout and swapin THP in on= e piece: https://lore.kernel.org/linux-mm/20181207054122.27822-1-ying.huang= @intel.com/. > >>> I am not sure whether the patchset makes into mainstream or not. It c= ould be a good technical reference > >>> for swapping in device private pages, although swapping in pages from= disk and from device private > >>> memory are two different scenarios. > >>> > >>> Since the device private memory swapin impacts core mm performance, w= e might want to discuss your patches > >>> with more people, like the ones from Ying=E2=80=99s patchset, in the = next version. > >> > >> I believe Ying will give you more insights about how THP swap works. > >> > >> But, IMHO device memory migration (migrate to system memory) seems > >> like THP CoW more than swap. > > > A fine point: overall, the desired behavior is "migrate", not CoW. > That's important. Migrate means that you don't leave a page behind, even > a read-only one. And that's exactly how device private migration is > specified. > > We should try to avoid any erosion of clarity here. Even if somehow > (really?) the underlying implementation calls this THP CoW, the actual > goal is to migrate pages over to the device (and back). > > > >> > >> When migrating in: > > > > Sorry for my fat finger, hit sent button inadvertently, let me finish h= ere. > > > > When migrating in: > > > > - if THP is enabled: allocate THP, but need handle allocation > > failure by falling back to base page > > - if THP is disabled: fallback to base page > > > > OK, but *all* page entries (base and huge/large pages) need to be cleared= , > when migrating to device memory, unless I'm really confused here. > So: not CoW. I realized the comment caused more confusion. I apologize for the confusion. Yes, the trigger condition for swap/migration and CoW are definitely different. Here I mean the fault handling part of migrating into system memory. Swap-in just needs to handle the base page case since THP swapin is not supported in upstream yet and the PMD is split in swap-out phase (see shrink_page_list). The patch adds THP migration support to device memory, but you need to handle migrate in (back to system memory) case correctly. The fault handling should look like THP CoW fault handling behavior (before 5.8): - if THP is enabled: allocate THP, fallback if allocation is failed - if THP is disabled: fallback to base page Swap fault handling doesn't look like the above. So, I said it seems like more THP CoW (fault handling part only before 5.8). I hope I articulate my mind. However, I didn't see such fallback is handled. It looks if THP allocation is failed, it just returns SIGBUS; and no check about THP status if I read the patches correctly. The THP might be disabled for the specific vma or system wide before migrating from device memory back to system memory. > > thanks, > -- > John Hubbard > NVIDIA