From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACA75C41604 for ; Fri, 2 Oct 2020 22:08:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ECA6720719 for ; Fri, 2 Oct 2020 22:08:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="arFps9Zm" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ECA6720719 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1D3346B0088; Fri, 2 Oct 2020 18:08:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1861A6B0089; Fri, 2 Oct 2020 18:08:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 023496B008A; Fri, 2 Oct 2020 18:08:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0058.hostedemail.com [216.40.44.58]) by kanga.kvack.org (Postfix) with ESMTP id C1A856B0088 for ; Fri, 2 Oct 2020 18:08:43 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 54F62181AE877 for ; Fri, 2 Oct 2020 22:08:43 +0000 (UTC) X-FDA: 77328375726.16.level50_5d14064271a7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 2E138100E54A8 for ; Fri, 2 Oct 2020 22:08:43 +0000 (UTC) X-HE-Tag: level50_5d14064271a7 X-Filterd-Recvd-Size: 19670 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Fri, 2 Oct 2020 22:08:42 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id b124so2266851pfg.13 for ; Fri, 02 Oct 2020 15:08:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=ZTsbIZbxk/VLPosRWvsRfPC9ZULIuxmEiMaEwM9spDY=; b=arFps9ZmNz+LxrAlpcG06dMwIARJR2irUoZbyHR+QUIhT4m3JXPL5Ay349I5q2Fo0V 0/sCkQhVxxIh5TpVOU8qj09EoY/m7jvscsKW8CIMW1Dypfj6TukHLwKdh0QGmJ/LwKrf g5aQNClXmo8UQCY9POADn0C/ZR+XPMVsGAlse1oRWM6tNjBqLOYCyAW3IxqzfbDZbwnT z91HvJd0a2dZpMDnTkuBS2LBKggMWlS5q1+OFnLPCFhVUVf2z9N3gorbyL21WHV4Axsv cee56sC4L4TqHh0JSotOulOM4L7IpTAmgSjUUJ+727oklMHWRF8R6iErW1O2c2QHUv0D GkOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=ZTsbIZbxk/VLPosRWvsRfPC9ZULIuxmEiMaEwM9spDY=; b=rHM8zmOnmfEAbzOffnS74wGi5XuwZohZdx303K7AVUY3Ry2T74vNcl1ROQ5p+7LXYr K4psasft8SF1rZbYTy5+pyVqBmMf3StvKVol03f9g7Wpr9c3H5mfmoyLqqMij9vbXeJ6 sqJrCmbfiWMdPoopvmeQvIott+A5EIgXFHsM6003+HEQGtuhcKZ+j3qTXHG/EauuZ90m FsxorPduQz/E2iHtdMbNHWaqwXsd/fR05LgYaH6caIWEyH3gh+6pzSmRWQxYCOXW6YL6 P4IiAHI08L2peBVcVf11TT8RqCBKQt6a1djES80Cw1i6gWaoOFVnvNVDc1lzxjOXfReA 0I7g== X-Gm-Message-State: AOAM5307JQ5ggG9Csi0j1MpbT0EB3N7M35CI2FIfegRAbNvDVwzd5eT5 IHrPwsHHeVTC/xiIIg6hdhN4F8nTylsGShcxPFOPFA== X-Google-Smtp-Source: ABdhPJzziu9jz1tm+VBXgbKFPycLeYOboaED5MZJg+7acHe57tXEkp/1yDAJ5qcnPCnqjWOrEb8bRS3OW7F7M3+LHp8= X-Received: by 2002:a65:5cc2:: with SMTP id b2mr4122419pgt.124.1601676521329; Fri, 02 Oct 2020 15:08:41 -0700 (PDT) MIME-Version: 1.0 References: <20201002162101.665549-1-kaleshsingh@google.com> <20201002162101.665549-4-kaleshsingh@google.com> <20201002165122.bm427a4ealmmmtqc@black.fi.intel.com> In-Reply-To: <20201002165122.bm427a4ealmmmtqc@black.fi.intel.com> From: Kalesh Singh Date: Fri, 2 Oct 2020 18:08:29 -0400 Message-ID: Subject: Re: [PATCH v2 3/6] mm: Speedup mremap on 1GB or larger regions To: "Kirill A. Shutemov" Cc: Suren Baghdasaryan , Minchan Kim , Joel Fernandes , Lokesh Gidra , "Cc: Android Kernel" , kernel test robot , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "the arch/x86 maintainers" , "H. Peter Anvin" , Andrew Morton , Shuah Khan , "Aneesh Kumar K.V" , Kees Cook , Peter Zijlstra , Sami Tolvanen , Masahiro Yamada , Arnd Bergmann , Frederic Weisbecker , Krzysztof Kozlowski , Hassan Naveed , Christian Brauner , Mark Rutland , Mike Rapoport , Gavin Shan , Chris von Recklinghausen , Jia He , Zhenyu Ye , John Hubbard , Thiago Jung Bauermann , William Kucharski , Ram Pai , Ralph Campbell , Mina Almasry , Sandipan Das , Dave Hansen , Masami Hiramatsu , SeongJae Park , Brian Geffon , LKML , "moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)" , "open list:MEMORY MANAGEMENT" , "open list:KERNEL SELFTEST FRAMEWORK" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Kirill, thank you for the feedback. On Fri, Oct 2, 2020 at 12:51 PM Kirill A. Shutemov wrote: > > On Fri, Oct 02, 2020 at 04:20:48PM +0000, Kalesh Singh wrote: > > Android needs to move large memory regions for garbage collection. > > The GC requires moving physical pages of multi-gigabyte heap > > using mremap. During this move, the application threads have to > > be paused for correctness. It is critical to keep this pause as > > short as possible to avoid jitters during user interaction. > > > > Optimize mremap for >=3D 1GB-sized regions by moving at the PUD/PGD > > level if the source and destination addresses are PUD-aligned. > > For CONFIG_PGTABLE_LEVELS =3D=3D 3, moving at the PUD level in effect m= oves > > PGD entries, since the PUD entry is =E2=80=9Cfolded back=E2=80=9D onto = the PGD entry. > > Add HAVE_MOVE_PUD so that architectures where moving at the PUD level > > isn't supported/tested can turn this off by not selecting the config. > > > > Fix build test error from v1 of this series reported by > > kernel test robot in [1]. > > > > [1] https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread= /CKPGL4FH4NG7TGH2CVYX2UX76L25BTA3/ > > > > Signed-off-by: Kalesh Singh > > Reported-by: kernel test robot > > --- > > Changes in v2: > > - Update commit message with description of Android GC's use case. > > - Move set_pud_at() to a separate patch. > > - Use switch() instead of ifs in move_pgt_entry() > > - Fix build test error reported by kernel test robot on x86_64 in [1]= . > > Guard move_huge_pmd() with IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE), > > since this section doesn't get optimized out in the kernel test > > robot's build test when HAVE_MOVE_PUD is enabled. > > - Keep WARN_ON_ONCE(1) instead of BUILD_BUG() for the aforementioned > > reason. > > Okay, but IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) on the caller side woul= d > do the trick, I believe. I tried moving this to the caller side in move_page_tables(), - if (extent =3D=3D HPAGE_PMD_SIZE && + if (extent =3D=3D HPAGE_PMD_SIZE && IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && but it produces the same error as reported by kernel test robot: ld.lld: error: undefined symbol: move_huge_pmd I'm not sure why these are different but the kernel test robot compiler complains. > > > > > arch/Kconfig | 7 ++ > > mm/mremap.c | 220 ++++++++++++++++++++++++++++++++++++++++++++------- > > 2 files changed, 197 insertions(+), 30 deletions(-) > > > > diff --git a/arch/Kconfig b/arch/Kconfig > > index af14a567b493..5eabaa00bf9b 100644 > > --- a/arch/Kconfig > > +++ b/arch/Kconfig > > @@ -602,6 +602,13 @@ config HAVE_IRQ_TIME_ACCOUNTING > > Archs need to ensure they use a high enough resolution clock to > > support irq time accounting and then call enable_sched_clock_ir= qtime(). > > > > +config HAVE_MOVE_PUD > > + bool > > + help > > + Architectures that select this are able to move page tables at = the > > + PUD level. If there are only 3 page table levels, the move effe= ctively > > + happens at the PGD level. > > + > > config HAVE_MOVE_PMD > > bool > > help > > diff --git a/mm/mremap.c b/mm/mremap.c > > index 138abbae4f75..c1d6ab667d70 100644 > > --- a/mm/mremap.c > > +++ b/mm/mremap.c > > @@ -249,14 +249,176 @@ static bool move_normal_pmd(struct vm_area_struc= t *vma, unsigned long old_addr, > > > > return true; > > } > > +#else > > +static inline bool move_normal_pmd(struct vm_area_struct *vma, unsigne= d long old_addr, > > + unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd) > > +{ > > + return false; > > +} > > #endif > > > > +#ifdef CONFIG_HAVE_MOVE_PUD > > +static pud_t *get_old_pud(struct mm_struct *mm, unsigned long addr) > > +{ > > + pgd_t *pgd; > > + p4d_t *p4d; > > + pud_t *pud; > > + > > + pgd =3D pgd_offset(mm, addr); > > + if (pgd_none_or_clear_bad(pgd)) > > + return NULL; > > + > > + p4d =3D p4d_offset(pgd, addr); > > + if (p4d_none_or_clear_bad(p4d)) > > + return NULL; > > + > > + pud =3D pud_offset(p4d, addr); > > + if (pud_none_or_clear_bad(pud)) > > + return NULL; > > + > > + return pud; > > +} > > + > > +static pud_t *alloc_new_pud(struct mm_struct *mm, struct vm_area_struc= t *vma, > > + unsigned long addr) > > +{ > > + pgd_t *pgd; > > + p4d_t *p4d; > > + pud_t *pud; > > + > > + pgd =3D pgd_offset(mm, addr); > > + p4d =3D p4d_alloc(mm, pgd, addr); > > + if (!p4d) > > + return NULL; > > + pud =3D pud_alloc(mm, p4d, addr); > > + if (!pud) > > + return NULL; > > + > > + return pud; > > +} > > Looks like a code duplication. > > Could you move these two helpers out of #ifdef CONFIG_HAVE_MOVE_PUD and > make get_old_pmd() and alloc_new_pmd() use them? Yes, that will be cleaner. I'll update it in the next version. > > > + > > +static bool move_normal_pud(struct vm_area_struct *vma, unsigned long = old_addr, > > + unsigned long new_addr, pud_t *old_pud, pud_t *new_pud) > > +{ > > + spinlock_t *old_ptl, *new_ptl; > > + struct mm_struct *mm =3D vma->vm_mm; > > + pud_t pud; > > + > > + /* > > + * The destination pud shouldn't be established, free_pgtables() > > + * should have released it. > > + */ > > + if (WARN_ON_ONCE(!pud_none(*new_pud))) > > + return false; > > + > > + /* > > + * We don't have to worry about the ordering of src and dst > > + * ptlocks because exclusive mmap_lock prevents deadlock. > > + */ > > + old_ptl =3D pud_lock(vma->vm_mm, old_pud); > > + new_ptl =3D pud_lockptr(mm, new_pud); > > + if (new_ptl !=3D old_ptl) > > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); > > + > > + /* Clear the pud */ > > + pud =3D *old_pud; > > + pud_clear(old_pud); > > + > > + VM_BUG_ON(!pud_none(*new_pud)); > > + > > + /* Set the new pud */ > > + set_pud_at(mm, new_addr, new_pud, pud); > > + flush_tlb_range(vma, old_addr, old_addr + PUD_SIZE); > > + if (new_ptl !=3D old_ptl) > > + spin_unlock(new_ptl); > > + spin_unlock(old_ptl); > > + > > + return true; > > +} > > +#else > > +static inline bool move_normal_pud(struct vm_area_struct *vma, unsigne= d long old_addr, > > + unsigned long new_addr, pud_t *old_pud, pud_t *new_pud) > > +{ > > + return false; > > +} > > +#endif > > + > > +enum pgt_entry { > > + NORMAL_PMD, > > + HPAGE_PMD, > > + NORMAL_PUD, > > +}; > > + > > +/* > > + * Returns an extent of the corresponding size for the pgt_entry speci= fied if valid. > > + * Else returns a smaller extent bounded by the end of the source and = destination > > + * pgt_entry. Returns 0 if an invalid pgt_entry is specified. > > + */ > > +static unsigned long get_extent(enum pgt_entry entry, unsigned long ol= d_addr, > > + unsigned long old_end, unsigned long new_addr) > > +{ > > + unsigned long next, extent, mask, size; > > + > > + if (entry =3D=3D NORMAL_PMD || entry =3D=3D HPAGE_PMD) { > > + mask =3D PMD_MASK; > > + size =3D PMD_SIZE; > > + } else if (entry =3D=3D NORMAL_PUD) { > > + mask =3D PUD_MASK; > > + size =3D PUD_SIZE; > > + } else > > + return 0; > > Em. Who would ever specify invalid pgt_entry? It's bug. > Again, switch()? Sounds good. I'll use BUG() and switch() instead. > > > + > > + next =3D (old_addr + size) & mask; > > + /* even if next overflowed, extent below will be ok */ > > + extent =3D (next > old_end) ? old_end - old_addr : next - old_add= r; > > + next =3D (new_addr + size) & mask; > > + if (extent > next - new_addr) > > + extent =3D next - new_addr; > > + return extent; > > +} > > + > > +/* > > + * Attempts to speedup the move by moving entry at the level correspon= ding to > > + * pgt_entry. Returns true if the move was successful, else false. > > + */ > > +static bool move_pgt_entry(enum pgt_entry entry, struct vm_area_struct= *vma, > > + unsigned long old_addr, unsigned long new_addr, v= oid *old_entry, > > + void *new_entry, bool need_rmap_locks) > > +{ > > + bool moved =3D false; > > + > > + /* See comment in move_ptes() */ > > + if (need_rmap_locks) > > + take_rmap_locks(vma); > > + > > + switch (entry) { > > + case NORMAL_PMD: > > + moved =3D move_normal_pmd(vma, old_addr, new_addr, old_e= ntry, new_entry); > > Nit: here and below, double space after '=3D'. Why? Sorry, editing mistake on my end. I'll clean this up before resending. > > > + break; > > + case NORMAL_PUD: > > + moved =3D move_normal_pud(vma, old_addr, new_addr, old_e= ntry, new_entry); > > + break; > > + case HPAGE_PMD: > > + moved =3D IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && > > + move_huge_pmd(vma, old_addr, new_addr, old_entry,= new_entry); > > + break; > > + default: > > + WARN_ON_ONCE(1); > > + break; > > + } > > + > > + if (need_rmap_locks) > > + drop_rmap_locks(vma); > > + > > + return moved; > > +} > > + > > unsigned long move_page_tables(struct vm_area_struct *vma, > > unsigned long old_addr, struct vm_area_struct *new_vma, > > unsigned long new_addr, unsigned long len, > > bool need_rmap_locks) > > { > > - unsigned long extent, next, old_end; > > + unsigned long extent, old_end; > > struct mmu_notifier_range range; > > pmd_t *old_pmd, *new_pmd; > > > > @@ -269,14 +431,27 @@ unsigned long move_page_tables(struct vm_area_str= uct *vma, > > > > for (; old_addr < old_end; old_addr +=3D extent, new_addr +=3D ex= tent) { > > cond_resched(); > > - next =3D (old_addr + PMD_SIZE) & PMD_MASK; > > - /* even if next overflowed, extent below will be ok */ > > - extent =3D next - old_addr; > > - if (extent > old_end - old_addr) > > - extent =3D old_end - old_addr; > > - next =3D (new_addr + PMD_SIZE) & PMD_MASK; > > - if (extent > next - new_addr) > > - extent =3D next - new_addr; > > +#ifdef CONFIG_HAVE_MOVE_PUD > > Any chance if (IS_ENABLED(CONFIG_HAVE_MOVE_PUD)) would work here? Once we move get_old_put() and alloc_new_pud() out of the #ifdefs as you suggested above, it should work. It would also now be possible to replace the #ifdef CONFIG_HAVE_MOVE_PMD in move_page_tables() with IS_ENABLED(CONFIG_HAVE_MOVE_PMD). Thanks, Kalesh > > > + /* > > + * If extent is PUD-sized try to speed up the move by mov= ing at the > > + * PUD level if possible. > > + */ > > + extent =3D get_extent(NORMAL_PUD, old_addr, old_end, new_= addr); > > + if (extent =3D=3D PUD_SIZE) { > > + pud_t *old_pud, *new_pud; > > + > > + old_pud =3D get_old_pud(vma->vm_mm, old_addr); > > + if (!old_pud) > > + continue; > > + new_pud =3D alloc_new_pud(vma->vm_mm, vma, new_ad= dr); > > + if (!new_pud) > > + break; > > + if (move_pgt_entry(NORMAL_PUD, vma, old_addr, new= _addr, > > + old_pud, new_pud, need_rmap_lo= cks)) > > + continue; > > + } > > +#endif > > + extent =3D get_extent(NORMAL_PMD, old_addr, old_end, new_= addr); > > old_pmd =3D get_old_pmd(vma->vm_mm, old_addr); > > if (!old_pmd) > > continue; > > @@ -284,18 +459,10 @@ unsigned long move_page_tables(struct vm_area_str= uct *vma, > > if (!new_pmd) > > break; > > if (is_swap_pmd(*old_pmd) || pmd_trans_huge(*old_pmd) || = pmd_devmap(*old_pmd)) { > > - if (extent =3D=3D HPAGE_PMD_SIZE) { > > - bool moved; > > - /* See comment in move_ptes() */ > > - if (need_rmap_locks) > > - take_rmap_locks(vma); > > - moved =3D move_huge_pmd(vma, old_addr, ne= w_addr, > > - old_pmd, new_pmd); > > - if (need_rmap_locks) > > - drop_rmap_locks(vma); > > - if (moved) > > - continue; > > - } > > + if (extent =3D=3D HPAGE_PMD_SIZE && > > + move_pgt_entry(HPAGE_PMD, vma, old_addr, new_= addr, old_pmd, > > + new_pmd, need_rmap_locks)) > > + continue; > > split_huge_pmd(vma, old_pmd, old_addr); > > if (pmd_trans_unstable(old_pmd)) > > continue; > > @@ -305,15 +472,8 @@ unsigned long move_page_tables(struct vm_area_stru= ct *vma, > > * If the extent is PMD-sized, try to speed the m= ove by > > * moving at the PMD level if possible. > > */ > > - bool moved; > > - > > - if (need_rmap_locks) > > - take_rmap_locks(vma); > > - moved =3D move_normal_pmd(vma, old_addr, new_addr= , > > - old_pmd, new_pmd); > > - if (need_rmap_locks) > > - drop_rmap_locks(vma); > > - if (moved) > > + if (move_pgt_entry(NORMAL_PMD, vma, old_addr, new= _addr, old_pmd, > > + new_pmd, need_rmap_locks)) > > continue; > > #endif > > } > > -- > > 2.28.0.806.g8561365e88-goog > > > > -- > Kirill A. Shutemov > > -- > To unsubscribe from this group and stop receiving emails from it, send an= email to kernel-team+unsubscribe@android.com. >