From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 765F2C433FE for ; Mon, 7 Dec 2020 11:34:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1934F233A0 for ; Mon, 7 Dec 2020 11:34:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1934F233A0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B32EC8D000F; Mon, 7 Dec 2020 06:34:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B08A18D0001; Mon, 7 Dec 2020 06:34:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A20C68D000F; Mon, 7 Dec 2020 06:34:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0203.hostedemail.com [216.40.44.203]) by kanga.kvack.org (Postfix) with ESMTP id 8B59F8D0001 for ; Mon, 7 Dec 2020 06:34:21 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4A28033CD for ; Mon, 7 Dec 2020 11:34:21 +0000 (UTC) X-FDA: 77566277922.29.cook68_250c4fc273de Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 28A19180868E3 for ; Mon, 7 Dec 2020 11:34:21 +0000 (UTC) X-HE-Tag: cook68_250c4fc273de X-Filterd-Recvd-Size: 10760 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Mon, 7 Dec 2020 11:34:20 +0000 (UTC) Received: by mail-pg1-f174.google.com with SMTP id q3so8672133pgr.3 for ; Mon, 07 Dec 2020 03:34:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ydAF98InOXvLOUHWcQbefHcTxGQjuTY7EYp8f1Ij7rU=; b=CbIO/lKMhlmMybymchHGdjYBFjooSqYR87ZMJNCpzamSy0rqGGupNjGohNmWYyuuVF +xqHe0wJyPOa8tX4wGwI9xsV0mLlIIhWZv7yi7zuh1i+ZjbHzI7SroHpaRk8tPLHv8j9 TY3DoXxPGxxbJvXcTFLjLpG9ptLpcORhfr4m/KQm4y0U00zVK1J9SxRLVrdKucZO6fgt Iq53lTb5pWIqLsYf0RPyKOtYavEZdCQ2SpdfqSEQDGGfienBvS0lHPdVmAi0he3F9oSM 2qO8yJB8UNJ6tM0GvVovraZIBAc/ZtLNNhUmoqFgjDcE70JCp5Sfe/0KBwOp0Vx9h2oD SRmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ydAF98InOXvLOUHWcQbefHcTxGQjuTY7EYp8f1Ij7rU=; b=q/4MzsIEpW4tabj8Px7LhoAL2XF7o/ZmzrzX7aBD0Y52Ny0oACYqAok7s7kteFyzng Y5XszABggwaxCqGwAm/lnAZ9s2dN07aYgbQluYVAFcsiwlx3eWl0POXQMxLvQIUrK7/V ReX8IlVB47BlfbXPI50jhwV+ob2CSo4+cfbHgWXaLT19TENP//UyryCoc6jkWOr3OZNB fv9eu6JM2C8EC3B2niaoZnvHueOyP7qZts8/wKomkwPdx4GYeKGMcBkrKSmfI8V1jhwp eEIS/Nzfqu2wLWrCk3kAEyFOSlgFMoOtraoA5IfQN8on7y4vb3bj+vix0QgXZwxx2UZA DLsw== X-Gm-Message-State: AOAM532Ze/FGWeLTFl3RVN+UWueSJ3qL+MAEyfjduSKPtaGxhMfpKzrn Gv2069Gqsvimb9lm4EpEbbzyVekssVY= X-Google-Smtp-Source: ABdhPJzAC/Gy2AUUTW9sU9fx/7AU0/diRQdzJN4dFsjfTKyWiDt0U8rw4OyPNmxFkhpistj53GkGWw== X-Received: by 2002:aa7:9244:0:b029:19a:b335:754b with SMTP id 4-20020aa792440000b029019ab335754bmr15898559pfp.29.1607340859523; Mon, 07 Dec 2020 03:34:19 -0800 (PST) Received: from localhost.localdomain ([203.205.141.39]) by smtp.gmail.com with ESMTPSA id d4sm14219822pfo.127.2020.12.07.03.34.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Dec 2020 03:34:19 -0800 (PST) From: yulei.kernel@gmail.com X-Google-Original-From: yuleixzhang@tencent.com To: linux-mm@kvack.org, akpm@linux-foundation.org, linux-fsdevel@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, naoya.horiguchi@nec.com, viro@zeniv.linux.org.uk, pbonzini@redhat.com Cc: joao.m.martins@oracle.com, rdunlap@infradead.org, sean.j.christopherson@intel.com, xiaoguangrong.eric@gmail.com, kernellwp@gmail.com, lihaiwei.kernel@gmail.com, Yulei Zhang , Haiwei Li Subject: [RFC V2 12/37] dmem: introduce mempolicy support Date: Mon, 7 Dec 2020 19:31:05 +0800 Message-Id: <28718e3b8886b9ec3e4700c2d55a9629ca9fc27c.1607332046.git.yuleixzhang@tencent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yulei Zhang It adds mempolicy support for dmem to allocates memory from mempolicy specified nodes. Signed-off-by: Haiwei Li Signed-off-by: Yulei Zhang --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 7 ++++ arch/x86/include/asm/pgtable_types.h | 13 +++++++- fs/dmemfs/Kconfig | 3 ++ include/linux/pgtable.h | 7 ++++ mm/Kconfig | 3 ++ mm/dmem.c | 63 ++++++++++++++++++++++++++++++= ++++-- 7 files changed, 94 insertions(+), 3 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f6946b8..9ccee76 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -73,6 +73,7 @@ config X86 select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 + select ARCH_HAS_PTE_DMEM if X86_64 select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_COPY_MC if X86_64 diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtabl= e.h index a02c672..dd4aff6 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -452,6 +452,13 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pmd_set_flags(pmd, _PAGE_DEVMAP); } =20 +#ifdef CONFIG_ARCH_HAS_PTE_DMEM +static inline pmd_t pmd_mkdmem(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SPECIAL | _PAGE_DMEM); +} +#endif + static inline pmd_t pmd_mkhuge(pmd_t pmd) { return pmd_set_flags(pmd, _PAGE_PSE); diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/= pgtable_types.h index 816b31c..ee4cae1 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -23,6 +23,15 @@ #define _PAGE_BIT_SOFTW2 10 /* " */ #define _PAGE_BIT_SOFTW3 11 /* " */ #define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */ +#define _PAGE_BIT_DMEM 57 /* Flag used to indicate dmem pmd. + * Since _PAGE_BIT_SPECIAL is defined + * same as _PAGE_BIT_CPA_TEST, we can + * not only use _PAGE_BIT_SPECIAL, so + * add _PAGE_BIT_DMEM to help + * indicate it. Since dmem pte will + * never be splitting, setting + * _PAGE_BIT_SPECIAL for pte is enough. + */ #define _PAGE_BIT_SOFTW4 58 /* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */ @@ -112,9 +121,11 @@ #if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE) #define _PAGE_NX (_AT(pteval_t, 1) << _PAGE_BIT_NX) #define _PAGE_DEVMAP (_AT(u64, 1) << _PAGE_BIT_DEVMAP) +#define _PAGE_DMEM (_AT(u64, 1) << _PAGE_BIT_DMEM) #else #define _PAGE_NX (_AT(pteval_t, 0)) #define _PAGE_DEVMAP (_AT(pteval_t, 0)) +#define _PAGE_DMEM (_AT(pteval_t, 0)) #endif =20 #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) @@ -128,7 +139,7 @@ #define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ _PAGE_SOFT_DIRTY | _PAGE_DEVMAP | _PAGE_ENC | \ - _PAGE_UFFD_WP) + _PAGE_UFFD_WP | _PAGE_DMEM) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) =20 /* diff --git a/fs/dmemfs/Kconfig b/fs/dmemfs/Kconfig index d2894a5..19ca391 100644 --- a/fs/dmemfs/Kconfig +++ b/fs/dmemfs/Kconfig @@ -1,5 +1,8 @@ config DMEM_FS tristate "Direct Memory filesystem support" + depends on DMEM + depends on TRANSPARENT_HUGEPAGE + depends on ARCH_HAS_PTE_DMEM help dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 71125a4..9e65694 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1157,6 +1157,13 @@ static inline int pud_trans_unstable(pud_t *pud) #endif } =20 +#ifndef CONFIG_ARCH_HAS_PTE_DMEM +static inline pmd_t pmd_mkdmem(pmd_t pmd) +{ + return pmd; +} +#endif + #ifndef pmd_read_atomic static inline pmd_t pmd_read_atomic(pmd_t *pmdp) { diff --git a/mm/Kconfig b/mm/Kconfig index 4dd8896..10fd7ff 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -794,6 +794,9 @@ config IDLE_PAGE_TRACKING config ARCH_HAS_PTE_DEVMAP bool =20 +config ARCH_HAS_PTE_DMEM + bool + config ZONE_DEVICE bool "Device memory (pmem, HMM, etc...) hotplug support" depends on MEMORY_HOTPLUG diff --git a/mm/dmem.c b/mm/dmem.c index 6992e57..2e61dbd 100644 --- a/mm/dmem.c +++ b/mm/dmem.c @@ -822,6 +822,56 @@ int dmem_alloc_init(unsigned long dpage_shift) } EXPORT_SYMBOL(dmem_alloc_pages_nodemask); =20 +/* Return a nodelist indicated for current node representing a mempolicy= */ +static int *policy_nodelist(struct mempolicy *policy) +{ + int nd =3D numa_node_id(); + + switch (policy->mode) { + case MPOL_PREFERRED: + if (!(policy->flags & MPOL_F_LOCAL)) + nd =3D policy->v.preferred_node; + break; + case MPOL_BIND: + if (unlikely(!node_isset(nd, policy->v.nodes))) + nd =3D first_node(policy->v.nodes); + break; + default: + WARN_ON(1); + } + return dmem_nodelist(nd); +} + +static nodemask_t *dmem_policy_nodemask(struct mempolicy *policy) +{ + if (unlikely(policy->mode =3D=3D MPOL_BIND) && + cpuset_nodemask_valid_mems_allowed(&policy->v.nodes)) + return &policy->v.nodes; + + return NULL; +} + +static void +get_mempolicy_nlist_and_nmask(struct mempolicy *pol, + struct vm_area_struct *vma, unsigned long addr, + int **nl, nodemask_t **nmask) +{ + if (pol->mode =3D=3D MPOL_INTERLEAVE) { + unsigned int nid; + + /* + * we use dpage_shift to interleave numa nodes although + * multiple dpages may be allocated + */ + nid =3D interleave_nid(pol, vma, addr, dmem_pool.dpage_shift); + *nl =3D dmem_nodelist(nid); + *nmask =3D NULL; + } else { + *nl =3D policy_nodelist(pol); + *nmask =3D dmem_policy_nodemask(pol); + } +} + /* * dmem_alloc_pages_vma - Allocate pages for a VMA. * @@ -830,6 +880,9 @@ int dmem_alloc_init(unsigned long dpage_shift) * @try_max: try to allocate @try_max dpages if possible * @result_nr: allocated dpage number returned to the caller * + * This function allocates pages from dmem pool and applies a NUMA polic= y + * associated with the VMA. + * * Return the physical address of the first dpage allocated from dmem * pool, or 0 on failure. The allocated dpage number is filled into * @result_nr @@ -839,13 +892,19 @@ int dmem_alloc_init(unsigned long dpage_shift) unsigned int try_max, unsigned int *result_nr) { phys_addr_t phys_addr; + struct mempolicy *pol; int *nl; + nodemask_t *nmask; unsigned int cpuset_mems_cookie; =20 retry_cpuset: - nl =3D dmem_nodelist(numa_node_id()); + pol =3D get_vma_policy(vma, addr); + cpuset_mems_cookie =3D read_mems_allowed_begin(); + + get_mempolicy_nlist_and_nmask(pol, vma, addr, &nl, &nmask); + mpol_cond_put(pol); =20 - phys_addr =3D dmem_alloc_pages_from_nodelist(nl, NULL, try_max, + phys_addr =3D dmem_alloc_pages_from_nodelist(nl, nmask, try_max, result_nr); if (unlikely(!phys_addr && read_mems_allowed_retry(cpuset_mems_cookie))= ) goto retry_cpuset; --=20 1.8.3.1