From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8464AC433EF for ; Tue, 19 Oct 2021 09:38:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 223ED611BF for ; Tue, 19 Oct 2021 09:38:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 223ED611BF Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id AF4926B006C; Tue, 19 Oct 2021 05:38:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AA5336B0071; Tue, 19 Oct 2021 05:38:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96DA6900002; Tue, 19 Oct 2021 05:38:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0139.hostedemail.com [216.40.44.139]) by kanga.kvack.org (Postfix) with ESMTP id 8963C6B006C for ; Tue, 19 Oct 2021 05:38:03 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4BD70180D870D for ; Tue, 19 Oct 2021 09:38:03 +0000 (UTC) X-FDA: 78712685646.11.E801283 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf26.hostedemail.com (Postfix) with ESMTP id B279620019C4 for ; Tue, 19 Oct 2021 09:38:03 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 95AD1219CC; Tue, 19 Oct 2021 09:38:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1634636281; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=wZDfGwlKY24k3bNyr+fz0tauAszGxMWggH9lo6OoW6E=; b=Nj676bDeubtY502YsO0DeoS4ZzdRb9E+FqlLkM9NbU3UP2WIZu1NBUqTfnwc0HxJK9jc5g oYr1FxnlK528UyGaqba5m4BA1qDgQbY5K57BbP9ZSdVSlgFUEpI75jrVO1jqDiZvPt2LAU FZKWgzZI66m5NJuO0RGuEzGpj3UcXeI= Received: from suse.cz (unknown [10.100.201.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 62889A3B81; Tue, 19 Oct 2021 09:38:01 +0000 (UTC) Date: Tue, 19 Oct 2021 11:38:00 +0200 From: Michal Hocko To: "Aneesh Kumar K.V" Cc: Andi Kleen , linux-mm@kvack.org, akpm@linux-foundation.org, Ben Widawsky , Dave Hansen , Feng Tang , Andrea Arcangeli , Mel Gorman , Mike Kravetz , Randy Dunlap , Vlastimil Babka , Dan Williams , Huang Ying Subject: Re: [RFC PATCH] mm/mempolicy: add MPOL_PREFERRED_STRICT memory policy Message-ID: References: <83483424-e617-51c4-d55c-6106e66e2659@linux.intel.com> <87pms8ymvl.fsf@linux.ibm.com> <49514c97-c540-48ee-0b2f-3cd7bd3dfcf9@linux.ibm.com> <249414f6-1bb7-b76c-5b5b-2b3ed8937d7b@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <249414f6-1bb7-b76c-5b5b-2b3ed8937d7b@linux.ibm.com> X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B279620019C4 X-Stat-Signature: t6pr4cqyg31zei1rpyjc88ip8xecqqza Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=Nj676bDe; spf=pass (imf26.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com X-HE-Tag: 1634636283-4667 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu 14-10-21 21:20:51, Aneesh Kumar K.V wrote: > On 10/14/21 20:26, Michal Hocko wrote: > > On Thu 14-10-21 18:59:14, Aneesh Kumar K.V wrote: > > > On 10/14/21 17:11, Michal Hocko wrote: > > > > On Thu 14-10-21 15:58:29, Aneesh Kumar K.V wrote: > > > > > On 10/14/21 15:08, Michal Hocko wrote: > > > > [...] > > > > > > Besides that it would be really great to finish the discussion about the > > > > > > usecase before suggesting a new userspace API. > > > > > > > > > > > > > > > > Application would like to hint a preferred node for allocating memory > > > > > backing a va range and at the same time wants to avoid fallback to some set > > > > > of nodes (in the use case I am interested don't fall back to slow memory > > > > > nodes). > > > > > > > > We do have means for that, right? You can set your memory policy and > > > > then set the cpu afffinity to the node you want to allocate from > > > > initially. You can migrate to a different cpu/node if this is not the > > > > preferred affinity. Why is that not usable? > > > > > > For the same reason you mentioned earlier, these nodes can be cpu less > > > nodes. > > > > It would have been easier if you were explicit about the usecase rather > > than let other guess. > > > > > > Also think about extensibility. Say I want to allocate from a set of > > > > nodes first before falling back to the rest of the nodemask? If you want > > > > to add a new API then think of other potential usecases. > > > > > > > > > > Describing the specific allocation details become hard with preferred node > > > being a nodemask. With the below interface > > > > > > SYSCALL_DEFINE5(preferred_mbind, unsigned long, start, unsigned long, len, > > > const unsigned long __user *, preferred_nmask, const unsigned long __user > > > *, fallback_nmask, > > > unsigned long, maxnode) > > > { > > > > > > > > > 1. The preferred node is the first node in the preferred node mask > > > 2. Then we try to allocate from nodes present in the preferred node mask > > > which is closer to the first node in the preferred node mask > > > 3. If the above fails, we try to allocate from nodes in the fallback node > > > mask which is closer to the first node in the preferred nodemask. > > > > > > Isn't that too complicated? Do we have a real usecase for that? > > > > No, I think this is a suboptimal interface. AFAIU you really want to > > define a "home" node(s) rather than any policy. Home node would > > effectively override the default local node whatever policy you have as > > it makes sense whether you have MPOL_PREFERRED_MANY or MPOL_BIND. > > > > > yes. I did describe it as below in an earlier email > > "We could do > set_mempolicy(MPOLD_PREFERRED, nodemask(nodeX))) > set_mempolicy(MPOLD_PREFFERED_EXTEND, nodemask(fallback nodemask for above > PREFERRED policy)) " > > But I agree that restricting this to virtual address range is much better. > Now I am wondering whether a nodemask is any better than a nodeid. The > concept of home nodes is confusing when compared to home node. > What would be the meaning of multiple nodes in a home nodes concept? If you go with a nodemask then I expect we will hit ordering requirement very quickly. A single home node for a range makes some sense to me for the cpu less nodes. I do not see why somebody might require them to be the first one to consider but I can imagine there might be some (semi)reasonable usecases out there. In any case, implementation wise this shouldn't be really restricted to any specific memorly policy and only override the local node when we use it currently. > Should we do > > SYSCALL_DEFINE4(home_node_mbind, unsigned long, start, unsigned long, len, > unsigned long, home_node, unsigned long, flags) > > > the flags is kept for future extension if any. > > > I guess this home node will only apply w.r.t MPOL_BIND and > MPOL_PREFFERED_MANY policy for now? Why to constrain that artificially. Interleaving has to start somewhere as well, right? Not that it matters much in practice as only the first allocation would be affected. > > Another potential interface would be set_nodeorder which would > > explicitly set the allocation fallback ordering. Again agnostic of the > > underlying memory policy. This would be more generic but the question is > > whether this is not too generic and whether there are usecases for that. > > > > I would suggest we wait for applications really wanting a fallback order > other than distance based one before adding this. Distance based fallback > order from a preferred node is well understood from application point of > view. Right, I am not pushing into that direction. The idea was that unlike home node this has more potential extensibility as a single home node cannot capture preferences for more nodes for example. -- Michal Hocko SUSE Labs