From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D6FBC04EB9 for ; Wed, 5 Dec 2018 17:42:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3DBBA20879 for ; Wed, 5 Dec 2018 17:42:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3DBBA20879 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=deltatee.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728152AbeLERmO (ORCPT ); Wed, 5 Dec 2018 12:42:14 -0500 Received: from ale.deltatee.com ([207.54.116.67]:34122 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727514AbeLERmN (ORCPT ); Wed, 5 Dec 2018 12:42:13 -0500 Received: from guinness.priv.deltatee.com ([172.16.1.162]) by ale.deltatee.com with esmtp (Exim 4.89) (envelope-from ) id 1gUbBP-0008HY-2X; Wed, 05 Dec 2018 10:42:00 -0700 To: Jerome Glisse Cc: Dan Williams , Andi Kleen , Linux MM , Andrew Morton , Linux Kernel Mailing List , "Rafael J. Wysocki" , Dave Hansen , Haggai Eran , balbirs@au1.ibm.com, "Aneesh Kumar K.V" , Benjamin Herrenschmidt , "Kuehling, Felix" , Philip.Yang@amd.com, "Koenig, Christian" , "Blinzer, Paul" , John Hubbard , rcampbell@nvidia.com References: <20181204192221.GG2937@redhat.com> <20181204201347.GK2937@redhat.com> <2f146730-1bf9-db75-911d-67809fc7afef@deltatee.com> <20181204205902.GM2937@redhat.com> <20181204215146.GO2937@redhat.com> <20181204235630.GQ2937@redhat.com> <20181205023116.GD3045@redhat.com> From: Logan Gunthorpe Message-ID: Date: Wed, 5 Dec 2018 10:41:56 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: <20181205023116.GD3045@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-CA Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: rcampbell@nvidia.com, jhubbard@nvidia.com, Paul.Blinzer@amd.com, christian.koenig@amd.com, Philip.Yang@amd.com, felix.kuehling@amd.com, benh@kernel.crashing.org, aneesh.kumar@linux.ibm.com, balbirs@au1.ibm.com, haggaie@mellanox.com, dave.hansen@intel.com, rafael@kernel.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, linux-mm@kvack.org, ak@linux.intel.com, dan.j.williams@intel.com, jglisse@redhat.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [RFC PATCH 02/14] mm/hms: heterogenenous memory system (HMS) documentation X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2018-12-04 7:31 p.m., Jerome Glisse wrote: > How can i express multiple link, or memory that is only accessible > by a subset of the devices/CPUs. In today model they are back in > assumption like everyone can access all the node which do not hold > in what i am trying to do. Well multiple links are easy when you have a 'link' bus. Just add another link device under the bus. Technically, the accessibility issue is already encoded in sysfs. For example, through the PCI tree you can determine which ACS bits are set and determine which devices are behind the same root bridge the same way we do in the kernel p2pdma subsystem. This is all bus specific which is fine, but if we want to change that, we should have a common way for existing buses to describe these attributes in the existing tree. The new 'link' bus devices would have to have some way to describe cases if memory isn't accessible in some way across it. But really, I would say the kernel is responsible for telling you when memory is accessible to a list of initiators, so it should be part of the checks in a theoretical hbind api. This is already the approach p2pdma takes in-kernel: we have functions that tell you if two PCI devices can talk to each other and we have functions to give you memory accessible by a set of devices. What we don't have is a special tree that p2pdma users have to walk through to determine accessibility. In my eye's, you are just conflating a bunch of different issues that are better solved independently in the existing frameworks we have. And if they were tackled individually, you'd have a much easier time getting them merged one by one. Logan