From mboxrd@z Thu Jan 1 00:00:00 1970 From: Logan Gunthorpe Subject: Re: [RFC PATCH 00/14] Heterogeneous Memory System (HMS) and hbind() Date: Thu, 6 Dec 2018 16:28:44 -0700 Message-ID: <935fc14d-91f2-bc2a-f8b5-665e4145e148@deltatee.com> References: <20181205001544.GR2937@redhat.com> <42006749-7912-1e97-8ccd-945e82cebdde@intel.com> <20181205021334.GB3045@redhat.com> <20181205175357.GG3536@redhat.com> <20181206192050.GC3544@redhat.com> <20181206223935.GG3544@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Content-Language: en-CA Sender: linux-kernel-owner@vger.kernel.org To: Dave Hansen , Jerome Glisse Cc: linux-mm@kvack.org, Andrew Morton , linux-kernel@vger.kernel.org, "Rafael J . Wysocki" , Matthew Wilcox , Ross Zwisler , Keith Busch , Dan Williams , Haggai Eran , Balbir Singh , "Aneesh Kumar K . V" , Benjamin Herrenschmidt , Felix Kuehling , Philip Yang , =?UTF-8?Q?Christian_K=c3=b6nig?= , Paul Blinzer , John Hubbard , Ralph Campbell , Michal Hocko List-Id: linux-acpi@vger.kernel.org On 2018-12-06 4:09 p.m., Dave Hansen wrote: > This looks great. But, we don't _have_ this kind of information for any > system that I know about or any system available in the near future. > > We basically have two different world views: > 1. The system is described point-to-point. A connects to B @ > 100GB/s. B connects to C at 50GB/s. Thus, C->A should be > 50GB/s. > * Less information to convey > * Potentially less precise if the properties are not perfectly > additive. If A->B=10ns and B->C=20ns, A->C might be >30ns. > * Costs must be calculated instead of being explicitly specified > 2. The system is described endpoint-to-endpoint. A->B @ 100GB/s > B->C @ 50GB/s, A->C @ 50GB/s. > * A *lot* more information to convey O(N^2)? > * Potentially more precise. > * Costs are explicitly specified, not calculated > > These patches are really tied to world view #1. But, the HMAT is really > tied to world view #1. I didn't think this was meant to describe actual real world performance between all of the links. If that's the case all of this seems like a pipe dream to me. Attributes like cache coherency, atomics, etc should fit well in world view #1... and, at best, some kind of flag saying whether or not to use a particular link if you care about transfer speed. -- But we don't need special "link" directories to describe the properties of existing buses. You're not *really* going to know bandwidth or latency for any of this unless you actually measure it on the system in question. Logan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDE93C04EB8 for ; Thu, 6 Dec 2018 23:29:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7BB1F20878 for ; Thu, 6 Dec 2018 23:29:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7BB1F20878 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=deltatee.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726236AbeLFX3d (ORCPT ); Thu, 6 Dec 2018 18:29:33 -0500 Received: from ale.deltatee.com ([207.54.116.67]:47074 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726134AbeLFX3d (ORCPT ); Thu, 6 Dec 2018 18:29:33 -0500 Received: from guinness.priv.deltatee.com ([172.16.1.162]) by ale.deltatee.com with esmtp (Exim 4.89) (envelope-from ) id 1gV34j-0006Wc-Cs; Thu, 06 Dec 2018 16:28:58 -0700 To: Dave Hansen , Jerome Glisse Cc: linux-mm@kvack.org, Andrew Morton , linux-kernel@vger.kernel.org, "Rafael J . Wysocki" , Matthew Wilcox , Ross Zwisler , Keith Busch , Dan Williams , Haggai Eran , Balbir Singh , "Aneesh Kumar K . V" , Benjamin Herrenschmidt , Felix Kuehling , Philip Yang , =?UTF-8?Q?Christian_K=c3=b6nig?= , Paul Blinzer , John Hubbard , Ralph Campbell , Michal Hocko , Jonathan Cameron , Mark Hairgrove , Vivek Kini , Mel Gorman , Dave Airlie , Ben Skeggs , Andrea Arcangeli , Rik van Riel , Ben Woodard , linux-acpi@vger.kernel.org References: <20181205001544.GR2937@redhat.com> <42006749-7912-1e97-8ccd-945e82cebdde@intel.com> <20181205021334.GB3045@redhat.com> <20181205175357.GG3536@redhat.com> <20181206192050.GC3544@redhat.com> <20181206223935.GG3544@redhat.com> From: Logan Gunthorpe Message-ID: <935fc14d-91f2-bc2a-f8b5-665e4145e148@deltatee.com> Date: Thu, 6 Dec 2018 16:28:44 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-CA Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: linux-acpi@vger.kernel.org, woodard@redhat.com, riel@surriel.com, aarcange@redhat.com, bskeggs@redhat.com, airlied@redhat.com, mgorman@techsingularity.net, vkini@nvidia.com, mhairgrove@nvidia.com, jonathan.cameron@huawei.com, mhocko@kernel.org, rcampbell@nvidia.com, jhubbard@nvidia.com, Paul.Blinzer@amd.com, christian.koenig@amd.com, Philip.Yang@amd.com, felix.kuehling@amd.com, benh@kernel.crashing.org, aneesh.kumar@linux.ibm.com, bsingharora@gmail.com, haggaie@mellanox.com, dan.j.williams@intel.com, keith.busch@intel.com, ross.zwisler@linux.intel.com, willy@infradead.org, rafael@kernel.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, linux-mm@kvack.org, jglisse@redhat.com, dave.hansen@intel.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [RFC PATCH 00/14] Heterogeneous Memory System (HMS) and hbind() X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2018-12-06 4:09 p.m., Dave Hansen wrote: > This looks great. But, we don't _have_ this kind of information for any > system that I know about or any system available in the near future. > > We basically have two different world views: > 1. The system is described point-to-point. A connects to B @ > 100GB/s. B connects to C at 50GB/s. Thus, C->A should be > 50GB/s. > * Less information to convey > * Potentially less precise if the properties are not perfectly > additive. If A->B=10ns and B->C=20ns, A->C might be >30ns. > * Costs must be calculated instead of being explicitly specified > 2. The system is described endpoint-to-endpoint. A->B @ 100GB/s > B->C @ 50GB/s, A->C @ 50GB/s. > * A *lot* more information to convey O(N^2)? > * Potentially more precise. > * Costs are explicitly specified, not calculated > > These patches are really tied to world view #1. But, the HMAT is really > tied to world view #1. I didn't think this was meant to describe actual real world performance between all of the links. If that's the case all of this seems like a pipe dream to me. Attributes like cache coherency, atomics, etc should fit well in world view #1... and, at best, some kind of flag saying whether or not to use a particular link if you care about transfer speed. -- But we don't need special "link" directories to describe the properties of existing buses. You're not *really* going to know bandwidth or latency for any of this unless you actually measure it on the system in question. Logan