From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94D3EC433F5 for ; Fri, 31 Aug 2018 15:51:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5735020839 for ; Fri, 31 Aug 2018 15:51:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5735020839 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=deltatee.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729219AbeHaT71 (ORCPT ); Fri, 31 Aug 2018 15:59:27 -0400 Received: from ale.deltatee.com ([207.54.116.67]:46966 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727345AbeHaT71 (ORCPT ); Fri, 31 Aug 2018 15:59:27 -0400 Received: from guinness.priv.deltatee.com ([172.16.1.162]) by ale.deltatee.com with esmtp (Exim 4.89) (envelope-from ) id 1fvlhT-0004Dw-Gd; Fri, 31 Aug 2018 09:51:08 -0600 To: =?UTF-8?Q?Christian_K=c3=b6nig?= , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org Cc: Stephen Bates , Christoph Hellwig , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt , Alex Williamson , Jonathan Corbet References: <20180830185352.3369-1-logang@deltatee.com> <20180830185352.3369-7-logang@deltatee.com> <98bff500-4e4c-3a34-6762-16ef4b076d90@amd.com> From: Logan Gunthorpe Message-ID: <6820d451-9591-1979-00bf-60c65fc4f136@deltatee.com> Date: Fri, 31 Aug 2018 09:51:06 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <98bff500-4e4c-3a34-6762-16ef4b076d90@amd.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: corbet@lwn.net, alex.williamson@redhat.com, benh@kernel.crashing.org, jglisse@redhat.com, dan.j.williams@intel.com, maxg@mellanox.com, jgg@mellanox.com, bhelgaas@google.com, sagi@grimberg.me, keith.busch@intel.com, hch@lst.de, sbates@raithlin.com, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, christian.koenig@amd.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v5 06/13] PCI/P2PDMA: Add P2P DMA driver writer's documentation X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 31/08/18 02:08 AM, Christian König wrote: >> +One of the biggest issues is that PCI doesn't require forwarding >> +transactions between hierarchy domains, and in PCIe, each Root Port >> +defines a separate hierarchy domain. To make things worse, there is no >> +simple way to determine if a given Root Complex supports this or not. >> +(See PCIe r4.0, sec 1.3.1). Therefore, as of this writing, the kernel >> +only supports doing P2P when the endpoints involved are all behind the >> +same PCI bridge, as such devices are all in the same PCI hierarchy >> +domain, and the spec guarantees that all transacations within the >> +hierarchy will be routable, but it does not require routing >> +between hierarchies. > > Can we add a kernel command line switch and a whitelist to enable P2P > between separate hierarchies? In future work, yes. But not for this patchset. This is definitely the way I see things going, but we've chosen to start with what we've presented. Logan