linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Logan Gunthorpe <logang@deltatee.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: "Stephen Bates" <sbates@raithlin.com>,
	"Sinan Kaya" <okaya@codeaurora.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"linux-nvdimm@lists.01.org" <linux-nvdimm@lists.01.org>,
	"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
	"Christoph Hellwig" <hch@lst.de>, "Jens Axboe" <axboe@kernel.dk>,
	"Keith Busch" <keith.busch@intel.com>,
	"Sagi Grimberg" <sagi@grimberg.me>,
	"Bjorn Helgaas" <bhelgaas@google.com>,
	"Jason Gunthorpe" <jgg@mellanox.com>,
	"Max Gurtovoy" <maxg@mellanox.com>,
	"Dan Williams" <dan.j.williams@intel.com>,
	"Jérôme Glisse" <jglisse@redhat.com>,
	"Benjamin Herrenschmidt" <benh@kernel.crashing.org>,
	"Alex Williamson" <alex.williamson@redhat.com>
Subject: Re: [PATCH v3 01/11] PCI/P2PDMA: Support peer-to-peer memory
Date: Wed, 14 Mar 2018 10:17:34 -0600	[thread overview]
Message-ID: <112493af-ccd0-455b-6600-b50764f7ab7e@deltatee.com> (raw)
In-Reply-To: <20180314025639.GA50067@bhelgaas-glaptop.roam.corp.google.com>



On 13/03/18 08:56 PM, Bjorn Helgaas wrote:
> I assume you want to exclude Root Ports because of multi-function
> devices and the "route to self" error.  I was hoping for a reference
> to that so I could learn more about it.

I haven't been able to find where in the spec it forbids route to self.
But I was told this by developers who work know switches. Hopefully
Stephen can find the reference.

But it's a bit of a moot point. Devices can DMA to themselves if they
are designed to do so. For example, some NVMe cards can read and write
their own CMB for certain types of DMA request. There is a register in
the spec (CMBSZ) which specifies which types of requests are supported.
(See 3.1.12 in NVMe 1.3a).

> I agree that peers need to have a common upstream bridge.  I think
> you're saying peers need to have *two* common upstream bridges.  If I
> understand correctly, requiring two common bridges is a way to ensure
> that peers directly below Root Ports don't try to DMA to each other.

No, I don't get where you think we need to have two common upstream
bridges. I'm not sure when such a case would ever happen. But you seem
to understand based on what you wrote below.

> So I guess the first order of business is to nail down whether peers
> below a Root Port are prohibited from DMAing to each other.  My
> assumption, based on 6.12.1.2 and the fact that I haven't yet found
> a prohibition, is that they can.

If you have a multifunction device designed to DMA to itself below a
root port, it can. But determining this is on a device by device basis,
just as determining whether a root complex can do peer to peer is on a
per device basis. So I'd say we don't want to allow it by default and
let someone who has such a device figure out what's necessary if and
when one comes along.

> You already have upstream_bridges_match(), which takes two pci_devs.
> I think it should walk up the PCI hierarchy from the first device,
> checking whether the bridge at each level is also a parent of the
> second device.

Yes, this is what I meant when I said walking the entire tree. I've been
kicking the can down the road on implementing this as getting ref
counting right and testing it is going to be quite tricky. The single
switch approach we implemented now is just a simplification which works
for a single switch. But I guess we can look at implementing it this way
for v4.

Logan

  parent reply	other threads:[~2018-03-14 16:18 UTC|newest]

Thread overview: 68+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-03-12 19:35 [PATCH v3 00/11] Copy Offload in NVMe Fabrics with P2P PCI Memory Logan Gunthorpe
2018-03-12 19:35 ` [PATCH v3 01/11] PCI/P2PDMA: Support peer-to-peer memory Logan Gunthorpe
2018-03-13  3:28   ` Sinan Kaya
2018-03-13 16:43     ` Logan Gunthorpe
2018-03-13 17:49       ` Sinan Kaya
2018-03-13 18:44         ` Logan Gunthorpe
2018-03-13 19:10           ` Sinan Kaya
2018-03-13 19:19             ` Logan Gunthorpe
2018-03-13 19:53               ` Sinan Kaya
2018-03-13 20:46                 ` Logan Gunthorpe
2018-03-13 21:22                   ` Sinan Kaya
2018-03-13 22:00                     ` Logan Gunthorpe
2018-03-13 22:29                       ` Sinan Kaya
2018-03-13 22:45                         ` Stephen  Bates
2018-03-13 22:48                         ` Logan Gunthorpe
2018-03-13 23:19                           ` Sinan Kaya
2018-03-13 23:45                             ` Logan Gunthorpe
2018-03-14 12:16                               ` David Laight
2018-03-14 16:23                                 ` Logan Gunthorpe
2018-03-13 22:31                       ` Stephen  Bates
2018-03-13 23:08                         ` Bjorn Helgaas
2018-03-13 23:21                           ` Logan Gunthorpe
2018-03-14  2:56                             ` Bjorn Helgaas
2018-03-14 14:05                               ` Stephen  Bates
2018-03-14 16:17                               ` Logan Gunthorpe [this message]
2018-03-14 18:51                                 ` Bjorn Helgaas
2018-03-14 19:03                                   ` Logan Gunthorpe
2018-03-14 19:28                                     ` Dan Williams
2018-03-14 19:30                                       ` Logan Gunthorpe
2018-03-14 19:34                                       ` Stephen  Bates
2018-03-15  4:00                                         ` Martin K. Petersen
2018-03-15  4:30                                         ` Dan Williams
2018-03-22 22:57                           ` Stephen  Bates
2018-03-23 21:50                             ` Bjorn Helgaas
2018-03-23 21:59                               ` Logan Gunthorpe
2018-03-24  3:49                                 ` Bjorn Helgaas
2018-03-24 15:28                                   ` Stephen  Bates
2018-03-26 15:43                                     ` Logan Gunthorpe
2018-03-26 11:11       ` Jonathan Cameron
2018-03-26 14:01         ` Bjorn Helgaas
2018-03-26 15:46           ` Logan Gunthorpe
2018-03-27  8:47             ` Jonathan Cameron
2018-03-27 15:37               ` Logan Gunthorpe
2018-04-13 21:56               ` Stephen  Bates
2018-03-26 16:41         ` Jason Gunthorpe
2018-03-26 17:30           ` Logan Gunthorpe
2018-03-26 19:35             ` Jason Gunthorpe
2018-03-26 20:42               ` Logan Gunthorpe
2018-03-13 18:40     ` Logan Gunthorpe
2018-03-12 19:35 ` [PATCH v3 02/11] PCI/P2PDMA: Add sysfs group to display p2pmem stats Logan Gunthorpe
2018-03-12 19:35 ` [PATCH v3 03/11] PCI/P2PDMA: Add PCI p2pmem dma mappings to adjust the bus offset Logan Gunthorpe
2018-03-12 19:35 ` [PATCH v3 04/11] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Logan Gunthorpe
2018-03-12 19:35 ` [PATCH v3 05/11] PCI/P2PDMA: Add P2P DMA driver writer's documentation Logan Gunthorpe
2018-03-12 19:41   ` Jonathan Corbet
2018-03-12 21:18     ` Logan Gunthorpe
2018-03-12 19:35 ` [PATCH v3 06/11] block: Introduce PCI P2P flags for request and request queue Logan Gunthorpe
2018-03-21  9:27   ` Christoph Hellwig
2018-03-12 19:35 ` [PATCH v3 07/11] IB/core: Ensure we map P2P memory correctly in rdma_rw_ctx_[init|destroy]() Logan Gunthorpe
2018-03-21  9:27   ` Christoph Hellwig
2018-03-12 19:35 ` [PATCH v3 08/11] nvme-pci: Use PCI p2pmem subsystem to manage the CMB Logan Gunthorpe
2018-03-13  1:55   ` Sinan Kaya
2018-03-13  1:58     ` Sinan Kaya
2018-03-12 19:35 ` [PATCH v3 09/11] nvme-pci: Add support for P2P memory in requests Logan Gunthorpe
2018-03-21  9:23   ` Christoph Hellwig
2018-03-12 19:35 ` [PATCH v3 10/11] nvme-pci: Add a quirk for a pseudo CMB Logan Gunthorpe
2018-03-12 19:35 ` [PATCH v3 11/11] nvmet: Optionally use PCI P2P memory Logan Gunthorpe
2018-03-21  9:27   ` Christoph Hellwig
2018-03-21 16:52     ` Logan Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=112493af-ccd0-455b-6600-b50764f7ab7e@deltatee.com \
    --to=logang@deltatee.com \
    --cc=alex.williamson@redhat.com \
    --cc=axboe@kernel.dk \
    --cc=benh@kernel.crashing.org \
    --cc=bhelgaas@google.com \
    --cc=dan.j.williams@intel.com \
    --cc=hch@lst.de \
    --cc=helgaas@kernel.org \
    --cc=jgg@mellanox.com \
    --cc=jglisse@redhat.com \
    --cc=keith.busch@intel.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nvdimm@lists.01.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=maxg@mellanox.com \
    --cc=okaya@codeaurora.org \
    --cc=sagi@grimberg.me \
    --cc=sbates@raithlin.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).