From: Logan Gunthorpe <logang@deltatee.com>
To: Sagi Grimberg <sagi@grimberg.me>,
Stephen Bates <sbates@raithlin.com>,
Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Cc: Christoph Hellwig <hch@lst.de>,
"James E.J. Bottomley" <jejb@linux.vnet.ibm.com>,
"Martin K. Petersen" <martin.petersen@oracle.com>,
Jens Axboe <axboe@kernel.dk>,
Steve Wise <swise@opengridcomputing.com>,
Max Gurtovoy <maxg@mellanox.com>,
Dan Williams <dan.j.williams@intel.com>,
Keith Busch <keith.busch@intel.com>,
"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
"linux-nvdimm@ml01.01.org" <linux-nvdimm@ml01.01.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem
Date: Mon, 10 Apr 2017 10:03:45 -0600 [thread overview]
Message-ID: <aeeaeb7b-9cec-4cc0-5d4d-e26aa6c02f80@deltatee.com> (raw)
In-Reply-To: <7fcc3ac8-8b96-90f5-3942-87f999c7499d@grimberg.me>
On 10/04/17 02:29 AM, Sagi Grimberg wrote:
> What you are saying is surprising to me. The switch needs to preserve
> ordering across different switch ports ??
> You are suggesting that there is a *switch-wide* state that tracks
> MemRds never pass MemWrs across all the switch ports? That is a very
> non-trivial statement...
Yes, it is a requirement of the PCIe spec for transactions to be
strongly ordered throughout the fabric so switches must have an internal
state across all ports. Without that, it would be impossible to have PCI
cards work together even if they are using system memory to do so. Also,
I believe, it was done this way to maintain maximum compatibility with
the legacy PCI bus. There is also a relaxed ordering bit that allows
specific transactions to ignore ordering which can help performance.
Obviously this becomes impossible if you have some kind of complex
multi-path fabric.
Logan
next prev parent reply other threads:[~2017-04-10 16:04 UTC|newest]
Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-30 22:12 [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Logan Gunthorpe
2017-03-30 22:12 ` [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Logan Gunthorpe
2017-03-31 18:49 ` Sinan Kaya
2017-03-31 21:23 ` Logan Gunthorpe
2017-03-31 21:38 ` Sinan Kaya
2017-03-31 22:42 ` Logan Gunthorpe
2017-03-31 23:51 ` Sinan Kaya
2017-04-01 1:57 ` Logan Gunthorpe
2017-04-01 2:17 ` okaya
2017-04-01 22:16 ` Logan Gunthorpe
2017-04-02 2:26 ` Sinan Kaya
2017-04-02 17:21 ` Logan Gunthorpe
2017-04-02 21:03 ` Sinan Kaya
2017-04-03 4:26 ` Logan Gunthorpe
2017-04-25 11:58 ` Marta Rybczynska
2017-04-25 16:58 ` Logan Gunthorpe
2017-03-30 22:12 ` [RFC 2/8] cxgb4: setup pcie memory window 4 and create p2pmem region Logan Gunthorpe
2017-04-04 10:42 ` Sagi Grimberg
2017-04-04 15:56 ` Logan Gunthorpe
2017-04-05 15:41 ` Steve Wise
2017-03-30 22:12 ` [RFC 3/8] nvmet: Use p2pmem in nvme target Logan Gunthorpe
2017-04-04 10:40 ` Sagi Grimberg
2017-04-04 16:16 ` Logan Gunthorpe
2017-04-06 5:47 ` Sagi Grimberg
2017-04-06 15:52 ` Logan Gunthorpe
2017-03-30 22:12 ` [RFC 4/8] p2pmem: Add debugfs "stats" file Logan Gunthorpe
2017-04-04 10:46 ` Sagi Grimberg
2017-04-04 17:25 ` Logan Gunthorpe
2017-04-05 15:43 ` Steve Wise
2017-03-30 22:12 ` [RFC 5/8] scatterlist: Modify SG copy functions to support io memory Logan Gunthorpe
2017-03-31 7:09 ` Christoph Hellwig
2017-03-31 15:41 ` Logan Gunthorpe
2017-04-03 21:20 ` Logan Gunthorpe
2017-04-03 21:44 ` Dan Williams
2017-04-03 22:10 ` Logan Gunthorpe
2017-04-03 22:47 ` Dan Williams
2017-04-03 23:12 ` Logan Gunthorpe
2017-04-04 0:07 ` Dan Williams
2017-04-07 17:59 ` Logan Gunthorpe
2017-03-30 22:12 ` [RFC 6/8] nvmet: Be careful about using iomem accesses when dealing with p2pmem Logan Gunthorpe
2017-04-04 10:59 ` Sagi Grimberg
2017-04-04 15:46 ` Jason Gunthorpe
2017-04-04 17:21 ` Logan Gunthorpe
2017-04-06 5:33 ` Sagi Grimberg
2017-04-06 16:02 ` Logan Gunthorpe
2017-04-06 16:35 ` Jason Gunthorpe
2017-04-07 11:19 ` Stephen Bates
2017-04-10 8:29 ` Sagi Grimberg
2017-04-10 16:03 ` Logan Gunthorpe [this message]
2017-03-30 22:12 ` [RFC 7/8] p2pmem: Support device removal Logan Gunthorpe
2017-03-30 22:12 ` [RFC 8/8] p2pmem: Added char device user interface Logan Gunthorpe
2017-04-12 5:22 ` [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Benjamin Herrenschmidt
2017-04-12 17:09 ` Logan Gunthorpe
2017-04-12 21:55 ` Benjamin Herrenschmidt
2017-04-13 21:22 ` Logan Gunthorpe
2017-04-13 22:37 ` Benjamin Herrenschmidt
2017-04-13 23:26 ` Bjorn Helgaas
2017-04-14 4:16 ` Jason Gunthorpe
2017-04-14 4:40 ` Logan Gunthorpe
2017-04-14 11:37 ` Benjamin Herrenschmidt
2017-04-14 11:39 ` Benjamin Herrenschmidt
2017-04-14 11:37 ` Benjamin Herrenschmidt
2017-04-14 17:30 ` Logan Gunthorpe
2017-04-14 19:04 ` Bjorn Helgaas
2017-04-14 22:07 ` Benjamin Herrenschmidt
2017-04-15 17:41 ` Logan Gunthorpe
2017-04-15 22:09 ` Dan Williams
2017-04-16 3:01 ` Benjamin Herrenschmidt
2017-04-16 4:46 ` Logan Gunthorpe
2017-04-16 15:53 ` Dan Williams
2017-04-16 16:34 ` Logan Gunthorpe
2017-04-16 22:31 ` Benjamin Herrenschmidt
2017-04-24 7:36 ` Knut Omang
2017-04-24 16:14 ` Logan Gunthorpe
2017-04-25 6:30 ` Knut Omang
2017-04-25 17:03 ` Logan Gunthorpe
2017-04-25 21:23 ` Stephen Bates
2017-04-25 21:23 ` Stephen Bates
2017-04-16 22:26 ` Benjamin Herrenschmidt
2017-04-15 22:17 ` Benjamin Herrenschmidt
2017-04-16 5:36 ` Logan Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=aeeaeb7b-9cec-4cc0-5d4d-e26aa6c02f80@deltatee.com \
--to=logang@deltatee.com \
--cc=axboe@kernel.dk \
--cc=dan.j.williams@intel.com \
--cc=hch@lst.de \
--cc=jejb@linux.vnet.ibm.com \
--cc=jgunthorpe@obsidianresearch.com \
--cc=keith.busch@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvdimm@ml01.01.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=maxg@mellanox.com \
--cc=sagi@grimberg.me \
--cc=sbates@raithlin.com \
--cc=swise@opengridcomputing.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).