From: 'Christoph Hellwig' <hch@infradead.org>
To: Tom Talpey <tom@talpey.com>
Cc: 'Christoph Hellwig' <hch@infradead.org>,
Jason Gunthorpe <jgunthorpe@obsidianresearch.com>,
Sagi Grimberg <sagig@dev.mellanox.co.il>,
Steve Wise <swise@opengridcomputing.com>,
dledford@redhat.com, sagig@mellanox.com, ogerlitz@mellanox.com,
roid@mellanox.com, linux-rdma@vger.kernel.org, eli@mellanox.com,
target-devel@vger.kernel.org, linux-nfs@vger.kernel.org,
trond.myklebust@primarydata.com, bfields@fieldses.org,
Oren Duer <oren@mellanox.com>
Subject: Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags
Date: Tue, 14 Jul 2015 00:37:56 -0700 [thread overview]
Message-ID: <20150714073756.GB7630@infradead.org> (raw)
In-Reply-To: <55A4134C.2040301@talpey.com>
On Mon, Jul 13, 2015 at 03:36:44PM -0400, Tom Talpey wrote:
> On 7/11/2015 6:25 AM, 'Christoph Hellwig' wrote:
> >I think what we need to support for now is FRMR as the primary target,
> >and FMR as a secondar[y].
>
> FMR is a *very* bad choice, for several reasons.
>
> 1) It's not supported by very many devices, in fact it might even
> be thought to be obsolete.
It's support by the Mellanox adapters, which might be a minority of
the drivers, but at least in the field I've been in the aboslute
majority of the deployed hardware.
It's the default in the SRP initiator and iSER initiators, and the
primary fallback in the NFS client if FRs aren't available.
I don't claim to be an expert on memory registrations models, as they
are horribly, horrible documents (baiscally not at all in the source
tree), so my knowledge is from looking at and using implementations as
well as this useful writeup from the NFS folks:
http://wiki.linux-nfs.org/wiki/index.php/NfsRdmaClient/MemRegModes
Which misses or downplays an important restriction of PHYS
registrations: They dont allow coalescing non-contiguous memory
into a single maping, which makes them totally unsuitable for iSER
which only allows a single rkey/offset/len pair and requires
additional RDMA READ/WRITE roundtrips and larger S/G lists for other
protocols.
Based on looking at the consumers and the above table I think FMR
are still the better fallback for devices that don't support FR,
but supporting PHYS MRs is easy enough that adding it to a common
layer seems easy enough if some cares enough to regularly test it.
next prev parent reply other threads:[~2015-07-14 7:37 UTC|newest]
Thread overview: 223+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-07-05 23:21 [PATCH V3 0/5] Transport-independent MRs Steve Wise
2015-07-05 23:22 ` [PATCH V3 1/5] RDMA/core: Transport-independent access flags Steve Wise
2015-07-06 5:25 ` Christoph Hellwig
2015-07-06 14:23 ` Steve Wise
2015-07-06 14:23 ` Steve Wise
2015-07-07 8:58 ` 'Christoph Hellwig'
[not found] ` <20150705232158.12029.25472.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-06 7:09 ` Haggai Eran
2015-07-06 7:09 ` Haggai Eran
[not found] ` <559A2991.3040304-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-07-06 14:29 ` Steve Wise
2015-07-06 14:29 ` Steve Wise
2015-07-07 14:17 ` Steve Wise
2015-07-07 14:17 ` Steve Wise
2015-07-07 14:34 ` Haggai Eran
2015-07-07 14:34 ` Haggai Eran
2015-07-07 14:46 ` Steve Wise
2015-07-07 14:46 ` Steve Wise
2015-07-07 15:07 ` Haggai Eran
2015-07-07 15:07 ` Haggai Eran
2015-07-06 7:53 ` Sagi Grimberg
2015-07-06 7:53 ` Sagi Grimberg
2015-07-06 14:37 ` Steve Wise
2015-07-06 14:37 ` Steve Wise
2015-07-06 16:17 ` Sagi Grimberg
2015-07-06 16:17 ` Sagi Grimberg
2015-07-06 16:55 ` Steve Wise
2015-07-06 16:55 ` Steve Wise
2015-07-07 9:00 ` Christoph Hellwig
[not found] ` <20150707090001.GB11736-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-07 9:14 ` Sagi Grimberg
2015-07-07 9:14 ` Sagi Grimberg
[not found] ` <559B9891.8060907-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-07 14:05 ` Steve Wise
2015-07-07 14:05 ` Steve Wise
2015-07-07 16:17 ` Jason Gunthorpe
2015-07-07 16:17 ` Jason Gunthorpe
[not found] ` <20150707161751.GA623-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-07 16:27 ` Sagi Grimberg
2015-07-07 16:27 ` Sagi Grimberg
[not found] ` <559BFE03.4020709-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-07 21:36 ` Jason Gunthorpe
2015-07-07 21:36 ` Jason Gunthorpe
2015-07-08 7:29 ` Sagi Grimberg
[not found] ` <559CD174.4040901-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-08 8:13 ` 'Christoph Hellwig'
2015-07-08 8:13 ` 'Christoph Hellwig'
[not found] ` <20150708081320.GB24203-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-08 10:05 ` Sagi Grimberg
2015-07-08 10:05 ` Sagi Grimberg
[not found] ` <559CF5E8.6080000-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-08 10:20 ` 'Christoph Hellwig'
2015-07-08 10:20 ` 'Christoph Hellwig'
[not found] ` <20150708102035.GA28421-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-08 11:08 ` Sagi Grimberg
2015-07-08 11:08 ` Sagi Grimberg
[not found] ` <559D0498.9050809-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-08 17:14 ` Hefty, Sean
2015-07-08 17:14 ` Hefty, Sean
2015-07-09 8:46 ` Sagi Grimberg
2015-07-09 13:52 ` Chuck Lever
2015-07-10 19:34 ` Christoph Hellwig
2015-07-12 7:49 ` Sagi Grimberg
2015-07-13 16:50 ` Jason Gunthorpe
2015-07-14 8:06 ` Sagi Grimberg
[not found] ` <55A4C2FA.9060707-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-14 12:24 ` Tom Talpey
2015-07-14 12:24 ` Tom Talpey
[not found] ` <55A4FF93.4090406-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org>
2015-07-14 13:21 ` Sagi Grimberg
2015-07-14 13:21 ` Sagi Grimberg
2015-07-23 0:43 ` Hefty, Sean
2015-07-23 0:43 ` Hefty, Sean
2015-07-08 19:08 ` Jason Gunthorpe
[not found] ` <20150708190842.GB11740-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-08 20:32 ` 'Christoph Hellwig'
2015-07-08 20:32 ` 'Christoph Hellwig'
[not found] ` <20150708203205.GA21847-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-08 20:37 ` 'Christoph Hellwig'
2015-07-08 20:37 ` 'Christoph Hellwig'
2015-07-09 8:47 ` Sagi Grimberg
2015-07-09 8:47 ` Sagi Grimberg
2015-07-09 0:03 ` Jason Gunthorpe
[not found] ` <20150709000337.GE16812-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-09 8:00 ` 'Christoph Hellwig'
2015-07-09 8:00 ` 'Christoph Hellwig'
2015-07-09 8:58 ` Sagi Grimberg
2015-07-09 8:58 ` Sagi Grimberg
2015-07-09 22:18 ` Doug Ledford
2015-07-09 22:18 ` Doug Ledford
2015-07-09 22:53 ` Jason Gunthorpe
2015-07-09 22:53 ` Jason Gunthorpe
2015-07-10 13:22 ` Tom Talpey
[not found] ` <559FC710.1050307-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org>
2015-07-10 16:11 ` Jason Gunthorpe
2015-07-10 16:11 ` Jason Gunthorpe
2015-07-10 17:56 ` Doug Ledford
2015-07-10 18:34 ` Chuck Lever
2015-07-10 18:42 ` Tom Talpey
[not found] ` <55A01225.9000000-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org>
2015-07-10 19:54 ` Jason Gunthorpe
2015-07-10 19:54 ` Jason Gunthorpe
2015-07-10 20:48 ` Jason Gunthorpe
[not found] ` <20150710195420.GA31500-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-10 22:33 ` Doug Ledford
2015-07-10 22:33 ` Doug Ledford
2015-07-11 10:17 ` 'Christoph Hellwig'
2015-07-11 10:17 ` 'Christoph Hellwig'
[not found] ` <20150711101736.GA14741-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-13 16:57 ` Jason Gunthorpe
2015-07-13 16:57 ` Jason Gunthorpe
2015-07-14 7:25 ` 'Christoph Hellwig'
2015-07-14 9:05 ` Sagi Grimberg
2015-07-14 15:35 ` 'Christoph Hellwig'
2015-07-14 17:26 ` Jason Gunthorpe
[not found] ` <20150714172655.GB24403-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-15 7:10 ` Sagi Grimberg
2015-07-15 7:10 ` Sagi Grimberg
2015-07-10 22:30 ` Doug Ledford
2015-07-10 22:30 ` Doug Ledford
2015-07-10 20:57 ` Jason Gunthorpe
[not found] ` <20150710205706.GA7883-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-10 22:27 ` Doug Ledford
2015-07-10 22:27 ` Doug Ledford
[not found] ` <20150710233417.GA8919@obsidianresearch.com>
[not found] ` <20150710233417.GA8919-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-11 3:10 ` Doug Ledford
2015-07-11 3:10 ` Doug Ledford
[not found] ` <55A0891F.4050105-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2015-07-13 17:18 ` Jason Gunthorpe
2015-07-13 17:18 ` Jason Gunthorpe
2015-07-13 22:23 ` Tom Talpey
2015-07-11 16:37 ` Steve Wise
[not found] ` <CCB6E837-88FF-4DBB-976D-4CF56396A1A1-/Yg/VP3ZvrM@public.gmane.org>
2015-07-12 10:46 ` Sagi Grimberg
2015-07-12 10:46 ` Sagi Grimberg
[not found] ` <55A24571.60902-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-14 19:25 ` Steve Wise
2015-07-14 19:25 ` Steve Wise
2015-07-14 19:29 ` Jason Gunthorpe
[not found] ` <20150714192941.GA26292-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-14 19:32 ` Steve Wise
2015-07-14 19:32 ` Steve Wise
2015-07-14 19:37 ` Jason Gunthorpe
2015-07-14 19:55 ` 'Christoph Hellwig'
2015-07-14 19:55 ` 'Christoph Hellwig'
2015-07-14 20:10 ` Steve Wise
2015-07-14 20:10 ` Steve Wise
[not found] ` <20150714195511.GB7716-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-14 20:29 ` Jason Gunthorpe
2015-07-14 20:29 ` Jason Gunthorpe
2015-07-15 6:50 ` 'Christoph Hellwig'
[not found] ` <20150715065057.GA22113-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-15 19:12 ` Jason Gunthorpe
2015-07-15 19:12 ` Jason Gunthorpe
[not found] ` <20150715191257.GF23588-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-16 6:41 ` Jason Gunthorpe
2015-07-16 6:41 ` Jason Gunthorpe
2015-07-16 8:04 ` 'Christoph Hellwig'
2015-07-16 8:04 ` 'Christoph Hellwig'
[not found] ` <20150716080402.GC9093-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-16 16:13 ` Jason Gunthorpe
2015-07-16 16:13 ` Jason Gunthorpe
[not found] ` <20150714202943.GB26927-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-14 20:40 ` Steve Wise
2015-07-14 20:40 ` Steve Wise
2015-07-14 20:44 ` Jason Gunthorpe
2015-07-14 20:44 ` Jason Gunthorpe
[not found] ` <20150714204442.GD26927-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-14 20:54 ` Steve Wise
2015-07-14 20:54 ` Steve Wise
2015-07-14 20:59 ` Jason Gunthorpe
2015-07-14 20:59 ` Jason Gunthorpe
2015-07-14 20:50 ` Tom Talpey
2015-07-14 20:50 ` Tom Talpey
2015-07-15 8:47 ` Sagi Grimberg
2015-07-15 8:47 ` Sagi Grimberg
[not found] ` <55A61E38.20201-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-15 12:19 ` 'Christoph Hellwig'
2015-07-15 12:19 ` 'Christoph Hellwig'
[not found] ` <20150715121926.GB14993-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-15 19:17 ` Jason Gunthorpe
2015-07-15 19:17 ` Jason Gunthorpe
2015-07-14 20:46 ` Tom Talpey
2015-07-14 19:45 ` 'Christoph Hellwig'
[not found] ` <20150714194512.GA25887-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-14 19:57 ` Jason Gunthorpe
2015-07-14 19:57 ` Jason Gunthorpe
2015-07-14 19:58 ` Steve Wise
2015-07-14 19:58 ` Steve Wise
2015-07-14 20:41 ` Jason Gunthorpe
2015-07-14 20:41 ` Jason Gunthorpe
[not found] ` <20150714204145.GC26927-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-14 20:51 ` Steve Wise
2015-07-14 20:51 ` Steve Wise
2015-07-14 21:01 ` Steve Wise
2015-07-14 21:01 ` Steve Wise
2015-07-14 21:14 ` Jason Gunthorpe
2015-07-14 21:14 ` Jason Gunthorpe
2015-07-23 18:53 ` Hefty, Sean
2015-07-23 18:53 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A82373A901C9A5-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2015-07-23 19:03 ` Steve Wise
2015-07-23 19:03 ` Steve Wise
2015-07-23 23:30 ` Hefty, Sean
2015-07-23 23:30 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A82373A901DB80-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2015-07-23 23:53 ` Jason Gunthorpe
2015-07-23 23:53 ` Jason Gunthorpe
[not found] ` <20150723235310.GA20537-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-24 0:18 ` Hefty, Sean
2015-07-24 0:18 ` Hefty, Sean
2015-07-24 4:46 ` Jason Gunthorpe
2015-07-08 21:38 ` Tom Talpey
2015-07-08 23:36 ` Jason Gunthorpe
2015-07-09 11:02 ` Sagi Grimberg
[not found] ` <559E54AB.2010905-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-09 17:01 ` Jason Gunthorpe
2015-07-09 17:01 ` Jason Gunthorpe
2015-07-09 20:00 ` Tom Talpey
2015-07-09 21:16 ` Jason Gunthorpe
[not found] ` <20150709170142.GA21921-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-10 8:55 ` kernel memory registration (was: RDMA/core: Transport-independent access flags) Sagi Grimberg
[not found] ` <559F8881.7070308-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-10 16:35 ` Jason Gunthorpe
2015-07-11 10:31 ` 'Christoph Hellwig'
[not found] ` <20150711103153.GC14741-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-13 16:46 ` Jason Gunthorpe
[not found] ` <20150713164652.GC23832-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-14 8:24 ` kernel memory registration Sagi Grimberg
[not found] ` <55A4C73A.7080001-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-14 17:24 ` Jason Gunthorpe
2015-07-11 10:25 ` [PATCH V3 1/5] RDMA/core: Transport-independent access flags 'Christoph Hellwig'
2015-07-13 16:35 ` Jason Gunthorpe
2015-07-13 19:36 ` Tom Talpey
2015-07-13 20:15 ` Jason Gunthorpe
[not found] ` <20150713201538.GA11681-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-07-14 9:10 ` Sagi Grimberg
2015-07-14 9:10 ` Sagi Grimberg
[not found] ` <55A4D20C.2000904-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-14 15:36 ` 'Christoph Hellwig'
2015-07-14 15:36 ` 'Christoph Hellwig'
[not found] ` <20150714153619.GC11026-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-14 15:47 ` Tom Talpey
2015-07-14 15:47 ` Tom Talpey
2015-07-14 16:22 ` Jason Gunthorpe
2015-07-14 16:22 ` Jason Gunthorpe
2015-07-14 7:37 ` 'Christoph Hellwig' [this message]
2015-07-14 9:22 ` Sagi Grimberg
[not found] ` <55A4D4C6.6040907-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-14 12:12 ` Tom Talpey
2015-07-14 12:12 ` Tom Talpey
2015-07-14 13:23 ` Sagi Grimberg
2015-07-14 14:45 ` Steve Wise
2015-07-14 14:45 ` Steve Wise
2015-07-14 15:40 ` 'Christoph Hellwig'
2015-07-14 15:40 ` 'Christoph Hellwig'
2015-07-08 8:11 ` 'Christoph Hellwig'
2015-07-06 7:58 ` Sagi Grimberg
2015-07-06 7:58 ` Sagi Grimberg
[not found] ` <559A3536.3020807-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-07-06 14:39 ` Steve Wise
2015-07-06 14:39 ` Steve Wise
2015-07-05 23:22 ` [PATCH V3 2/5] RDMA/iser: Use transport independent MR allocation Steve Wise
[not found] ` <20150705231831.12029.80307.stgit-PBeJgSbIpn97NCTnQtmixQ@public.gmane.org>
2015-07-05 23:22 ` [PATCH V3 3/5] RDMA/isert: " Steve Wise
2015-07-05 23:22 ` Steve Wise
2015-07-05 23:22 ` [PATCH V3 4/5] svcrdma: " Steve Wise
2015-07-05 23:22 ` Steve Wise
2015-07-05 23:22 ` [PATCH V3 5/5] xprtrdma: " Steve Wise
2015-07-05 23:22 ` Steve Wise
2015-07-06 5:25 ` [PATCH V3 0/5] Transport-independent MRs Christoph Hellwig
2015-07-06 5:25 ` Christoph Hellwig
[not found] ` <20150706052542.GB1109-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-07-06 14:24 ` Steve Wise
2015-07-06 14:24 ` Steve Wise
2015-07-07 9:01 ` 'Christoph Hellwig'
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150714073756.GB7630@infradead.org \
--to=hch@infradead.org \
--cc=bfields@fieldses.org \
--cc=dledford@redhat.com \
--cc=eli@mellanox.com \
--cc=jgunthorpe@obsidianresearch.com \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=ogerlitz@mellanox.com \
--cc=oren@mellanox.com \
--cc=roid@mellanox.com \
--cc=sagig@dev.mellanox.co.il \
--cc=sagig@mellanox.com \
--cc=swise@opengridcomputing.com \
--cc=target-devel@vger.kernel.org \
--cc=tom@talpey.com \
--cc=trond.myklebust@primarydata.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.