From: Sagi Grimberg <sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> To: Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>, Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> Cc: Linux NFS Mailing List <linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: Re: [PATCH v1 00/16] NFS/RDMA patches proposed for 4.1 Date: Wed, 06 May 2015 15:15:57 +0300 [thread overview] Message-ID: <554A05FD.5080605@dev.mellanox.co.il> (raw) In-Reply-To: <20150505172540.GA19442-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> On 5/5/2015 8:25 PM, Christoph Hellwig wrote: > On Tue, May 05, 2015 at 12:04:00PM -0400, Chuck Lever wrote: >>> Just curious if you ever though of moving this into the generic >>> rdma layer? >> >> Not really. The new files are really just shims that adapt the RPC/RDMA >> transport to memory registration verbs. There?s only a few hundred lines >> per registration mode, and it?s all fairly specific to RPC/RDMA. > > While it's using RPC/RDMA specific data structures it basically > abstracts out the action of mapping a number of pages onto the rdma > hardware. There isn't a whole lot of interaction with the actual > sunrpc layer except for a few hardcoded limits. > > Btw, this is not a critique of the code, it's an obvious major > improvement of what was there before, it justs seems like it would be > nice to move it up to a higher layer. > >>> And from I see we litterly dont use them much different from the generic >>> dma mapping API helpers (at a very high level) so it seems it should >>> be easy to move a slightly expanded version of your API to the core >>> code. >> >> IMO FRWR is the only registration mode that has legs for the long term, >> and is specifically designed for storage. >> >> If you are not working on a legacy piece of code that has to support >> older HCAs, why not stay with FRWR? Hey Christoph, I agree here, FMRs (and FMR pools) are not supported over VFs. Also, I've seen some unpredictable performance in certain workloads because the fmr pool maintains a flush thread that executes a HW sync (terribly blocking on mlx4 devices) when hitting some dirty_watermark... If you are writing a driver, I suggest to stick FRWR as well. > > The raw FRWR API seems like an absolute nightmare, and I'm bound to > get it wrong at first :) This is only half joking, but despite that > it's the first target for sure. It's just very frustrating that there > is no usable common API. The FRWR API is a work request interface. The advantage here is the option to concatenate it with a send/rdma work request and save an extra send queue lock and more importantly a doorbell. This matters in high workloads. The iser target is doing this and I have a patch for the initiator code as well. I'm not sure that providing an API that allows you to do post-lists might be simpler... Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Sagi Grimberg <sagig@dev.mellanox.co.il> To: Christoph Hellwig <hch@infradead.org>, Chuck Lever <chuck.lever@oracle.com> Cc: Linux NFS Mailing List <linux-nfs@vger.kernel.org>, linux-rdma@vger.kernel.org Subject: Re: [PATCH v1 00/16] NFS/RDMA patches proposed for 4.1 Date: Wed, 06 May 2015 15:15:57 +0300 [thread overview] Message-ID: <554A05FD.5080605@dev.mellanox.co.il> (raw) In-Reply-To: <20150505172540.GA19442@infradead.org> On 5/5/2015 8:25 PM, Christoph Hellwig wrote: > On Tue, May 05, 2015 at 12:04:00PM -0400, Chuck Lever wrote: >>> Just curious if you ever though of moving this into the generic >>> rdma layer? >> >> Not really. The new files are really just shims that adapt the RPC/RDMA >> transport to memory registration verbs. There?s only a few hundred lines >> per registration mode, and it?s all fairly specific to RPC/RDMA. > > While it's using RPC/RDMA specific data structures it basically > abstracts out the action of mapping a number of pages onto the rdma > hardware. There isn't a whole lot of interaction with the actual > sunrpc layer except for a few hardcoded limits. > > Btw, this is not a critique of the code, it's an obvious major > improvement of what was there before, it justs seems like it would be > nice to move it up to a higher layer. > >>> And from I see we litterly dont use them much different from the generic >>> dma mapping API helpers (at a very high level) so it seems it should >>> be easy to move a slightly expanded version of your API to the core >>> code. >> >> IMO FRWR is the only registration mode that has legs for the long term, >> and is specifically designed for storage. >> >> If you are not working on a legacy piece of code that has to support >> older HCAs, why not stay with FRWR? Hey Christoph, I agree here, FMRs (and FMR pools) are not supported over VFs. Also, I've seen some unpredictable performance in certain workloads because the fmr pool maintains a flush thread that executes a HW sync (terribly blocking on mlx4 devices) when hitting some dirty_watermark... If you are writing a driver, I suggest to stick FRWR as well. > > The raw FRWR API seems like an absolute nightmare, and I'm bound to > get it wrong at first :) This is only half joking, but despite that > it's the first target for sure. It's just very frustrating that there > is no usable common API. The FRWR API is a work request interface. The advantage here is the option to concatenate it with a send/rdma work request and save an extra send queue lock and more importantly a doorbell. This matters in high workloads. The iser target is doing this and I have a patch for the initiator code as well. I'm not sure that providing an API that allows you to do post-lists might be simpler... Sagi.
next prev parent reply other threads:[~2015-05-06 12:15 UTC|newest] Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-03-13 21:21 [PATCH v1 00/16] NFS/RDMA patches proposed for 4.1 Chuck Lever 2015-03-13 21:21 ` [PATCH v1 01/16] xprtrdma: Display IPv6 addresses and port numbers correctly Chuck Lever 2015-03-13 21:21 ` [PATCH v1 02/16] xprtrdma: Perform a full marshal on retransmit Chuck Lever 2015-03-13 21:21 ` [PATCH v1 03/16] xprtrdma: Add vector of ops for each memory registration strategy Chuck Lever 2015-03-13 21:21 ` [PATCH v1 04/16] xprtrdma: Add a "max_payload" op for each memreg mode Chuck Lever 2015-03-13 21:22 ` [PATCH v1 05/16] xprtrdma: Add a "register_external" " Chuck Lever 2015-03-13 21:22 ` [PATCH v1 06/16] xprtrdma: Add a "deregister_external" " Chuck Lever 2015-03-17 14:37 ` Anna Schumaker 2015-03-17 15:04 ` Chuck Lever 2015-03-13 21:22 ` [PATCH v1 07/16] xprtrdma: Add "init MRs" memreg op Chuck Lever 2015-03-13 21:22 ` [PATCH v1 08/16] xprtrdma: Add "reset " Chuck Lever 2015-03-13 21:22 ` [PATCH v1 09/16] xprtrdma: Add "destroy " Chuck Lever 2015-03-13 21:22 ` [PATCH v1 10/16] xprtrdma: Add "open" " Chuck Lever 2015-03-17 15:16 ` Anna Schumaker 2015-03-17 15:19 ` Chuck Lever 2015-03-13 21:23 ` [PATCH v1 11/16] xprtrdma: Handle non-SEND completions via a callout Chuck Lever 2015-03-13 21:23 ` [PATCH v1 12/16] xprtrdma: Acquire FMRs in rpcrdma_fmr_register_external() Chuck Lever 2015-03-13 21:23 ` [PATCH v1 13/16] xprtrdma: Acquire MRs in rpcrdma_register_external() Chuck Lever 2015-03-13 21:23 ` [PATCH v1 14/16] xprtrdma: Remove rpcrdma_ia::ri_memreg_strategy Chuck Lever 2015-03-13 21:23 ` [PATCH v1 15/16] xprtrdma: Make rpcrdma_{un}map_one() into inline functions Chuck Lever 2015-03-13 21:23 ` [PATCH v1 16/16] xprtrdma: Split rb_lock Chuck Lever [not found] ` <20150313211124.22471.14517.stgit-FYjufvaPoItvLzlybtyyYzGyq/o6K9yX@public.gmane.org> 2015-05-05 15:44 ` [PATCH v1 00/16] NFS/RDMA patches proposed for 4.1 Christoph Hellwig 2015-05-05 15:44 ` Christoph Hellwig [not found] ` <20150505154411.GA16729-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2015-05-05 16:04 ` Chuck Lever 2015-05-05 16:04 ` Chuck Lever [not found] ` <5E1B32EA-9803-49AA-856D-BF0E1A5DFFF4-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2015-05-05 17:25 ` Christoph Hellwig 2015-05-05 17:25 ` Christoph Hellwig [not found] ` <20150505172540.GA19442-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2015-05-05 18:14 ` Tom Talpey 2015-05-05 18:14 ` Tom Talpey [not found] ` <55490886.4070502-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2015-05-05 19:10 ` Christoph Hellwig 2015-05-05 19:10 ` Christoph Hellwig [not found] ` <20150505191012.GA21164-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2015-05-05 20:57 ` Tom Talpey 2015-05-05 20:57 ` Tom Talpey [not found] ` <55492ED3.7000507-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2015-05-05 21:06 ` Christoph Hellwig 2015-05-05 21:06 ` Christoph Hellwig [not found] ` <20150505210627.GA5941-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2015-05-05 21:32 ` Tom Talpey 2015-05-05 21:32 ` Tom Talpey [not found] ` <554936E5.80607-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2015-05-05 22:38 ` Jason Gunthorpe 2015-05-05 22:38 ` Jason Gunthorpe [not found] ` <20150505223855.GA7696-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2015-05-06 0:16 ` Tom Talpey 2015-05-06 0:16 ` Tom Talpey [not found] ` <55495D41.5090502-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2015-05-06 16:20 ` Jason Gunthorpe 2015-05-06 16:20 ` Jason Gunthorpe 2015-05-06 7:01 ` Bart Van Assche 2015-05-06 7:01 ` Bart Van Assche [not found] ` <5549BC33.30905-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> 2015-05-06 16:38 ` Jason Gunthorpe 2015-05-06 16:38 ` Jason Gunthorpe 2015-05-06 7:33 ` Christoph Hellwig 2015-05-06 7:33 ` Christoph Hellwig 2015-05-06 7:09 ` Bart Van Assche 2015-05-06 7:09 ` Bart Van Assche [not found] ` <5549BE30.8020505-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> 2015-05-06 7:29 ` Christoph Hellwig 2015-05-06 7:29 ` Christoph Hellwig 2015-05-06 12:15 ` Sagi Grimberg [this message] 2015-05-06 12:15 ` Sagi Grimberg 2015-03-13 21:26 Chuck Lever
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=554A05FD.5080605@dev.mellanox.co.il \ --to=sagig-ldsdmyg8hgv8yrgs2mwiifqbs+8scbdb@public.gmane.org \ --cc=chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org \ --cc=hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org \ --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.