From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bart Van Assche Subject: Re: linux rdma 3.14 merge plans Date: Thu, 30 Jan 2014 11:07:33 +0100 Message-ID: <52EA2465.8010907@acm.org> References: <52CD1C68.4050406@mellanox.com> <1389645171.5567.459.camel@haakon3.risingtidesystems.com> <1389820541.5567.543.camel@haakon3.risingtidesystems.com> <1389906852.5567.668.camel@haakon3.risingtidesystems.com> <1390102949.5567.749.camel@haakon3.risingtidesystems.com> <52DBB4F1.4020400@mellanox.com> <52DF93D3.6030509@dev.mellanox.co.il> <52E90C81.9040800@acm.org> <52E91906.70802@dev.mellanox.co.il> <52E940B2.3070901@acm.org> <52EA0AFA.40100@mellanox.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <52EA0AFA.40100-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Or Gerlitz , Sagi Grimberg , Or Gerlitz , Roland Dreier , Hefty Sean , Sagi Grimberg Cc: linux-rdma , Mike Christie , Oren Duer List-Id: linux-rdma@vger.kernel.org On 01/30/14 09:19, Or Gerlitz wrote: > On 29/01/2014 19:56, Bart Van Assche wrote: >> On 01/29/14 16:06, Sagi Grimberg wrote: >>> Didn't understand why should it matter where the copy is done >>> (iser/block)? >> In the Linux kernel community it is considered important to avoid code >> duplication. Hence the proposal to keep code that copies data buffers >> in the block layer core and to avoid that such functionality has to be >> reimplemented in every block driver or SCSI LLD. > > Thanks for narrowing this down, I see your point, however the solution I > propose if to remove this copy altogether... for those rare cases where > fast-registration can't be done -- in SRP I think the code goes to > indirect mode under which there no copy (right?). As for iSER, we will > add to our TODO enhancing the code to avoid using bounce-buffer, few > designs might be possible, one way would be to create set of > FMRS/Fast-reg element which are capable to map chunks which are < > PAGE_SIZE, e.g in the order of single block, and hence can map what ever > arbitrary set of blocks provided by the upper layer. I'm not sure how important this issue is on x86 systems since the page size on these systems is 4 KB and since many applications use an I/O size >= 4 KB. So if the I/O buffers are aligned on page boundaries the buffer memory can be registered as a single memory region without having to copy any data. However, on PPC systems, which have a page size of 64 KB, if e.g. a database issues a readv() or writev() call or if an I/O scheduler coalesces several small I/O requests into a single I/O request the data buffer of these I/O requests is discontiguous. I think it is important that data copying can be avoided for such I/O requests. By the way, is it documented somewhere what the alignment requirements are for FMR and FR requests ? As far as I can see the mlx4 driver requires page alignment for FMR requests but not for FR requests (see e.g. mlx4_check_fmr()). I haven't found a page aligment requirement in e.g. ehca_fmr_check_page_list(). Does this mean that alignment restrictions for FMR are HCA-dependent ? Would it be a good idea to add an attribute in e.g. ib_device_attr that allows ULP drivers to retrieve FMR alignment requirements in a generic way ? Thanks, Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html