From: Jason Gunthorpe <jgg@nvidia.com> To: Matthew Wilcox <willy@infradead.org> Cc: linux-kernel@vger.kernel.org, Christoph Hellwig <hch@lst.de>, Joao Martins <joao.m.martins@oracle.com>, John Hubbard <jhubbard@nvidia.com>, Logan Gunthorpe <logang@deltatee.com>, Ming Lei <ming.lei@redhat.com>, linux-block@vger.kernel.org, netdev@vger.kernel.org, linux-mm@kvack.org, linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org, nvdimm@lists.linux.dev Subject: Re: Phyr Starter Date: Wed, 12 Jan 2022 15:08:02 -0400 [thread overview] Message-ID: <20220112190802.GW2328285@nvidia.com> (raw) In-Reply-To: <Yd8fz4bY/aMMk24h@casper.infradead.org> On Wed, Jan 12, 2022 at 06:37:03PM +0000, Matthew Wilcox wrote: > On Tue, Jan 11, 2022 at 06:53:06PM -0400, Jason Gunthorpe wrote: > > IOMMU is not common in those cases, it is slow. > > > > So you end up with 16 bytes per entry then another 24 bytes in the > > entirely redundant scatter list. That is now 40 bytes/page for typical > > HPC case, and I can't see that being OK. > > Ah, I didn't realise what case you wanted to optimise for. It is pretty common, even systems with the iommu turned on will run the kernel drivers with an identity map due to the performance delta.. > Since you want to get to the same destination as I do (a > 16-byte-per-entry dma_addr+dma_len struct), but need to get there sooner > than "make all sg users stop using it wrongly", let's introduce a > (hopefully temporary) "struct dma_range". > > But let's go further than that (which only brings us to 32 bytes per > range). For the systems you care about which use an identity mapping, > and have sizeof(dma_addr_t) == sizeof(phys_addr_t), we can simply > point the dma_range pointer to the same memory as the phyr. We just > have to not free it too early. That gets us down to 16 bytes per range, > a saving of 33%. Yes, that is more or less what I suggested. I'm not sure I understand your "make all sg users stop using it wrongly" I suspect trying to change scatterlist is a tar pit. Thanks, Jason
WARNING: multiple messages have this Message-ID (diff)
From: Jason Gunthorpe <jgg@nvidia.com> To: Matthew Wilcox <willy@infradead.org> Cc: nvdimm@lists.linux.dev, linux-rdma@vger.kernel.org, John Hubbard <jhubbard@nvidia.com>, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Ming Lei <ming.lei@redhat.com>, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, Joao Martins <joao.m.martins@oracle.com>, Logan Gunthorpe <logang@deltatee.com>, Christoph Hellwig <hch@lst.de> Subject: Re: Phyr Starter Date: Wed, 12 Jan 2022 15:08:02 -0400 [thread overview] Message-ID: <20220112190802.GW2328285@nvidia.com> (raw) In-Reply-To: <Yd8fz4bY/aMMk24h@casper.infradead.org> On Wed, Jan 12, 2022 at 06:37:03PM +0000, Matthew Wilcox wrote: > On Tue, Jan 11, 2022 at 06:53:06PM -0400, Jason Gunthorpe wrote: > > IOMMU is not common in those cases, it is slow. > > > > So you end up with 16 bytes per entry then another 24 bytes in the > > entirely redundant scatter list. That is now 40 bytes/page for typical > > HPC case, and I can't see that being OK. > > Ah, I didn't realise what case you wanted to optimise for. It is pretty common, even systems with the iommu turned on will run the kernel drivers with an identity map due to the performance delta.. > Since you want to get to the same destination as I do (a > 16-byte-per-entry dma_addr+dma_len struct), but need to get there sooner > than "make all sg users stop using it wrongly", let's introduce a > (hopefully temporary) "struct dma_range". > > But let's go further than that (which only brings us to 32 bytes per > range). For the systems you care about which use an identity mapping, > and have sizeof(dma_addr_t) == sizeof(phys_addr_t), we can simply > point the dma_range pointer to the same memory as the phyr. We just > have to not free it too early. That gets us down to 16 bytes per range, > a saving of 33%. Yes, that is more or less what I suggested. I'm not sure I understand your "make all sg users stop using it wrongly" I suspect trying to change scatterlist is a tar pit. Thanks, Jason
next prev parent reply other threads:[~2022-01-12 19:08 UTC|newest] Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-01-10 19:34 Phyr Starter Matthew Wilcox 2022-01-10 19:34 ` Matthew Wilcox 2022-01-11 0:41 ` Jason Gunthorpe 2022-01-11 0:41 ` Jason Gunthorpe 2022-01-11 4:32 ` Matthew Wilcox 2022-01-11 4:32 ` Matthew Wilcox 2022-01-11 15:01 ` Jason Gunthorpe 2022-01-11 15:01 ` Jason Gunthorpe 2022-01-11 18:33 ` Matthew Wilcox 2022-01-11 18:33 ` Matthew Wilcox 2022-01-11 20:21 ` Jason Gunthorpe 2022-01-11 20:21 ` Jason Gunthorpe 2022-01-11 21:25 ` Matthew Wilcox 2022-01-11 21:25 ` Matthew Wilcox 2022-01-11 22:09 ` Logan Gunthorpe 2022-01-11 22:09 ` Logan Gunthorpe 2022-01-11 22:57 ` Jason Gunthorpe 2022-01-11 22:57 ` Jason Gunthorpe 2022-01-11 23:02 ` Logan Gunthorpe 2022-01-11 23:02 ` Logan Gunthorpe 2022-01-11 22:53 ` Jason Gunthorpe 2022-01-11 22:53 ` Jason Gunthorpe 2022-01-11 22:57 ` Logan Gunthorpe 2022-01-11 22:57 ` Logan Gunthorpe 2022-01-11 23:02 ` Jason Gunthorpe 2022-01-11 23:02 ` Jason Gunthorpe 2022-01-11 23:08 ` Logan Gunthorpe 2022-01-11 23:08 ` Logan Gunthorpe 2022-01-12 18:37 ` Matthew Wilcox 2022-01-12 18:37 ` Matthew Wilcox 2022-01-12 19:08 ` Jason Gunthorpe [this message] 2022-01-12 19:08 ` Jason Gunthorpe 2022-01-20 14:03 ` Christoph Hellwig 2022-01-20 17:17 ` Jason Gunthorpe 2022-01-20 17:17 ` Jason Gunthorpe 2022-01-20 14:00 ` Christoph Hellwig 2022-01-11 9:05 ` Daniel Vetter 2022-01-11 9:05 ` Daniel Vetter 2022-01-11 20:26 ` Jason Gunthorpe 2022-01-11 20:26 ` Jason Gunthorpe 2022-01-20 14:09 ` Christoph Hellwig 2022-01-20 13:56 ` Christoph Hellwig 2022-01-20 15:27 ` Keith Busch 2022-01-20 15:27 ` Keith Busch 2022-01-20 15:28 ` Christoph Hellwig 2022-01-20 17:54 ` Robin Murphy 2022-01-11 8:17 ` John Hubbard 2022-01-11 8:17 ` John Hubbard 2022-01-11 14:01 ` Matthew Wilcox 2022-01-11 14:01 ` Matthew Wilcox 2022-01-11 15:02 ` Jason Gunthorpe 2022-01-11 15:02 ` Jason Gunthorpe 2022-01-11 17:31 ` Logan Gunthorpe 2022-01-11 17:31 ` Logan Gunthorpe 2022-01-20 14:12 ` Christoph Hellwig 2022-01-20 21:35 ` John Hubbard 2022-01-20 21:35 ` John Hubbard 2022-01-11 11:40 ` Thomas Zimmermann 2022-01-11 13:56 ` Matthew Wilcox 2022-01-11 13:56 ` Matthew Wilcox 2022-01-11 14:10 ` Thomas Zimmermann 2022-01-20 13:39 ` Christoph Hellwig
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20220112190802.GW2328285@nvidia.com \ --to=jgg@nvidia.com \ --cc=dri-devel@lists.freedesktop.org \ --cc=hch@lst.de \ --cc=jhubbard@nvidia.com \ --cc=joao.m.martins@oracle.com \ --cc=linux-block@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux-rdma@vger.kernel.org \ --cc=logang@deltatee.com \ --cc=ming.lei@redhat.com \ --cc=netdev@vger.kernel.org \ --cc=nvdimm@lists.linux.dev \ --cc=willy@infradead.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.