From: Keith Busch <keith.busch@intel.com>
To: Logan Gunthorpe <logang@deltatee.com>
Cc: "Jens Axboe" <axboe@kernel.dk>,
linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org,
linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-nvme@lists.infradead.org, linux-block@vger.kernel.org,
"Jérôme Glisse" <jglisse@redhat.com>,
"Jason Gunthorpe" <jgg@mellanox.com>,
"Benjamin Herrenschmidt" <benh@kernel.crashing.org>,
"Bjorn Helgaas" <bhelgaas@google.com>,
"Max Gurtovoy" <maxg@mellanox.com>,
"Christoph Hellwig" <hch@lst.de>
Subject: Re: [PATCH 09/12] nvme-pci: Use PCI p2pmem subsystem to manage the CMB
Date: Fri, 5 Jan 2018 12:01:35 -0700 [thread overview]
Message-ID: <20180105190135.GB10657@localhost.localdomain> (raw)
In-Reply-To: <6c89e4b4-1854-c251-a5ec-7e54bc8085fc@deltatee.com>
On Fri, Jan 05, 2018 at 11:19:28AM -0700, Logan Gunthorpe wrote:
> Although it is not explicitly stated anywhere, pci_alloc_p2pmem() should
> always be at least 4k aligned. This is because the gen_pool that implements
> it is created with PAGE_SHIFT for its min_alloc_order.
Ah, I see that now. Thanks for the explanation.
Does it need to be created with page sized minimum alloc order? That
granularity makes it difficult to fit SQs in CMB on archs with larger
pages when we only needed 4k alignment.
I was also hoping to extend this for PRP/SGL in CMB where even 4k is
too high a granularity to make it really useful.
It looks like creating the gen pool with a smaller minimum and
gen_pool_first_fit_order_align algo would satisfy my use cases, but I'm
not sure if there's another reason you've set it to page alignment.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
WARNING: multiple messages have this Message-ID (diff)
From: Keith Busch <keith.busch@intel.com>
To: Logan Gunthorpe <logang@deltatee.com>
Cc: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org,
linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
linux-nvdimm@lists.01.org, linux-block@vger.kernel.org,
"Stephen Bates" <sbates@raithlin.com>,
"Christoph Hellwig" <hch@lst.de>, "Jens Axboe" <axboe@kernel.dk>,
"Sagi Grimberg" <sagi@grimberg.me>,
"Bjorn Helgaas" <bhelgaas@google.com>,
"Jason Gunthorpe" <jgg@mellanox.com>,
"Max Gurtovoy" <maxg@mellanox.com>,
"Dan Williams" <dan.j.williams@intel.com>,
"Jérôme Glisse" <jglisse@redhat.com>,
"Benjamin Herrenschmidt" <benh@kernel.crashing.org>
Subject: Re: [PATCH 09/12] nvme-pci: Use PCI p2pmem subsystem to manage the CMB
Date: Fri, 5 Jan 2018 12:01:35 -0700 [thread overview]
Message-ID: <20180105190135.GB10657@localhost.localdomain> (raw)
In-Reply-To: <6c89e4b4-1854-c251-a5ec-7e54bc8085fc@deltatee.com>
On Fri, Jan 05, 2018 at 11:19:28AM -0700, Logan Gunthorpe wrote:
> Although it is not explicitly stated anywhere, pci_alloc_p2pmem() should
> always be at least 4k aligned. This is because the gen_pool that implements
> it is created with PAGE_SHIFT for its min_alloc_order.
Ah, I see that now. Thanks for the explanation.
Does it need to be created with page sized minimum alloc order? That
granularity makes it difficult to fit SQs in CMB on archs with larger
pages when we only needed 4k alignment.
I was also hoping to extend this for PRP/SGL in CMB where even 4k is
too high a granularity to make it really useful.
It looks like creating the gen pool with a smaller minimum and
gen_pool_first_fit_order_align algo would satisfy my use cases, but I'm
not sure if there's another reason you've set it to page alignment.
WARNING: multiple messages have this Message-ID (diff)
From: Keith Busch <keith.busch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
To: Logan Gunthorpe <logang-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
Cc: "Jens Axboe" <axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
"Jérôme Glisse" <jglisse-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
"Jason Gunthorpe" <jgg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
"Benjamin Herrenschmidt"
<benh-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org>,
"Bjorn Helgaas"
<bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>,
"Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
"Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>
Subject: Re: [PATCH 09/12] nvme-pci: Use PCI p2pmem subsystem to manage the CMB
Date: Fri, 5 Jan 2018 12:01:35 -0700 [thread overview]
Message-ID: <20180105190135.GB10657@localhost.localdomain> (raw)
In-Reply-To: <6c89e4b4-1854-c251-a5ec-7e54bc8085fc-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
On Fri, Jan 05, 2018 at 11:19:28AM -0700, Logan Gunthorpe wrote:
> Although it is not explicitly stated anywhere, pci_alloc_p2pmem() should
> always be at least 4k aligned. This is because the gen_pool that implements
> it is created with PAGE_SHIFT for its min_alloc_order.
Ah, I see that now. Thanks for the explanation.
Does it need to be created with page sized minimum alloc order? That
granularity makes it difficult to fit SQs in CMB on archs with larger
pages when we only needed 4k alignment.
I was also hoping to extend this for PRP/SGL in CMB where even 4k is
too high a granularity to make it really useful.
It looks like creating the gen pool with a smaller minimum and
gen_pool_first_fit_order_align algo would satisfy my use cases, but I'm
not sure if there's another reason you've set it to page alignment.
WARNING: multiple messages have this Message-ID (diff)
From: keith.busch@intel.com (Keith Busch)
Subject: [PATCH 09/12] nvme-pci: Use PCI p2pmem subsystem to manage the CMB
Date: Fri, 5 Jan 2018 12:01:35 -0700 [thread overview]
Message-ID: <20180105190135.GB10657@localhost.localdomain> (raw)
In-Reply-To: <6c89e4b4-1854-c251-a5ec-7e54bc8085fc@deltatee.com>
On Fri, Jan 05, 2018@11:19:28AM -0700, Logan Gunthorpe wrote:
> Although it is not explicitly stated anywhere, pci_alloc_p2pmem() should
> always be at least 4k aligned. This is because the gen_pool that implements
> it is created with PAGE_SHIFT for its min_alloc_order.
Ah, I see that now. Thanks for the explanation.
Does it need to be created with page sized minimum alloc order? That
granularity makes it difficult to fit SQs in CMB on archs with larger
pages when we only needed 4k alignment.
I was also hoping to extend this for PRP/SGL in CMB where even 4k is
too high a granularity to make it really useful.
It looks like creating the gen pool with a smaller minimum and
gen_pool_first_fit_order_align algo would satisfy my use cases, but I'm
not sure if there's another reason you've set it to page alignment.
next prev parent reply other threads:[~2018-01-05 18:53 UTC|newest]
Thread overview: 198+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-04 19:01 [PATCH 00/11] Copy Offload in NVMe Fabrics with P2P PCI Memory Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 01/12] pci-p2p: Support peer to peer memory Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 21:40 ` Bjorn Helgaas
2018-01-04 21:40 ` Bjorn Helgaas
2018-01-04 21:40 ` Bjorn Helgaas
2018-01-04 23:06 ` Logan Gunthorpe
2018-01-04 23:06 ` Logan Gunthorpe
2018-01-04 23:06 ` Logan Gunthorpe
2018-01-04 23:06 ` Logan Gunthorpe
2018-01-04 21:59 ` Bjorn Helgaas
2018-01-04 21:59 ` Bjorn Helgaas
2018-01-04 21:59 ` Bjorn Helgaas
2018-01-04 21:59 ` Bjorn Helgaas
2018-01-05 0:20 ` Logan Gunthorpe
2018-01-05 0:20 ` Logan Gunthorpe
2018-01-05 0:20 ` Logan Gunthorpe
2018-01-05 0:20 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 02/12] pci-p2p: Add sysfs group to display p2pmem stats Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 21:50 ` Bjorn Helgaas
2018-01-04 21:50 ` Bjorn Helgaas
2018-01-04 21:50 ` Bjorn Helgaas
2018-01-04 22:25 ` Jason Gunthorpe
2018-01-04 22:25 ` Jason Gunthorpe
2018-01-04 22:25 ` Jason Gunthorpe
2018-01-04 22:25 ` Jason Gunthorpe
2018-01-04 23:13 ` Logan Gunthorpe
2018-01-04 23:13 ` Logan Gunthorpe
2018-01-04 23:13 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 03/12] pci-p2p: Add PCI p2pmem dma mappings to adjust the bus offset Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 04/12] pci-p2p: Clear ACS P2P flags for all client devices Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 21:57 ` Bjorn Helgaas
2018-01-04 21:57 ` Bjorn Helgaas
2018-01-04 21:57 ` Bjorn Helgaas
2018-01-04 21:57 ` Bjorn Helgaas
2018-01-04 22:35 ` Alex Williamson
2018-01-04 22:35 ` Alex Williamson
2018-01-04 22:35 ` Alex Williamson
2018-01-05 0:00 ` Logan Gunthorpe
2018-01-05 0:00 ` Logan Gunthorpe
2018-01-05 0:00 ` Logan Gunthorpe
2018-01-05 0:00 ` Logan Gunthorpe
2018-01-05 1:09 ` Logan Gunthorpe
2018-01-05 1:09 ` Logan Gunthorpe
2018-01-05 1:09 ` Logan Gunthorpe
2018-01-05 3:33 ` Alex Williamson
2018-01-05 3:33 ` Alex Williamson
2018-01-05 3:33 ` Alex Williamson
2018-01-05 6:47 ` Jerome Glisse
2018-01-05 6:47 ` Jerome Glisse
2018-01-05 6:47 ` Jerome Glisse
2018-01-05 6:47 ` Jerome Glisse
2018-01-05 15:41 ` Alex Williamson
2018-01-05 15:41 ` Alex Williamson
2018-01-05 15:41 ` Alex Williamson
2018-01-05 17:10 ` Logan Gunthorpe
2018-01-05 17:10 ` Logan Gunthorpe
2018-01-05 17:10 ` Logan Gunthorpe
2018-01-05 17:10 ` Logan Gunthorpe
2018-01-05 17:18 ` Alex Williamson
2018-01-05 17:18 ` Alex Williamson
2018-01-05 17:18 ` Alex Williamson
2018-01-05 17:18 ` Alex Williamson
2018-01-04 19:01 ` [PATCH 05/12] block: Introduce PCI P2P flags for request and request queue Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]() Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:22 ` Jason Gunthorpe
2018-01-04 19:22 ` Jason Gunthorpe
2018-01-04 19:52 ` Logan Gunthorpe
2018-01-04 19:52 ` Logan Gunthorpe
2018-01-04 19:52 ` Logan Gunthorpe
2018-01-04 19:52 ` Logan Gunthorpe
2018-01-04 22:13 ` Jason Gunthorpe
2018-01-04 22:13 ` Jason Gunthorpe
2018-01-04 23:44 ` Logan Gunthorpe
2018-01-04 23:44 ` Logan Gunthorpe
2018-01-04 23:44 ` Logan Gunthorpe
2018-01-05 4:50 ` Jason Gunthorpe
2018-01-05 4:50 ` Jason Gunthorpe
2018-01-08 14:59 ` Christoph Hellwig
2018-01-08 14:59 ` Christoph Hellwig
2018-01-08 14:59 ` Christoph Hellwig
2018-01-08 18:09 ` Jason Gunthorpe
2018-01-08 18:09 ` Jason Gunthorpe
2018-01-08 18:17 ` Logan Gunthorpe
2018-01-08 18:17 ` Logan Gunthorpe
2018-01-08 18:17 ` Logan Gunthorpe
2018-01-08 18:17 ` Logan Gunthorpe
2018-01-08 18:29 ` Jason Gunthorpe
2018-01-08 18:29 ` Jason Gunthorpe
2018-01-08 18:29 ` Jason Gunthorpe
2018-01-08 18:34 ` Christoph Hellwig
2018-01-08 18:34 ` Christoph Hellwig
2018-01-08 18:34 ` Christoph Hellwig
2018-01-08 18:34 ` Christoph Hellwig
2018-01-08 18:44 ` Logan Gunthorpe
2018-01-08 18:44 ` Logan Gunthorpe
2018-01-08 18:44 ` Logan Gunthorpe
2018-01-08 18:44 ` Logan Gunthorpe
2018-01-08 18:57 ` Christoph Hellwig
2018-01-08 18:57 ` Christoph Hellwig
2018-01-08 18:57 ` Christoph Hellwig
2018-01-08 18:57 ` Christoph Hellwig
2018-01-08 19:05 ` Logan Gunthorpe
2018-01-08 19:05 ` Logan Gunthorpe
2018-01-08 19:05 ` Logan Gunthorpe
2018-01-08 19:05 ` Logan Gunthorpe
2018-01-09 16:47 ` Christoph Hellwig
2018-01-09 16:47 ` Christoph Hellwig
2018-01-09 16:47 ` Christoph Hellwig
2018-01-09 16:47 ` Christoph Hellwig
2018-01-08 19:49 ` Jason Gunthorpe
2018-01-08 19:49 ` Jason Gunthorpe
2018-01-09 16:46 ` Christoph Hellwig
2018-01-09 16:46 ` Christoph Hellwig
2018-01-09 16:46 ` Christoph Hellwig
2018-01-09 17:10 ` Jason Gunthorpe
2018-01-09 17:10 ` Jason Gunthorpe
2018-01-08 19:01 ` Jason Gunthorpe
2018-01-08 19:01 ` Jason Gunthorpe
2018-01-08 19:01 ` Jason Gunthorpe
2018-01-09 16:55 ` Christoph Hellwig
2018-01-09 16:55 ` Christoph Hellwig
2018-01-09 16:55 ` Christoph Hellwig
2018-01-09 16:55 ` Christoph Hellwig
2018-01-04 19:01 ` [PATCH 07/12] nvme-pci: clean up CMB initialization Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 08/12] nvme-pci: clean up SMBSZ bit definitions Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:08 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 09/12] nvme-pci: Use PCI p2pmem subsystem to manage the CMB Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-05 15:30 ` Marta Rybczynska
2018-01-05 15:30 ` Marta Rybczynska
2018-01-05 15:30 ` Marta Rybczynska
2018-01-05 15:30 ` Marta Rybczynska
2018-01-05 18:14 ` Logan Gunthorpe
2018-01-05 18:14 ` Logan Gunthorpe
2018-01-05 18:14 ` Logan Gunthorpe
2018-01-05 18:14 ` Logan Gunthorpe
2018-01-05 18:11 ` Keith Busch
2018-01-05 18:11 ` Keith Busch
2018-01-05 18:11 ` Keith Busch
2018-01-05 18:11 ` Keith Busch
2018-01-05 18:19 ` Logan Gunthorpe
2018-01-05 18:19 ` Logan Gunthorpe
2018-01-05 18:19 ` Logan Gunthorpe
2018-01-05 19:01 ` Keith Busch [this message]
2018-01-05 19:01 ` Keith Busch
2018-01-05 19:01 ` Keith Busch
2018-01-05 19:01 ` Keith Busch
2018-01-05 19:04 ` Logan Gunthorpe
2018-01-05 19:04 ` Logan Gunthorpe
2018-01-05 19:04 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 10/12] nvme-pci: Add support for P2P memory in requests Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 11/12] nvme-pci: Add a quirk for a pseudo CMB Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` [PATCH 12/12] nvmet: Optionally use PCI P2P memory Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
2018-01-04 19:01 ` Logan Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180105190135.GB10657@localhost.localdomain \
--to=keith.busch@intel.com \
--cc=axboe@kernel.dk \
--cc=benh@kernel.crashing.org \
--cc=bhelgaas@google.com \
--cc=hch@lst.de \
--cc=jgg@mellanox.com \
--cc=jglisse@redhat.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=logang@deltatee.com \
--cc=maxg@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.