From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f196.google.com (mail-wr0-f196.google.com [209.85.128.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id A3A4321942327 for ; Wed, 5 Apr 2017 22:47:27 -0700 (PDT) Received: by mail-wr0-f196.google.com with SMTP id t20so8035349wra.2 for ; Wed, 05 Apr 2017 22:47:27 -0700 (PDT) Subject: Re: [RFC 3/8] nvmet: Use p2pmem in nvme target References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1490911959-5146-4-git-send-email-logang@deltatee.com> From: Sagi Grimberg Message-ID: <0689e764-bf04-6da2-3b7d-2cbf0b6b94a0@grimberg.me> Date: Thu, 6 Apr 2017 08:47:22 +0300 MIME-Version: 1.0 In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Christoph Hellwig , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe Cc: linux-scsi@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, Sinan Kaya List-ID: > I hadn't done this yet but I think a simple closest device in the tree > would solve the issue sufficiently. However, I originally had it so the > user has to pick the device and I prefer that approach. But if the user > picks the device, then why bother restricting what he picks? Because the user can get it wrong, and its our job to do what we can in order to prevent the user from screwing itself. > Per the > thread with Sinan, I'd prefer to use what the user picks. You were one > of the biggest opponents to that so I'd like to hear your opinion on > removing the restrictions. I wasn't against it that much, I'm all for making things "just work" with minimal configuration steps, but I'm not sure we can get it right without it. >>> Ideally, we'd want to use an NVME CMB buffer as p2p memory. This would >>> save an extra PCI transfer as the NVME card could just take the data >>> out of it's own memory. However, at this time, cards with CMB buffers >>> don't seem to be available. >> >> Even if it was available, it would be hard to make real use of this >> given that we wouldn't know how to pre-post recv buffers (for in-capsule >> data). But let's leave this out of the scope entirely... > > I don't understand what you're referring to. We'd simply use the CMB > buffer as a p2pmem device, why does that change anything? I'm referring to the in-capsule data buffers pre-posts that we do. Because we prepare a buffer that would contain in-capsule data, we have no knowledge to which device the incoming I/O is directed to, which means we can (and will) have I/O where the data lies in CMB of device A but it's really targeted to device B - which sorta defeats the purpose of what we're trying to optimize here... >> Why do you need this? you have a reference to the >> queue itself. > > This keeps track of whether the response was actually allocated with > p2pmem or not. It's needed for when we free the SGL because the queue > may have a p2pmem device assigned to it but, if the alloc failed and it > fell back on system memory then we need to know how to free it. I'm > currently looking at having SGLs having an iomem flag. In which case, > this would no longer be needed as the flag in the SGL could be used. That would be better, maybe... [...] >> This is a problem. namespaces can be added at any point in time. No one >> guarantee that dma_devs are all the namepaces we'll ever see. > > Yeah, well restricting p2pmem based on all the devices in use is hard. > So we'd need a call into the transport every time an ns is added and > we'd have to drop the p2pmem if they add one that isn't supported. This > complexity is just one of the reasons I prefer just letting the user chose. Still the user can get it wrong. Not sure we can get a way without keeping track of this as new devices join the subsystem. >>> + >>> + if (queue->p2pmem) >>> + pr_debug("using %s for rdma nvme target queue", >>> + dev_name(&queue->p2pmem->dev)); >>> + >>> + kfree(dma_devs); >>> +} >>> + >>> static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, >>> struct rdma_cm_event *event) >>> { >>> @@ -1199,6 +1271,8 @@ static int nvmet_rdma_queue_connect(struct >>> rdma_cm_id *cm_id, >>> } >>> queue->port = cm_id->context; >>> >>> + nvmet_rdma_queue_setup_p2pmem(queue); >>> + >> >> Why is all this done for each queue? looks completely redundant to me. > > A little bit. Where would you put it? I think we'll need a representation of a controller in nvmet-rdma for that. we sort of got a way without it so far, but I don't think we can anymore with this. >>> ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn); >>> if (ret) >>> goto release_queue; >> >> You seemed to skip the in-capsule buffers for p2pmem (inline_page), I'm >> curious why? > > Yes, the thinking was that these transfers were small anyway so there > would not be significant benefit to pushing them through p2pmem. There's > really no reason why we couldn't do that if it made sense to though. I don't see an urgent reason for it too. I was just curious... _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sagi Grimberg Subject: Re: [RFC 3/8] nvmet: Use p2pmem in nvme target Date: Thu, 6 Apr 2017 08:47:22 +0300 Message-ID: <0689e764-bf04-6da2-3b7d-2cbf0b6b94a0@grimberg.me> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1490911959-5146-4-git-send-email-logang@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Christoph Hellwig , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe Cc: linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Sinan Kaya List-Id: linux-rdma@vger.kernel.org > I hadn't done this yet but I think a simple closest device in the tree > would solve the issue sufficiently. However, I originally had it so the > user has to pick the device and I prefer that approach. But if the user > picks the device, then why bother restricting what he picks? Because the user can get it wrong, and its our job to do what we can in order to prevent the user from screwing itself. > Per the > thread with Sinan, I'd prefer to use what the user picks. You were one > of the biggest opponents to that so I'd like to hear your opinion on > removing the restrictions. I wasn't against it that much, I'm all for making things "just work" with minimal configuration steps, but I'm not sure we can get it right without it. >>> Ideally, we'd want to use an NVME CMB buffer as p2p memory. This would >>> save an extra PCI transfer as the NVME card could just take the data >>> out of it's own memory. However, at this time, cards with CMB buffers >>> don't seem to be available. >> >> Even if it was available, it would be hard to make real use of this >> given that we wouldn't know how to pre-post recv buffers (for in-capsule >> data). But let's leave this out of the scope entirely... > > I don't understand what you're referring to. We'd simply use the CMB > buffer as a p2pmem device, why does that change anything? I'm referring to the in-capsule data buffers pre-posts that we do. Because we prepare a buffer that would contain in-capsule data, we have no knowledge to which device the incoming I/O is directed to, which means we can (and will) have I/O where the data lies in CMB of device A but it's really targeted to device B - which sorta defeats the purpose of what we're trying to optimize here... >> Why do you need this? you have a reference to the >> queue itself. > > This keeps track of whether the response was actually allocated with > p2pmem or not. It's needed for when we free the SGL because the queue > may have a p2pmem device assigned to it but, if the alloc failed and it > fell back on system memory then we need to know how to free it. I'm > currently looking at having SGLs having an iomem flag. In which case, > this would no longer be needed as the flag in the SGL could be used. That would be better, maybe... [...] >> This is a problem. namespaces can be added at any point in time. No one >> guarantee that dma_devs are all the namepaces we'll ever see. > > Yeah, well restricting p2pmem based on all the devices in use is hard. > So we'd need a call into the transport every time an ns is added and > we'd have to drop the p2pmem if they add one that isn't supported. This > complexity is just one of the reasons I prefer just letting the user chose. Still the user can get it wrong. Not sure we can get a way without keeping track of this as new devices join the subsystem. >>> + >>> + if (queue->p2pmem) >>> + pr_debug("using %s for rdma nvme target queue", >>> + dev_name(&queue->p2pmem->dev)); >>> + >>> + kfree(dma_devs); >>> +} >>> + >>> static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, >>> struct rdma_cm_event *event) >>> { >>> @@ -1199,6 +1271,8 @@ static int nvmet_rdma_queue_connect(struct >>> rdma_cm_id *cm_id, >>> } >>> queue->port = cm_id->context; >>> >>> + nvmet_rdma_queue_setup_p2pmem(queue); >>> + >> >> Why is all this done for each queue? looks completely redundant to me. > > A little bit. Where would you put it? I think we'll need a representation of a controller in nvmet-rdma for that. we sort of got a way without it so far, but I don't think we can anymore with this. >>> ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn); >>> if (ret) >>> goto release_queue; >> >> You seemed to skip the in-capsule buffers for p2pmem (inline_page), I'm >> curious why? > > Yes, the thinking was that these transfers were small anyway so there > would not be significant benefit to pushing them through p2pmem. There's > really no reason why we couldn't do that if it made sense to though. I don't see an urgent reason for it too. I was just curious... From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756084AbdDFFrh (ORCPT ); Thu, 6 Apr 2017 01:47:37 -0400 Received: from mail-wr0-f195.google.com ([209.85.128.195]:33759 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755720AbdDFFr2 (ORCPT ); Thu, 6 Apr 2017 01:47:28 -0400 Subject: Re: [RFC 3/8] nvmet: Use p2pmem in nvme target To: Logan Gunthorpe , Christoph Hellwig , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1490911959-5146-4-git-send-email-logang@deltatee.com> Cc: linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@ml01.01.org, linux-kernel@vger.kernel.org, Sinan Kaya From: Sagi Grimberg Message-ID: <0689e764-bf04-6da2-3b7d-2cbf0b6b94a0@grimberg.me> Date: Thu, 6 Apr 2017 08:47:22 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > I hadn't done this yet but I think a simple closest device in the tree > would solve the issue sufficiently. However, I originally had it so the > user has to pick the device and I prefer that approach. But if the user > picks the device, then why bother restricting what he picks? Because the user can get it wrong, and its our job to do what we can in order to prevent the user from screwing itself. > Per the > thread with Sinan, I'd prefer to use what the user picks. You were one > of the biggest opponents to that so I'd like to hear your opinion on > removing the restrictions. I wasn't against it that much, I'm all for making things "just work" with minimal configuration steps, but I'm not sure we can get it right without it. >>> Ideally, we'd want to use an NVME CMB buffer as p2p memory. This would >>> save an extra PCI transfer as the NVME card could just take the data >>> out of it's own memory. However, at this time, cards with CMB buffers >>> don't seem to be available. >> >> Even if it was available, it would be hard to make real use of this >> given that we wouldn't know how to pre-post recv buffers (for in-capsule >> data). But let's leave this out of the scope entirely... > > I don't understand what you're referring to. We'd simply use the CMB > buffer as a p2pmem device, why does that change anything? I'm referring to the in-capsule data buffers pre-posts that we do. Because we prepare a buffer that would contain in-capsule data, we have no knowledge to which device the incoming I/O is directed to, which means we can (and will) have I/O where the data lies in CMB of device A but it's really targeted to device B - which sorta defeats the purpose of what we're trying to optimize here... >> Why do you need this? you have a reference to the >> queue itself. > > This keeps track of whether the response was actually allocated with > p2pmem or not. It's needed for when we free the SGL because the queue > may have a p2pmem device assigned to it but, if the alloc failed and it > fell back on system memory then we need to know how to free it. I'm > currently looking at having SGLs having an iomem flag. In which case, > this would no longer be needed as the flag in the SGL could be used. That would be better, maybe... [...] >> This is a problem. namespaces can be added at any point in time. No one >> guarantee that dma_devs are all the namepaces we'll ever see. > > Yeah, well restricting p2pmem based on all the devices in use is hard. > So we'd need a call into the transport every time an ns is added and > we'd have to drop the p2pmem if they add one that isn't supported. This > complexity is just one of the reasons I prefer just letting the user chose. Still the user can get it wrong. Not sure we can get a way without keeping track of this as new devices join the subsystem. >>> + >>> + if (queue->p2pmem) >>> + pr_debug("using %s for rdma nvme target queue", >>> + dev_name(&queue->p2pmem->dev)); >>> + >>> + kfree(dma_devs); >>> +} >>> + >>> static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, >>> struct rdma_cm_event *event) >>> { >>> @@ -1199,6 +1271,8 @@ static int nvmet_rdma_queue_connect(struct >>> rdma_cm_id *cm_id, >>> } >>> queue->port = cm_id->context; >>> >>> + nvmet_rdma_queue_setup_p2pmem(queue); >>> + >> >> Why is all this done for each queue? looks completely redundant to me. > > A little bit. Where would you put it? I think we'll need a representation of a controller in nvmet-rdma for that. we sort of got a way without it so far, but I don't think we can anymore with this. >>> ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn); >>> if (ret) >>> goto release_queue; >> >> You seemed to skip the in-capsule buffers for p2pmem (inline_page), I'm >> curious why? > > Yes, the thinking was that these transfers were small anyway so there > would not be significant benefit to pushing them through p2pmem. There's > really no reason why we couldn't do that if it made sense to though. I don't see an urgent reason for it too. I was just curious... From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f195.google.com ([209.85.128.195]:33759 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755720AbdDFFr2 (ORCPT ); Thu, 6 Apr 2017 01:47:28 -0400 Subject: Re: [RFC 3/8] nvmet: Use p2pmem in nvme target To: Logan Gunthorpe , Christoph Hellwig , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , Jason Gunthorpe References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1490911959-5146-4-git-send-email-logang@deltatee.com> Cc: linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org, Sinan Kaya From: Sagi Grimberg Message-ID: <0689e764-bf04-6da2-3b7d-2cbf0b6b94a0@grimberg.me> Date: Thu, 6 Apr 2017 08:47:22 +0300 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Sender: linux-pci-owner@vger.kernel.org List-ID: > I hadn't done this yet but I think a simple closest device in the tree > would solve the issue sufficiently. However, I originally had it so the > user has to pick the device and I prefer that approach. But if the user > picks the device, then why bother restricting what he picks? Because the user can get it wrong, and its our job to do what we can in order to prevent the user from screwing itself. > Per the > thread with Sinan, I'd prefer to use what the user picks. You were one > of the biggest opponents to that so I'd like to hear your opinion on > removing the restrictions. I wasn't against it that much, I'm all for making things "just work" with minimal configuration steps, but I'm not sure we can get it right without it. >>> Ideally, we'd want to use an NVME CMB buffer as p2p memory. This would >>> save an extra PCI transfer as the NVME card could just take the data >>> out of it's own memory. However, at this time, cards with CMB buffers >>> don't seem to be available. >> >> Even if it was available, it would be hard to make real use of this >> given that we wouldn't know how to pre-post recv buffers (for in-capsule >> data). But let's leave this out of the scope entirely... > > I don't understand what you're referring to. We'd simply use the CMB > buffer as a p2pmem device, why does that change anything? I'm referring to the in-capsule data buffers pre-posts that we do. Because we prepare a buffer that would contain in-capsule data, we have no knowledge to which device the incoming I/O is directed to, which means we can (and will) have I/O where the data lies in CMB of device A but it's really targeted to device B - which sorta defeats the purpose of what we're trying to optimize here... >> Why do you need this? you have a reference to the >> queue itself. > > This keeps track of whether the response was actually allocated with > p2pmem or not. It's needed for when we free the SGL because the queue > may have a p2pmem device assigned to it but, if the alloc failed and it > fell back on system memory then we need to know how to free it. I'm > currently looking at having SGLs having an iomem flag. In which case, > this would no longer be needed as the flag in the SGL could be used. That would be better, maybe... [...] >> This is a problem. namespaces can be added at any point in time. No one >> guarantee that dma_devs are all the namepaces we'll ever see. > > Yeah, well restricting p2pmem based on all the devices in use is hard. > So we'd need a call into the transport every time an ns is added and > we'd have to drop the p2pmem if they add one that isn't supported. This > complexity is just one of the reasons I prefer just letting the user chose. Still the user can get it wrong. Not sure we can get a way without keeping track of this as new devices join the subsystem. >>> + >>> + if (queue->p2pmem) >>> + pr_debug("using %s for rdma nvme target queue", >>> + dev_name(&queue->p2pmem->dev)); >>> + >>> + kfree(dma_devs); >>> +} >>> + >>> static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, >>> struct rdma_cm_event *event) >>> { >>> @@ -1199,6 +1271,8 @@ static int nvmet_rdma_queue_connect(struct >>> rdma_cm_id *cm_id, >>> } >>> queue->port = cm_id->context; >>> >>> + nvmet_rdma_queue_setup_p2pmem(queue); >>> + >> >> Why is all this done for each queue? looks completely redundant to me. > > A little bit. Where would you put it? I think we'll need a representation of a controller in nvmet-rdma for that. we sort of got a way without it so far, but I don't think we can anymore with this. >>> ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn); >>> if (ret) >>> goto release_queue; >> >> You seemed to skip the in-capsule buffers for p2pmem (inline_page), I'm >> curious why? > > Yes, the thinking was that these transfers were small anyway so there > would not be significant benefit to pushing them through p2pmem. There's > really no reason why we couldn't do that if it made sense to though. I don't see an urgent reason for it too. I was just curious... From mboxrd@z Thu Jan 1 00:00:00 1970 From: sagi@grimberg.me (Sagi Grimberg) Date: Thu, 6 Apr 2017 08:47:22 +0300 Subject: [RFC 3/8] nvmet: Use p2pmem in nvme target In-Reply-To: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1490911959-5146-4-git-send-email-logang@deltatee.com> Message-ID: <0689e764-bf04-6da2-3b7d-2cbf0b6b94a0@grimberg.me> > I hadn't done this yet but I think a simple closest device in the tree > would solve the issue sufficiently. However, I originally had it so the > user has to pick the device and I prefer that approach. But if the user > picks the device, then why bother restricting what he picks? Because the user can get it wrong, and its our job to do what we can in order to prevent the user from screwing itself. > Per the > thread with Sinan, I'd prefer to use what the user picks. You were one > of the biggest opponents to that so I'd like to hear your opinion on > removing the restrictions. I wasn't against it that much, I'm all for making things "just work" with minimal configuration steps, but I'm not sure we can get it right without it. >>> Ideally, we'd want to use an NVME CMB buffer as p2p memory. This would >>> save an extra PCI transfer as the NVME card could just take the data >>> out of it's own memory. However, at this time, cards with CMB buffers >>> don't seem to be available. >> >> Even if it was available, it would be hard to make real use of this >> given that we wouldn't know how to pre-post recv buffers (for in-capsule >> data). But let's leave this out of the scope entirely... > > I don't understand what you're referring to. We'd simply use the CMB > buffer as a p2pmem device, why does that change anything? I'm referring to the in-capsule data buffers pre-posts that we do. Because we prepare a buffer that would contain in-capsule data, we have no knowledge to which device the incoming I/O is directed to, which means we can (and will) have I/O where the data lies in CMB of device A but it's really targeted to device B - which sorta defeats the purpose of what we're trying to optimize here... >> Why do you need this? you have a reference to the >> queue itself. > > This keeps track of whether the response was actually allocated with > p2pmem or not. It's needed for when we free the SGL because the queue > may have a p2pmem device assigned to it but, if the alloc failed and it > fell back on system memory then we need to know how to free it. I'm > currently looking at having SGLs having an iomem flag. In which case, > this would no longer be needed as the flag in the SGL could be used. That would be better, maybe... [...] >> This is a problem. namespaces can be added at any point in time. No one >> guarantee that dma_devs are all the namepaces we'll ever see. > > Yeah, well restricting p2pmem based on all the devices in use is hard. > So we'd need a call into the transport every time an ns is added and > we'd have to drop the p2pmem if they add one that isn't supported. This > complexity is just one of the reasons I prefer just letting the user chose. Still the user can get it wrong. Not sure we can get a way without keeping track of this as new devices join the subsystem. >>> + >>> + if (queue->p2pmem) >>> + pr_debug("using %s for rdma nvme target queue", >>> + dev_name(&queue->p2pmem->dev)); >>> + >>> + kfree(dma_devs); >>> +} >>> + >>> static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, >>> struct rdma_cm_event *event) >>> { >>> @@ -1199,6 +1271,8 @@ static int nvmet_rdma_queue_connect(struct >>> rdma_cm_id *cm_id, >>> } >>> queue->port = cm_id->context; >>> >>> + nvmet_rdma_queue_setup_p2pmem(queue); >>> + >> >> Why is all this done for each queue? looks completely redundant to me. > > A little bit. Where would you put it? I think we'll need a representation of a controller in nvmet-rdma for that. we sort of got a way without it so far, but I don't think we can anymore with this. >>> ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn); >>> if (ret) >>> goto release_queue; >> >> You seemed to skip the in-capsule buffers for p2pmem (inline_page), I'm >> curious why? > > Yes, the thinking was that these transfers were small anyway so there > would not be significant benefit to pushing them through p2pmem. There's > really no reason why we couldn't do that if it made sense to though. I don't see an urgent reason for it too. I was just curious...