From: Logan Gunthorpe <logang@deltatee.com> To: Sagi Grimberg <sagi@grimberg.me>, Hannes Reinecke <hare@suse.de>, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org Cc: Christoph Hellwig <hch@lst.de>, Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@fb.com>, Chaitanya Kulkarni <Chaitanya.Kulkarni@wdc.com>, Max Gurtovoy <maxg@mellanox.com>, Stephen Bates <sbates@raithlin.com> Subject: Re: [PATCH v6 00/16] nvmet: add target passthru commands support Date: Fri, 26 Jul 2019 18:09:30 -0600 [thread overview] Message-ID: <1260e01c-6731-52f7-ae83-0b90e0345c68@deltatee.com> (raw) In-Reply-To: <e166c392-1548-f0bb-02bc-ced3dd85f301@grimberg.me> On 2019-07-26 5:13 p.m., Sagi Grimberg wrote: > >>> Why? if nvmet is capable, why shouldn't we support it? >> >> I'm saying that passthru is exporting a specific controller and submits >> commands (both admin and IO) straight to the nvme_ctrl's queues. It's >> not exporting an nvme_subsys and I think it would be troublesome to do >> so; for example, if the target receives an admin command which ctrl of >> the subsystem should it send it to? > > Its the same controller in the backend, what is the difference from > which fabrics controller the admin command came from? This is not my understanding. It's not really the same controller in the back end and there are admin commands that operate on the controller like the namespace attachment command which takes a list of cntlids (though admittedly is not something I'm too familiar with because I don't have any hardware to play around with). Though that command is already a bit problematic for passthru because we have different cntlid address spaces. > I haven't thought this through so its very possible that I'm missing > something, but why can't the host see multiple controllers if it has > more than one path to the target? Well a target controller is created for each connection. So if the host wanted to see multiple controllers it would have to do multiple "nvme connects" and some how need to address the individual controllers for each connection. Right now a connect is based on subsysnqn which would be the same for every multipath controller. > What specific admin commands are you concerned about? What exactly > would clash? Namespace attach comes to mind. > And I'm suggesting to allow more than a single controller given that all > controller allocations match a single hostnqn. It wouldn't make sense to > expose this controller to multiple hosts (although that might be doable > but but definitely requires non-trivial infrastructure around it). > Look, when it comes to fabrics, multipath is a fundamental piece of the > puzzle. Not supporting multipathing significantly diminishes the value > of this in my mind (assuming this answers a real-world use-case). I'd agree with that. But it's the multipath through different ports that seems important for fabrics. ie. If I have a host with a path through RDMA and a path through TCP they should both work and allow fail over. This is quite orthogonal to passthru and would be easily supported if we dropped the multiple hosts restriction (I'm not sure what the objection really is to that). This is different from multipath on say a multi-controller PCI device and trying to expose both those controllers through passthru. this is where the problems we are discussing come up. Supporting this is what is hard and I think the sensible answer is if users want to do something like that, they use non-passthru NVME-of and the multipath code will just work as designed. Our real-world use case is to support our PCI device which has a bunch of vendor unique commands and isn't likely to support multiple controllers in the foreseeable future. Logan
WARNING: multiple messages have this Message-ID (diff)
From: logang@deltatee.com (Logan Gunthorpe) Subject: [PATCH v6 00/16] nvmet: add target passthru commands support Date: Fri, 26 Jul 2019 18:09:30 -0600 [thread overview] Message-ID: <1260e01c-6731-52f7-ae83-0b90e0345c68@deltatee.com> (raw) In-Reply-To: <e166c392-1548-f0bb-02bc-ced3dd85f301@grimberg.me> On 2019-07-26 5:13 p.m., Sagi Grimberg wrote: > >>> Why? if nvmet is capable, why shouldn't we support it? >> >> I'm saying that passthru is exporting a specific controller and submits >> commands (both admin and IO) straight to the nvme_ctrl's queues. It's >> not exporting an nvme_subsys and I think it would be troublesome to do >> so; for example, if the target receives an admin command which ctrl of >> the subsystem should it send it to? > > Its the same controller in the backend, what is the difference from > which fabrics controller the admin command came from? This is not my understanding. It's not really the same controller in the back end and there are admin commands that operate on the controller like the namespace attachment command which takes a list of cntlids (though admittedly is not something I'm too familiar with because I don't have any hardware to play around with). Though that command is already a bit problematic for passthru because we have different cntlid address spaces. > I haven't thought this through so its very possible that I'm missing > something, but why can't the host see multiple controllers if it has > more than one path to the target? Well a target controller is created for each connection. So if the host wanted to see multiple controllers it would have to do multiple "nvme connects" and some how need to address the individual controllers for each connection. Right now a connect is based on subsysnqn which would be the same for every multipath controller. > What specific admin commands are you concerned about? What exactly > would clash? Namespace attach comes to mind. > And I'm suggesting to allow more than a single controller given that all > controller allocations match a single hostnqn. It wouldn't make sense to > expose this controller to multiple hosts (although that might be doable > but but definitely requires non-trivial infrastructure around it). > Look, when it comes to fabrics, multipath is a fundamental piece of the > puzzle. Not supporting multipathing significantly diminishes the value > of this in my mind (assuming this answers a real-world use-case). I'd agree with that. But it's the multipath through different ports that seems important for fabrics. ie. If I have a host with a path through RDMA and a path through TCP they should both work and allow fail over. This is quite orthogonal to passthru and would be easily supported if we dropped the multiple hosts restriction (I'm not sure what the objection really is to that). This is different from multipath on say a multi-controller PCI device and trying to expose both those controllers through passthru. this is where the problems we are discussing come up. Supporting this is what is hard and I think the sensible answer is if users want to do something like that, they use non-passthru NVME-of and the multipath code will just work as designed. Our real-world use case is to support our PCI device which has a bunch of vendor unique commands and isn't likely to support multiple controllers in the foreseeable future. Logan
next prev parent reply other threads:[~2019-07-27 0:09 UTC|newest] Thread overview: 122+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-07-25 17:23 [PATCH v6 00/16] nvmet: add target passthru commands support Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 01/16] chardev: factor out cdev_lookup() helper Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 02/16] chardev: introduce cdev_get_by_path() Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:40 ` Greg Kroah-Hartman 2019-07-25 17:40 ` Greg Kroah-Hartman 2019-07-25 17:53 ` Logan Gunthorpe 2019-07-25 17:53 ` Logan Gunthorpe 2019-07-25 17:58 ` Matthew Wilcox 2019-07-25 17:58 ` Matthew Wilcox 2019-07-25 18:08 ` Logan Gunthorpe 2019-07-25 18:08 ` Logan Gunthorpe 2019-07-25 18:08 ` Greg Kroah-Hartman 2019-07-25 18:08 ` Greg Kroah-Hartman 2019-07-25 18:14 ` Logan Gunthorpe 2019-07-25 18:14 ` Logan Gunthorpe 2019-07-25 18:27 ` Greg Kroah-Hartman 2019-07-25 18:27 ` Greg Kroah-Hartman 2019-07-25 18:36 ` Logan Gunthorpe 2019-07-25 18:36 ` Logan Gunthorpe 2019-07-25 19:02 ` Sagi Grimberg 2019-07-25 19:02 ` Sagi Grimberg 2019-07-25 19:34 ` Greg Kroah-Hartman 2019-07-25 19:34 ` Greg Kroah-Hartman 2019-07-25 19:37 ` Sagi Grimberg 2019-07-25 19:37 ` Sagi Grimberg 2019-07-25 19:43 ` Greg Kroah-Hartman 2019-07-25 19:43 ` Greg Kroah-Hartman 2019-07-25 19:45 ` Sagi Grimberg 2019-07-25 19:45 ` Sagi Grimberg 2019-07-25 19:43 ` Sagi Grimberg 2019-07-25 19:43 ` Sagi Grimberg 2019-07-25 19:41 ` Logan Gunthorpe 2019-07-25 19:41 ` Logan Gunthorpe 2019-07-25 19:00 ` Matthew Wilcox 2019-07-25 19:00 ` Matthew Wilcox 2019-07-25 19:05 ` Sagi Grimberg 2019-07-25 19:05 ` Sagi Grimberg 2019-07-25 19:11 ` Matthew Wilcox 2019-07-25 19:11 ` Matthew Wilcox 2019-07-25 19:24 ` Logan Gunthorpe 2019-07-25 19:24 ` Logan Gunthorpe 2019-07-25 19:26 ` Matthew Wilcox 2019-07-25 19:26 ` Matthew Wilcox 2019-07-25 19:31 ` Logan Gunthorpe 2019-07-25 19:31 ` Logan Gunthorpe 2019-07-25 23:55 ` Al Viro 2019-07-25 23:55 ` Al Viro 2019-07-26 4:29 ` Sagi Grimberg 2019-07-26 4:29 ` Sagi Grimberg 2019-07-26 7:13 ` Greg Kroah-Hartman 2019-07-26 7:13 ` Greg Kroah-Hartman 2019-07-26 15:46 ` Logan Gunthorpe 2019-07-26 15:46 ` Logan Gunthorpe 2019-07-25 19:31 ` Sagi Grimberg 2019-07-25 19:31 ` Sagi Grimberg 2019-07-25 18:10 ` Greg Kroah-Hartman 2019-07-25 18:10 ` Greg Kroah-Hartman 2019-07-25 18:16 ` Logan Gunthorpe 2019-07-25 18:16 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 03/16] chardev: export cdev_put() Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 04/16] nvme-core: introduce nvme_get_by_path() Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:50 ` Matthew Wilcox 2019-07-25 17:50 ` Matthew Wilcox 2019-07-25 17:54 ` Logan Gunthorpe 2019-07-25 17:54 ` Logan Gunthorpe 2019-07-25 19:58 ` Keith Busch 2019-07-25 19:58 ` Keith Busch 2019-07-25 20:12 ` Sagi Grimberg 2019-07-25 20:12 ` Sagi Grimberg 2019-07-25 20:28 ` Logan Gunthorpe 2019-07-25 20:28 ` Logan Gunthorpe 2019-07-25 20:31 ` Keith Busch 2019-07-25 20:31 ` Keith Busch 2019-07-25 20:37 ` Logan Gunthorpe 2019-07-25 20:37 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 05/16] nvme-core: export existing ctrl and ns interfaces Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 06/16] nvmet: add return value to nvmet_add_async_event() Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 07/16] nvmet: make nvmet_copy_ns_identifier() non-static Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 08/16] nvmet-passthru: update KConfig with config passthru option Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 09/16] nvmet-passthru: add passthru code to process commands Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 10/16] nvmet-passthru: add enable/disable helpers Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 11/16] nvmet-core: allow one host per passthru-ctrl Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 12/16] nvmet-core: don't check the data len for pt-ctrl Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 13/16] nvmet-configfs: introduce passthru configfs interface Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 14/16] block: don't check blk_rq_is_passthrough() in blk_do_io_stat() Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 15/16] block: call blk_account_io_start() in blk_execute_rq_nowait() Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-25 17:23 ` [PATCH v6 16/16] nvmet-passthru: support block accounting Logan Gunthorpe 2019-07-25 17:23 ` Logan Gunthorpe 2019-07-26 6:23 ` [PATCH v6 00/16] nvmet: add target passthru commands support Hannes Reinecke 2019-07-26 6:23 ` Hannes Reinecke 2019-07-26 17:07 ` Logan Gunthorpe 2019-07-26 17:07 ` Logan Gunthorpe 2019-07-26 22:21 ` Sagi Grimberg 2019-07-26 22:21 ` Sagi Grimberg 2019-07-26 22:37 ` Logan Gunthorpe 2019-07-26 22:37 ` Logan Gunthorpe 2019-07-26 23:13 ` Sagi Grimberg 2019-07-26 23:13 ` Sagi Grimberg 2019-07-27 0:09 ` Logan Gunthorpe [this message] 2019-07-27 0:09 ` Logan Gunthorpe 2019-07-27 0:50 ` Stephen Bates 2019-07-27 0:50 ` Stephen Bates 2019-07-29 16:15 ` Sagi Grimberg 2019-07-29 16:15 ` Sagi Grimberg 2019-07-29 16:17 ` Logan Gunthorpe 2019-07-29 16:17 ` Logan Gunthorpe
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1260e01c-6731-52f7-ae83-0b90e0345c68@deltatee.com \ --to=logang@deltatee.com \ --cc=Chaitanya.Kulkarni@wdc.com \ --cc=axboe@fb.com \ --cc=hare@suse.de \ --cc=hch@lst.de \ --cc=kbusch@kernel.org \ --cc=linux-block@vger.kernel.org \ --cc=linux-fsdevel@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-nvme@lists.infradead.org \ --cc=maxg@mellanox.com \ --cc=sagi@grimberg.me \ --cc=sbates@raithlin.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.