From: Jason Gunthorpe <jgg@ziepe.ca>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Oded Gabbay <oded.gabbay@gmail.com>,
Leon Romanovsky <leon@kernel.org>,
Gal Pressman <galpress@amazon.com>,
Jakub Kicinski <kuba@kernel.org>,
"Linux-Kernel@Vger. Kernel. Org" <linux-kernel@vger.kernel.org>,
netdev@vger.kernel.org, SW_Drivers <SW_Drivers@habana.ai>,
"David S. Miller" <davem@davemloft.net>,
Andrew Lunn <andrew@lunn.ch>,
Florian Fainelli <f.fainelli@gmail.com>,
linux-rdma@vger.kernel.org
Subject: Re: [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver
Date: Mon, 21 Sep 2020 08:52:39 -0300 [thread overview]
Message-ID: <20200921115239.GC8409@ziepe.ca> (raw)
In-Reply-To: <20200920084702.GA533114@kroah.com>
On Sun, Sep 20, 2020 at 10:47:02AM +0200, Greg Kroah-Hartman wrote:
> > If not, what open source userspace are you going to ask them to
> > present to merge the kernel side into misc?
>
> I don't think that they have a userspace api to their rdma feature from
> what I understand, but I could be totally wrong as I do not know their
> hardware at all, so I'll let them answer this question.
I thought Oded was pretty clear, the goal of this series is to expose
their RDMA HW to userspace. This problem space requires co-mingling
networking and compute at extremely high speed/low overhead. This is
all done in userspace.
We are specifically talking about this in
include/uapi/misc/habanalabs.h:
/*
* NIC
*
* This IOCTL allows the user to manage and configure the device's NIC ports.
* The following operations are available:
* - Create a completion queue
* - Destroy a completion queue
* - Wait on completion queue
* - Poll a completion queue
* - Update consumed completion queue entries
* - Set a work queue
* - Unset a work queue
*
* For all operations, the user should provide a pointer to an input structure
* with the context parameters. Some of the operations also require a pointer to
* driver regarding how many of the available CQEs were actually
* processed/consumed. Only then the driver will override them with newer
* entries.
* The set WQ operation should provide the device virtual address of the WQ with
* a matching size for the number of WQs and entries per WQ.
*
*/
#define HL_IOCTL_NIC _IOWR('H', 0x07, struct hl_nic_args)
Which is ibv_create_qp, ibv_create_cq, ibv_poll_cq, etc, etc
Habana has repeatedly described their HW as having multiple 100G RoCE
ports. RoCE is one of the common industry standards that ibverbs
unambiguously is responsible for.
I would be much less annoyed if they were not actively marketing their
product as RoCE RDMA.
Sure there is some argument that their RoCE isn't spec compliant, but
I don't think it excuses the basic principle of our subsystem:
RDMA HW needs to demonstrate some basic functionality using the
standard open source userspace software stack.
I don't like this idea of backdooring a bunch of proprietary closed
source RDMA userspace through drivers/misc, and if you don't have a
clear idea how to get something equal for drivers/misc you should not
accept the H_IOCTL_NIC.
Plus RoCE is complicated, there is a bunch of interaction with netdev
and rules related to that that really needs to be respected.
> For anything that _has_ to have a userspace RMDA interface, sure ibverbs
> are the one we are stuck with, but I didn't think that was the issue
> here at all, which is why I wrote the above comments.
I think you should look at the patches #8 through 11:
https://lore.kernel.org/lkml/20200915171022.10561-9-oded.gabbay@gmail.com/
Jason
next prev parent reply other threads:[~2020-09-21 11:52 UTC|newest]
Thread overview: 83+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-15 17:10 [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 02/14] habanalabs/gaudi: add NIC firmware-related definitions Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 03/14] habanalabs/gaudi: add NIC security configuration Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 04/14] habanalabs/gaudi: add support for NIC QMANs Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 05/14] habanalabs/gaudi: add NIC Ethernet support Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 06/14] habanalabs/gaudi: add NIC PHY code Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 07/14] habanalabs/gaudi: allow user to get MAC addresses in INFO IOCTL Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 08/14] habanalabs/gaudi: add a new IOCTL for NIC control operations Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 09/14] habanalabs/gaudi: add CQ " Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 10/14] habanalabs/gaudi: add WQ " Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 11/14] habanalabs/gaudi: add QP error handling Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 12/14] habanalabs/gaudi: Add ethtool support using coresight Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 13/14] habanalabs/gaudi: support DCB protocol Oded Gabbay
2020-09-15 17:10 ` [PATCH v3 14/14] habanalabs/gaudi: add NIC init/fini calls from common code Oded Gabbay
2020-09-15 20:35 ` [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver Jakub Kicinski
2020-09-15 20:46 ` Oded Gabbay
2020-09-15 21:04 ` Jakub Kicinski
2020-09-15 21:20 ` Oded Gabbay
2020-09-15 21:37 ` Andrew Lunn
2020-09-15 21:43 ` Oded Gabbay
2020-09-15 22:35 ` David Miller
2020-09-15 22:36 ` David Miller
2020-09-15 22:34 ` David Miller
2020-09-16 4:26 ` Oded Gabbay
2020-09-17 17:18 ` Jason Gunthorpe
2020-09-18 11:36 ` Gal Pressman
2020-09-18 11:52 ` Leon Romanovsky
2020-09-18 11:56 ` Oded Gabbay
2020-09-18 12:03 ` Leon Romanovsky
2020-09-18 12:07 ` Oded Gabbay
2020-09-18 12:19 ` Leon Romanovsky
2020-09-18 12:31 ` Oded Gabbay
2020-09-18 13:09 ` Leon Romanovsky
2020-09-19 6:40 ` Greg Kroah-Hartman
2020-09-19 8:20 ` Leon Romanovsky
2020-09-19 8:30 ` Greg Kroah-Hartman
2020-09-19 8:58 ` Leon Romanovsky
2020-09-19 16:43 ` Oded Gabbay
2020-09-19 17:27 ` Greg Kroah-Hartman
2020-09-19 19:22 ` Jason Gunthorpe
2020-09-20 8:47 ` Greg Kroah-Hartman
2020-09-20 19:05 ` Oded Gabbay
2020-09-21 10:39 ` Leon Romanovsky
2020-09-21 11:52 ` Jason Gunthorpe [this message]
2020-09-21 21:20 ` Jakub Kicinski
2020-09-22 11:49 ` Jason Gunthorpe
2020-09-19 18:49 ` Andrew Lunn
2020-09-18 11:56 ` Jason Gunthorpe
2020-09-18 11:59 ` Oded Gabbay
2020-09-18 12:16 ` Jason Gunthorpe
2020-09-18 12:34 ` Oded Gabbay
2020-09-18 12:50 ` Jason Gunthorpe
2020-09-18 13:02 ` Oded Gabbay
2020-09-18 13:26 ` Jason Gunthorpe
2020-09-18 13:49 ` Oded Gabbay
2020-09-18 13:59 ` Jason Gunthorpe
2020-09-18 14:12 ` Oded Gabbay
2020-09-18 14:19 ` Jason Gunthorpe
2020-09-18 14:45 ` Oded Gabbay
2020-09-18 15:07 ` Jason Gunthorpe
2020-09-18 15:15 ` Oded Gabbay
2020-09-18 15:28 ` Jason Gunthorpe
2020-09-21 11:22 ` Gal Pressman
2020-09-21 11:49 ` Leon Romanovsky
2020-09-22 11:41 ` Jason Gunthorpe
2020-09-22 12:46 ` Gal Pressman
2020-09-22 16:14 ` Jason Gunthorpe
2020-09-22 16:30 ` Gal Pressman
2020-09-22 16:52 ` Jason Gunthorpe
2020-09-18 12:10 ` Oded Gabbay
2020-09-15 20:42 ` David Miller
2020-09-15 20:49 ` Oded Gabbay
2020-09-16 6:26 ` Greg Kroah-Hartman
2020-09-16 6:36 ` Oded Gabbay
2020-09-16 7:42 ` Greg Kroah-Hartman
2020-09-16 8:02 ` Oded Gabbay
2020-09-16 8:22 ` Greg Kroah-Hartman
2020-09-16 8:47 ` Oded Gabbay
2020-09-16 12:00 ` Greg Kroah-Hartman
2020-09-20 16:45 ` Daniel Vetter
2020-09-16 23:04 ` Williams, Dan J
2020-09-18 12:00 ` Jason Gunthorpe
2020-09-18 12:01 ` Oded Gabbay
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200921115239.GC8409@ziepe.ca \
--to=jgg@ziepe.ca \
--cc=SW_Drivers@habana.ai \
--cc=andrew@lunn.ch \
--cc=davem@davemloft.net \
--cc=f.fainelli@gmail.com \
--cc=galpress@amazon.com \
--cc=gregkh@linuxfoundation.org \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=oded.gabbay@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).