Re: [PATCH 00/15] Adding GAUDI NIC code to habanalabs driver

From: Greg KH <gregkh@linuxfoundation.org>
To: Oded Gabbay <oded.gabbay@gmail.com>
Cc: "Linux-Kernel@Vger. Kernel. Org" <linux-kernel@vger.kernel.org>,
	SW_Drivers <SW_Drivers@habana.ai>
Subject: Re: [PATCH 00/15] Adding GAUDI NIC code to habanalabs driver
Date: Thu, 10 Sep 2020 18:08:18 +0200	[thread overview]
Message-ID: <20200910160818.GA1158578@kroah.com> (raw)
In-Reply-To: <CAFCwf132vzvEQ2Jrzme8SuK8OCSB8-QTXiwr5Rj96_hwi91TKQ@mail.gmail.com>

On Thu, Sep 10, 2020 at 06:59:41PM +0300, Oded Gabbay wrote:
> On Thu, Sep 10, 2020 at 6:54 PM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Sep 10, 2020 at 06:03:13PM +0300, Oded Gabbay wrote:
> > > This patch-set adds support for initializing and using the GAUDI NIC ports,
> > > functioning as scale-out interconnect when doing distributed Deep Learning
> > > training. The training can be performed over tens of thousands of GAUDIs
> > > and it is done using the RDMA-over-converged-Ethernet (RoCE) v2 protocol.
> > >
> > > Each GAUDI exposes 10x100GbE ports that are designed to scale-out the
> > > inter-GAUDI communication by integrating a complete communication engine
> > > on-die. This native integration allows users to use the same scaling
> > > technology, both inside the server and rack (termed as scale-up), as well
> > > as for scaling across racks (scale-out). The racks can be connected
> > > directly between GAUDI processors, or through any number of standard
> > > Ethernet switches.
> > >
> > > The driver exposes the NIC ports to the user as standard Ethernet ports by
> > > registering each port to the networking subsystem. This allows the user to
> > > manage the ports with standard tools such as ifconfig, ethtool, etc. It
> > > also enables us to connect to the Linux networking stack and thus support
> > > standard networking protocols, such as IPv4, IPv6, TCP, etc. In addition,
> > > we can also leverage protocols such as DCB for dynamically configuring
> > > priorities to avoid congestion.
> > >
> > > For each NIC port there is a matching QMAN entity. For RoCE, the user
> > > submits workloads to the NIC through the QMAN, same as he does for the
> > > compute engines. For regular Ethernet, the user sends and receives packets
> > > through the standard Ethernet sockets. Those sockets are used only as a
> > > control path. The data path that is used for AI training goes through the
> > > RoCE interface.
> > >
> > > It is important to note that there are some limitations and uniqueness
> > > in GAUDI's NIC H/W, compared to other networking adapters that enforced us
> > > to use a less-than-common driver design:
> > >
> > > 1. The NIC functionality is NOT exposed as different PCI Physical
> > >    Functions. There is a single PF which is used for compute and
> > >    networking, as the main goal of the NIC ports is to be used as
> > >    intra-communication and not as standard network interfaces. This
> > >    implies we can't connect different drivers to handle the networking
> > >    ports because it is the same device, from the kernel POV, as the
> > >    compute. Therefore, we must integrate the networking code into the
> > >    main habanalabs driver.
> >
> > That's kind of common, see the long threads on the netdev and IB mailing
> > lists about this type of issue on other networking cards today.  The
> > whole "virtual bus" code should help solve this, if Intel ever gets
> > around to posting a new version of that patch series one day...
> >
> > But, because you are writing networking driver code here, you really
> > should run all of this by the netdev@vger.kernel.org maintainers and
> > developers, as they know how to review this interaction with the network
> > stack better than anyone else.
> >
> > Care to resend it and cc: them too?
> >
> > thanks,
> >
> > greg k-h
> 
> Sure, np.
> Do you have someone specific or should I just send it to that mailing list ?
> I don't know anyone there.

the MAINTAINERS file should help you:
	NETWORKING DRIVERS
	M:      "David S. Miller" <davem@davemloft.net>
	M:      Jakub Kicinski <kuba@kernel.org>
	L:      netdev@vger.kernel.org
	S:      Maintained
	W:      http://www.linuxfoundation.org/en/Net
	Q:      http://patchwork.ozlabs.org/project/netdev/list/
	T:      git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
	T:      git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
	F:      Documentation/devicetree/bindings/net/
	F:      drivers/net/
	F:      include/linux/etherdevice.h
	F:      include/linux/fcdevice.h
	F:      include/linux/fddidevice.h
	F:      include/linux/hippidevice.h
	F:      include/linux/if_*
	F:      include/linux/inetdevice.h
	F:      include/linux/netdevice.h
	F:      include/uapi/linux/if_*
	F:      include/uapi/linux/netdevice.h

:)