Re: [for-next 4/6] net/mlx5: FPGA, Add basic support for Innova

From: Alexei Starovoitov <alexei.starovoitov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
To: Jason Gunthorpe
	<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
Cc: Ilan Tayari <ilant-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
	Saeed Mahameed
	<saeedm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>,
	"David S. Miller" <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>,
	Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
	"netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	"linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	"jsorensen-b10kYP2dOMg@public.gmane.org"
	<jsorensen-b10kYP2dOMg@public.gmane.org>,
	Andy Shevchenko
	<andy.shevchenko-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
	"linux-fpga-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<linux-fpga-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	Alan Tull
	<atull-yzvPICuk2ABMcg4IHK0kFoH6Mc4MB0Vx@public.gmane.org>,
	"yi1.li-VuQAYsv1563Yd54FQh9/CA@public.gmane.org"
	<yi1.li-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>,
	Boris Pismenny <borisp-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Subject: Re: [for-next 4/6] net/mlx5: FPGA, Add basic support for Innova
Date: Tue, 6 Jun 2017 10:42:35 -0700	[thread overview]
Message-ID: <20170606174233.w377ctwtapzccsk7@ast-mbp.dhcp.thefacebook.com> (raw)
In-Reply-To: <20170606161709.GA8671-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>

On Tue, Jun 06, 2017 at 10:17:09AM -0600, Jason Gunthorpe wrote:
> On Tue, Jun 06, 2017 at 06:52:15AM +0000, Ilan Tayari wrote:
> 
> > So neither the host stack nor the network are aware of them.
> > They exist momentarily only on the internal traces on the board and not
> > anywhere else.
> 
> Is that really true? If you are creating rocee QPs' then the RDMA
> stack sees this stuff and now we have buried a RDMA ULP inside an
> ethernet driver which seems really wonky..
> 
> > I don't mind explaining further, but I think you will just see it in the
> > patchset when we submit.
> 
> You described exactly what I thought.. I just disagree with you that
> an ethernet connected and controlled IP accelerator is 'part of the
> NIC', even if it happens to be colocated on the same circuit board.

+1

what Ilan described is a kernel bypass done by hw.
This is non starter in production. Same as eswitch this fpga is not
represented as a kernel object, there is no way to debug it.
NIC crafts roce packets back and forth?!
so it's like rdma, but without using kernel rdma stack?
When hw ipsec or tls will mysteriously drop or mangle the packets
how this can be debugged? Does fpga have attached ddr to
store/forward the packets? How memory issues will be reported?
No MCE errors ever? Buffer overflow? How many receive queues inside fpga?
How health check of fgpa itself will be done? Through roce packets?
I would buy the lack of kernel visibility if this fpga+nic combo
was a prototype, but it's being presented as a production device
with subsequent changes to core networking stack and that's where
I have a problem with its sw architecture.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html