From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 696BCC4727D for ; Tue, 22 Sep 2020 12:46:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 22A9C2399A for ; Tue, 22 Sep 2020 12:46:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="TfEOQVNe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726640AbgIVMqv (ORCPT ); Tue, 22 Sep 2020 08:46:51 -0400 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]:20978 "EHLO smtp-fw-6001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726563AbgIVMqu (ORCPT ); Tue, 22 Sep 2020 08:46:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1600778810; x=1632314810; h=subject:to:cc:references:from:message-id:date: mime-version:in-reply-to:content-transfer-encoding; bh=nziyF0gCHf+QbUZWkcNumlIIbcw7LDkmtCIhMVokIFM=; b=TfEOQVNe2a5SEJoigXV06ZCR+NEP1ZuUl27rQCymWfl8DCwejcAq4FJ8 5bc57ErZ70/WjpfAJxPpRepwHb3ogep+EODSbgn9NJ+VJ39yw58T8+CL0 s3e5hFLbBYgfNQVtfxRNe9D3kSdT70BOzmUNYMYR/j8AsFDjNXlL/POsK I=; X-IronPort-AV: E=Sophos;i="5.77,290,1596499200"; d="scan'208";a="56952007" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1e-a70de69e.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 22 Sep 2020 12:46:48 +0000 Received: from EX13D19EUB003.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-1e-a70de69e.us-east-1.amazon.com (Postfix) with ESMTPS id E7FC8A1869; Tue, 22 Sep 2020 12:46:43 +0000 (UTC) Received: from 8c85908914bf.ant.amazon.com (10.43.161.237) by EX13D19EUB003.ant.amazon.com (10.43.166.69) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 22 Sep 2020 12:46:34 +0000 Subject: Re: [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver To: Jason Gunthorpe CC: Oded Gabbay , Greg Kroah-Hartman , , Jakub Kicinski , "Linux-Kernel@Vger. Kernel. Org" , , SW_Drivers , "David S. Miller" , Andrew Lunn , Florian Fainelli , , Olof Johansson References: <20200918132645.GS8409@ziepe.ca> <20200918135915.GT8409@ziepe.ca> <20200918141909.GU8409@ziepe.ca> <20200918150735.GV8409@ziepe.ca> <20200918152852.GW8409@ziepe.ca> <20200922114101.GE8409@ziepe.ca> From: Gal Pressman Message-ID: Date: Tue, 22 Sep 2020 15:46:29 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.2.2 MIME-Version: 1.0 In-Reply-To: <20200922114101.GE8409@ziepe.ca> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [10.43.161.237] X-ClientProxiedBy: EX13D17UWB004.ant.amazon.com (10.43.161.132) To EX13D19EUB003.ant.amazon.com (10.43.166.69) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 22/09/2020 14:41, Jason Gunthorpe wrote: > On Mon, Sep 21, 2020 at 02:22:02PM +0300, Gal Pressman wrote: > >> What is considered a RoCE port in this case if it's not compliant with RoCE? >> Sounds like it's an implementation of RDMA over ethernet, not RoCE. >> Does GAUDI support UD/RC/.. QPs? Is it using a proprietary wire protocol? >> (BTW, Oded claims it's similar to nvlink, how is nvlink's implementation >> exposed? Or is it closed source?) > > I think Oded was drawing a parallel to how nvlink is integral with the > compute element. From Oded's descriptions I don't think it is much > like nvlink at all. > >> Jason, how do you imagine GAUDI in the RDMA subsystem? Userspace control path >> verbs (used by hl-thunk?) and all data path verbs exposed as kverbs (used by >> habanalabs driver)? >> So neither any userspace verbs apps could use it nor kernel ULPs? > > Based on what Oded described it seems like a reasonable RDMA device > with some limitations around MR IOVA. > > Looks like the desire is to create a RDMA WR and CQ ring in userspace, > and then co-mingle that with the compute side of the device. > > So instead of doing the special IOCTL and mmap against the compute FD > it would create a RDMA QP and RDMA CQ, use dv to access the raw > internals, and the propritary stack would have exactly the same stuff > it would have had with the misc ioctl. > > But, completely separately, they'd also have to implement some of > verbs which serves as the open source userspace showing how this HW > works. What that is depends largely on what their HW can do, and if > they want to connect to UCX/mpi/libfabric/etc > > A bunch of ioctl stubs or a few tests is far below our standard in > RDMA. > > There may have been some argument that the compute side of this device > has no industry standards so should be a drivers/misc, but HPC > networking *does* have extensive standards and extensive open source > software stacks. It is very hard for me to see how a device in this > market could be competitive without integrating with that stuff. I agree, that makes sense. But assuming Oded actually goes and implements all the needed verbs to get a basic functional libibverbs provider (assuming their HW can do it somehow), is it really useful if no one is going to use it? It doesn't sound like habanalabs want people to use GAUDI as an RDMA adapter, and I'm assuming the only real world use case is going to be using the hl stack, which means we're left with a lot of dead code that's not used/tested by anyone. Genuine question, wouldn't it be better if they only implement what's actually going to be used and tested by their customers?