From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E23FDC433E2 for ; Thu, 17 Sep 2020 17:19:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9F4BC221EC for ; Thu, 17 Sep 2020 17:19:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="UgmNS1ft" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726239AbgIQRTp (ORCPT ); Thu, 17 Sep 2020 13:19:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726470AbgIQRSg (ORCPT ); Thu, 17 Sep 2020 13:18:36 -0400 Received: from mail-qt1-x844.google.com (mail-qt1-x844.google.com [IPv6:2607:f8b0:4864:20::844]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73B17C061788 for ; Thu, 17 Sep 2020 10:18:36 -0700 (PDT) Received: by mail-qt1-x844.google.com with SMTP id b2so2479432qtp.8 for ; Thu, 17 Sep 2020 10:18:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=jfL5I8+SAEaL2OQyBySvbdAEfPXTRZQx7kR/t+e9RqA=; b=UgmNS1ftT004DWJhMLLmAaHR2BO1naPrzFegZa5rZnf3rGpWH8LMd1xTTJKbRlPhqt 5ze07Yq/IeeCzRJkU8FCSG9qU7RVm34s/t37biz6aQFB8nlZATJJz83vJ272XMHqXboE RvOhYFgszZRwlejnC4o2y865WVl/4NTQWeYhnVX7pacrsGys8HLaK9NFowueO0pvOP8+ bgB0BQA3PQNHnQ2s1nzRiqfajZFklpSFbi3PXOxNyYvluJygKcOCGzqQU72aqH2ef6zd FApi0ug9pTRrGURqNOZz5Tueh7aalLAgRaDYuAgXIMZOzigN2jCcaDeoLYOY5CpBfEgx npSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=jfL5I8+SAEaL2OQyBySvbdAEfPXTRZQx7kR/t+e9RqA=; b=OpKeiIaDYJXeMjMwNeV1TZ1cHyJRRsF77BvpmF3jy+BBaHCkNl+cQF6usaB0/8yOxq ZAaCX513PfFFL2jgjUeWesCbCNduD/1/Jo6VFjvKlnH/5YtMAk2W2LjBAqLhzeHmxWuN OBMydxQdqhjz9uKk+mUxyvSYAVIE8UiJSIlUcaEcDLE4MrpLrHBzh4IQESthK+0Jvl/J G/kmD4uF+Z6zke6PLQrdZCv/HkcaH6nQ0EEShe9vHIWuiqXKDckCSfsKaooLW2S1yp+C iOdEpunDzm2yTvW4ypzJAUyOB7X9RZwr7DfthKGCj0MVeCWa9goFWFuiiNez1j26Qp+7 qzbg== X-Gm-Message-State: AOAM531p8DfZdtFuoAgSsu+t0BhdSrxAKZtC66jhTwsMjCC7W3kxzO6I El/yJYERDPnslS3t2U6chHU0Gg== X-Google-Smtp-Source: ABdhPJxyBiY7E2PwXlRdxQJ94MHJWynfjteZ/S0FWxpEdsMUTXsd97vU2/ByViQ4IYP6HH7MEiQ61g== X-Received: by 2002:ac8:2383:: with SMTP id q3mr16956309qtq.230.1600363115224; Thu, 17 Sep 2020 10:18:35 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-48-30.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.48.30]) by smtp.gmail.com with ESMTPSA id i187sm280659qke.43.2020.09.17.10.18.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Sep 2020 10:18:34 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kIxYH-000afx-Uu; Thu, 17 Sep 2020 14:18:33 -0300 Date: Thu, 17 Sep 2020 14:18:33 -0300 From: Jason Gunthorpe To: Oded Gabbay Cc: Jakub Kicinski , "Linux-Kernel@Vger. Kernel. Org" , netdev@vger.kernel.org, SW_Drivers , Greg Kroah-Hartman , "David S. Miller" , Andrew Lunn , Florian Fainelli , linux-rdma@vger.kernel.org Subject: Re: [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver Message-ID: <20200917171833.GJ8409@ziepe.ca> References: <20200915171022.10561-1-oded.gabbay@gmail.com> <20200915133556.21268811@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Sep 15, 2020 at 11:46:58PM +0300, Oded Gabbay wrote: > infrastructure for communication between multiple accelerators. Same > as Nvidia uses NVlink, we use RDMA that we have inside our ASIC. > The RDMA implementation we did does NOT support some basic RDMA > IBverbs (such as MR and PD) and therefore, we can't use the rdma-core > library or to connect to the rdma infrastructure in the kernel. You can't create a parallel RDMA subsystem in netdev, or in misc, and you can't add random device offloads as IOCTL to nedevs. RDMA is the proper home for all the networking offloads that don't fit into netdev. EFA was able to fit into rdma-core/etc and it isn't even RoCE at all. I'm sure this can too. > wanted to do it but when we analyzed it, we saw we wouldn't be able to > support basic stuff and therefore we had to revert to our IOCTLs. Try again. Ask for help. Your patches add CQs, WQ, and other RDMA objects. This is very clearly not an appropriate functionality for netdev. > To sum it up, because our NIC is used for intra-communication, we > don't expose nor intend users to use it as a NIC per-se. However, to > be able to get statistics and manage them in a standard way, and > support control plane over Ethernet, we do register each port to the > net subsystem (i.e. create netdev per port). Sure, the basic ethernet side is conceptually fine. > > Please make sure to CC linux-rdma. You clearly stated that the device > > does RDMA-like transfers. > > We don't use the RDMA infrastructure in the kernel and we can't > connect to it due to the lack of H/W support we have so I don't see > why we need to CC linux-rdma. Because you can't put RDMA like concepts under net. Jakub, NAK from me on this series. Jason