From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56956C43464 for ; Fri, 18 Sep 2020 12:50:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0B90021D7B for ; Fri, 18 Sep 2020 12:50:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="QyJDbsqV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726365AbgIRMuR (ORCPT ); Fri, 18 Sep 2020 08:50:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726129AbgIRMuQ (ORCPT ); Fri, 18 Sep 2020 08:50:16 -0400 Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E898DC061788 for ; Fri, 18 Sep 2020 05:50:16 -0700 (PDT) Received: by mail-qk1-x744.google.com with SMTP id o5so5831669qke.12 for ; Fri, 18 Sep 2020 05:50:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=KAoPuTEzUmIe2wdxaRxMUqqp3YdtmLOZgXAXpjKRNsk=; b=QyJDbsqV+Clftle8r/t/pW701Y/xrp6nUJ3MlzmLFCL32GerNaErkVHXxBR5FtLcuE DjRIlx9mDzvwBV1qDo2wZUwIjWI27lrH2c1lCsovl4kYcbSdvNnau+k8cP18rJSDvG1L sGi0QcjouM6J0NHgJEdxsmuLb3yLqhb99Dq2cRUle5JcDjD22B1rCwyCq9YY86oIVbCw QqJzm+r2mcrE75S6SO/bTsMGVaezkZTEKr9uRuGfKtn3NSzQPYnRQGxjpOKopN4HgsfS kFr7dzmb/BHaUb7btWWeU259YSwwO8x+I1AuoJLMaULNgrqkMdz0EImj6xmYaymMrEzv g1CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=KAoPuTEzUmIe2wdxaRxMUqqp3YdtmLOZgXAXpjKRNsk=; b=Vy8DRmBHuRY+iA+ZE2oIW3jLTTkyrZHFPXvuJCV/cETY7vGWuoK9DT05FF8+jit//k vrMPWFnBIOk4fIb6UnsyLswtNKuTLN6f357YsdhsOYn8iwZxOBh8xFDlFSNBYHA1oWYZ kNnHrUfkaeMR+SMJHXmfa7faLX1EtlziAgRE4W3UNYTJy01TBZLjf0i7mxTqOYHuNiiU JJMUok+oeA/ZfQLBF/3YXzlJmVJMjzvSUgf6qoAK0x8RyFsCvxLHZ6CMnnurCFxS35Q5 26gYHiaVYvxEl7Nb8gf+8PvqMlGtiZ1H+su8f9xwuq4AVtG6eu3srTPM5qgwP4j58ZZM dWbA== X-Gm-Message-State: AOAM532H1cAOJusUX6GLPFzNPMFWXl4HECv5LP9+Klzeu+L4tVEbCKhv orH6CvA1AGQrUai+jgaOfUk9PA== X-Google-Smtp-Source: ABdhPJw907l05mss1ldyLf/9wiOK6C5YwGtGpEhTGyHGuXCXNATWIp2kIEIi+FlDBdV5O02aKx9WRQ== X-Received: by 2002:a37:4a57:: with SMTP id x84mr32144214qka.17.1600433416085; Fri, 18 Sep 2020 05:50:16 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-48-30.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.48.30]) by smtp.gmail.com with ESMTPSA id i5sm1927930qko.86.2020.09.18.05.50.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2020 05:50:15 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kJFqA-000z9a-Ua; Fri, 18 Sep 2020 09:50:14 -0300 Date: Fri, 18 Sep 2020 09:50:14 -0300 From: Jason Gunthorpe To: Oded Gabbay Cc: izur@habana.ai, Gal Pressman , Jakub Kicinski , "Linux-Kernel@Vger. Kernel. Org" , netdev@vger.kernel.org, SW_Drivers , Greg Kroah-Hartman , "David S. Miller" , Andrew Lunn , Florian Fainelli , linux-rdma@vger.kernel.org Subject: Re: [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver Message-ID: <20200918125014.GR8409@ziepe.ca> References: <20200915171022.10561-1-oded.gabbay@gmail.com> <20200915133556.21268811@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> <20200917171833.GJ8409@ziepe.ca> <0b21db8d-1061-6453-960b-8043951b3bad@amazon.com> <20200918115601.GP8409@ziepe.ca> <20200918121621.GQ8409@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Fri, Sep 18, 2020 at 03:34:54PM +0300, Oded Gabbay wrote: > > > Another example is that the submission of WQ is done through our QMAN > > > mechanism and is NOT mapped to userspace (due to the restrictions you > > > mentioned above and other restrictions). > > > > Sure, other RDMA drivers also require a kernel ioctl for command > > execution. > > > > In this model the MR can be a software construct, again representing a > > security authorization: > > > > - A 'full process' MR, in which case the kernel command excution > > handles dma map and pinning at command execution time > > - A 'normal' MR, in which case the DMA list is pre-created and the > > command execution just re-uses this data > > > > The general requirement for RDMA is the same as DRM, you must provide > > enough code in rdma-core to show how the device works, and minimally > > test it. EFA uses ibv_ud_pingpong, and some pyverbs tests IIRC. > > > > So you'll want to arrange something where the default MR and PD > > mechanisms do something workable on this device, like auto-open the > > misc FD when building the PD, and support the 'normal' MR flow for > > command execution. > > I don't know how we can support MR because we can't support any > virtual address on the host. Our internal MMU doesn't support 64-bits. > We investigated in the past, very much wanted to use IBverbs but > didn't figure out how to make it work. > I'm adding Itay here and he can also shed more details on that. I'm not sure what that means, if the driver intends to DMA from process memory then it certainly has a MR concept. MRs can control the IOVA directly so if you say the HW needs a MR IOVA < 2**32 then that is still OK. Jason