From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40E10C433E6 for ; Thu, 18 Feb 2021 22:52:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA715601FD for ; Thu, 18 Feb 2021 22:52:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229784AbhBRWwO (ORCPT ); Thu, 18 Feb 2021 17:52:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229752AbhBRWwN (ORCPT ); Thu, 18 Feb 2021 17:52:13 -0500 Received: from mail-qv1-xf36.google.com (mail-qv1-xf36.google.com [IPv6:2607:f8b0:4864:20::f36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99D32C061574 for ; Thu, 18 Feb 2021 14:51:33 -0800 (PST) Received: by mail-qv1-xf36.google.com with SMTP id dr7so1802562qvb.1 for ; Thu, 18 Feb 2021 14:51:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=f4o2UaWR0IaBZ9I3kkpgTMwRbNwJtqHh4PFVjtr2gyI=; b=TAnM2mqFsfYh+QjgqIZGJwM4sOU/VBMwhtX2PbGKVVAmq73Qmo3I3H6ol+mSl3Su3j R+aKl2eR4DsYGvEcHR6ElvHCZnRpVx5qseWU6K3v76M2LvuvpOKYzuYPo2eXONx53Eqk cbmUWt0dGqfN1Ru/TgyuTBt03512o92v0v1AzcnTuB0TaeWT9cCNaasqZaG0lTiJ+LJh o5ObZCL1+nr5szNKUDUz7uuKHCJpou2qx+Huof6ywZYdJy0F922NIs9vcFVi1iI9CaKn ZUfbtu0xI+5KLiZ8n0iPKEDHYecI7lCdwTJqss1c3DSnfMOGF/qmsy5tTdiCNZBaCxvS gtOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=f4o2UaWR0IaBZ9I3kkpgTMwRbNwJtqHh4PFVjtr2gyI=; b=WlEVxv0MEO/KpESaVFmfLXNTxHr0mo37U10om0itBgKwGusMLp08C+VRT2eqMDc9rn HOONyiK9q2nWbGKU4g9CDNDjcSXrPokwIkKPG3G10IWLZMc24yn48rXJsgWT2/BodzR0 XIUT6tL9yYk2ZZ79ngePNcNVSGZ6NwpI0e0z+Bq0l6PyOtIbmBqd0RC72FlzkETCk74L 1XokSUSAI9JIHm+7hxEWW3uWgfDZ30KFJ9tJdc4aP2FJRaMOP5gDpneyHwTKH3S9gaMM uTU+W9L4w5Z9VPgqsJXrCpr0O4Eim7Ykkt6iIgLS7gTJKJ0vpugmf4AbC806PrUFWzYJ 72Ng== X-Gm-Message-State: AOAM5301LutfhH9UxGH1LiYvRzWxryVIzY5S2K1+rn4SHbs3u5ti3Yud OF63AqI6Y1ekJ2cpSkAbaaHfjg== X-Google-Smtp-Source: ABdhPJxk1+QiLVDPFFx7j2AaNZWKgMKA77nmEYDXwYTxZ9s/nFDNit48B172/0LuQVOVq4czmxMWEA== X-Received: by 2002:a0c:e18a:: with SMTP id p10mr6869493qvl.0.1613688692807; Thu, 18 Feb 2021 14:51:32 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-162-115-133.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.115.133]) by smtp.gmail.com with ESMTPSA id t6sm4901409qkd.127.2021.02.18.14.51.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Feb 2021 14:51:32 -0800 (PST) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lCs8x-00BeS6-FC; Thu, 18 Feb 2021 18:51:31 -0400 Date: Thu, 18 Feb 2021 18:51:31 -0400 From: Jason Gunthorpe To: Tom Talpey Cc: Gal Pressman , RDMA mailing list Subject: Re: ibv_req_notify_cq clarification Message-ID: <20210218225131.GB2643399@ziepe.ca> References: <20210218125339.GY4718@ziepe.ca> <5287c059-3d8c-93f4-6be4-a6da07ccdb8a@amazon.com> <20210218162329.GZ4718@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Thu, Feb 18, 2021 at 05:22:31PM -0500, Tom Talpey wrote: > On 2/18/2021 11:23 AM, Jason Gunthorpe wrote: > > On Thu, Feb 18, 2021 at 05:52:16PM +0200, Gal Pressman wrote: > > > On 18/02/2021 14:53, Jason Gunthorpe wrote: > > > > On Thu, Feb 18, 2021 at 11:13:43AM +0200, Gal Pressman wrote: > > > > > I'm a bit confused about the meaning of the ibv_req_notify_cq() verb: > > > > > "Upon the addition of a new CQ entry (CQE) to cq, a completion event will be > > > > > added to the completion channel associated with the CQ." > > > > > > > > > > What is considered a new CQE in this case? > > > > > The next CQE from the user's perspective, i.e. any new CQE that wasn't consumed > > > > > by the user's poll cq? > > > > > Or any new CQE from the device's perspective? > > > > > > > > new CQE from the device perspective. > > > > > > > > > For example, if at the time of ibv_req_notify_cq() call the CQ has received 100 > > > > > completions, but the user hasn't polled his CQ yet, when should he be notified? > > > > > On the 101 completion or immediately (since there are completions waiting on the > > > > > CQ)? > > > > > > > > 101 completion > > > > > > > > It is only meaningful to call it when the CQ is empty. > > > > > > Thanks, so there's an inherent race between the user's CQ poll and the next arm? > > > > I think the specs or man pages talk about this, the application has to > > observe empty, do arm, then poll again then sleep on the cq if empty. > > > > > Do you know what's the purpose of the consumer index in the arm doorbell that's > > > implemented by many providers? > > > > The consumer index is needed by HW to prevent CQ overflow, presumably > > the drivers push to reduce the cases where the HW has to read it from > > PCI > > Prevent CQ overflow? There's no such requirement that I'm aware of. > If the consumer doesn't provide a large-enough CQ, then it reaps the > consequences. Same thing for WQ depth, although I am aware that some > verbs implementations attempt to return a kind of EAGAIN when posting > to a send WQ. > > What can the provider do if the CQ is "full" anyway? Buffer the CQE > and go into some type of polling loop attempting to redeliver? Ouch! QP goes to error, CQE is discarded, IIRC. Wrapping and overflowing the CQ is not acceptable, it would mean reading CQEs could never be done reliably. Jason