From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B20E4C433E0 for ; Fri, 12 Mar 2021 02:54:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8BD4164F95 for ; Fri, 12 Mar 2021 02:54:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229530AbhCLCxe (ORCPT ); Thu, 11 Mar 2021 21:53:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231439AbhCLCx2 (ORCPT ); Thu, 11 Mar 2021 21:53:28 -0500 Received: from mail-io1-xd29.google.com (mail-io1-xd29.google.com [IPv6:2607:f8b0:4864:20::d29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77DEFC061574; Thu, 11 Mar 2021 18:53:28 -0800 (PST) Received: by mail-io1-xd29.google.com with SMTP id y20so5905934iot.4; Thu, 11 Mar 2021 18:53:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3r2VmurgsZAdK+mJ/SSuYF0DS4iZlEJSrc+E9dLuQLw=; b=WMBIg04QmQOavkO2RGio47QGbtUWuB/YTAjHEpShOqSl0D10HYLGZDVeyOm6yT2QEa yiBL+G3ItjoJOr2nPTjKSWW7XQn0w5/WKkVwEvydsLFLEuRFRstpPAIi/9OtVnWOgC4W BFTuocVS47QR0KLlJ2X5ivBlfj5VLti/7mXcUpuwtl4omkeXe5AaetjAIufeLuC0YSjr g8vfm7HhiK+QYf2oV4XCWWBWdCOcBd3jMs/4ORBIvBOQZxr/93fHBUHqEDR/5rWMghRb 4Ekrbeabdy/3qAcQ8dkb3tSy90TqP9gcjaufsj+GhJ9OAb84/noIh8MnIJbyBg4GqlEZ UBiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3r2VmurgsZAdK+mJ/SSuYF0DS4iZlEJSrc+E9dLuQLw=; b=t36LDTOpSCZNm+RU/7MsMXjWqnpD+HSdfc70Qr5ej06sPNaFsjo7jlQhC/NifthDiZ mmRtAd/NFHuYb1gtnnJ/6eUjXSSJeoN/EduZIn4JpLUhuZaY0c7DlEpq7ku5ig0j9S/0 mRUts1c1IkvLdsAkCUrVE6STXOzf+t9Yb1NMixHRwwvBZ/yqFx7sSwwr6Cft04K/h85i w5g5kIHJeaAZ83ubJi7JOGbsHdeu74SFP/c532sJxo+AfPi9HwyXlvE7HlD92H025tZ3 68zU6qyXWliItMBdbtLjZ4v3zQhOUd/9ec58XO8oTp+OZj0B682vt+g8Os1jLVnSBw9G q26Q== X-Gm-Message-State: AOAM530WmTBAodQnhix7PRwewUopSoCl5vtMzfx8nnHnSitb1FO4cYaO TlTHli8d6eZKHaMlQdFA0g1A2BtGJCbV6VLHovI= X-Google-Smtp-Source: ABdhPJxVIUgAKs03auIcdOplshnX6MKD+0QQ6cYavZhwX6ZokUV7aSSkmZejE50Sf5kZZqECq9Or+iMRrVBA9Z9xpVA= X-Received: by 2002:a6b:f909:: with SMTP id j9mr8705979iog.138.1615517607777; Thu, 11 Mar 2021 18:53:27 -0800 (PST) MIME-Version: 1.0 References: <20210311181729.GA2148230@bjorn-Precision-5520> <20210311201929.GN2356281@nvidia.com> <20210311232059.GR2356281@nvidia.com> In-Reply-To: <20210311232059.GR2356281@nvidia.com> From: Alexander Duyck Date: Thu, 11 Mar 2021 18:53:16 -0800 Message-ID: Subject: Re: [PATCH mlx5-next v7 0/4] Dynamically assign MSI-X vectors count To: Jason Gunthorpe Cc: Bjorn Helgaas , Leon Romanovsky , Bjorn Helgaas , Saeed Mahameed , Leon Romanovsky , Jakub Kicinski , linux-pci , linux-rdma@vger.kernel.org, Netdev , Don Dutile , Alex Williamson , "David S . Miller" , Greg Kroah-Hartman Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On Thu, Mar 11, 2021 at 3:21 PM Jason Gunthorpe wrote: > > On Thu, Mar 11, 2021 at 01:49:24PM -0800, Alexander Duyck wrote: > > > We don't need to invent new locks and new complexity for something > > > that is trivially solved already. > > > > I am not wanting a new lock. What I am wanting is a way to mark the VF > > as being stale/offline while we are performing the update. With that > > we would be able to apply similar logic to any changes in the future. > > I think we should hold off doing this until someone comes up with HW > that needs it. The response time here is microseconds, it is not worth > any complexity I disagree. Take a look at section 8.5.3 in the NVMe document that was linked to earlier: https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4a-2020.03.09-Ratified.pdf This is exactly what they are doing and I think it makes a ton of sense. Basically the VF has to be taken "offline" before you are allowed to start changing resources on it. It would basically consist of one extra sysfs file and has additional uses beyond just the configuration of MSI-X vectors. We would just have to add one additional sysfs file, maybe modify the "dead" device flag to be "offline", and we could make this work with minimal changes to the patch set you already have. We could probably toggle to "offline" while holding just the VF lock. To toggle the VF back to being "online" we might need to take the PF device lock since it is ultimately responsible for guaranteeing we have the resources. Another way to think of this is that we are essentially pulling a device back after we have already allocated the VFs and we are reconfiguring it before pushing it back out for usage. Having a flag that we could set on the VF device to say it is "under construction"/modification/"not ready for use" would be quite useful I would think.