From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by gmr-mx.google.com with ESMTPS id h12si82264lfv.4.2021.12.01.13.21.19 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Dec 2021 13:21:20 -0800 (PST) Message-ID: Date: Wed, 1 Dec 2021 14:21:15 -0700 MIME-Version: 1.0 Subject: Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc() Content-Language: en-US References: <20211126230957.239391799@linutronix.de> <20211126232735.547996838@linutronix.de> <7daba0e2-73a3-4980-c3a5-a71f6b597b22@deltatee.com> <874k7ueldt.ffs@tglx> <6ba084d6-2b26-7c86-4526-8fcd3d921dfd@deltatee.com> <87ilwacwp8.ffs@tglx> <87v909bf2k.ffs@tglx> <20211130202800.GE4670@nvidia.com> <87o861banv.ffs@tglx> <20211201001748.GF4670@nvidia.com> <87mtlkaauo.ffs@tglx> <8c2262ba-173e-0007-bc4c-94ec54b2847d@intel.com> <87pmqg88xq.ffs@tglx> <87k0go8432.ffs@tglx> From: Dave Jiang In-Reply-To: <87k0go8432.ffs@tglx> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit To: Thomas Gleixner , Jason Gunthorpe Cc: Logan Gunthorpe , LKML , Bjorn Helgaas , Marc Zygnier , Alex Williamson , Kevin Tian , Megha Dey , Ashok Raj , linux-pci@vger.kernel.org, Greg Kroah-Hartman , Jon Mason , Allen Hubbe , linux-ntb@googlegroups.com, linux-s390@vger.kernel.org, Heiko Carstens , Christian Borntraeger , x86@kernel.org, Joerg Roedel , iommu@lists.linux-foundation.org List-ID: On 12/1/2021 1:25 PM, Thomas Gleixner wrote: > On Wed, Dec 01 2021 at 11:47, Dave Jiang wrote: >> On 12/1/2021 11:41 AM, Thomas Gleixner wrote: >>>> Hi Thomas. This is actually the IDXD usage for a mediated device passed >>>> to a guest kernel when we plumb the pass through of IMS to the guest >>>> rather than doing previous implementation of having a MSIX vector on >>>> guest backed by IMS. >>> Which makes a lot of sense. >>> >>>> The control block for the mediated device is emulated and therefore an >>>> emulated MSIX vector will be surfaced as vector 0. However the queues >>>> will backed by IMS vectors. So we end up needing MSIX and IMS coexist >>>> running on the guest kernel for the same device. >>> Why? What's wrong with using straight MSI-X for all of them? >> The hardware implementation does not have enough MSIX vectors for >> guests. There are only 9 MSIX vectors total (8 for queues) and 2048 IMS >> vectors. So if we are to do MSI-X for all of them, then we need to do >> the IMS backed MSIX scheme rather than passthrough IMS to guests. > Confused. Are you talking about passing a full IDXD device to the guest > or about passing a carved out subdevice, aka. queue? I'm talking about carving out a subdevice. I had the impression of you wanting IMS passed through for all variations. But it sounds like for a sub-device, you are ok with the implementation of MSIX backed by IMS? > > Thanks, > > tglx > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 378E4C433EF for ; Wed, 1 Dec 2021 21:21:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id B5EB9402C5; Wed, 1 Dec 2021 21:21:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nZxQMzqO8MeX; Wed, 1 Dec 2021 21:21:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 22D2740004; Wed, 1 Dec 2021 21:21:23 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F33C5C000A; Wed, 1 Dec 2021 21:21:22 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id EDB69C000A for ; Wed, 1 Dec 2021 21:21:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id E1A50607A5 for ; Wed, 1 Dec 2021 21:21:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Gf4imkpNVsD2 for ; Wed, 1 Dec 2021 21:21:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp3.osuosl.org (Postfix) with ESMTPS id CBFB96073D for ; Wed, 1 Dec 2021 21:21:18 +0000 (UTC) X-IronPort-AV: E=McAfee;i="6200,9189,10185"; a="236502703" X-IronPort-AV: E=Sophos;i="5.87,280,1631602800"; d="scan'208";a="236502703" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Dec 2021 13:21:18 -0800 X-IronPort-AV: E=Sophos;i="5.87,280,1631602800"; d="scan'208";a="602309136" Received: from djiang5-mobl1.amr.corp.intel.com (HELO [10.212.64.69]) ([10.212.64.69]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Dec 2021 13:21:16 -0800 Message-ID: Date: Wed, 1 Dec 2021 14:21:15 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.3.1 Subject: Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc() Content-Language: en-US To: Thomas Gleixner , Jason Gunthorpe References: <20211126230957.239391799@linutronix.de> <20211126232735.547996838@linutronix.de> <7daba0e2-73a3-4980-c3a5-a71f6b597b22@deltatee.com> <874k7ueldt.ffs@tglx> <6ba084d6-2b26-7c86-4526-8fcd3d921dfd@deltatee.com> <87ilwacwp8.ffs@tglx> <87v909bf2k.ffs@tglx> <20211130202800.GE4670@nvidia.com> <87o861banv.ffs@tglx> <20211201001748.GF4670@nvidia.com> <87mtlkaauo.ffs@tglx> <8c2262ba-173e-0007-bc4c-94ec54b2847d@intel.com> <87pmqg88xq.ffs@tglx> <87k0go8432.ffs@tglx> From: Dave Jiang In-Reply-To: <87k0go8432.ffs@tglx> Cc: Allen Hubbe , linux-s390@vger.kernel.org, Kevin Tian , x86@kernel.org, Ashok Raj , Greg Kroah-Hartman , Marc Zygnier , Heiko Carstens , LKML , iommu@lists.linux-foundation.org, Christian Borntraeger , Alex Williamson , Joerg Roedel , Bjorn Helgaas , linux-pci@vger.kernel.org, linux-ntb@googlegroups.com, Logan Gunthorpe , Megha Dey X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On 12/1/2021 1:25 PM, Thomas Gleixner wrote: > On Wed, Dec 01 2021 at 11:47, Dave Jiang wrote: >> On 12/1/2021 11:41 AM, Thomas Gleixner wrote: >>>> Hi Thomas. This is actually the IDXD usage for a mediated device passed >>>> to a guest kernel when we plumb the pass through of IMS to the guest >>>> rather than doing previous implementation of having a MSIX vector on >>>> guest backed by IMS. >>> Which makes a lot of sense. >>> >>>> The control block for the mediated device is emulated and therefore an >>>> emulated MSIX vector will be surfaced as vector 0. However the queues >>>> will backed by IMS vectors. So we end up needing MSIX and IMS coexist >>>> running on the guest kernel for the same device. >>> Why? What's wrong with using straight MSI-X for all of them? >> The hardware implementation does not have enough MSIX vectors for >> guests. There are only 9 MSIX vectors total (8 for queues) and 2048 IMS >> vectors. So if we are to do MSI-X for all of them, then we need to do >> the IMS backed MSIX scheme rather than passthrough IMS to guests. > Confused. Are you talking about passing a full IDXD device to the guest > or about passing a carved out subdevice, aka. queue? I'm talking about carving out a subdevice. I had the impression of you wanting IMS passed through for all variations. But it sounds like for a sub-device, you are ok with the implementation of MSIX backed by IMS? > > Thanks, > > tglx > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu