linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Nicolin Chen <nicolinc@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	Robin Murphy <robin.murphy@arm.com>,
	"will@kernel.org" <will@kernel.org>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"shameerali.kolothum.thodi@huawei.com" 
	<shameerali.kolothum.thodi@huawei.com>,
	"jean-philippe@linaro.org" <jean-philippe@linaro.org>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v1 14/14] iommu/arm-smmu-v3: Add arm_smmu_cache_invalidate_user
Date: Wed, 22 Mar 2023 10:11:33 -0700	[thread overview]
Message-ID: <ZBs2xaa7Ow1pnB3C@Asurada-Nvidia> (raw)
In-Reply-To: <ZBr3/1NzY5VvJrJQ@nvidia.com>

On Wed, Mar 22, 2023 at 09:43:43AM -0300, Jason Gunthorpe wrote:
> On Tue, Mar 21, 2023 at 11:42:25PM -0700, Nicolin Chen wrote:
> > On Tue, Mar 21, 2023 at 08:48:31AM -0300, Jason Gunthorpe wrote:
> > > On Tue, Mar 21, 2023 at 08:34:00AM +0000, Tian, Kevin wrote:
> > > 
> > > > > > Rephrasing that to put into a design: the IOCTL would pass a
> > > > > > user pointer to the queue, the size of the queue, then a head
> > > > > > pointer and a tail pointer? Then the kernel reads out all the
> > > > > > commands between the head and the tail and handles all those
> > > > > > invalidation commands only?
> > > > > 
> > > > > Yes, that is one possible design
> > > > 
> > > > If we cannot have the short path in the kernel then I'm not sure the
> > > > value of using native format and queue in the uAPI. Batching can
> > > > be enabled over any format.
> > > 
> > > SMMUv3 will have a hardware short path where the HW itself runs the
> > > VM's command queue and does this logic.
> > > 
> > > So I like the symmetry of the SW path being close to that.
> > 
> > A tricky thing here that I just realized:
> > 
> > With VCMDQ, the guest will have two CMDQs. One is the vSMMU's
> > CMDQ handling all non-TLBI commands like CMD_CFGI_STE via the
> > invalidation IOCTL, and the other hardware accelerated VCMDQ
> > handling all TLBI commands by the HW. In this setup, we will
> > need a VCMDQ kernel driver to dispatch commands into the two
> > different queues.
> 
> You mean a VM kernel driver? Yes that was always the point, the VM
> would use the extra CMDQ's only for invalidation

Yes, I was saying the guest kernel driver would dispatch the
commands.

> The main CMDQ would work as today through a trap.

Yes.

> > Yet, it feels a bit different with this SW path exposing the
> > entire SMMU CMDQ, since now theoretically non-TLBI and TLBI
> > commands can be interlaced in one batch, so the hypervisor
> > should go through the queue first to handle and delete all
> > non-TLBI commands, and then forward the CMDQ to the host to
> > run remaining TLBI commands, if there's any.
> 
> Yes, there are a few different ways to handle this and still preserve
> batching. It is part of the reason it would be hard to make the kernel
> natively parse the commandq

Yea. I think the way I described above might be the cleanest,
since the host kernel would only handle all the leftover TLBI
commands? I am open for other better idea, if there's any.

> On the other hand, we could add some more native kernel support for a
> SW emulated vCMDQ and that might be interesting for performance.

That's something I have thought about too. But it would feel
like changing the "hardware" of the VM, right? If the host
kernel enables nesting, then we'd have this extra queue for
TLBI commands. From the driver prospective, it would feels
like detecting an extra feature bit in the HW register, but
there's no such bit in the SMMU HW spec :)

Yet, would you please elaborate how it impacts performance?
I can only see the benefit of isolation, from having a SW
emulated VCMDQ exclusively for TLBI commands v.s. having a
single CMDQ interlacing different commands, because both of
them requires trapping and some sort of dispatching.

> One of the biggest reasons to use nesting is to get to vSVA and
> invalidation performance is very important in a vSVA environment. We
> should not ignore this in the design.
> 
> > > If the VMID is tied to the entire iommufd_ctx then it can flow
> > > independently.
> > 
> > One more thing about the VMID unification is that SMMU might
> > have limitation on the VMID range:
> > 	smmu->vmid_bits = reg & IDR0_VMID16 ? 16 : 8;
> > 	...
> > 	vmid = arm_smmu_bitmap_alloc(smmu->vmid_map, smmu->vmid_bits);
> > 
> > So, we'd likely need a CAP for that, to apply some limitation
> > with the iommufd_ctx too?
> 
> I'd imagine the driver would have to allocate its internal data
> against the iommufd_ctx
> 
> I'm not sure how best to organize that if it is the way to go.
> 
> Do we have a use case for more than one S2 iommu_domain on ARM?

In the previous VFIO solution from Eric, a nested iommu_domain
represented an S1+S2 two-stage setup. Since every CMD_CFGI_STE
could trigger an iommu_domain allocation of that, there could
be multiple S2 domains, when we have 2+ passthrough devices.
That's why I had quite a few patch for VMID unification in the
old VCMDQ series.

But now, we have only one S2 domain that works well with multi-
devices. So, I can't really think of a use case that needs two
S2 domains. Yet, I am not very sure.

Btw, just to confirm my understanding, a use case having two
or more iommu_domains means an S2 iommu_domain replacement,
right? I.e. a running S2 iommu_domain gets replaced on the fly
by a different S2 iommu_domain holding a different VMID, while
the IOAS still has the previous mappings? When would that
actually happen in the real world?

Thanks
Nic

  reply	other threads:[~2023-03-22 17:11 UTC|newest]

Thread overview: 165+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-03-09 10:53 [PATCH v1 00/14] Add Nested Translation Support for SMMUv3 Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 01/14] iommu: Add iommu_get_unmanaged_domain helper Nicolin Chen
2023-03-09 12:51   ` Robin Murphy
2023-03-09 14:19     ` Jason Gunthorpe
2023-03-09 19:04       ` Robin Murphy
2023-03-10  0:23         ` Jason Gunthorpe
2023-03-10  8:41     ` Eric Auger
2023-03-10 15:55       ` Jason Gunthorpe
2023-03-16  1:21         ` Nicolin Chen
2023-03-16 18:42           ` Robin Murphy
2023-03-16 20:01             ` Nicolin Chen
2023-03-20 12:51             ` Jason Gunthorpe
2023-03-10 10:14   ` Eric Auger
2023-03-10 15:33     ` Jason Gunthorpe
2023-03-10 15:44       ` Shameerali Kolothum Thodi
2023-03-10 15:56         ` Jason Gunthorpe
2023-03-10 16:07           ` Shameerali Kolothum Thodi
2023-03-10 16:21             ` Jason Gunthorpe
2023-03-10 16:30               ` Shameerali Kolothum Thodi
2023-03-10 17:03                 ` Jason Gunthorpe
2023-03-22 16:07                   ` Eric Auger
2023-03-22 17:02                     ` Jason Gunthorpe
2023-03-22 17:41                       ` Eric Auger
2023-03-22 18:07                         ` Jason Gunthorpe
2023-03-16 19:51                 ` Nicolin Chen
2023-03-16 19:56                   ` Shameerali Kolothum Thodi
2023-03-22 15:44                     ` Eric Auger
2023-03-09 10:53 ` [PATCH v1 02/14] iommufd: Add nesting related data structures for ARM SMMUv3 Nicolin Chen
2023-03-09 13:42   ` Jean-Philippe Brucker
2023-03-09 14:48     ` Jason Gunthorpe
2023-03-09 18:26       ` Jean-Philippe Brucker
2023-03-09 21:01         ` Jason Gunthorpe
2023-03-10 12:16           ` Jean-Philippe Brucker
2023-03-10 14:52           ` Robin Murphy
2023-03-10 15:25             ` Jason Gunthorpe
2023-03-10 15:57               ` Robin Murphy
2023-03-10 16:03                 ` Jason Gunthorpe
2023-03-17 10:10                   ` Tian, Kevin
2023-03-17 10:04                 ` Tian, Kevin
2023-03-10  4:50       ` Nicolin Chen
2023-03-10 12:54         ` Jean-Philippe Brucker
2023-03-10 14:00           ` Jason Gunthorpe
2023-03-10 16:06         ` Jason Gunthorpe
2023-03-16  0:59           ` Nicolin Chen
2023-03-09 15:26     ` Shameerali Kolothum Thodi
2023-03-09 15:40       ` Jason Gunthorpe
2023-03-09 15:51         ` Shameerali Kolothum Thodi
2023-03-09 15:59           ` Jason Gunthorpe
2023-03-09 16:07             ` Shameerali Kolothum Thodi
2023-03-10  5:26               ` Nicolin Chen
2023-03-10  5:36                 ` Nicolin Chen
2023-03-10 12:55                   ` Jason Gunthorpe
2023-03-10  5:18         ` Nicolin Chen
2023-03-10  5:04     ` Nicolin Chen
2023-03-10 11:33     ` Eric Auger
2023-03-10 12:51       ` Jason Gunthorpe
2023-03-17 10:17         ` Tian, Kevin
2023-03-09 10:53 ` [PATCH v1 03/14] iommufd/device: Setup MSI on kernel-managed domains Nicolin Chen
2023-03-10 16:45   ` Eric Auger
2023-03-11  0:17     ` Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 04/14] iommu/arm-smmu-v3: Add arm_smmu_hw_info Nicolin Chen
2023-03-09 13:03   ` Robin Murphy
2023-03-10  1:17     ` Nicolin Chen
2023-03-10 15:28       ` Robin Murphy
2023-03-16  0:13         ` Nicolin Chen
2023-03-16 15:19           ` Robin Murphy
2023-03-16 20:06             ` Nicolin Chen
2023-04-12  7:47               ` Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 05/14] iommu/arm-smmu-v3: Remove ARM_SMMU_DOMAIN_NESTED Nicolin Chen
2023-03-10 16:39   ` Eric Auger
2023-03-10 17:05     ` Jason Gunthorpe
2023-03-11  0:24       ` Nicolin Chen
2023-03-11  0:23     ` Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 06/14] iommu/arm-smmu-v3: Unset corresponding STE fields when s2_cfg is NULL Nicolin Chen
2023-03-09 13:13   ` Robin Murphy
2023-03-09 18:24     ` Shameerali Kolothum Thodi
2023-03-10  1:54       ` Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 07/14] iommu/arm-smmu-v3: Add STRTAB_STE_0_CFG_NESTED for 2-stage translation Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 08/14] iommu/arm-smmu-v3: Prepare for nested domain support Nicolin Chen
2023-03-10 20:39   ` Robin Murphy
2023-03-11 12:40     ` Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 09/14] iommu/arm-smmu-v3: Implement arm_smmu_get_unmanaged_domain Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 10/14] iommu/arm-smmu-v3: Pass in user_cfg to arm_smmu_domain_finalise Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 11/14] iommu/arm-smmu-v3: Add arm_smmu_domain_alloc_user Nicolin Chen
2023-03-24 15:28   ` Eric Auger
2023-03-24 17:40     ` Nicolin Chen
2023-03-24 17:50       ` Jason Gunthorpe
2023-03-24 18:00         ` Nicolin Chen
2023-03-24 15:33   ` Eric Auger
2023-03-24 17:43     ` Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 12/14] iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED type of allocations Nicolin Chen
2023-03-09 13:20   ` Robin Murphy
2023-03-09 14:28     ` Robin Murphy
2023-03-10  1:34       ` Nicolin Chen
2023-03-24 15:44   ` Eric Auger
2023-03-24 16:30     ` Jason Gunthorpe
2023-03-24 17:50     ` Nicolin Chen
2023-03-24 17:51       ` Jason Gunthorpe
2023-03-24 17:55         ` Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 13/14] iommu/arm-smmu-v3: Add CMDQ_OP_TLBI_NH_VAA and CMDQ_OP_TLBI_NH_ALL Nicolin Chen
2023-03-09 13:44   ` Robin Murphy
2023-03-10  1:19     ` Nicolin Chen
2023-03-09 10:53 ` [PATCH v1 14/14] iommu/arm-smmu-v3: Add arm_smmu_cache_invalidate_user Nicolin Chen
2023-03-09 14:49   ` Robin Murphy
2023-03-09 15:31     ` Jason Gunthorpe
2023-03-10  4:20       ` Nicolin Chen
2023-03-10 16:19         ` Jason Gunthorpe
2023-03-11 11:56           ` Nicolin Chen
2023-03-11 12:53             ` Nicolin Chen
2023-03-20 13:03             ` Jason Gunthorpe
2023-03-20 15:56               ` Nicolin Chen
2023-03-20 16:04                 ` Jason Gunthorpe
2023-03-20 16:59                   ` Nicolin Chen
2023-03-20 18:45                     ` Jason Gunthorpe
2023-03-20 21:22                       ` Nicolin Chen
2023-03-20 22:19                         ` Jason Gunthorpe
2023-03-22 20:57                           ` Nicolin Chen
2023-03-23 12:17                             ` Jason Gunthorpe
2023-03-17  9:41           ` Tian, Kevin
2023-03-17 14:24             ` Nicolin Chen
2023-03-20 12:59             ` Jason Gunthorpe
2023-03-20 16:12               ` Nicolin Chen
2023-03-20 18:00                 ` Jason Gunthorpe
2023-03-21  8:34                   ` Tian, Kevin
2023-03-21 11:48                     ` Jason Gunthorpe
2023-03-22  6:42                       ` Nicolin Chen
2023-03-22 12:43                         ` Jason Gunthorpe
2023-03-22 17:11                           ` Nicolin Chen [this message]
2023-03-22 17:28                             ` Jason Gunthorpe
2023-03-22 19:21                               ` Nicolin Chen
2023-03-22 19:41                                 ` Jason Gunthorpe
2023-03-22 20:43                                   ` Nicolin Chen
2023-03-23 12:16                                     ` Jason Gunthorpe
2023-03-23 18:13                                       ` Nicolin Chen
2023-03-23 18:27                                         ` Jason Gunthorpe
2023-03-24  9:02                         ` Tian, Kevin
2023-03-24 14:57                           ` Jason Gunthorpe
2023-03-24 17:35                             ` Nicolin Chen
2023-03-28  3:03                               ` Tian, Kevin
2023-03-24  8:47                       ` Tian, Kevin
2023-03-24 14:44                         ` Jason Gunthorpe
2023-03-28  2:48                           ` Tian, Kevin
2023-03-28 12:26                             ` Jason Gunthorpe
2023-03-31  8:09                               ` Tian, Kevin
2023-03-17  9:24       ` Tian, Kevin
2023-03-10  3:51     ` Nicolin Chen
2023-03-10 17:53       ` Robin Murphy
2023-03-10 18:49         ` Jason Gunthorpe
2023-03-11 12:38         ` Nicolin Chen
2023-03-13 13:07           ` Robin Murphy
2023-03-16  0:01             ` Nicolin Chen
2023-03-16 14:58               ` Robin Murphy
2023-03-16 21:09                 ` Nicolin Chen
2023-03-20  1:32                   ` Nicolin Chen
2023-03-20 13:11                     ` Jason Gunthorpe
2023-03-20 15:28                       ` Nicolin Chen
2023-03-20 16:01                         ` Jason Gunthorpe
2023-03-20 16:35                           ` Nicolin Chen
2023-03-20 18:07                             ` Jason Gunthorpe
2023-03-20 20:46                               ` Nicolin Chen
2023-03-20 22:14                                 ` Jason Gunthorpe
2023-03-22  5:14                                   ` Nicolin Chen
2023-03-24  8:55                                     ` Tian, Kevin
2023-03-17  9:47     ` Tian, Kevin
2023-03-17 14:16       ` Nicolin Chen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ZBs2xaa7Ow1pnB3C@Asurada-Nvidia \
    --to=nicolinc@nvidia.com \
    --cc=baolu.lu@linux.intel.com \
    --cc=eric.auger@redhat.com \
    --cc=iommu@lists.linux.dev \
    --cc=jean-philippe@linaro.org \
    --cc=jgg@nvidia.com \
    --cc=joro@8bytes.org \
    --cc=kevin.tian@intel.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=robin.murphy@arm.com \
    --cc=shameerali.kolothum.thodi@huawei.com \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).