From: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
To: Oleksandr Tyshchenko <olekstysh@gmail.com>,
"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>
Cc: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>,
"julien.grall@arm.com" <julien.grall@arm.com>,
"sstabellini@kernel.org" <sstabellini@kernel.org>
Subject: Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Date: Wed, 7 Aug 2019 02:41:06 +0000 [thread overview]
Message-ID: <OSBPR01MB453664F7A6D6AA717761BC07D8D40@OSBPR01MB4536.jpnprd01.prod.outlook.com> (raw)
In-Reply-To: <1564763985-20312-7-git-send-email-olekstysh@gmail.com>
Hi Oleksandr-san,
I can access the datasheet of this hardware, so that I reviewed this patch.
I'm not familar about Xen development rulus, so that some comments might
be not good fit. If so, please ignore :)
> From: Oleksandr Tyshchenko, Sent: Saturday, August 3, 2019 1:40 AM
>
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>
> The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU)
> which provides address translation and access protection functionalities
> to processing units and interconnect networks.
>
> Please note, current driver is supposed to work only with newest
> Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation
This should be "R-Car Gen3 SoCs", instead of "Gen3 SoCs".
> table format and is able to use CPU's P2M table as is if one is
> 3-level page table (up to 40 bit IPA).
>
> The major differences compare to the Linux driver are:
>
> 1. Stage 1/Stage 2 translation. Linux driver supports Stage 1
> translation only (with Stage 1 translation table format). It manages
> page table by itself. But Xen driver supports Stage 2 translation
> (with Stage 2 translation table format) to be able to share the P2M
> with the CPU. Stage 1 translation is always bypassed in Xen driver.
>
> So, Xen driver is supposed to be used with newest Gen3 SoC revisions only
Same here.
> (H3 ES3.0, M3 ES3.0, etc.) which IPMMU H/W supports stage 2 translation
According to the latest manual, M3 ES3.0 is named as "M3-W+".
> table format.
<snip>
> diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> new file mode 100644
> index 0000000..a34a8f8
> --- /dev/null
> +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> @@ -0,0 +1,1342 @@
> +/*
> + * xen/drivers/passthrough/arm/ipmmu-vmsa.c
> + *
> + * Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs.
> + *
> + * The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU)
> + * which provides address translation and access protection functionalities
> + * to processing units and interconnect networks.
> + *
> + * Please note, current driver is supposed to work only with newest Gen3 SoCs
> + * revisions which IPMMU hardware supports stage 2 translation table format and
> + * is able to use CPU's P2M table as is.
> + *
> + * Based on Linux's IPMMU-VMSA driver from Renesas BSP:
> + * drivers/iommu/ipmmu-vmsa.c
So, I think the Linux's Copyrights should be described here.
> + * you can found at:
> + * url: git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
> + * branch: v4.14.75-ltsi/rcar-3.9.6
> + * commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044
> + * and Xen's SMMU driver:
> + * xen/drivers/passthrough/arm/smmu.c
> + *
> + * Copyright (C) 2016-2019 EPAM Systems Inc.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms and conditions of the GNU General Public
> + * License, version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public
> + * License along with this program; If not, see <http://www.gnu.org/licenses/>.
I don't know that Xen license description rule, but since a few source files have
SPDX-License-Identifier, can we also use it on the driver?
> + */
> +
> +#include <xen/delay.h>
> +#include <xen/err.h>
> +#include <xen/irq.h>
> +#include <xen/lib.h>
> +#include <xen/list.h>
I don't know that Xen passthrough driver rule though, doesn't here need
#include <xen/iommu.h>? (The xen/sched.h seems to have it so that
no compile error happens though.)
<snip>
> +/* Registers Definition */
> +#define IM_CTX_SIZE 0x40
> +
> +#define IMCTR 0x0000
> +/*
> + * These fields are implemented in IPMMU-MM only. So, can be set for
> + * Root IPMMU only.
> + */
> +#define IMCTR_VA64 (1 << 29)
> +#define IMCTR_TRE (1 << 17)
> +#define IMCTR_AFE (1 << 16)
> +#define IMCTR_RTSEL_MASK (3 << 4)
> +#define IMCTR_RTSEL_SHIFT 4
> +#define IMCTR_TREN (1 << 3)
> +/*
> + * These fields are common for all IPMMU devices. So, can be set for
> + * Cache IPMMUs as well.
> + */
> +#define IMCTR_INTEN (1 << 2)
> +#define IMCTR_FLUSH (1 << 1)
> +#define IMCTR_MMUEN (1 << 0)
> +#define IMCTR_COMMON_MASK (7 << 0)
> +
> +#define IMCAAR 0x0004
> +
> +#define IMTTBCR 0x0008
> +#define IMTTBCR_EAE (1 << 31)
> +#define IMTTBCR_PMB (1 << 30)
> +#define IMTTBCR_SH1_NON_SHAREABLE (0 << 28)
> +#define IMTTBCR_SH1_OUTER_SHAREABLE (2 << 28)
> +#define IMTTBCR_SH1_INNER_SHAREABLE (3 << 28)
> +#define IMTTBCR_SH1_MASK (3 << 28)
> +#define IMTTBCR_ORGN1_NC (0 << 26)
> +#define IMTTBCR_ORGN1_WB_WA (1 << 26)
> +#define IMTTBCR_ORGN1_WT (2 << 26)
> +#define IMTTBCR_ORGN1_WB (3 << 26)
> +#define IMTTBCR_ORGN1_MASK (3 << 26)
> +#define IMTTBCR_IRGN1_NC (0 << 24)
> +#define IMTTBCR_IRGN1_WB_WA (1 << 24)
> +#define IMTTBCR_IRGN1_WT (2 << 24)
> +#define IMTTBCR_IRGN1_WB (3 << 24)
> +#define IMTTBCR_IRGN1_MASK (3 << 24)
> +#define IMTTBCR_TSZ1_MASK (1f << 16)
At the moment, no one uses it though, this should be (0x1f << 16).
<snip>
+/* Xen IOMMU ops */
> +static int __must_check ipmmu_iotlb_flush_all(struct domain *d)
> +{
> + struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
> +
> + if ( !xen_domain || !xen_domain->root_domain )
> + return 0;
> +
> + spin_lock(&xen_domain->lock);
Is local irq is already disabled here?
If no, you should use spin_lock_irqsave() because the ipmmu_irq() also
gets the lock.
# To be honest, in normal case, any irq on the current implementation
# should not happen though.
> + ipmmu_tlb_invalidate(xen_domain->root_domain);
> + spin_unlock(&xen_domain->lock);
> +
> + return 0;
> +}
> +
<snip>
> +static int ipmmu_assign_device(struct domain *d, u8 devfn, struct device *dev,
> + uint32_t flag)
> +{
> + struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
> + struct ipmmu_vmsa_domain *domain;
> + int ret;
> +
> + if ( !xen_domain )
> + return -EINVAL;
> +
> + if ( !to_ipmmu(dev) )
> + return -ENODEV;
> +
> + spin_lock(&xen_domain->lock);
Same here.
> + /*
> + * The IPMMU context for the Xen domain is not allocated beforehand
> + * (at the Xen domain creation time), but on demand only, when the first
> + * master device being attached to it.
> + * Create Root IPMMU domain which context will be mapped to this Xen domain
> + * if not exits yet.
> + */
> + if ( !xen_domain->root_domain )
> + {
> + domain = ipmmu_alloc_root_domain(d);
> + if ( IS_ERR(domain) )
> + {
> + ret = PTR_ERR(domain);
> + goto out;
> + }
> +
> + xen_domain->root_domain = domain;
> + }
> +
> + if ( to_domain(dev) )
> + {
> + dev_err(dev, "Already attached to IPMMU domain\n");
> + ret = -EEXIST;
> + goto out;
> + }
> +
> + /*
> + * Master devices behind the same Cache IPMMU can be attached to the same
> + * Cache IPMMU domain.
> + * Before creating new IPMMU domain check to see if the required one
> + * already exists for this Xen domain.
> + */
> + domain = ipmmu_get_cache_domain(d, dev);
> + if ( !domain )
> + {
> + /* Create new IPMMU domain this master device will be attached to. */
> + domain = ipmmu_alloc_cache_domain(d);
> + if ( IS_ERR(domain) )
> + {
> + ret = PTR_ERR(domain);
> + goto out;
> + }
> +
> + /* Chain new IPMMU domain to the Xen domain. */
> + list_add(&domain->list, &xen_domain->cache_domains);
> + }
> +
> + ret = ipmmu_attach_device(domain, dev);
> + if ( ret )
> + {
> + /*
> + * Destroy Cache IPMMU domain only if there are no master devices
> + * attached to it.
> + */
> + if ( !domain->refcount )
> + ipmmu_free_cache_domain(domain);
> + }
> + else
> + {
> + domain->refcount++;
> + set_domain(dev, domain);
> + }
> +
> +out:
> + spin_unlock(&xen_domain->lock);
> +
> + return ret;
> +}
> +
> +static int ipmmu_deassign_device(struct domain *d, struct device *dev)
> +{
> + struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
> + struct ipmmu_vmsa_domain *domain = to_domain(dev);
> +
> + if ( !domain || domain->d != d )
> + {
> + dev_err(dev, "Not attached to %pd\n", d);
> + return -ESRCH;
> + }
> +
> + spin_lock(&xen_domain->lock);
Same here.
> + ipmmu_detach_device(domain, dev);
> + set_domain(dev, NULL);
> + domain->refcount--;
> +
> + /*
> + * Destroy Cache IPMMU domain only if there are no master devices
> + * attached to it.
> + */
> + if ( !domain->refcount )
> + ipmmu_free_cache_domain(domain);
> +
> + spin_unlock(&xen_domain->lock);
> +
> + return 0;
> +}
<snip>
> +static void __hwdom_init ipmmu_iommu_hwdom_init(struct domain *d)
> +{
> + /* Set to false options not supported on ARM. */
> + if ( iommu_hwdom_inclusive )
> + printk(XENLOG_WARNING "ipmmu: map-inclusive dom0-iommu option is not supported on ARM\n");
> + iommu_hwdom_inclusive = false;
> + if ( iommu_hwdom_reserved == 1 )
> + printk(XENLOG_WARNING "ipmmu: map-reserved dom0-iommu option is not supported on ARM\n");
> + iommu_hwdom_reserved = 0;
> +
> + arch_iommu_hwdom_init(d);
> +}
> +
> +static void ipmmu_iommu_domain_teardown(struct domain *d)
> +{
> + struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
> +
> + if ( !xen_domain )
> + return;
> +
> + spin_lock(&xen_domain->lock);
Same here.
> + /*
> + * Destroy Root IPMMU domain which context is mapped to this Xen domain
> + * if exits.
> + */
> + if ( xen_domain->root_domain )
> + ipmmu_free_root_domain(xen_domain->root_domain);
> +
> + spin_unlock(&xen_domain->lock);
> +
> + /*
> + * We assume that all master devices have already been detached from
> + * this Xen domain and there must be no associated Cache IPMMU domains
> + * in use.
> + */
> + ASSERT(list_empty(&xen_domain->cache_domains));
I think this should be in the spin lock held by &xen_domain->lock.
> + xfree(xen_domain);
> + dom_iommu(d)->arch.priv = NULL;
> +}
> +
> +static const struct iommu_ops ipmmu_iommu_ops =
> +{
> + .init = ipmmu_iommu_domain_init,
> + .hwdom_init = ipmmu_iommu_hwdom_init,
> + .teardown = ipmmu_iommu_domain_teardown,
> + .iotlb_flush = ipmmu_iotlb_flush,
> + .iotlb_flush_all = ipmmu_iotlb_flush_all,
> + .assign_device = ipmmu_assign_device,
> + .reassign_device = ipmmu_reassign_device,
> + .map_page = arm_iommu_map_page,
> + .unmap_page = arm_iommu_unmap_page,
> + .add_device = ipmmu_add_device,
> +};
> +
> +/* RCAR GEN3 product and cut information. */
"R-Car Gen3 SoCs" is better than "RCAR GEN3".
> +#define RCAR_PRODUCT_MASK 0x00007F00
> +#define RCAR_PRODUCT_H3 0x00004F00
> +#define RCAR_PRODUCT_M3 0x00005200
At least, I think we should be M3W, instead of M3.
# FYI, M3-W and M3-W+ are the same value.
<snip>
Best regards,
Yoshihiro Shimoda
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
next prev parent reply other threads:[~2019-08-07 2:41 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-02 16:39 [Xen-devel] [PATCH V2 0/6] iommu/arm: Add Renesas IPMMU-VMSA support + Linux's iommu_fwspec Oleksandr Tyshchenko
2019-08-02 16:39 ` [Xen-devel] [PATCH V2 1/6] iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff Oleksandr Tyshchenko
2019-08-09 17:35 ` Julien Grall
2019-08-09 18:10 ` Oleksandr
2019-08-02 16:39 ` [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request Oleksandr Tyshchenko
2019-08-12 11:11 ` Julien Grall
2019-08-12 12:01 ` Oleksandr
2019-08-12 19:46 ` Julien Grall
2019-08-13 12:35 ` Oleksandr
2019-08-14 17:34 ` Julien Grall
2019-08-14 19:25 ` Stefano Stabellini
2019-08-15 9:29 ` Julien Grall
2019-08-15 12:54 ` Julien Grall
2019-08-15 13:14 ` Oleksandr
2019-08-15 16:39 ` Oleksandr
2019-08-02 16:39 ` [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function Oleksandr Tyshchenko
2019-08-05 10:02 ` Jan Beulich
2019-08-06 18:50 ` Oleksandr
2019-08-07 6:22 ` Jan Beulich
2019-08-07 17:31 ` Oleksandr
2019-08-06 19:51 ` Volodymyr Babchuk
2019-08-07 6:26 ` Jan Beulich
2019-08-07 18:36 ` Oleksandr
2019-08-08 6:08 ` Jan Beulich
2019-08-08 7:05 ` Jan Beulich
2019-08-08 11:05 ` Oleksandr
2019-08-02 16:39 ` [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support Oleksandr Tyshchenko
2019-08-13 12:39 ` Julien Grall
2019-08-13 15:17 ` Oleksandr
2019-08-13 15:28 ` Julien Grall
2019-08-13 16:18 ` Oleksandr
2019-08-13 13:40 ` Julien Grall
2019-08-13 16:28 ` Oleksandr
2019-08-02 16:39 ` [Xen-devel] [PATCH V2 5/6] iommu/arm: Introduce iommu_add_dt_device API Oleksandr Tyshchenko
2019-08-13 13:49 ` Julien Grall
2019-08-13 16:05 ` Oleksandr
2019-08-13 17:13 ` Julien Grall
2019-08-02 16:39 ` [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support Oleksandr Tyshchenko
2019-08-07 2:41 ` Yoshihiro Shimoda [this message]
2019-08-07 16:01 ` Oleksandr
2019-08-07 19:15 ` Julien Grall
2019-08-07 20:28 ` Oleksandr Tyshchenko
2019-08-08 9:05 ` Julien Grall
2019-08-08 10:14 ` Oleksandr
2019-08-08 12:44 ` Julien Grall
2019-08-08 15:04 ` Oleksandr
2019-08-08 17:16 ` Julien Grall
2019-08-08 19:29 ` Oleksandr
2019-08-08 20:32 ` Julien Grall
2019-08-08 23:32 ` Oleksandr Tyshchenko
2019-08-09 9:56 ` Julien Grall
2019-08-09 18:38 ` Oleksandr
2019-08-08 12:28 ` Oleksandr
2019-08-08 14:23 ` Lars Kurth
2019-08-08 4:05 ` Yoshihiro Shimoda
2019-08-14 17:38 ` Julien Grall
2019-08-14 18:45 ` Oleksandr
2019-08-05 7:58 ` [Xen-devel] [PATCH V2 0/6] iommu/arm: Add Renesas IPMMU-VMSA support + Linux's iommu_fwspec Oleksandr
2019-08-05 8:29 ` Julien Grall
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=OSBPR01MB453664F7A6D6AA717761BC07D8D40@OSBPR01MB4536.jpnprd01.prod.outlook.com \
--to=yoshihiro.shimoda.uh@renesas.com \
--cc=julien.grall@arm.com \
--cc=oleksandr_tyshchenko@epam.com \
--cc=olekstysh@gmail.com \
--cc=sstabellini@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).